airship
/
armada
Code Issues Proposed changes

[focal] Python modules sync with Airship project 

- armada-airskiff-deploy is voting gate again
- fixed falcon.API deprecation - -> falcon.App
- fixed collections.abc.defaultdict not found error
- fixed tox4 requirements
- implemented requirements-frozen.txt approach to make allike as other
  Airship projects
- uplifted docker version in the image building and publishing gate

Change-Id: I337ec07cd6d082acabd9ad65dd9eefb728a43b12
This commit is contained in:
Sergiy Markin 2023-04-05 18:53:45 +00:00
parent 0e9d828fe6
commit 386a686e69
32 changed files with 684 additions and 475 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -20,7 +20,6 @@ lib64/
parts/
sdist/
var/
*.egg-info/
.installed.cfg
*.egg
etc/*.sample

35
.zuul.yaml
View File

@ -20,7 +20,6 @@
- openstack-tox-pep8
- armada-chart-build-gate
- armada-chart-build-latest-htk
# - armada-docker-build-gate-ubuntu_bionic
- armada-docker-build-gate-ubuntu_focal
- armada-airskiff-deploy
gate:
@ -28,7 +27,6 @@
- openstack-tox-docs
- openstack-tox-pep8
- armada-chart-build-gate
# - armada-docker-build-gate-ubuntu_bionic
- armada-docker-build-gate-ubuntu_focal
- armada-airskiff-deploy
post:
@ -65,24 +63,12 @@
description: |
Builds charts using latest Helm toolkit.
timeout: 900
voting: false
voting: true
run: tools/gate/playbooks/build-charts.yaml
nodeset: armada-single-node-focal
vars:
HTK_COMMIT: master
# - job:
# name: armada-docker-build-gate-ubuntu_bionic
# timeout: 1800
# run: tools/gate/playbooks/docker-image-build.yaml
# nodeset: armada-single-node
# vars:
# publish: false
# distro: ubuntu_bionic
# tags:
# dynamic:
# patch_set: true
- job:
name: armada-docker-build-gate-ubuntu_focal
timeout: 1800
@ -101,7 +87,7 @@
description: |
Deploy Memcached using Airskiff and submitted Armada changes.
timeout: 9600
voting: false
voting: true
pre-run:
- tools/gate/playbooks/airskiff-reduce-site.yaml
- tools/gate/playbooks/git-config.yaml
@ -111,7 +97,6 @@
- airship/treasuremap
vars:
CLONE_ARMADA: false
HTK_COMMIT: master
irrelevant-files:
- ^.*\.rst$
- ^doc/.*$
@ -119,22 +104,6 @@
- ^releasenotes/.*$
- ^swagger/.*$
# - job:
# name: armada-docker-publish-ubuntu_bionic
# timeout: 1800
# run: tools/gate/playbooks/docker-image-build.yaml
# nodeset: armada-single-node
# secrets:
# - airship_armada_quay_creds
# vars:
# publish: true
# distro: ubuntu_bionic
# tags:
# dynamic:
# branch: true
# commit: true
# static:
# - latest
- job:
name: armada-docker-publish-ubuntu_focal

142
Armada.egg-info/PKG-INFO Normal file
View File

@ -0,0 +1,142 @@
Metadata-Version: 2.1
Name: Armada
Version: 1.1.0.dev687
Summary: Tool for managing multiple Helm charts with dependencies by centralizing all configurations in a single Armada YAML and providing life-cycle hooks for all Helm releases.
Home-page: https://airshipit.org
Author: The Airship Authors
Author-email: airship-discuss@lists.airshipit.org
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.10
License-File: LICENSE
Armada
======
|Docker Repository on Quay|
Armada is a tool for managing multiple Helm charts with dependencies by
centralizing all configurations in a single Armada YAML and providing
life-cycle hooks for all Helm releases.
Find more documentation for Armada in the `Armada documentation <https://docs.airshipit.org/armada>`_.
Overview
--------
The Armada Python library and command line tool provide a way to
synchronize a Helm target with an operator's intended state,
consisting of several charts, dependencies, and overrides using a single file
or directory with a collection of files. This allows operators to define many
charts, potentially with different namespaces for those releases, and their
overrides in a central place. With a single command, deploy and/or upgrade them
where applicable.
Armada also supports fetching Helm chart source and then building charts from
source from various local and remote locations, such as Git endpoints, tarballs
or local directories.
It will also give the operator some indication of what is about to change by
assisting with diffs for both values, values overrides, and actual template
changes.
Its functionality extends beyond Helm, assisting in interacting with Kubernetes
directly to perform basic pre- and post-steps, such as removing completed or
failed jobs, running backup jobs, blocking on chart readiness, or deleting
resources that do not support upgrades. However, primarily, it is an interface
to support orchestrating Helm.
Components
----------
Armada consists of two separate but complementary components:
#. CLI component (**mandatory**) which interfaces directly with `Helm`_.
#. API component (**optional**) which services user requests through a wsgi
server (which in turn communicates with the `Helm`_ CLI) and provides
the following additional functionality:
* Role-Based Access Control.
* Limiting projects to specific functionality by leveraging
project-scoping provided by `Keystone`_.
Installation
------------
Quick Start (via Container)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Armada can be most easily installed as a container, which requires Docker to be
executed. To install Docker, please reference the following
`install guide <https://docs.docker.com/engine/installation/>`_.
Afterward, you can launch the Armada container by executing:
.. code-block:: bash
$ sudo docker run -d --net host -p 8000:8000 --name armada \
-v ~/.kube/config:/armada/.kube/config \
-v $(pwd)/examples/:/examples quay.io/airshipit/armada:latest-ubuntu_bionic
Manual Installation
^^^^^^^^^^^^^^^^^^^
For a comprehensive manual installation guide, please
see `Manual Install Guide`_.
Usage
^^^^^
To run Armada, simply supply it with your YAML-based intention for any
number of charts::
$ armada apply examples/openstack-helm.yaml [ --debug ]
Which should output something like this::
$ armada apply examples/openstack-helm.yaml 2017-02-10 09:42:36,753
armada INFO Cloning git:
...
For more information on how to install and use Armada, please reference:
`Armada Quickstart`_.
Integration Points
------------------
Armada CLI component has the following integration points:
* `Helm`_ manages Armada chart installations.
* `Deckhand`_ is one of the supported control document sources for Armada.
* `Prometheus`_ exporter is provided for metric data related to application
of charts and collections of charts. See `metrics`_.
In addition, Armada's API component has the following integration points:
* `Keystone`_ (OpenStack's identity service) provides authentication and
support for role-based authorization.
Further Reading
---------------
`Airship <https://airshipit.org>`_.
.. _Manual Install Guide: https://docs.airshipit.org/armada/development/getting-started.html#developer-install-guide
.. _Armada Quickstart: https://docs.airshipit.org/armada/operations/guide-use-armada.html
.. _metrics: https://docs.airshipit.org/armada/operations/metrics.html#metrics
.. _kubectl: https://kubernetes.io/docs/user-guide/kubectl/kubectl_config/
.. _Helm: https://docs.helm.sh
.. _Deckhand: https://opendev.org/airship/deckhand
.. _Prometheus: https://prometheus.io
.. _Keystone: https://github.com/openstack/keystone
.. |Docker Repository on Quay| image:: https://quay.io/repository/airshipit/armada/status
:target: https://quay.io/repository/airshipit/armada

237
Armada.egg-info/SOURCES.txt Normal file
View File

@ -0,0 +1,237 @@
.coveragerc
.dockerignore
.editorconfig
.readthedocs.yaml
.stestr.conf
.style.yapf
.zuul.yaml
CONTRIBUTING.rst
LICENSE
Makefile
README.rst
controller.sh
entrypoint.sh
plugin.yaml
requirements-direct.txt
requirements-frozen.txt
requirements.txt
setup.cfg
setup.py
test-requirements.txt
tox.ini
.github/SECURITY.md
Armada.egg-info/PKG-INFO
Armada.egg-info/SOURCES.txt
Armada.egg-info/dependency_links.txt
Armada.egg-info/entry_points.txt
Armada.egg-info/not-zip-safe
Armada.egg-info/pbr.json
Armada.egg-info/requires.txt
Armada.egg-info/top_level.txt
armada/__init__.py
armada/const.py
armada/shell.py
armada/version.py
armada/api/__init__.py
armada/api/middleware.py
armada/api/server.py
armada/api/controller/armada.py
armada/api/controller/health.py
armada/api/controller/metrics.py
armada/api/controller/releases.py
armada/api/controller/test.py
armada/api/controller/tiller.py
armada/api/controller/validation.py
armada/api/controller/versions.py
armada/cli/__init__.py
armada/cli/apply.py
armada/cli/test.py
armada/cli/validate.py
armada/common/__init__.py
armada/common/client.py
armada/common/i18n.py
armada/common/policy.py
armada/common/session.py
armada/common/policies/__init__.py
armada/common/policies/base.py
armada/common/policies/service.py
armada/common/policies/tiller.py
armada/conf/__init__.py
armada/conf/default.py
armada/conf/opts.py
armada/conf/utils.py
armada/exceptions/__init__.py
armada/exceptions/api_exceptions.py
armada/exceptions/armada_exceptions.py
armada/exceptions/base_exception.py
armada/exceptions/chartbuilder_exceptions.py
armada/exceptions/helm_exceptions.py
armada/exceptions/k8s_exceptions.py
armada/exceptions/manifest_exceptions.py
armada/exceptions/override_exceptions.py
armada/exceptions/source_exceptions.py
armada/exceptions/validate_exceptions.py
armada/handlers/__init__.py
armada/handlers/armada.py
armada/handlers/chart_delete.py
armada/handlers/chart_deploy.py
armada/handlers/chart_download.py
armada/handlers/chartbuilder.py
armada/handlers/document.py
armada/handlers/helm.py
armada/handlers/k8s.py
armada/handlers/lock.py
armada/handlers/manifest.py
armada/handlers/metrics.py
armada/handlers/override.py
armada/handlers/pre_update_actions.py
armada/handlers/release_diff.py
armada/handlers/schema.py
armada/handlers/test.py
armada/handlers/wait.py
armada/schemas/armada-chart-schema-v1.yaml
armada/schemas/armada-chart-schema-v2.yaml
armada/schemas/armada-chartgroup-schema-v1.yaml
armada/schemas/armada-chartgroup-schema-v2.yaml
armada/schemas/armada-manifest-schema-v1.yaml
armada/schemas/armada-manifest-schema-v2.yaml
armada/tests/__init__.py
armada/tests/test_utils.py
armada/tests/unit/__init__.py
armada/tests/unit/base.py
armada/tests/unit/fake_policy.py
armada/tests/unit/fixtures.py
armada/tests/unit/api/__init__.py
armada/tests/unit/api/base.py
armada/tests/unit/api/test_api_initialization.py
armada/tests/unit/api/test_armada_controller.py
armada/tests/unit/api/test_health_controller.py
armada/tests/unit/api/test_releases_controller.py
armada/tests/unit/api/test_test_controller.py
armada/tests/unit/api/test_tiller_controller.py
armada/tests/unit/api/test_validation_controller.py
armada/tests/unit/api/test_versions_controller.py
armada/tests/unit/common/__init__.py
armada/tests/unit/common/test_policy.py
armada/tests/unit/common/test_session.py
armada/tests/unit/handlers/__init__.py
armada/tests/unit/handlers/test_armada.py
armada/tests/unit/handlers/test_chartbuilder.py
armada/tests/unit/handlers/test_lock.py
armada/tests/unit/handlers/test_manifest.py
armada/tests/unit/handlers/test_override.py
armada/tests/unit/handlers/test_release_diff.py
armada/tests/unit/handlers/test_test.py
armada/tests/unit/handlers/test_wait.py
armada/tests/unit/handlers/templates/base.yaml
armada/tests/unit/handlers/templates/override-01-expected.yaml
armada/tests/unit/handlers/templates/override-01.yaml
armada/tests/unit/handlers/templates/override-02-expected.yaml
armada/tests/unit/handlers/templates/override-03-expected.yaml
armada/tests/unit/resources/keystone-manifest.yaml
armada/tests/unit/resources/valid_armada_document.yaml
armada/tests/unit/utils/__init__.py
armada/tests/unit/utils/schema.py
armada/tests/unit/utils/test_release.py
armada/tests/unit/utils/test_source.py
armada/tests/unit/utils/test_validate.py
armada/utils/__init__.py
armada/utils/helm.py
armada/utils/keystone.py
armada/utils/release.py
armada/utils/source.py
armada/utils/validate.py
armada/utils/validation_message.py
charts/armada/.helmignore
charts/armada/Chart.yaml
charts/armada/requirements.yaml
charts/armada/values.yaml
charts/armada/templates/configmap-bin.yaml
charts/armada/templates/configmap-etc.yaml
charts/armada/templates/deployment-api.yaml
charts/armada/templates/ingress-api.yaml
charts/armada/templates/job-image-repo-sync.yaml
charts/armada/templates/job-ks-endpoints.yaml
charts/armada/templates/job-ks-service.yaml
charts/armada/templates/job-ks-user.yaml
charts/armada/templates/network_policy.yaml
charts/armada/templates/secret-ingress-tls.yaml
charts/armada/templates/secret-keystone-env.yaml
charts/armada/templates/service-ingress.yaml
charts/armada/templates/service.yaml
charts/armada/templates/tests/test-armada-api.yaml
charts/deps/.gitkeep
doc/requirements.txt
doc/source/conf.py
doc/source/index.rst
doc/source/_static/.placeholder
doc/source/_static/airship.logo.white.svg
doc/source/_static/armada.conf.sample
doc/source/_static/armada.policy.yaml.sample
doc/source/commands/apply.rst
doc/source/commands/index.rst
doc/source/commands/test.rst
doc/source/commands/validate.rst
doc/source/development/contributing.rst
doc/source/development/getting-started.rst
doc/source/development/index.rst
doc/source/operations/guide-configure.rst
doc/source/operations/guide-helm-plugin.rst
doc/source/operations/guide-troubleshooting.rst
doc/source/operations/guide-use-armada.rst
doc/source/operations/index.rst
doc/source/operations/metrics.rst
doc/source/operations/sampleconf.rst
doc/source/operations/samplepolicy.rst
doc/source/operations/documents/index.rst
doc/source/operations/documents/migration-v1-v2.rst
doc/source/operations/documents/v1/document-authoring.rst
doc/source/operations/documents/v1/index.rst
doc/source/operations/documents/v1/schemas.rst
doc/source/operations/documents/v2/document-authoring.rst
doc/source/operations/documents/v2/index.rst
doc/source/operations/documents/v2/schemas.rst
doc/source/operations/exceptions/api-exceptions.inc
doc/source/operations/exceptions/armada-exceptions.inc
doc/source/operations/exceptions/base-exceptions.inc
doc/source/operations/exceptions/chartbuilder-exceptions.inc
doc/source/operations/exceptions/guide-exceptions.rst
doc/source/operations/exceptions/index.rst
doc/source/operations/exceptions/k8s-exceptions.inc
doc/source/operations/exceptions/manifest-exceptions.inc
doc/source/operations/exceptions/override-exceptions.inc
doc/source/operations/exceptions/source-exceptions.inc
doc/source/operations/exceptions/validate-exceptions.inc
etc/armada/api-paste.ini
etc/armada/armada.conf.sample
etc/armada/config-generator.conf
etc/armada/policy-generator.conf
etc/armada/policy.yaml
examples/armada-keystone-manifest.yaml
examples/keystone-manifest.yaml
examples/podinfo.yaml
examples/simple-ovr-values.yaml
examples/simple.yaml
examples/tar_example.yaml
images/armada/Dockerfile.ubuntu_bionic
images/armada/Dockerfile.ubuntu_focal
releasenotes/notes/.placeholder
releasenotes/source/conf.py
releasenotes/source/index.rst
releasenotes/source/unreleased.rst
releasenotes/source/_static/.placeholder
releasenotes/source/_templates/.placeholder
tools/armada_image_run.sh
tools/helm_install.sh
tools/helm_tk.sh
tools/image_tags.py
tools/keystone-account.sh
tools/whitespace-linter.sh
tools/gate/playbooks/airskiff-deploy.yaml
tools/gate/playbooks/airskiff-reduce-site.yaml
tools/gate/playbooks/build-charts.yaml
tools/gate/playbooks/debug-report.yaml
tools/gate/playbooks/docker-image-build.yaml
tools/gate/playbooks/git-config.yaml
tools/gate/playbooks/vars.yaml
tools/gate/playbooks/files/docker-systemd.conf

1
Armada.egg-info/dependency_links.txt Normal file
View File

@ -0,0 +1 @@

8
Armada.egg-info/entry_points.txt Normal file
View File

@ -0,0 +1,8 @@
[console_scripts]
armada = armada.shell:main
[oslo.config.opts]
armada.conf = armada.conf.opts:list_opts
[oslo.policy.policies]
armada = armada.common.policies:list_rules

1
Armada.egg-info/not-zip-safe Normal file
View File

@ -0,0 +1 @@

1
Armada.egg-info/pbr.json Normal file
View File

@ -0,0 +1 @@
{"git_version": "5658b3d", "is_release": false}

30
Armada.egg-info/requires.txt Normal file
View File

@ -0,0 +1,30 @@
click
configparser
deepdiff<=5.8.1
docutils
falcon
importlib_metadata
jsonschema<=3.2.0
keystoneauth1<=5.1.1
kubernetes==26.1.0
MarkupSafe<2.1.0,>=0.9.2
nose
oslo.config<=8.7.1
oslo.i18n
oslo.log<=4.6.0
oslo.policy<=3.10.1
oslo.utils<=4.12.3
prometheus-client==0.8.0
pylibyaml==0.1.0
python_dateutil==2.8.2
PyYAML<=5.4.1
reno
requests==2.27.0
retry
setuptools<=45.2.0
six
Sphinx
sphinx_rtd_theme==0.5.0
urllib3<=1.25.11,>=1.21.1
uWSGI==2.0.21
wheel

1
Armada.egg-info/top_level.txt Normal file
View File

@ -0,0 +1 @@
armada

4
armada/api/server.py
View File

@ -45,7 +45,7 @@ def create(enable_middleware=CONF.middleware):
"""
if enable_middleware:
api = falcon.API(
api = falcon.App(
request_type=ArmadaRequest,
middleware=[
AuthMiddleware(),
@ -53,7 +53,7 @@ def create(enable_middleware=CONF.middleware):
LoggingMiddleware(),
])
else:
api = falcon.API(request_type=ArmadaRequest)
api = falcon.App(request_type=ArmadaRequest)
logging.set_defaults(default_log_levels=CONF.default_log_levels)
logging.setup(CONF, 'armada')

4
armada/conf/opts.py
View File

@ -28,7 +28,7 @@ package. It is assumed that:
"""
import collections.abc
import collections
import importlib
import os
import pkgutil
@ -38,7 +38,7 @@ IGNORED_MODULES = ('opts', 'constants', 'utils')
def list_opts():
opts = collections.abc.defaultdict(list)
opts = collections.defaultdict(list)
module_names = _list_module_names()
imported_modules = _import_modules(module_names)
_append_config_options(imported_modules, opts)

2
armada/handlers/wait.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
from abc import ABC, abstractmethod
import collections.abc
import collections
import copy
import math
import re

2
armada/tests/unit/api/test_api_initialization.py
View File

@ -23,4 +23,4 @@ class TestApi(test_base.BaseControllerTest):
def test_init_application(self):
server = importlib.import_module('armada.api.server')
api = server.create()
self.assertIsInstance(api, falcon.API)
self.assertIsInstance(api, falcon.App)

10
bindep.txt Normal file
View File

@ -0,0 +1,10 @@
# This file contains runtime (non-python) dependencies
# More info at: https://docs.openstack.org/infra/bindep/readme.html
libffi-dev [test platform:dpkg]
libkrb5-dev [platform:dpkg]
libpq-dev [platform:dpkg]
libsasl2-dev [platform:dpkg]
libssl-dev [platform:dpkg]
libre2-dev [platform:dpkg]
ethtool [platform:dpkg]

14
doc/requirements.txt
View File

@ -1,7 +1,13 @@
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
sphinx!=1.6.6,!=1.6.7,>=1.6.2 # BSD
sphinxcontrib-apidoc>=0.2.0 # BSD
sphinx_rtd_theme>=0.2.4
reno>=2.5.0 # Apache-2.0
Sphinx==6.1.3
sphinx-rtd-theme==0.5.0
reno
pylibyaml==0.1.0
plantuml
sphinxcontrib-apidoc==0.3.0 # BSD
sphinxcontrib-plantuml==0.25
jinja2
MarkupSafe<2.1.0, >=0.9.2
docutils

280
doc/source/_static/armada.conf.sample
View File

@ -202,66 +202,6 @@
#fatal_deprecations = false
[cors]
#
# From oslo.middleware
#
# Indicate whether this resource may be shared with the domain received in the
# requests "origin" header. Format: "<protocol>://<host>[:<port>]", no trailing
# slash. Example: https://horizon.example.com (list value)
#allowed_origin = <None>
# Indicate that the actual request can include user credentials (boolean value)
#allow_credentials = true
# Indicate which headers are safe to expose to the API. Defaults to HTTP Simple
# Headers. (list value)
#expose_headers =
# Maximum cache age of CORS preflight requests. (integer value)
#max_age = 3600
# Indicate which methods can be used during the actual request. (list value)
#allow_methods = OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,PATCH
# Indicate which header field names may be used during the actual request.
# (list value)
#allow_headers =
[healthcheck]
#
# From oslo.middleware
#
# DEPRECATED: The path to respond to healtcheck requests on. (string value)
# This option is deprecated for removal.
# Its value may be silently ignored in the future.
#path = /healthcheck
# Show more detailed information as part of the response. Security note:
# Enabling this option may expose sensitive details about the service being
# monitored. Be sure to verify that it will not violate your security policies.
# (boolean value)
#detailed = false
# Additional backends that can perform health checks and report that
# information back as part of a request. (list value)
#backends =
# Check the presence of a file to determine if an application is running on a
# port. Used by DisableByFileHealthcheck plugin. (string value)
#disable_by_file_path = <None>
# Check the presence of a file based on a port to determine if an application
# is running on a port. Expects a "port:path" list of strings. Used by
# DisableByFilesPortsHealthcheck plugin. (list value)
#disable_by_file_paths =
[keystone_authtoken]
#
@ -321,7 +261,7 @@
# Domain name containing project (string value)
#project_domain_name = <None>
# Trust ID (string value)
# ID of the trust to use as a trustee use (string value)
#trust_id = <None>
# Optional domain ID to use with v3 and v2 parameters. It will be used for both
@ -350,219 +290,6 @@
# User's password (string value)
#password = <None>
#
# From keystonemiddleware.auth_token
#
# Complete "public" Identity API endpoint. This endpoint should not be an
# "admin" endpoint, as it should be accessible by all end users.
# Unauthenticated clients are redirected to this endpoint to authenticate.
# Although this endpoint should ideally be unversioned, client support in the
# wild varies. If you're using a versioned v2 endpoint here, then this should
# *not* be the same endpoint the service user utilizes for validating tokens,
# because normal end users may not be able to reach that endpoint. (string
# value)
# Deprecated group/name - [keystone_authtoken]/auth_uri
#www_authenticate_uri = <None>
# DEPRECATED: Complete "public" Identity API endpoint. This endpoint should not
# be an "admin" endpoint, as it should be accessible by all end users.
# Unauthenticated clients are redirected to this endpoint to authenticate.
# Although this endpoint should ideally be unversioned, client support in the
# wild varies. If you're using a versioned v2 endpoint here, then this should
# *not* be the same endpoint the service user utilizes for validating tokens,
# because normal end users may not be able to reach that endpoint. This option
# is deprecated in favor of www_authenticate_uri and will be removed in the S
# release. (string value)
# This option is deprecated for removal since Queens.
# Its value may be silently ignored in the future.
# Reason: The auth_uri option is deprecated in favor of www_authenticate_uri
# and will be removed in the S release.
#auth_uri = <None>
# API version of the admin Identity API endpoint. (string value)
#auth_version = <None>
# Do not handle authorization requests within the middleware, but delegate the
# authorization decision to downstream WSGI components. (boolean value)
#delay_auth_decision = false
# Request timeout value for communicating with Identity API server. (integer
# value)
#http_connect_timeout = <None>
# How many times are we trying to reconnect when communicating with Identity
# API Server. (integer value)
#http_request_max_retries = 3
# Request environment key where the Swift cache object is stored. When
# auth_token middleware is deployed with a Swift cache, use this option to have
# the middleware share a caching backend with swift. Otherwise, use the
# ``memcached_servers`` option instead. (string value)
#cache = <None>
# Required if identity server requires client certificate (string value)
#certfile = <None>
# Required if identity server requires client certificate (string value)
#keyfile = <None>
# A PEM encoded Certificate Authority to use when verifying HTTPs connections.
# Defaults to system CAs. (string value)
#cafile = <None>
# Verify HTTPS connections. (boolean value)
#insecure = false
# The region in which the identity server can be found. (string value)
#region_name = <None>
# DEPRECATED: Directory used to cache files related to PKI tokens. This option
# has been deprecated in the Ocata release and will be removed in the P
# release. (string value)
# This option is deprecated for removal since Ocata.
# Its value may be silently ignored in the future.
# Reason: PKI token format is no longer supported.
#signing_dir = <None>
# Optionally specify a list of memcached server(s) to use for caching. If left
# undefined, tokens will instead be cached in-process. (list value)
# Deprecated group/name - [keystone_authtoken]/memcache_servers
#memcached_servers = <None>
# In order to prevent excessive effort spent validating tokens, the middleware
# caches previously-seen tokens for a configurable duration (in seconds). Set
# to -1 to disable caching completely. (integer value)
#token_cache_time = 300
# DEPRECATED: Determines the frequency at which the list of revoked tokens is
# retrieved from the Identity service (in seconds). A high number of revocation
# events combined with a low cache duration may significantly reduce
# performance. Only valid for PKI tokens. This option has been deprecated in
# the Ocata release and will be removed in the P release. (integer value)
# This option is deprecated for removal since Ocata.
# Its value may be silently ignored in the future.
# Reason: PKI token format is no longer supported.
#revocation_cache_time = 10
# (Optional) If defined, indicate whether token data should be authenticated or
# authenticated and encrypted. If MAC, token data is authenticated (with HMAC)
# in the cache. If ENCRYPT, token data is encrypted and authenticated in the
# cache. If the value is not one of these options or empty, auth_token will
# raise an exception on initialization. (string value)
# Possible values:
# None - <No description provided>
# MAC - <No description provided>
# ENCRYPT - <No description provided>
#memcache_security_strategy = None
# (Optional, mandatory if memcache_security_strategy is defined) This string is
# used for key derivation. (string value)
#memcache_secret_key = <None>
# (Optional) Number of seconds memcached server is considered dead before it is
# tried again. (integer value)
#memcache_pool_dead_retry = 300
# (Optional) Maximum total number of open connections to every memcached
# server. (integer value)
#memcache_pool_maxsize = 10
# (Optional) Socket timeout in seconds for communicating with a memcached
# server. (integer value)
#memcache_pool_socket_timeout = 3
# (Optional) Number of seconds a connection to memcached is held unused in the
# pool before it is closed. (integer value)
#memcache_pool_unused_timeout = 60
# (Optional) Number of seconds that an operation will wait to get a memcached
# client connection from the pool. (integer value)
#memcache_pool_conn_get_timeout = 10
# (Optional) Use the advanced (eventlet safe) memcached client pool. The
# advanced pool will only work under python 2.x. (boolean value)
#memcache_use_advanced_pool = false
# (Optional) Indicate whether to set the X-Service-Catalog header. If False,
# middleware will not ask for service catalog on token validation and will not
# set the X-Service-Catalog header. (boolean value)
#include_service_catalog = true
# Used to control the use and type of token binding. Can be set to: "disabled"
# to not check token binding. "permissive" (default) to validate binding
# information if the bind type is of a form known to the server and ignore it
# if not. "strict" like "permissive" but if the bind type is unknown the token
# will be rejected. "required" any form of token binding is needed to be
# allowed. Finally the name of a binding method that must be present in tokens.
# (string value)
#enforce_token_bind = permissive
# DEPRECATED: If true, the revocation list will be checked for cached tokens.
# This requires that PKI tokens are configured on the identity server. (boolean
# value)
# This option is deprecated for removal since Ocata.
# Its value may be silently ignored in the future.
# Reason: PKI token format is no longer supported.
#check_revocations_for_cached = false
# DEPRECATED: Hash algorithms to use for hashing PKI tokens. This may be a
# single algorithm or multiple. The algorithms are those supported by Python
# standard hashlib.new(). The hashes will be tried in the order given, so put
# the preferred one first for performance. The result of the first hash will be
# stored in the cache. This will typically be set to multiple values only while
# migrating from a less secure algorithm to a more secure one. Once all the old
# tokens are expired this option should be set to a single value for better
# performance. (list value)
# This option is deprecated for removal since Ocata.
# Its value may be silently ignored in the future.
# Reason: PKI token format is no longer supported.
#hash_algorithms = md5
# A choice of roles that must be present in a service token. Service tokens are
# allowed to request that an expired token can be used and so this check should
# tightly control that only actual services should be sending this token. Roles
# here are applied as an ANY check so any role in this list must be present.
# For backwards compatibility reasons this currently only affects the
# allow_expired check. (list value)
#service_token_roles = service
# For backwards compatibility reasons we must let valid service tokens pass
# that don't pass the service_token_roles check as valid. Setting this true
# will become the default in a future release and should be enabled if
# possible. (boolean value)
#service_token_roles_required = false
# Authentication type to load (string value)
# Deprecated group/name - [keystone_authtoken]/auth_plugin
#auth_type = <None>
# Config Section from which to load plugin specific options (string value)
#auth_section = <None>
[oslo_middleware]
#
# From oslo.middleware
#
# The maximum body size for each request, in bytes. (integer value)
# Deprecated group/name - [DEFAULT]/osapi_max_request_body_size
# Deprecated group/name - [DEFAULT]/max_request_body_size
#max_request_body_size = 114688
# DEPRECATED: The HTTP Header that will be used to determine what the original
# request protocol scheme was, even if it was hidden by a SSL termination
# proxy. (string value)
# This option is deprecated for removal.
# Its value may be silently ignored in the future.
#secure_proxy_ssl_header = X-Forwarded-Proto
# Whether the application is behind a proxy or not. This determines if the
# middleware should parse the headers or not. (boolean value)
#enable_proxy_headers_parsing = false
[oslo_policy]
@ -583,7 +310,10 @@
# to be evaluated. This means if any existing token is allowed for old defaults
# but is disallowed for new defaults, it will be disallowed. It is encouraged
# to enable this flag along with the ``enforce_scope`` flag so that you can get
# the benefits of new defaults and ``scope_type`` together (boolean value)
# the benefits of new defaults and ``scope_type`` together. If ``False``, the
# deprecated policy check string is logically OR'd with the new policy check
# string, allowing for a graceful upgrade experience between releases with new
# policies, which is the default behavior. (boolean value)
#enforce_new_defaults = false
# The relative or absolute path of a file that maps roles to permissions for a

4
images/armada/Dockerfile.ubuntu_bionic
View File

@ -41,7 +41,7 @@ RUN useradd -u 1000 -g users -d $(pwd) armada
ENTRYPOINT ["./entrypoint.sh"]
CMD ["server"]
COPY requirements.txt ./
COPY requirements-frozen.txt ./
ENV LD_LIBRARY_PATH=/usr/local/lib
@ -71,7 +71,7 @@ RUN set -ex \
&& cd .. \
&& rm -fr libyaml \
&& python3 -m pip install -U pip \
&& pip3 install -r requirements.txt --no-cache-dir \
&& pip3 install -r requirements-frozen.txt --no-cache-dir \
&& curl -fSSL -O ${HELM_ARTIFACT_URL} \
&& tar -xvf $(basename ${HELM_ARTIFACT_URL}) \
&& mv linux-amd64/helm /usr/local/bin \

4
images/armada/Dockerfile.ubuntu_focal
View File

@ -41,7 +41,7 @@ RUN useradd -u 1000 -g users -d $(pwd) armada
ENTRYPOINT ["./entrypoint.sh"]
CMD ["server"]
COPY requirements.txt ./
COPY requirements-frozen.txt ./
ENV LD_LIBRARY_PATH=/usr/local/lib
@ -71,7 +71,7 @@ RUN set -ex \
&& cd .. \
&& rm -fr libyaml \
&& python3 -m pip install -U pip \
&& pip3 install -r requirements.txt --no-cache-dir \
&& pip3 install -r requirements-frozen.txt --no-cache-dir \
&& curl -fSSL -O ${HELM_ARTIFACT_URL} \
&& tar -xvf $(basename ${HELM_ARTIFACT_URL}) \
&& mv linux-amd64/helm /usr/local/bin \

31
requirements-direct.txt Normal file
View File

@ -0,0 +1,31 @@
click
configparser
deepdiff<=5.8.1
docutils
falcon
gitpython
importlib_metadata
jsonschema<=3.2.0
keystoneauth1<=5.1.1
kubernetes==26.1.0
MarkupSafe<2.1.0, >=0.9.2
nose
oslo.config<=8.7.1
oslo.i18n
oslo.log<=4.6.0
oslo.policy<=3.10.1
oslo.utils<=4.12.3
prometheus-client==0.8.0
pylibyaml==0.1.0
python_dateutil==2.8.2
PyYAML<=5.4.1
reno
requests==2.27.0
retry
setuptools<=45.2.0
six
Sphinx
sphinx_rtd_theme==0.5.0
urllib3 >= 1.21.1, <= 1.25.11
uWSGI==2.0.21
wheel

79
requirements-frozen.txt Normal file
View File

@ -0,0 +1,79 @@
alabaster==0.7.13
attrs==23.1.0
Babel==2.12.1
cachetools==5.3.0
certifi==2022.12.7
charset-normalizer==2.0.12
click==8.1.3
configparser==5.3.0
debtcollector==2.5.0
decorator==5.1.1
deepdiff==5.8.1
docutils==0.19
dulwich==0.21.3
falcon==3.1.1
gitdb==4.0.10
GitPython==3.1.31
google-auth==2.17.3
idna==3.4
imagesize==1.4.1
importlib-metadata==6.5.0
iso8601==1.1.0
Jinja2==3.1.2
jsonschema==3.2.0
keystoneauth1==5.1.1
kubernetes==26.1.0
MarkupSafe==2.0.1
msgpack==1.0.5
netaddr==0.8.0
netifaces==0.11.0
nose==1.3.7
oauthlib==3.2.2
ordered-set==4.1.0
os-service-types==1.7.0
oslo.config==8.7.1
oslo.context==5.1.1
oslo.i18n==6.0.0
oslo.log==4.6.0
oslo.policy==3.10.1
oslo.serialization==5.1.1
oslo.utils==4.12.3
packaging==23.1
pbr==5.11.1
pip==23.0.1
prometheus-client==0.8.0
py==1.11.0
pyasn1==0.5.0
pyasn1-modules==0.3.0
Pygments==2.15.1
pylibyaml==0.1.0
pyparsing==3.0.9
pyrsistent==0.19.3
python-dateutil==2.8.2
pytz==2023.3
PyYAML==5.4.1
reno==4.0.0
requests==2.27.0
requests-oauthlib==1.3.1
retry==0.9.2
rfc3986==2.0.0
rsa==4.9
setuptools==45.2.0
six==1.16.0
smmap==5.0.0
snowballstemmer==2.2.0
Sphinx==6.1.3
sphinx-rtd-theme==0.5.0
sphinxcontrib-applehelp==1.0.4
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.1
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
stevedore==5.0.0
urllib3==1.25.11
uWSGI==2.0.21
websocket-client==1.5.1
wheel==0.40.0
wrapt==1.15.0
zipp==3.15.0

39
requirements.txt
View File

@ -1,36 +1,3 @@
amqp==5.0.8
deepdiff>=3.3.0
gitpython
jsonschema>=3.0.1,<4
keystoneauth1>=3.18.0
keystonemiddleware>=5.3.0
kombu==5.1.0
kubernetes~=26.1.0; python_version >= '3.6'
Paste>=2.0.3
PasteDeploy>=1.5.2
pylibyaml~=0.1
pyyaml~=5.1
requests
retry
setuptools>=40.4.3
prometheus_client<0.13.0,>=0.7.0
# API
falcon
uwsgi>=2.0.15
# CLI
click>=6.7
# Oslo
oslo.cache>=1.38.1 # Apache-2.0
oslo.concurrency>=3.8.0 # Apache-2.0
oslo.config>=7.0.0 # Apache-2.0
oslo.context>=2.14.0 # Apache-2.0
oslo.i18n!=3.15.2,>=2.1.0 # Apache-2.0
oslo.log>=3.45.2 # Apache-2.0
oslo.messaging!=5.25.0,>=5.24.2 # Apache-2.0
oslo.middleware>=3.27.0 # Apache-2.0
oslo.policy>=1.23.0 # Apache-2.0
oslo.serialization>=2.29.2 # Apache-2.0
oslo.utils>=3.42.1 # Apache-2.0
# Warning: This file should be empty.
# Specify direct dependencies in requirements-direct.txt instead.
-r requirements-direct.txt

2
setup.cfg
View File

@ -1,6 +1,6 @@
[metadata]
name = Armada
version = 1.0
version = 1.1
summary = Tool for managing multiple Helm charts with dependencies by centralizing all configurations in a single Armada YAML and providing life-cycle hooks for all Helm releases.
description_file = README.rst
author = The Airship Authors

41
test-requirements.txt
View File

@ -1,23 +1,20 @@
# General
pyflakes>=2.1.1
tox
# Docs
sphinx>=1.6.2
sphinx_rtd_theme>=0.2.4
# Testing
bandit
coverage!=4.4,>=4.5.1 # Apache-2.0
testtools>=2.3.0 # MIT
os-testr>=1.0.0 # Apache-2.0
flake8>=3.3.0
mock
responses>=0.8.1
yapf==0.27.0
flake8-import-order>=0.18.1
grpcio-tools>=1.16.0
typing-extensions>=3.7.2
bandit==1.6.0
cmd2>=1.5.0
stestr
coverage==5.3
flake8-import-order>=0.18.1
flake8==3.8.4
grpcio-tools>=1.16.0
mock==4.0.2
os-testr>=1.0.0 # Apache-2.0
pyflakes==2.2.0
responses==0.12.1
sphinx-rtd-theme==0.5.0
stestr==3.2.0
testrepository==0.0.20
testresources==2.0.1
testscenarios==0.5.0
testtools==2.5.0
testtools==2.5.0
tox>=3.28.0, <4.0.0
typing-extensions>=3.7.2
yapf==0.27.0

42
tools/gate/playbooks/airskiff-deploy.yaml
View File

@ -13,41 +13,19 @@
# limitations under the License.
- hosts: primary
vars:
env:
HTTP_PROXY: ""
HTTPS_PROXY: ""
NO_PROXY: ""
DISTRO: "{{ distro }}"
become: true
roles:
- bindep
- clear-firewall
- ensure-docker
- ensure-python
- ensure-pip
- disable-systemd-resolved
tasks:
# Stop systemd-resolved service before starting docker.
- name: stop systemd-resolved service
systemd:
state: stopped
enabled: no
masked: yes
daemon_reload: yes
name: systemd-resolved
become: yes
- name: ensure pip3 is installed
apt:
name: "{{ item }}"
with_items:
- python3-pip
- python3-setuptools
when: ansible_os_family == 'Debian'
become: true
- name: Clone Required Repositories
shell: |
export CLONE_ARMADA={{ CLONE_ARMADA }}
export OSH_INFRA_COMMIT={{ HTK_COMMIT }}
export OSH_INFRA_COMMIT=master
./tools/deployment/airskiff/developer/000-clone-dependencies.sh
args:
chdir: "{{ zuul.projects['opendev.org/airship/treasuremap'].src_dir }}"
@ -71,9 +49,15 @@
chdir: "{{ zuul.project.src_dir }}"
become: yes
- name: Setup clients
shell: |
./tools/deployment/airskiff/developer/020-setup-client.sh
args:
chdir: "{{ zuul.projects['opendev.org/airship/treasuremap'].src_dir }}"
- name: Deploy Airship components using Armada
shell: |
mkdir ~/.kube
mkdir -p ~/.kube
cp -rp /home/zuul/.kube/config ~/.kube/config
./tools/deployment/airskiff/developer/030-armada-bootstrap.sh
args:

52
tools/gate/playbooks/docker-image-build.yaml
View File

@ -13,6 +13,12 @@
# limitations under the License.
- hosts: primary
roles:
- bindep
- ensure-docker
- ensure-python
- ensure-pip
tasks:
- include_vars: vars.yaml
@ -42,54 +48,12 @@
debug:
var: image_tags
- name: Install Docker (Debian)
when: ansible_os_family == 'Debian'
- name: Install Docker python module for ansible docker login
block:
- file:
path: "{{ item }}"
state: directory
with_items:
- /etc/docker/
- /etc/systemd/system/docker.service.d/
- /var/lib/docker/
- mount:
path: /var/lib/docker/
src: tmpfs
fstype: tmpfs
opts: size=25g
state: mounted
- copy: "{{ item }}"
with_items:
- content: "{{ docker_daemon | to_json }}"
dest: /etc/docker/daemon.json
- src: files/docker-systemd.conf
dest: /etc/systemd/system/docker.service.d/
- apt_key:
url: https://download.docker.com/linux/ubuntu/gpg
- apt_repository:
repo: deb http://{{ zuul_site_mirror_fqdn }}/deb-docker/{{ ansible_distribution_release }} {{ ansible_distribution_release }} stable
- apt:
name: "{{ item }}"
allow_unauthenticated: True
with_items:
- apparmor
- docker-ce
- docker-ce-cli
- docker-buildx-plugin
- containerd.io
- python3-pip
- python3-setuptools
- pip:
name: docker
version: 2.7.0
version: 4.4.4
executable: pip3
# NOTE(SamYaple): Allow all connections from containers to host so the
# containers can access the http server for git and wheels
- iptables:
action: insert
chain: INPUT
in_interface: docker0
jump: ACCEPT
become: True
- name: Make images

8
tools/gate/playbooks/files/docker-systemd.conf
View File

@ -1,8 +0,0 @@
# NOTE(SamYaple): CentOS cannot be build with userns-remap enabled. httpd uses
# cap_set_file capability and there is no way to pass that in at build as of
# docker 17.06.
# TODO(SamYaple): Periodically check to see if this is possible in newer
# versions of Docker
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd

1
tools/gate/playbooks/roles Symbolic link
View File

@ -0,0 +1 @@
../roles

37
tools/gate/roles/disable-systemd-resolved/tasks/disable-systemd-resolved.yaml Normal file
View File

@ -0,0 +1,37 @@
# Copyright 2020 AT&T Intellectual Property. All other rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
- name: Disable systemd-resolved service
systemd:
state: stopped
enabled: no
masked: yes
daemon_reload: yes
name: systemd-resolved
become: yes
- name: Remove local stub dns from resolv.conf, if it exists
lineinfile:
path: /etc/resolv.conf
state: absent
regexp: '^nameserver.*127.0.0.1'
become: yes
- name: Add upstream nameservers in resolv.conf
blockinfile:
path: /etc/resolv.conf
block: |
nameserver 8.8.8.8
nameserver 8.8.4.4
become: yes

15
tools/gate/roles/disable-systemd-resolved/tasks/main.yaml Normal file
View File

@ -0,0 +1,15 @@
# Copyright 2020 AT&T Intellectual Property. All other rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
- include: disable-systemd-resolved.yaml

2
tools/helm_tk.sh
View File

@ -17,7 +17,7 @@
set -eux
HTK_REPO=${HTK_REPO:-"https://opendev.org/openstack/openstack-helm-infra.git"}
HTK_STABLE_COMMIT=${HTK_COMMIT:-"a7cd689280cdbc0acd04a7a1b745941260e8700b"}
HTK_STABLE_COMMIT=${HTK_COMMIT:-"7b8d459d14a751021265cd29dbe9920ceac71f3a"}
TMP_DIR=$(mktemp -d)

29
tox.ini
View File

@ -1,11 +1,11 @@
[tox]
skipsdist = True
minversion = 2.3.1
minversion = 3.18.0
envlist = py38, pep8, cover, bandit
[testenv]
deps=
-r{toxinidir}/requirements.txt
-r{toxinidir}/requirements-frozen.txt
-r{toxinidir}/test-requirements.txt
passenv=
HTTP_PROXY
@ -30,6 +30,21 @@ commands =
commands =
{posargs}
[testenv:freeze]
basepython=python3
recreate = True
allowlist_externals=
rm
sh
deps=
-r{toxinidir}/requirements-direct.txt
commands=
rm -f requirements-frozen.txt
sh -c "pip freeze --all | grep -vE 'deckhand|pyinotify|pkg-resources==0.0.0' > requirements-frozen.txt"
[testenv:py38]
commands =
{[testenv]commands}
@ -37,33 +52,28 @@ commands =
stestr slowest
[testenv:docs]
basepython = python3
deps=
-r{toxinidir}/requirements.txt
-r{toxinidir}/requirements-frozen.txt
-r{toxinidir}/doc/requirements.txt
commands =
rm -rf doc/build
sphinx-build -W -b html doc/source doc/build/html
[testenv:genconfig]
basepython = python3
commands =
oslo-config-generator --config-file=etc/armada/config-generator.conf
[testenv:genpolicy]
basepython = python3
commands =
oslopolicy-sample-generator --config-file=etc/armada/policy-generator.conf
[testenv:releasenotes]
basepython = python3
deps = -r{toxinidir}/doc/requirements.txt
commands =
rm -rf releasenotes/build
sphinx-build -a -E -W -d releasenotes/build/doctrees -b html releasenotes/source releasenotes/build/html
[testenv:pep8]
basepython = python3
deps =
.[bandit]
{[testenv]deps}
@ -76,12 +86,10 @@ commands =
bandit -r armada -n 5 -x armada/tests/*
[testenv:bandit]
basepython = python3
commands =
bandit -r armada -n 5 -x armada/tests/*
[testenv:cover]
basepython = python3
setenv = {[testenv]setenv}
PYTHON=coverage run --source armada --parallel-mode
commands =
@ -94,7 +102,6 @@ commands =
coverage report
[testenv:fmt]
basepython = python3
deps = {[testenv]deps}
commands =
yapf -ir {toxinidir}/armada {toxinidir}/setup.py