This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2
Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
- upgraded Airflow to 1.10.15 -
https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html
- disabled xenial, bionic and opensuse images build gates
- added focal image build gate
- added focal zuul build node
- adjusted Makefile for focal
- added bindep.txt to utilize bindep zuul base role for zuul build node
pre-setup
- added focal Dockerfile
- implemented freeze requirements.txt approach like in other Airship
projects
- removed specific requirements.txt for airflow in favor of using
requirements-frozen.txt from shipyard_airflow project when building
airflow docker image
- fixed docker image publishing to Quay
- replaces deprecated LOG.warn with new LOG.warning call
- replaced deprecated body attribute in responce wiht responce.text
attribute
- update of falcon module deprecated .API call - replaced wiht
falcon.App call
- deprecated routing.create_http_method_map method replaced with
routing.map_http_methods
- re-formatted code tabulations based on yapf recommendations
- replaced deprecated protocol attribute in Pytest create_environ() with
http_version attribute
- replaced deprecated app attribute in Pytest create_environ() with
root_path attribute
- fixed airflow CLI commands to match 1.10.15 version
- updated zuul gates to work on focal nodes and added focal specific
node setup items by adding appriate ansible tasks and roles
- uplifted Helm to 3.9.4
- uplifted stable HTK commit id
- updated tox.in to with with tox v4
- uplifted dependences references to other Airship projects
- common python dependences were syncronized with other Airship
projects(Promenade, Deckhand, Armada, Drydock)
- fixed airskiff deployment gate
- fixed genconfig* profiles in shipyard-airflow tox.ini responsible for
maintanance of policy.yaml.sample and shipyard.conf.sample
Change-Id: I0c85187dc9bacf0849382563dd5ff7e9b2814c59
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].
These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.
Change-Id: Idd9ea0d57b5be063b133036cfc9ebaa69956f4fc
Now that the Shipyard gates are running on Focal(20.04), flake8, grpcio,
and pyflakes need to use a version compatible with python3.8 and Ubuntu
20.04 which is the default python version in Ubuntu focal. Also unpinning
setuptools so that it is compatible with python3.8.
Additionally, address pep8 violations that arise from using a newer
version of flake8
Change-Id: Idc3c5d66b48fc9e4497a71d1b640bcd2872c22eb
Uplifted celery to 4.4.2 to address memory leak issue raised
in: https://github.com/celery/celery/issues/4843.
This issue is addressed in celery 4.4.2
Change-Id: I26c403513ba48308044d69130d33561b314fd209
Adds an option to create configdocs as an empty colleciton. This is done
as an explicit flag (--empty-collection) on the command as opposed to
using empty files to prevent accidental emtpy file loads leading to
frustration.
Since this introduced a new flag value for the CLI, the CLIs using flag
values were updated to use the standard is_flag=True instead of the
flag_value=True or some other value when a boolean flag is expected.
Minor updates to CLI tests due to moving to responses 0.10.2
Depends-On: https://review.openstack.org/#/c/614421/
Change-Id: I489b0e1183335cbfbaa2014c1458a84dadf6bb0b
Refactors the uwsgi-based logging filter to encapsulate
better and adds a redaction filter for api logging.
This change now also includes bringing a stray test back into
the correct location.
Change-Id: I53de7da6bd4182e824366fdbb5221c7c9ae5be09
Refactor Shipyard to be better able to leverage common
packages and conform with the target UCP standard layout.
This change supports the same tox entrypoints at
the root level, but the preferred approach is to use make
targets defined in the Makefile such as 'make tests' and
'make lint'
The previous tox.ini has moved and been
tailored to the specifics of each subproject at
src/bin/*/tox.ini
Autotmatic generation of the policy and configuration
files has been removed from the sphinx build for now
but these files will be automatically generated locally
into the docs source by using a 'make docs' command.
This may need to be revisited later to re-enable the
automatic generation of these files such that readthedocs
would still support the project layout.
Change-Id: Ifdc1cd4cf35fb3c5923414c677b781a60a9bae42
Sergiy Markin