This PS bumps up Airflow version to the latest
2.8.2 and also bumps up openstack dependences to
Antelope 2023.1
Change-Id: I7bbcbc8f9e6334100f47702f4546f0963d11b2ca
This PS fixes one more SubDAG, makes sure that decide_airflow_upgrade
task works correctly after we previously replaced armada_build SubDAG
with corresponding TaskGroup. Also a related unit test has been fixed.
Change-Id: Ic8304cc1985f8197d363861c588b8b48e13fe5d4
This PS is mainly fixing SubDAGs timing issues when they
started along with main DAG and not at the time the main
DAG needs them. In Airflow 2.6.2 SubDAGs are deprecated
in favor of TaskGroups. So in this PS all SubDAGS were
replaced with TaskGroups.
Also task level logging config was extended by adding
py-console to obtain logs from tasks like it was
configured in Airflow 1.10.5.
Change-Id: I3f6d3961b1511e3b7cd2f7aab9810d033cfc14a3
This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2
Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.
Change-Id: I4c4c7186e67c771f8249b2017b2507aac17760b7
This PS adds default values for chart values and resolves some issues
in python code that utilizes these values:
validation_connect_timeout: 20
validation_read_timeout: 300
deckhand_client_connect_timeout: 20
deckhand_client_read_timeout: 300
drydock_client_connect_timeout: 20
drydock_client_read_timeout: 300
Change-Id: Ic5b1920257859239613a3ce77134e6b05bd7e9dd
This PS is created to :
- roll back apache-airflow back to 1.10.5
- change default chart values from bionic to ubuntu_focal
- save freshly generated config and policy files samples
- in focal Dockerfile postgres client version is pubped up to v15
- change airflow docker image requirements from limited to fully frozen for shipyard-airflow project
- adjusted requirements-direct.txt for shipyard_airflow
- adjusted requirements-direct.txt for shipyard_client
- regenerated requirements-frozen for both projects
- fixed unit tests after upgrading click module
- gen_all tox profile processing has been moved over to py8 gate because it requires focal zuul node to run
- upgraded airskiff gate playbook to include latest treasuremap patchset with updated airskiff site admada manifests
Change-Id: I47e44f5cfa19b2649697e7cc5a31557a6f4fcfea
- upgraded Airflow to 1.10.15 -
https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html
- disabled xenial, bionic and opensuse images build gates
- added focal image build gate
- added focal zuul build node
- adjusted Makefile for focal
- added bindep.txt to utilize bindep zuul base role for zuul build node
pre-setup
- added focal Dockerfile
- implemented freeze requirements.txt approach like in other Airship
projects
- removed specific requirements.txt for airflow in favor of using
requirements-frozen.txt from shipyard_airflow project when building
airflow docker image
- fixed docker image publishing to Quay
- replaces deprecated LOG.warn with new LOG.warning call
- replaced deprecated body attribute in responce wiht responce.text
attribute
- update of falcon module deprecated .API call - replaced wiht
falcon.App call
- deprecated routing.create_http_method_map method replaced with
routing.map_http_methods
- re-formatted code tabulations based on yapf recommendations
- replaced deprecated protocol attribute in Pytest create_environ() with
http_version attribute
- replaced deprecated app attribute in Pytest create_environ() with
root_path attribute
- fixed airflow CLI commands to match 1.10.15 version
- updated zuul gates to work on focal nodes and added focal specific
node setup items by adding appriate ansible tasks and roles
- uplifted Helm to 3.9.4
- uplifted stable HTK commit id
- updated tox.in to with with tox v4
- uplifted dependences references to other Airship projects
- common python dependences were syncronized with other Airship
projects(Promenade, Deckhand, Armada, Drydock)
- fixed airskiff deployment gate
- fixed genconfig* profiles in shipyard-airflow tox.ini responsible for
maintanance of policy.yaml.sample and shipyard.conf.sample
Change-Id: I0c85187dc9bacf0849382563dd5ff7e9b2814c59
Connections from airflow-worker can be long-lived (e.g. when applying
large Armada manifests), and are prone to disconnection due to network
inactivity.
This change uplifts the Armada client to enable socket keepalives in
order to avoid this issue. (Promenade is also uplifted to address a
requirements conflict.)
Change-Id: I067a5edbbdff06f45394e772ebb16d5012987b6c
Update Zuul config and tox settings to not run python3.5 jobs, and to
make the xenial image jobs non-voting.
Change-Id: Icc9f71e60e6b48254917ea1d41ce9bc823cce512
* Fixed typo for jsonschema version
* Setuptools version forced to <58 to allow usage of use_2to3, which
some dependencies require
* Fixed scripting error that allowed Airflow installation to silently
fail
Change-Id: I237801488795df07f2d4de1cde6cd4ec41182e31
* Install older version of pip<21.0 for ubuntu_xenial images
* Install setuptools via pip for ubuntu_xenial images
* Pin typing-extensions to 3.7.2 and apache-airflow to 1.10.5
* Move promenade Dependencies under UCP components in requirements.txt
* With apache-airflow=1.10.5, strip ANSI escape sequences
in test_deployment_group_manager.py
* Update tox.ini to support apache-airflow=1.10.5
* airskiff gate fixes
- Pin treasuremap to v1.9 branch
- Pin openstack-helm-infra to master
- Remove openstack client setup as it's not used
Change-Id: Iee4ce59fdceacb165120a69d11c44e6e47feaea8
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].
These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.
Change-Id: Idd9ea0d57b5be063b133036cfc9ebaa69956f4fc
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.
https://pypi.org/project/pylibyaml/
Change-Id: Ic3f7ba1ce6c4db1a4bc18fe1aef8e0675f7cd69e
Now that the Shipyard gates are running on Focal(20.04), flake8, grpcio,
and pyflakes need to use a version compatible with python3.8 and Ubuntu
20.04 which is the default python version in Ubuntu focal. Also unpinning
setuptools so that it is compatible with python3.8.
Additionally, address pep8 violations that arise from using a newer
version of flake8
Change-Id: Idc3c5d66b48fc9e4497a71d1b640bcd2872c22eb
In the case where a Drydock task either fails or times out, it is not very
clear which Drydock task is being referred to. This commit updates the log
messages to include the Drydock task-id that has either failed or timed out.
Additionally, debug logging is added to log the Drydock task status. The reason
for this is because there are situations where a Drydock task is still running,
but on the Shipyard side the task has timed out and therefore, Shipyard shows
the task (i.e. prepare_nodes or deploy_nodes) as failed. It is good to get a point
in time reference to the task state for a couple of reasons:
1) To verify via logging that Shipyard is reporting Drydock tasks (success,
failures, partial_successes, etc) accurately
2) Later on if Drydock shows a node as deployed, then the task can be
queried and the state can be checked to see if the task was indeed a success.
Change-Id: I7050338e2f92ad548e639e2ea4059a520c27e686
- Overrode uwsgi default configs to improve stability and performance.
- Increased mas number of worker processes to increase capacity and
performance.
- Enabled uwsgi cheaper subsystem to scale worker processes dynamically.
- Uplifted uwsgi to the latest release to bring bug fixes and
improvements since 2018.
Upgraded uwsgi to bring in bug fixes since 2018.
For background information for this change please see:
https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html
Change-Id: If067e9786e9dbbd39ef832dea6f51aa5523af4d7
Uplifted celery to 4.4.2 to address memory leak issue raised
in: https://github.com/celery/celery/issues/4843.
This issue is addressed in celery 4.4.2
Change-Id: I26c403513ba48308044d69130d33561b314fd209
The current Armada DAG allows for 3 retries, reduced from 10 here:
bef8eecac1
This is sometimes insufficient, especially in cases where chart updates
of underlying Airship or Kubernetes components are upgraded and pods are
restarted underneath the airflow workers. The chart installation may be
successful, but an Armada retry may still be consumed.
This change increases the number of retries to 5. This will allow Armada
to progress further through the manifest if there is a disruption after
a chart is successfully installed. The tradeoff is that Armada may try
to repeatedly install a chart that keeps failing in the same way,
delaying the ultimate failure of the deployment.
Change-Id: I1fad7b1d95af061595680a76d24c6d323b365a67
1. Locked the WTForms to 2.2.1 to address the import issue with
wtforms.widgets.HTMLString. WTForms 2.3.0 was released on
April 21/2020. This release causes shipyard gate fail with
import error for wtforms.widgets.HTMLString.
2. Deleted psycopg2==2.7.7, which is installed as a dependency of
apache-airflow extra package postgres, and resoted the newer
release psycopg2-binary==2.8.4, to be used instead.
Change-Id: I303a2c94ec409e97af1192ae892b8148fcdbb8d5
- With bionic image based shipyard docker images, uwsgi crashes
with segmentation fault, when it tries to load the psycopg2 library,
causing the api become unreachable on both shipyard docker images.
This happens because psycopg2 2.7.x and uwsgi binary wheels are built
with incompatible ssl libraries. This patch upgrades psycopg2 to the
latest release to address this issue.
- The existing image build script cannot run in a docker or a pod,
based pipeline because of two reasons:
- The build script runs a docker (docker-in-docker) and mounts a
volume.
In a dind case, volume bind mounts will not work, because the nested
container will need the host file system's path for the source path.
- The shipyard service listens to its exposed service port in the
nested docker network namespace, which is not reachable from the host
pod/container.
This patch address both of the above issues. It first creates the
container, copies needed config files to the container and then starts
it. Also it execs into the nested docker to access the shipyard services
in a dind (docker-in-dcoker) case.
Change-Id: Ifdfed539babab01608bfaef37001bb79cd3a080d
The latest Werkzeug package release 1.0.0, released on February 8,
is not backwards compatible with the earler releases of this package,
which is used in Flask, used by apache airflow.
This causes shipyard make image job fail, with missing import errors
from the Werkzeug library.
This change locks the Werkzeuz package release to the last compatible
release with the apache airflow in shipyard.
Change-Id: I54dad4ccc1858f4d5986c6e8e9fbf8f5d9847158
Added support to buid shipyard and airflow images using either a xenial
or Ubuntu bionic base image.
The default base image is set to bionic.
Change-Id: I6ad4d42dede081586b3ccea87a42e250979ac106
Change URLs from git.openstack.org & github.com to opendev.org due to the
migration; wrap multiple LABELs into one.
Change-Id: I240fa6f746bd1e424e5a2b7fd381903c46059ae5
- Adds the information related to the test_site action.
- Reformats, slightly, the output from 'shipyard help actions'
- Adds tests that use an externalized list of actions to keep the help
documentation in alignment with the actions supported in the API.
Change-Id: I2efd473da0dbf6c8cbadfc9fae575c303996c43b
Updated deckhand commit id, because apache airflow release 1.10.3 and
later use flask >=1.1.0, which needs Werkzeug library version >=0.15.0.
see: https://issues.apache.org/jira/browse/AIRFLOW-4900
The updated deckhand commit lets the Werkzeug package float to >=0.15.0.
Change-Id: I62d7f4e5eecb2f05035a1c9552544aa65e70ee3f
- Allow a Docker build-arg to specify the source of the Airflow
python package in any format supported by pip
Change-Id: Ifa2dd62d91570035cff91ff07868b0fcf659add8
Pegleg Promenade and Shipyard should all use same versions of
packages when able. Requests currently is giving a warning
in Pegleg:
ERROR: deckhand 0.0.1.dev657 has requirement urllib3==1.24, but
you'll have urllib3 1.24.3 which is incompatible.
Change-Id: I9f21203b4109e7542a952d68c7a6ffcdb2653026
Setuptools is currently listed as a requirement for shipyard-client, but
it is not used. Pegging setuptools forces downgrading when installing
shipyard-client, which can cause issues for packages that attempt to
utilize features found in newer versions of setuptools. Removing the
setuptools requirements will allow installing users to choose their own
version of setuptools to use.
Change-Id: I88570b69346ec7019e753bdd8681fd63d55dc8c8
This change is going to be done in conjunction with Pegleg's shipyard
helper code [0] such that an auth token can be set as an environment
variable. If present it will be used rather than authenticating
against Keystone. This is a requested change by Darren Dejaeger so
that when using automation such as Jenkins pipelines the user is not
forced to provide their credentials to have shipyard automatically
obtain the token - instead they can obtain their token separately and
provide that token for shipyard to use.
[0]: https://opendev.org/airship/pegleg/src/branch/master/pegleg/engine/util/shipyard_helper.py
Change-Id: I859bff978069e8cc3502b952273237e84de83456
Shipyard Client's version of PyYAML (3.13) conflicts with the version
used by Pegleg (5.1). This conflict can cause issues when trying to
install Pegleg as it uses Shipyard Client as a dependency. This update
to Shipyard Client's PyYAML version will fix the dependency conflict
with Pegleg and will also fix a possible security vulnerability
associated with older versions of PyYAML.
Upgrades PyYAML to version 5.1, which makes several changes:
- Deprecates yaml.load in favor of FullLoader and UnsafeLoader due to
yaml.load's vulnerability with code injection [0]
- Drops support for py33 and py26
- Changes to default_flow_style=False
- Several other bug fixes and additional changes
[0] https://nvd.nist.gov/vuln/detail/CVE-2017-18342
Change-Id: I8eb4bb02dcc99eab3d983640d6c769cadb3b97e8
Deckhand's requirement for keystoneauth1 (3.11.1) does not match with
the version used by Shipyard Client (3.11.0). In addition, Shipyard
Client uses requests-2.19.1 which is an insecure version of requests.
Both of these requirements conflict with other Airship project
dependencies. These conflicts become apparent in Pegleg, which uses both
Shipyard Client and Deckhand. The updates made in this change make
Shipyard Client compatible with Pegleg's own dependencies and other
projects it imports from.
Upticks keystoneauth1 to 3.11.1, a bugfix which should not impact
Shipyard Client behavior.
Upticks requests to 2.20.0, which contains several bug fixes and
deprecates support for py26.
Change-Id: I8716c56351e90bca8e7dbdf672638a45f15c655a