- upgraded Airflow to 1.10.15 -
https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html
- disabled xenial, bionic and opensuse images build gates
- added focal image build gate
- added focal zuul build node
- adjusted Makefile for focal
- added bindep.txt to utilize bindep zuul base role for zuul build node
pre-setup
- added focal Dockerfile
- implemented freeze requirements.txt approach like in other Airship
projects
- removed specific requirements.txt for airflow in favor of using
requirements-frozen.txt from shipyard_airflow project when building
airflow docker image
- fixed docker image publishing to Quay
- replaces deprecated LOG.warn with new LOG.warning call
- replaced deprecated body attribute in responce wiht responce.text
attribute
- update of falcon module deprecated .API call - replaced wiht
falcon.App call
- deprecated routing.create_http_method_map method replaced with
routing.map_http_methods
- re-formatted code tabulations based on yapf recommendations
- replaced deprecated protocol attribute in Pytest create_environ() with
http_version attribute
- replaced deprecated app attribute in Pytest create_environ() with
root_path attribute
- fixed airflow CLI commands to match 1.10.15 version
- updated zuul gates to work on focal nodes and added focal specific
node setup items by adding appriate ansible tasks and roles
- uplifted Helm to 3.9.4
- uplifted stable HTK commit id
- updated tox.in to with with tox v4
- uplifted dependences references to other Airship projects
- common python dependences were syncronized with other Airship
projects(Promenade, Deckhand, Armada, Drydock)
- fixed airskiff deployment gate
- fixed genconfig* profiles in shipyard-airflow tox.ini responsible for
maintanance of policy.yaml.sample and shipyard.conf.sample
Change-Id: I0c85187dc9bacf0849382563dd5ff7e9b2814c59
Connections from airflow-worker can be long-lived (e.g. when applying
large Armada manifests), and are prone to disconnection due to network
inactivity.
This change uplifts the Armada client to enable socket keepalives in
order to avoid this issue. (Promenade is also uplifted to address a
requirements conflict.)
Change-Id: I067a5edbbdff06f45394e772ebb16d5012987b6c
* Fixed typo for jsonschema version
* Setuptools version forced to <58 to allow usage of use_2to3, which
some dependencies require
* Fixed scripting error that allowed Airflow installation to silently
fail
Change-Id: I237801488795df07f2d4de1cde6cd4ec41182e31
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].
These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.
Change-Id: Idd9ea0d57b5be063b133036cfc9ebaa69956f4fc
1. Locked the WTForms to 2.2.1 to address the import issue with
wtforms.widgets.HTMLString. WTForms 2.3.0 was released on
April 21/2020. This release causes shipyard gate fail with
import error for wtforms.widgets.HTMLString.
2. Deleted psycopg2==2.7.7, which is installed as a dependency of
apache-airflow extra package postgres, and resoted the newer
release psycopg2-binary==2.8.4, to be used instead.
Change-Id: I303a2c94ec409e97af1192ae892b8148fcdbb8d5
- With bionic image based shipyard docker images, uwsgi crashes
with segmentation fault, when it tries to load the psycopg2 library,
causing the api become unreachable on both shipyard docker images.
This happens because psycopg2 2.7.x and uwsgi binary wheels are built
with incompatible ssl libraries. This patch upgrades psycopg2 to the
latest release to address this issue.
- The existing image build script cannot run in a docker or a pod,
based pipeline because of two reasons:
- The build script runs a docker (docker-in-docker) and mounts a
volume.
In a dind case, volume bind mounts will not work, because the nested
container will need the host file system's path for the source path.
- The shipyard service listens to its exposed service port in the
nested docker network namespace, which is not reachable from the host
pod/container.
This patch address both of the above issues. It first creates the
container, copies needed config files to the container and then starts
it. Also it execs into the nested docker to access the shipyard services
in a dind (docker-in-dcoker) case.
Change-Id: Ifdfed539babab01608bfaef37001bb79cd3a080d
Change URLs from git.openstack.org & github.com to opendev.org due to the
migration; wrap multiple LABELs into one.
Change-Id: I240fa6f746bd1e424e5a2b7fd381903c46059ae5
Updated deckhand commit id, because apache airflow release 1.10.3 and
later use flask >=1.1.0, which needs Werkzeug library version >=0.15.0.
see: https://issues.apache.org/jira/browse/AIRFLOW-4900
The updated deckhand commit lets the Werkzeug package float to >=0.15.0.
Change-Id: I62d7f4e5eecb2f05035a1c9552544aa65e70ee3f
- Allow a Docker build-arg to specify the source of the Airflow
python package in any format supported by pip
Change-Id: Ifa2dd62d91570035cff91ff07868b0fcf659add8
A recent Shipyard change [0] introduced user-context tracing support in
Shipyard, which requires the passing of an additional parameter to the
Deckhand and Drydock API clients. The Deckhand [1] and Drydock [2]
clients were updated to handle the additional parameter, but the
versions of those components used by Shipyard was not updated. This
change uplifts the versions of Deckhand and Drydock in requirements.txt
for the Airflow component of Shipyard to a version that supports the
aforementioned, additional, user-context parameter.
[0] https://review.openstack.org/633873
[1] https://review.openstack.org/634068
[2] https://review.openstack.org/638115
Change-Id: I096191f436c5123b6c263fe92c0d69b9e4085c66
The airflow image has a requirements file that is separate from the one
located in src/bin/shipyard_airflow, which was missed in a previous
uplift patch [0]. This change updates all versions in the image
requirements file to match the source code requirements.
[0] https://review.openstack.org/652736
Change-Id: I238bba4038e5cd8364711c2c5d1f78f51fc76948
Updates the dependencies on Armada, Drydock, and Deckhand to current
values. The primary reason for this is to take advantage of a new set of
Armada requirements that allows for a reduced image size (no longer
includes grpc tools)
Change-Id: I8f208ab91664479a7af96c543a4e72d8f6d154ef
Updates to Airflow 1.10.1; See (1), (2) for some notes
Related, and additionally: configures Airflow to restore logging of
workflow steps to a console/sdtout logger, supporting the desired
ability to attach logging and monitoring to standard container
mechanisms. This does not change the behavior of also logging to the
airflow-arranged log files for steps and DAG runs.
A side effect of updating to 1.10.1 includes a major decrease in
resource usage by the Airflow scheudler process (reducing from ~ 1 core
fully consumed to less than 5% of a core consumed YMMV, but significant)
Additional adjustment downward of resources allocated, threads produced,
and frequency of polling leads to an overall significant reduction in
resource usage.
Airship note:
Because Airflow 1.10.0 and 1.10.1 use compatible versions of celery and
dag_run information, updating from 1.10.0 - 1.10.1 in place is possible
if airflow-worker pods are allowed to continue to run.
(1) https://github.com/apache/incubator-airflow/blob/master/UPDATING.md
(2) https://github.com/apache/incubator-airflow/releases/tag/1.10.1
Change-Id: I9b024e3996c528c7b74e2888191d48c7a45a1f04
Now that Deckhand supports rolling back to revision 0 with
https://review.openstack.org/#/c/592725/ it is preferrable to
exercise this functionality instead of deleting all revisions
in Deckhand's database, so that previous revision history
isn't lost.
Deckhand is fast-forwarded to
0b5aa2e98a1ab5ab8a58c9dec3c1f88ef00d17a9 which contains
the commit required for SY to be able to make use of this
new change in Deckhand's client, as well as to align with
Promenade's Deckhand fast-forward of Deckhand in
https://review.openstack.org/#/c/604459/
Depends-On: https://review.openstack.org/#/c/592725/
Change-Id: Iebbfeb5e6c74b3e2f09b2137ec14482cc8fa4f44
Updates the image building to Airflow 1.10, including necessary
configuration changes and a general update of dependencies.
Airflow 1.10 includes many enhancements and bugfixes since 1.9 [0]
This change introduces many "unused" configuration parameters to satsify
Airflow's expectations[1]. An ugly, but likely harmless change to the
log output with interleaved newline characters from Airflow steps[2].
Changes to the chart and other dependendencies have also been introduced
to match this update.
[0] https://github.com/apache/incubator-airflow/blob/master/CHANGELOG.txt
[1] https://issues.apache.org/jira/browse/AIRFLOW-3099
[2] https://issues.apache.org/jira/browse/AIRFLOW-1917
Change-Id: I179dcf1f0369650b8c4519f704abb7fb495f4248
1) UCP -> Airship
2) readthedocs.org -> readthedocs.io (there is redirect)
3) http -> https
4) attcomdev -> airshipit (repo on quay.io)
5) att-comdev -> openstack/airship-* (repo on github/openstack git)
6) many URLs have been verified and adjusted to be current
7) no need for 'en/latest/' path in URL of the RTD
8) added more info to some setup.cfg and setup.py files
9) ucp-integration docs are now in airship-in-a-bottle
10) various other minor fixes
Change-Id: I4b8cc6ddf491e35d600a83f5f82d7717108e31dd
This patchset uplifts Deckhand commit to include fix
for document replacement.
This uplift is the same one used in Promenade [0].
The Shipyard README.md was renamed to README.rst for
2 reasons:
* To fix an issue with [1] breaking.
* To update content for OpenStack migration.
[0] I32d22110749b334d1fbf19f910e41ab0b7ff3a16
[1] http://logs.openstack.org/96/571296/1/check/airship-shipyard-pep8/76f6cb1/job-output.txt.gz#_2018-05-30_20_19_47_616403
Change-Id: Ie15f4234d504223e961c525339c87fca3a883ffc
Adds the validation of the existence of a deployment
configuration and a deployment strategy document to
the checks before submitting a site action for
processing.
Change-Id: I61bf67759bd919dcc31208370cb1be5a777baf54
Refactor Shipyard to be better able to leverage common
packages and conform with the target UCP standard layout.
This change supports the same tox entrypoints at
the root level, but the preferred approach is to use make
targets defined in the Makefile such as 'make tests' and
'make lint'
The previous tox.ini has moved and been
tailored to the specifics of each subproject at
src/bin/*/tox.ini
Autotmatic generation of the policy and configuration
files has been removed from the sphinx build for now
but these files will be automatically generated locally
into the docs source by using a 'make docs' command.
This may need to be revisited later to re-enable the
automatic generation of these files such that readthedocs
would still support the project layout.
Change-Id: Ifdc1cd4cf35fb3c5923414c677b781a60a9bae42
This reduces the number of layers built by the Dockerfile and puts
more of hte dependency management into a common place.
Change-Id: I416ac11749f3fe8803919994e807f46d64493b5a
Airflow version 1.9.0 was released a few weeks ago
This Patch Set will bump the version from 1.8.2 to 1.9.0
Change-Id: Ief9e33f64d0bacea4628dedc1a94342f60cfecea
This PS migrates the Shipyard and Airflow Dockerfiles
into this repo and adds a Makefile
We will run the following command from the root directory
to build the airflow and shipyard images
$ sudo make build_airflow
$ sudo make build_shipyard
Change-Id: I9a9fb761ce193b1c5b9c5d9589982366eb73e396
Sergiy Markin