This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2
Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
This PS makes the following changes:
- makes sure the airskiff-deploy playbook is using 80Gb partition if
available
- adds available security updates to docker images
Change-Id: Ibcb8fcddb319c3032483fdb73b55b725ea7737d4
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.
Change-Id: I4c4c7186e67c771f8249b2017b2507aac17760b7
This PS is created to :
- roll back apache-airflow back to 1.10.5
- change default chart values from bionic to ubuntu_focal
- save freshly generated config and policy files samples
- in focal Dockerfile postgres client version is pubped up to v15
- change airflow docker image requirements from limited to fully frozen for shipyard-airflow project
- adjusted requirements-direct.txt for shipyard_airflow
- adjusted requirements-direct.txt for shipyard_client
- regenerated requirements-frozen for both projects
- fixed unit tests after upgrading click module
- gen_all tox profile processing has been moved over to py8 gate because it requires focal zuul node to run
- upgraded airskiff gate playbook to include latest treasuremap patchset with updated airskiff site admada manifests
Change-Id: I47e44f5cfa19b2649697e7cc5a31557a6f4fcfea
- upgraded Airflow to 1.10.15 -
https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html
- disabled xenial, bionic and opensuse images build gates
- added focal image build gate
- added focal zuul build node
- adjusted Makefile for focal
- added bindep.txt to utilize bindep zuul base role for zuul build node
pre-setup
- added focal Dockerfile
- implemented freeze requirements.txt approach like in other Airship
projects
- removed specific requirements.txt for airflow in favor of using
requirements-frozen.txt from shipyard_airflow project when building
airflow docker image
- fixed docker image publishing to Quay
- replaces deprecated LOG.warn with new LOG.warning call
- replaced deprecated body attribute in responce wiht responce.text
attribute
- update of falcon module deprecated .API call - replaced wiht
falcon.App call
- deprecated routing.create_http_method_map method replaced with
routing.map_http_methods
- re-formatted code tabulations based on yapf recommendations
- replaced deprecated protocol attribute in Pytest create_environ() with
http_version attribute
- replaced deprecated app attribute in Pytest create_environ() with
root_path attribute
- fixed airflow CLI commands to match 1.10.15 version
- updated zuul gates to work on focal nodes and added focal specific
node setup items by adding appriate ansible tasks and roles
- uplifted Helm to 3.9.4
- uplifted stable HTK commit id
- updated tox.in to with with tox v4
- uplifted dependences references to other Airship projects
- common python dependences were syncronized with other Airship
projects(Promenade, Deckhand, Armada, Drydock)
- fixed airskiff deployment gate
- fixed genconfig* profiles in shipyard-airflow tox.ini responsible for
maintanance of policy.yaml.sample and shipyard.conf.sample
Change-Id: I0c85187dc9bacf0849382563dd5ff7e9b2814c59
* Install older version of pip<21.0 for ubuntu_xenial images
* Install setuptools via pip for ubuntu_xenial images
* Pin typing-extensions to 3.7.2 and apache-airflow to 1.10.5
* Move promenade Dependencies under UCP components in requirements.txt
* With apache-airflow=1.10.5, strip ANSI escape sequences
in test_deployment_group_manager.py
* Update tox.ini to support apache-airflow=1.10.5
* airskiff gate fixes
- Pin treasuremap to v1.9 branch
- Pin openstack-helm-infra to master
- Remove openstack client setup as it's not used
Change-Id: Iee4ce59fdceacb165120a69d11c44e6e47feaea8
Updates Dockerfiles to build the LibYAML library, which can provide much
faster YAML parsing and emitting than the native Python library.
https://pyyaml.org/wiki/LibYAML
Change-Id: I6ebafa1eb2af8841b86b64fd119dcb581e6c18a7
Added support to buid shipyard and airflow images using either a xenial
or Ubuntu bionic base image.
The default base image is set to bionic.
Change-Id: I6ad4d42dede081586b3ccea87a42e250979ac106
Change URLs from git.openstack.org & github.com to opendev.org due to the
migration; wrap multiple LABELs into one.
Change-Id: I240fa6f746bd1e424e5a2b7fd381903c46059ae5
Adding DISTRO parameter for makefile to invoke distribution specific
Dockerfile and build image accordingly.
Also qualifying image job name to distinguish ubuntu vs opensuse
check jobs and passing distro specific variable.
Similar logic for pushing distro specific images.
Added new document section for multiple distro support.
Change-Id: I215f8a107cab8770181c0472e0cc29053b2b978e
Change to use ubuntu base image instead of python
Refactor Shipyard Dockerfile to reduce image size significantly
BREAKING CHANGE: The `make images` PYTHON_BASE_IMAGE arg is now renamed to BASE_IMAGE.
Change-Id: I3338dfbbb91b5514fa4fd205bdfc4136d0abc2e5
This PS enables shipyard to use PBR as rest
of the other OpenStack project.
Currently if shipyard is used as a requirement in another
project, the dependencies for shipyard doesn't get installed.
This PS fixes this bug.
Change-Id: I34d452649a886af9d865d4ad51eeab70399fe395
1) Use OCI Image Specs for labels instead of custom 'commit-id=xxxxx'
or legacy "Label Schema"
2) Fix missing git commit id labels on images (.revision)
3) Add human-readable title (.title) of the image, URL (.url), and
a few other properties (annotations) according to the latest Specs
Change-Id: Iedd64fa70016880d5df241d27d8dad101e10130b
l is to let user customize the base image of the component
by passing FROM=myimage during the build process. This would let any
project leveraging Airship ensure that the base image is matching the
security requirements for that project and still use the same Dockerfile.
This will also ease the control of the /etc/apt/source.list
and thereby the result of apt-get update/upgrade procedure.
2. The above goal is achievable by using docker-ce feature such as:
ARG FROM="defaultbaseimage:xx"
FROM ${FROM}
For this reason, the installation of docker.io in the Zuul gating is beeing
replaced by docker-ce.
3. Third Goal is to bring consistency with the other compoenents leveraging
Helm such as the openstack-helm and potentially use bindep the same way
the LOCI images are to ensure
4. The new syntax in the Dockerfile is still commented out until the associated
image builder have been updated to use docker-ce as they have been for the LOCI
images.
Change-Id: Ife7e1be53c7c139bdc42dee42f0798e83f4fd271
Refactor Shipyard to be better able to leverage common
packages and conform with the target UCP standard layout.
This change supports the same tox entrypoints at
the root level, but the preferred approach is to use make
targets defined in the Makefile such as 'make tests' and
'make lint'
The previous tox.ini has moved and been
tailored to the specifics of each subproject at
src/bin/*/tox.ini
Autotmatic generation of the policy and configuration
files has been removed from the sphinx build for now
but these files will be automatically generated locally
into the docs source by using a 'make docs' command.
This may need to be revisited later to re-enable the
automatic generation of these files such that readthedocs
would still support the project layout.
Change-Id: Ifdc1cd4cf35fb3c5923414c677b781a60a9bae42
As mentioned in [0], the best practice is NOT to use ENV for
DEBIAN_FRONTEND as it will persists in the final image. It
appears that the common consensus is to use ARG instead so
that noninteractive DEBIAN_FRONTEND is set only during build.
[0] https://github.com/moby/moby/issues/4032
Change-Id: I6980e140ee5a51596a2f9ba4e1471e2c98599846
Updates dockerfile to use python 3.5 base image and
reorders to take advantage of image layering caching better
Removes some outdated and unused files
adds .editorconfig file
adds .vscode to gitignore
Change-Id: I08571b7176f0cb9fab42f072326c6c14d86d518a
This P.S. will allow user to execute Shipyard CLI using a
Shipyard docker container.
The updates made to the Dockerfile and entrypoint.sh are
aligned with those in Armada and Drydock.
Change-Id: I59ccd2d41fab19781f51334cb1a69b69a48d3c49
This PS migrates the Shipyard and Airflow Dockerfiles
into this repo and adds a Makefile
We will run the following command from the root directory
to build the airflow and shipyard images
$ sudo make build_airflow
$ sudo make build_shipyard
Change-Id: I9a9fb761ce193b1c5b9c5d9589982366eb73e396
Sergiy Markin