This PS replaces old airskiff integration gate
with new kubeadm based airskiff integration gate.
The main goal of this gate is to test new Shipyard
and Airflow images and chart together with other
Airship components.
Change-Id: I7a2df36a89f92f720eabbc5783b935aeeebb3afc
This PS bumps up Airflow version to the latest
2.8.2 and also bumps up openstack dependences to
Antelope 2023.1
Change-Id: I7bbcbc8f9e6334100f47702f4546f0963d11b2ca
Update helm toolkit reference
Update helm toolkit ref so that ingress will be created with a pathType
of Prefix.
https: //review.opendev.org/c/openstack/openstack-helm-infra/+/905757
Change-Id: I233194b2bdda8b8f909592cd1c1f7cffe135a315
This PS enable TLS connection from celery to rabbitmq
when TLS connection is enabled
Change-Id: I49ccf159ca73e0764703a6d3c686c108143f12e2
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
Upgrading htk to version 0.2.55, which deprecates the ingress class
annotation (kubernetes.io/ingress.class) with .spec.ingressClassName
https://review.opendev.org/c/openstack/openstack-helm-infra/+/891720
Change-Id: I392b9ebaebc696ae07a97b0a5d34c1a433b442ae
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
This PS adds deployment with Airflow webserver UI in viewer mode
protected by base http autorization and exposed via ingress.
Change-Id: I5692eecf5a9af2930f8cce98b7a1e430f26b5a1b
Signed-off-by: Sergiy Markin <smarkin@mirantis.com>
This PS fixes one more SubDAG, makes sure that decide_airflow_upgrade
task works correctly after we previously replaced armada_build SubDAG
with corresponding TaskGroup. Also a related unit test has been fixed.
Change-Id: Ic8304cc1985f8197d363861c588b8b48e13fe5d4
This PS is mainly fixing SubDAGs timing issues when they
started along with main DAG and not at the time the main
DAG needs them. In Airflow 2.6.2 SubDAGs are deprecated
in favor of TaskGroups. So in this PS all SubDAGS were
replaced with TaskGroups.
Also task level logging config was extended by adding
py-console to obtain logs from tasks like it was
configured in Airflow 1.10.5.
Change-Id: I3f6d3961b1511e3b7cd2f7aab9810d033cfc14a3
This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2
Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
This PS makes the following changes:
- makes sure the airskiff-deploy playbook is using 80Gb partition if
available
- adds available security updates to docker images
Change-Id: Ibcb8fcddb319c3032483fdb73b55b725ea7737d4
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.
Change-Id: I4c4c7186e67c771f8249b2017b2507aac17760b7
This PS adds default values for chart values and resolves some issues
in python code that utilizes these values:
validation_connect_timeout: 20
validation_read_timeout: 300
deckhand_client_connect_timeout: 20
deckhand_client_read_timeout: 300
drydock_client_connect_timeout: 20
drydock_client_read_timeout: 300
Change-Id: Ic5b1920257859239613a3ce77134e6b05bd7e9dd
This PS is created to :
- roll back apache-airflow back to 1.10.5
- change default chart values from bionic to ubuntu_focal
- save freshly generated config and policy files samples
- in focal Dockerfile postgres client version is pubped up to v15
- change airflow docker image requirements from limited to fully frozen for shipyard-airflow project
- adjusted requirements-direct.txt for shipyard_airflow
- adjusted requirements-direct.txt for shipyard_client
- regenerated requirements-frozen for both projects
- fixed unit tests after upgrading click module
- gen_all tox profile processing has been moved over to py8 gate because it requires focal zuul node to run
- upgraded airskiff gate playbook to include latest treasuremap patchset with updated airskiff site admada manifests
Change-Id: I47e44f5cfa19b2649697e7cc5a31557a6f4fcfea
- upgraded Airflow to 1.10.15 -
https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html
- disabled xenial, bionic and opensuse images build gates
- added focal image build gate
- added focal zuul build node
- adjusted Makefile for focal
- added bindep.txt to utilize bindep zuul base role for zuul build node
pre-setup
- added focal Dockerfile
- implemented freeze requirements.txt approach like in other Airship
projects
- removed specific requirements.txt for airflow in favor of using
requirements-frozen.txt from shipyard_airflow project when building
airflow docker image
- fixed docker image publishing to Quay
- replaces deprecated LOG.warn with new LOG.warning call
- replaced deprecated body attribute in responce wiht responce.text
attribute
- update of falcon module deprecated .API call - replaced wiht
falcon.App call
- deprecated routing.create_http_method_map method replaced with
routing.map_http_methods
- re-formatted code tabulations based on yapf recommendations
- replaced deprecated protocol attribute in Pytest create_environ() with
http_version attribute
- replaced deprecated app attribute in Pytest create_environ() with
root_path attribute
- fixed airflow CLI commands to match 1.10.15 version
- updated zuul gates to work on focal nodes and added focal specific
node setup items by adding appriate ansible tasks and roles
- uplifted Helm to 3.9.4
- uplifted stable HTK commit id
- updated tox.in to with with tox v4
- uplifted dependences references to other Airship projects
- common python dependences were syncronized with other Airship
projects(Promenade, Deckhand, Armada, Drydock)
- fixed airskiff deployment gate
- fixed genconfig* profiles in shipyard-airflow tox.ini responsible for
maintanance of policy.yaml.sample and shipyard.conf.sample
Change-Id: I0c85187dc9bacf0849382563dd5ff7e9b2814c59
Airskiff gates are setting up a k8s cluster by minikube, so test node needs a clear firewall rules in order to let calico to get initialized. So a task that flushes firewall rules and deploy-package role have been added to ansible in gates.
Change-Id: I19f9c8a28b394fa239313bda8b008d79cc048469
Connections from airflow-worker can be long-lived (e.g. when applying
large Armada manifests), and are prone to disconnection due to network
inactivity.
This change uplifts the Armada client to enable socket keepalives in
order to avoid this issue. (Promenade is also uplifted to address a
requirements conflict.)
Change-Id: I067a5edbbdff06f45394e772ebb16d5012987b6c
Update Zuul config and tox settings to not run python3.5 jobs, and to
make the xenial image jobs non-voting.
Change-Id: Icc9f71e60e6b48254917ea1d41ce9bc823cce512
* Fixed typo for jsonschema version
* Setuptools version forced to <58 to allow usage of use_2to3, which
some dependencies require
* Fixed scripting error that allowed Airflow installation to silently
fail
Change-Id: I237801488795df07f2d4de1cde6cd4ec41182e31
Updates the helm installation script to download and install v2.17.0
from get.helm.sh (instead of v2.16.9 from storage.googleapis.com).
Change-Id: Ib08d39cec82c850b2308880f92f268e4cbf8cb66
* Install older version of pip<21.0 for ubuntu_xenial images
* Install setuptools via pip for ubuntu_xenial images
* Pin typing-extensions to 3.7.2 and apache-airflow to 1.10.5
* Move promenade Dependencies under UCP components in requirements.txt
* With apache-airflow=1.10.5, strip ANSI escape sequences
in test_deployment_group_manager.py
* Update tox.ini to support apache-airflow=1.10.5
* airskiff gate fixes
- Pin treasuremap to v1.9 branch
- Pin openstack-helm-infra to master
- Remove openstack client setup as it's not used
Change-Id: Iee4ce59fdceacb165120a69d11c44e6e47feaea8
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].
These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.
Change-Id: Idd9ea0d57b5be063b133036cfc9ebaa69956f4fc
Since we introduced chart version check in gates, requirements are not
satisfied with strict check of 0.1.0
Change-Id: I0b52182baa9a0541b4e79a66e64829f2619e91b5