This PS replaces old airskiff integration gate
with new kubeadm based airskiff integration gate.
The main goal of this gate is to test new Shipyard
and Airflow images and chart together with other
Airship components.
Change-Id: I7a2df36a89f92f720eabbc5783b935aeeebb3afc
Upgrading htk to version 0.2.55, which deprecates the ingress class
annotation (kubernetes.io/ingress.class) with .spec.ingressClassName
https://review.opendev.org/c/openstack/openstack-helm-infra/+/891720
Change-Id: I392b9ebaebc696ae07a97b0a5d34c1a433b442ae
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
This PS adds deployment with Airflow webserver UI in viewer mode
protected by base http autorization and exposed via ingress.
Change-Id: I5692eecf5a9af2930f8cce98b7a1e430f26b5a1b
Signed-off-by: Sergiy Markin <smarkin@mirantis.com>
This PS is mainly fixing SubDAGs timing issues when they
started along with main DAG and not at the time the main
DAG needs them. In Airflow 2.6.2 SubDAGs are deprecated
in favor of TaskGroups. So in this PS all SubDAGS were
replaced with TaskGroups.
Also task level logging config was extended by adding
py-console to obtain logs from tasks like it was
configured in Airflow 1.10.5.
Change-Id: I3f6d3961b1511e3b7cd2f7aab9810d033cfc14a3
This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2
Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
This PS makes the following changes:
- makes sure the airskiff-deploy playbook is using 80Gb partition if
available
- adds available security updates to docker images
Change-Id: Ibcb8fcddb319c3032483fdb73b55b725ea7737d4
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.
Change-Id: I4c4c7186e67c771f8249b2017b2507aac17760b7
This PS is created to :
- roll back apache-airflow back to 1.10.5
- change default chart values from bionic to ubuntu_focal
- save freshly generated config and policy files samples
- in focal Dockerfile postgres client version is pubped up to v15
- change airflow docker image requirements from limited to fully frozen for shipyard-airflow project
- adjusted requirements-direct.txt for shipyard_airflow
- adjusted requirements-direct.txt for shipyard_client
- regenerated requirements-frozen for both projects
- fixed unit tests after upgrading click module
- gen_all tox profile processing has been moved over to py8 gate because it requires focal zuul node to run
- upgraded airskiff gate playbook to include latest treasuremap patchset with updated airskiff site admada manifests
Change-Id: I47e44f5cfa19b2649697e7cc5a31557a6f4fcfea
- upgraded Airflow to 1.10.15 -
https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html
- disabled xenial, bionic and opensuse images build gates
- added focal image build gate
- added focal zuul build node
- adjusted Makefile for focal
- added bindep.txt to utilize bindep zuul base role for zuul build node
pre-setup
- added focal Dockerfile
- implemented freeze requirements.txt approach like in other Airship
projects
- removed specific requirements.txt for airflow in favor of using
requirements-frozen.txt from shipyard_airflow project when building
airflow docker image
- fixed docker image publishing to Quay
- replaces deprecated LOG.warn with new LOG.warning call
- replaced deprecated body attribute in responce wiht responce.text
attribute
- update of falcon module deprecated .API call - replaced wiht
falcon.App call
- deprecated routing.create_http_method_map method replaced with
routing.map_http_methods
- re-formatted code tabulations based on yapf recommendations
- replaced deprecated protocol attribute in Pytest create_environ() with
http_version attribute
- replaced deprecated app attribute in Pytest create_environ() with
root_path attribute
- fixed airflow CLI commands to match 1.10.15 version
- updated zuul gates to work on focal nodes and added focal specific
node setup items by adding appriate ansible tasks and roles
- uplifted Helm to 3.9.4
- uplifted stable HTK commit id
- updated tox.in to with with tox v4
- uplifted dependences references to other Airship projects
- common python dependences were syncronized with other Airship
projects(Promenade, Deckhand, Armada, Drydock)
- fixed airskiff deployment gate
- fixed genconfig* profiles in shipyard-airflow tox.ini responsible for
maintanance of policy.yaml.sample and shipyard.conf.sample
Change-Id: I0c85187dc9bacf0849382563dd5ff7e9b2814c59
Airskiff gates are setting up a k8s cluster by minikube, so test node needs a clear firewall rules in order to let calico to get initialized. So a task that flushes firewall rules and deploy-package role have been added to ansible in gates.
Change-Id: I19f9c8a28b394fa239313bda8b008d79cc048469
* Install older version of pip<21.0 for ubuntu_xenial images
* Install setuptools via pip for ubuntu_xenial images
* Pin typing-extensions to 3.7.2 and apache-airflow to 1.10.5
* Move promenade Dependencies under UCP components in requirements.txt
* With apache-airflow=1.10.5, strip ANSI escape sequences
in test_deployment_group_manager.py
* Update tox.ini to support apache-airflow=1.10.5
* airskiff gate fixes
- Pin treasuremap to v1.9 branch
- Pin openstack-helm-infra to master
- Remove openstack client setup as it's not used
Change-Id: Iee4ce59fdceacb165120a69d11c44e6e47feaea8
Updated to pip to pip3 to address zuul gate issue.
Set the ensure_global_symlinks to true for zuul-jobs to set
symlink for tox path."
Change-Id: I1b0634ef18328bccc4d6929072f53db779d70ef1
Added support to buid shipyard and airflow images using either a xenial
or Ubuntu bionic base image.
The default base image is set to bionic.
Change-Id: I6ad4d42dede081586b3ccea87a42e250979ac106
The location of the Airskiff site recently changed [0], causing the
Airskiff job to fail. This change updates the "reduce site" playbook to
match the job in treasuremap, thereby avoiding the directory which no
longer exists.
The change also removes the usage of install-packages script, because
it was removed [1].
[0] https://review.opendev.org/674963
[1] https://review.opendev.org/672540
Change-Id: I7b20ff7c50cfa085039e893558df2cf022c4333c
Signed-off-by: Drew Walters <andrew.walters@att.com>
This removes several of the the required-projects from the Airskiff
gate as the scripts for the gate can already clone down the necessary
repositories and the required-projects can overwrite any pins that may
have been included in the Airskiff scripts.
Change-Id: Ia70169bc89b493562e4a0db9d703a46fb2d2f87d
Adding DISTRO parameter for makefile to invoke distribution specific
Dockerfile and build image accordingly.
Also qualifying image job name to distinguish ubuntu vs opensuse
check jobs and passing distro specific variable.
Similar logic for pushing distro specific images.
Added new document section for multiple distro support.
Change-Id: I215f8a107cab8770181c0472e0cc29053b2b978e
This commit introduces a non-voting job to lint Helm charts against the
latest version of Helm toolkit from OpenStack-Helm Infra. This job
should serve as an indicator of when it's safe to advance the version of
Helm toolkit used by Airship.
Additionally, this commit modifies all Helm chart lint jobs to run on
each commit, regardless of the files modified by a change. This should
not introduce a noticeable difference in CI runtime, as these jobs
execute quicker than the tox jobs.
Change-Id: Ife3b285a2f9a160df9724e9dcb230243d06be3ab
- Zuul updated ansible to 2.7, no longer allows missing variables.
- Using default value when it isn't available.
Based on Aaron Sheffield's PS for Pegleg: https://review.openstack.org/#/c/645631/
Change-Id: I02495bc793021b429e0be62ecac0ed45e930484c
This commit introduces a chart build gate that triggers when changes are
made to charts in the charts/ directory.
Change-Id: Iec1c6da785d26e2e7273b37fd1028f3bf47ec199
Currently, the airship-shipyard-ubuntu job deploys K8s using outdated
playbooks. The job fails due to a change [0] that introduced an AppArmor
requirement to Calico. This change removes the failing job, as its
success does not depend on changes in Shipyard.
In the future, jobs requiring K8s should be deployed using the new
Minikube gate scripts from the OpenStack-Helm infra repository. They
will be added in future changes once they are required. Additionally, a
future change will introduce a Helm linting gate for charts hosted in
the Shipyard repository.
[0] https://review.openstack.org/614805
[1] https://git.openstack.org/cgit/openstack/openstack-helm-infra/tree/tools/deployment/common/005-deploy-k8s.sh
Change-Id: I72aaff9050eab844ef0856794b059de14e2a4a28
This change starts setting up mirrored queues by default.
If there is only one rabbitmq pod present this will still
work, though will cause some performance overhead.
Depends-On: https://review.openstack.org/617812
Depends-On: https://review.openstack.org/617817
Change-Id: I8982aed699185f9b7fb4962e108eb76377643f25
Zuul post-merge job fails to run docker login, complaining about
missing docker-py module. This patch adds installation of this module
to the relevant job.
Change-Id: I3523eedd9abb56a9cabbfb72391ed9f1471a4f4c
1. Removed osh-infra-upgrade-host playbook from image building jobs,
as non-relevant
2. Removed osh-infra-deploy-docker playbook from image building jobs,
as docker-ce is deployed in run-image playbook
3. Removed osh-infra-build, osh-infra-deploy-k8s and security-bandit
playbooks - they are not referenced anywhere; bandit has already
been merged with pep8 job
Change-Id: I27f0fd8f346d0b52c55870b6005be525a1fec671
This patch set uses zuul templates for python35 and python36
because:
1) python35 template is maintained by project-config; don't need
to maintain separately in SY
2) python36 is now what SY will be building off in the Dockerfile
so it is better to also test this actively in CICD
Change-Id: I98230b51db69f1054d11428f48bd09dc2547f8b6
1) Use OCI Image Specs for labels instead of custom 'commit-id=xxxxx'
or legacy "Label Schema"
2) Fix missing git commit id labels on images (.revision)
3) Add human-readable title (.title) of the image, URL (.url), and
a few other properties (annotations) according to the latest Specs
Change-Id: Iedd64fa70016880d5df241d27d8dad101e10130b
During the publish of an image post-merge, an error was being rasied due
to a conflict between the ansible-supplied docker-py and the pip
installed docker. This removes the pip installed docker to allow
publishing to proceed.
Change-Id: I6a1ff54ed2d2af85cbe44ed50b1c637dee9adf54
l is to let user customize the base image of the component
by passing FROM=myimage during the build process. This would let any
project leveraging Airship ensure that the base image is matching the
security requirements for that project and still use the same Dockerfile.
This will also ease the control of the /etc/apt/source.list
and thereby the result of apt-get update/upgrade procedure.
2. The above goal is achievable by using docker-ce feature such as:
ARG FROM="defaultbaseimage:xx"
FROM ${FROM}
For this reason, the installation of docker.io in the Zuul gating is beeing
replaced by docker-ce.
3. Third Goal is to bring consistency with the other compoenents leveraging
Helm such as the openstack-helm and potentially use bindep the same way
the LOCI images are to ensure
4. The new syntax in the Dockerfile is still commented out until the associated
image builder have been updated to use docker-ce as they have been for the LOCI
images.
Change-Id: Ife7e1be53c7c139bdc42dee42f0798e83f4fd271
This patchset updates osh-infra-deploy-docker.yaml playbook
to align with role rename change in osh-infra causing pre-run
to fail: https://review.openstack.org/#/c/578703/6
Change-Id: Ie119f953e77fc748cc784b8cd9f509892e55927d
This patch set adds tox tests to .zuul.yaml
Prometheus scrapers are not running in the airship-shipyard
and hence removed to prevent airship-shipyard-ubuntu job
from failing.
Change-Id: Ib805946ea76664b7448e02b4c7e4adb40c4dbaf2
This PS adds the skeleton for a set of zuul checks and gates for
Airship, using the framework from OpenStack-Helm.
Change-Id: I620f75dd0d450168f5781a870ef8f29d4fd3a0cb
Signed-off-by: Pete Birley <pete@port.direct>
Sergiy Markin