Commit Graph

32 Commits

Author SHA1 Message Date
Sergiy Markin bd67c08136 Disable airflow webserver UI by default
This PS disables Airflow Webserver UI by default.

Change-Id: Ic4734e8159db8b44675fcbced2abb1858f1813fc
2024-03-19 18:01:29 +00:00
Sergiy Markin 6fc09a6713 Kubeadm based Airskiff gate
This PS replaces old airskiff integration gate
with new kubeadm based airskiff integration gate.
The main goal of this gate is to test new Shipyard
and Airflow images and chart together with other
Airship components.

Change-Id: I7a2df36a89f92f720eabbc5783b935aeeebb3afc
2024-03-13 16:17:58 +00:00
Sergiy Markin 580c42e0ac Airflow stable 2.8.2
This PS bumps up Airflow version to the latest
2.8.2 and also bumps up openstack dependences to
Antelope 2023.1

Change-Id: I7bbcbc8f9e6334100f47702f4546f0963d11b2ca
2024-02-27 19:04:31 +00:00
Sergiy Markin 20593bc746 Airflow stable 2.8.1
Change-Id: Iac800d9bc6e53a5a60e11c649d05dc663e8dd8e9
2024-02-23 19:43:03 +00:00
Anselme, Schubert (sa246v) f3ea82daec Deprecating the Ingress Class Annotation
Upgrading htk to version 0.2.55, which deprecates the ingress class
annotation (kubernetes.io/ingress.class) with .spec.ingressClassName

https://review.opendev.org/c/openstack/openstack-helm-infra/+/891720

Change-Id: I392b9ebaebc696ae07a97b0a5d34c1a433b442ae
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
2023-10-05 22:56:53 +00:00
Sergiy Markin 81066ae98f Airflow stable 2.6.2
This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:

- bionic py36 gates  were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
  airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
  items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
  schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
  new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2

Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
2023-08-30 16:04:47 +00:00
Sergiy Markin 7ea83c25ef Restored ubuntu_bionic image build
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.

Change-Id: I4c4c7186e67c771f8249b2017b2507aac17760b7
2023-05-30 22:39:59 +00:00
Sergiy Markin 154a099b28 Shipyard upgrade for focal
- upgraded Airflow to 1.10.15  -
  https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html
- disabled xenial, bionic and opensuse images build  gates
- added focal image build  gate
- added focal zuul build node
- adjusted Makefile for focal
- added bindep.txt to utilize bindep zuul base role for zuul build node
  pre-setup
- added focal Dockerfile
- implemented freeze requirements.txt approach like in other Airship
  projects
- removed specific requirements.txt for airflow in favor of using
  requirements-frozen.txt from shipyard_airflow project when building
  airflow docker image
- fixed docker image publishing to Quay
- replaces deprecated LOG.warn with new LOG.warning call
- replaced deprecated body attribute in responce wiht responce.text
  attribute
- update of falcon module deprecated .API call - replaced wiht
  falcon.App call
- deprecated routing.create_http_method_map method replaced with
  routing.map_http_methods
- re-formatted code tabulations based on yapf recommendations
- replaced deprecated protocol attribute in Pytest create_environ() with
  http_version attribute
- replaced deprecated app attribute in Pytest create_environ() with
  root_path attribute
- fixed airflow CLI commands to match 1.10.15 version
- updated zuul gates to work on focal nodes and added focal specific
  node setup items by adding appriate ansible tasks and roles
- uplifted Helm to 3.9.4
- uplifted stable HTK commit id
- updated tox.in to with with tox v4
- uplifted dependences references to other Airship projects
- common python dependences were syncronized with other Airship
  projects(Promenade, Deckhand, Armada, Drydock)
- fixed airskiff deployment gate
- fixed genconfig* profiles in shipyard-airflow tox.ini responsible for
  maintanance of policy.yaml.sample and shipyard.conf.sample

Change-Id: I0c85187dc9bacf0849382563dd5ff7e9b2814c59
2023-04-28 20:40:50 +00:00
Phil Sphicas 0b985d60a9 Gate fixes: remove py35, make xenial non-voting
Update Zuul config and tox settings to not run python3.5 jobs, and to
make the xenial image jobs non-voting.

Change-Id: Icc9f71e60e6b48254917ea1d41ce9bc823cce512
2021-10-18 12:43:13 -07:00
anthony.bellino 00200ef98d Gate fixes for Shipyard
* Install older version of pip<21.0 for ubuntu_xenial images
* Install setuptools via pip for ubuntu_xenial images
* Pin typing-extensions to 3.7.2 and apache-airflow to 1.10.5
* Move promenade Dependencies under UCP components in requirements.txt
* With apache-airflow=1.10.5, strip ANSI escape sequences
  in test_deployment_group_manager.py
* Update tox.ini to support apache-airflow=1.10.5
* airskiff gate fixes
  - Pin treasuremap to v1.9 branch
  - Pin openstack-helm-infra to master
  - Remove openstack client setup as it's not used

Change-Id: Iee4ce59fdceacb165120a69d11c44e6e47feaea8
2021-06-25 13:15:31 -07:00
Ahmad Mahmoudi fb9cc9a65e (fix) - updated to pip3
Updated to pip to pip3 to address zuul gate issue.
Set the ensure_global_symlinks to true for zuul-jobs to set
symlink for tox path."

Change-Id: I1b0634ef18328bccc4d6929072f53db779d70ef1
2020-06-18 19:51:47 +00:00
Ahmad Mahmoudi 24f6a01e0b Add support for Ubuntu bionic base image
Added support to buid shipyard and airflow images using either a xenial
or Ubuntu bionic base image.
The default base image is set to bionic.

Change-Id: I6ad4d42dede081586b3ccea87a42e250979ac106
2020-02-04 13:38:39 -06:00
Michael Beaver 6fa4ce6ee0 Remove required-projects from Airskiff gate
This removes several of the the required-projects from the Airskiff
gate as the scripts for the gate can already clone down the necessary
repositories and the required-projects can overwrite any pins that may
have been included in the Airskiff scripts.

Change-Id: Ia70169bc89b493562e4a0db9d703a46fb2d2f87d
2019-06-08 00:42:08 +00:00
Kaspars Skels a9181dcc60 Encrypt git mirroring ssh_key to specific project
Change-Id: Icc1e3077a9e0efcfd53e92eab325fdfe264abb65
2019-05-23 13:08:56 -05:00
Kaspars Skels 3bce3c41ca Add Zuul job for mirroring to GitHub
Change-Id: I318020a92d0d81fa8fb206ee9e7366cc8982255e
2019-05-22 16:21:20 -05:00
Andreas Jaeger 8bfa323814 Fix rtd publishing
The trigger URL is [0].

The tooling sets rtd_project_name to "airmada" but the project is
named "airship-xxxx" in RTD. Introduce rtd_project_name and set it to
correct value.

See also [1].

[0] https://readthedocs.org/api/v2/webhook/{{ rtd_project_name }}/{{ rtd_webhook_id }}
[1] https://zuul-ci.org/docs/zuul-jobs/general-roles.html#rolevar-trigger-readthedocs.rtd_project_name

Change-Id: I57ab07ac8b5cf891437b41cea74b09640df026be
2019-05-17 16:04:43 +02:00
Arun Kant cd6f154da5 Adding opensuse support in image building of airflow and shipyard
Adding DISTRO parameter for makefile to invoke distribution specific
Dockerfile and build image accordingly.

Also qualifying image job name to distinguish ubuntu vs opensuse
check jobs and passing distro specific variable.
Similar logic for pushing distro specific images.

Added new document section for multiple distro support.

Change-Id: I215f8a107cab8770181c0472e0cc29053b2b978e
2019-05-14 09:52:24 -07:00
Drew Walters 5556227634 CI: Add Airskiff check
This commit adds a non-voting Zuul job that deploys Memcached using the
Airship-Treasuremap airskiff site and submitted Shipyard changes.

[0] https://airship-treasuremap.readthedocs.io/en/latest/airskiff.html

Change-Id: Ibe9c6c94406bcdac4c39493b694535baf8ad2719
2019-04-22 16:39:58 +00:00
Drew Walters 95179f53ff CI: Add chart build job for latest Helm toolkit
This commit introduces a non-voting job to lint Helm charts against the
latest version of Helm toolkit from OpenStack-Helm Infra. This job
should serve as an indicator of when it's safe to advance the version of
Helm toolkit used by Airship.

Additionally, this commit modifies all Helm chart lint jobs to run on
each commit, regardless of the files modified by a change. This should
not introduce a noticeable difference in CI runtime, as these jobs
execute quicker than the tox jobs.

Change-Id: Ife3b285a2f9a160df9724e9dcb230243d06be3ab
2019-04-03 14:26:37 +00:00
Drew Walters 5a32c2d1fd CI: Add chart build gate
This commit introduces a chart build gate that triggers when changes are
made to charts in the charts/ directory.

Change-Id: Iec1c6da785d26e2e7273b37fd1028f3bf47ec199
2019-01-28 11:27:26 -06:00
Drew Walters ebf4ee4785 CI: Remove OpenStack-Helm-Infra CI job
Currently, the airship-shipyard-ubuntu job deploys K8s using outdated
playbooks. The job fails due to a change [0] that introduced an AppArmor
requirement to Calico. This change removes the failing job, as its
success does not depend on changes in Shipyard.

In the future, jobs requiring K8s should be deployed using the new
Minikube gate scripts from the OpenStack-Helm infra repository. They
will be added in future changes once they are required. Additionally, a
future change will introduce a Helm linting gate for charts hosted in
the Shipyard repository.

[0] https://review.openstack.org/614805
[1] https://git.openstack.org/cgit/openstack/openstack-helm-infra/tree/tools/deployment/common/005-deploy-k8s.sh

Change-Id: I72aaff9050eab844ef0856794b059de14e2a4a28
2019-01-28 09:20:17 -06:00
Roman Gorshunov 1c08239287 Zuul gates cleanup and fixes
1. Removed osh-infra-upgrade-host playbook from image building jobs,
   as non-relevant
2. Removed osh-infra-deploy-docker playbook from image building jobs,
   as docker-ce is deployed in run-image playbook
3. Removed osh-infra-build, osh-infra-deploy-k8s and security-bandit
   playbooks - they are not referenced anywhere; bandit has already
   been merged with pep8 job

Change-Id: I27f0fd8f346d0b52c55870b6005be525a1fec671
2018-11-12 15:13:56 +00:00
Felipe Monteiro d3b282468f zuul: Use zuul templates for unit tests
This patch set uses zuul templates for python35 and python36
because:

1) python35 template is maintained by project-config; don't need
   to maintain separately in SY
2) python36 is now what SY will be building off in the Dockerfile
   so it is better to also test this actively in CICD

Change-Id: I98230b51db69f1054d11428f48bd09dc2547f8b6
2018-10-23 17:58:51 -04:00
Roman Gorshunov 785c4ca5f5 Set up publishing of docs
Set up publishing of docs to the readthedocs.

Change-Id: Idfafa228e9136de3cec72d9df82b537ebd8fc8d3
2018-09-14 21:32:41 +02:00
Gage Hugo 0de9ffcbf2 Consolidate pep8/bandit zuul gating
This change adds the global zuul pep8 tox job, which runs both
bandit and pep8 using tox. This also removes the two other airship
specific lint-pep8 and bandit zuul jobs since they are both covered
by the default openstack global one.

Change-Id: Iebea5b872f78762d6f401b574d53965b2e1c090b
2018-08-21 20:53:24 +00:00
Tin Lam 87ae709fb2 fix(gate): fix airship-shipyard-ubuntu job
Change-Id: I146c57bf39c7588eca3df486b4505584c333bf74
Signed-off-by: Tin Lam <tin@irrational.io>
2018-07-06 04:17:13 +00:00
Scott Hussey c4b6c179e9 (Zuul) Recreate quay credentials
- Recreate quay credential secrets with stripped
  whitespace

Change-Id: I4de4bbbdfebd6c55f9ed82a25c78dc302e3897ee
2018-06-07 21:41:50 -05:00
Scott Hussey 5b1189db6e (zuul) Seperate publish job
- It seems that parent job cannot utilize
  secrets defined in the child job

Change-Id: I1c2b0077cab0ef5b6ef6c32e0d3b7c8ff3f54f4f
2018-06-07 16:13:42 -05:00
anthony.lin 8a393dfc80 Add docker image publishing
1) Refactor shipyard image build
2) Add shipyard docker image publishing

Change-Id: I1a007ae21a88d6468bf42724e5def5cd04ebe48e
2018-06-07 14:39:38 -05:00
Scott Hussey 4d0bbb546b (zuul) Add a bandit scanning gate
- Add a gate for bandit code scanning

Change-Id: I27bf7e6fb5ddc5bf854796c4d15ef0e40eebfbff
2018-05-31 16:21:20 -05:00
anthony.lin b87569abe8 Add tox tests to .zuul.yaml
This patch set adds tox tests to .zuul.yaml

Prometheus scrapers are not running in the airship-shipyard
and hence removed to prevent airship-shipyard-ubuntu job
from failing.

Change-Id: Ib805946ea76664b7448e02b4c7e4adb40c4dbaf2
2018-05-25 10:59:40 +08:00
Pete Birley ec5bb5ac2c Zuul: Initial Airship-Shipyard checks
This PS adds the skeleton for a set of zuul checks and gates for
Airship, using the framework from OpenStack-Helm.

Change-Id: I620f75dd0d450168f5781a870ef8f29d4fd3a0cb
Signed-off-by: Pete Birley <pete@port.direct>
2018-05-17 20:48:29 +01:00