Commit Graph

807 Commits

Author SHA1 Message Date
Sergiy Markin bd67c08136 Disable airflow webserver UI by default
This PS disables Airflow Webserver UI by default.

Change-Id: Ic4734e8159db8b44675fcbced2abb1858f1813fc
2024-03-19 18:01:29 +00:00
Sergiy Markin 6fc09a6713 Kubeadm based Airskiff gate
This PS replaces old airskiff integration gate
with new kubeadm based airskiff integration gate.
The main goal of this gate is to test new Shipyard
and Airflow images and chart together with other
Airship components.

Change-Id: I7a2df36a89f92f720eabbc5783b935aeeebb3afc
2024-03-13 16:17:58 +00:00
Sergiy Markin 220d1a5936 Version fix in dockerfile
Change-Id: Ib3786729dcb6e55c3f0e578a8fb1afd1c79bdbc9
2024-03-04 17:01:29 +00:00
Sergiy Markin 580c42e0ac Airflow stable 2.8.2
This PS bumps up Airflow version to the latest
2.8.2 and also bumps up openstack dependences to
Antelope 2023.1

Change-Id: I7bbcbc8f9e6334100f47702f4546f0963d11b2ca
2024-02-27 19:04:31 +00:00
Sergiy Markin 20593bc746 Airflow stable 2.8.1
Change-Id: Iac800d9bc6e53a5a60e11c649d05dc663e8dd8e9
2024-02-23 19:43:03 +00:00
Ritchie, Frank (fr801x) 84cd4e524d Update helm toolkit reference
Update helm toolkit reference

Update helm toolkit ref so that ingress will be created with a pathType
of Prefix.

https: //review.opendev.org/c/openstack/openstack-helm-infra/+/905757
Change-Id: I233194b2bdda8b8f909592cd1c1f7cffe135a315
2024-01-23 17:21:23 -05:00
Anselme, Schubert (sa246v) 2a6c028a41
Enable TLS for celery
This PS enable TLS connection from celery to rabbitmq
when TLS connection is enabled

Change-Id: I49ccf159ca73e0764703a6d3c686c108143f12e2
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
2023-12-06 15:48:29 -05:00
Anselme, Schubert (sa246v) f571611f3c
Enable TLS connection to rabbitmq
Change-Id: Ia4d65393ad0112ae63433a7f67a8c8706a15b216
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
2023-11-20 10:42:30 -05:00
Anselme, Schubert (sa246v) f3ea82daec Deprecating the Ingress Class Annotation
Upgrading htk to version 0.2.55, which deprecates the ingress class
annotation (kubernetes.io/ingress.class) with .spec.ingressClassName

https://review.opendev.org/c/openstack/openstack-helm-infra/+/891720

Change-Id: I392b9ebaebc696ae07a97b0a5d34c1a433b442ae
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
2023-10-05 22:56:53 +00:00
Sergiy Markin 181da83d34 Airflow webserver UI
This PS adds deployment with Airflow webserver UI in viewer mode
protected by base http autorization and exposed via ingress.

Change-Id: I5692eecf5a9af2930f8cce98b7a1e430f26b5a1b
Signed-off-by: Sergiy Markin <smarkin@mirantis.com>
2023-09-25 22:01:00 +00:00
Sergiy Markin 1177707fae Airflow fix #2
This PS fixes one more SubDAG, makes sure that decide_airflow_upgrade
task works correctly after we previously replaced armada_build SubDAG
with corresponding TaskGroup. Also a related unit test has been fixed.

Change-Id: Ic8304cc1985f8197d363861c588b8b48e13fe5d4
2023-09-15 22:37:22 +00:00
Sergiy Markin 99c2da745a Airflow fix
This PS is mainly fixing SubDAGs timing issues when they
started along with main DAG and not at the time the main
DAG needs them. In Airflow 2.6.2 SubDAGs are deprecated
in favor of TaskGroups. So in this PS all SubDAGS were
replaced with TaskGroups.

Also task level logging config was extended by adding
py-console to obtain logs from tasks like it was
configured in Airflow 1.10.5.

Change-Id: I3f6d3961b1511e3b7cd2f7aab9810d033cfc14a3
2023-09-15 19:14:39 +00:00
Sergiy Markin 1ba0e72628 Shipyard fix
This PS has to fix select tasks by id action.

Change-Id: I6503a48fb43793b2aae91dc7e5549f6bc3818d4a
2023-08-31 19:40:13 +00:00
Sergiy Markin 81066ae98f Airflow stable 2.6.2
This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:

- bionic py36 gates  were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
  airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
  items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
  schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
  new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2

Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
2023-08-30 16:04:47 +00:00
Ritchie, Frank (fr801x) 3fcc6e0d65 Use helm-toolkit for worker readiness probes
Use helm-toolkit for worker readiness probes.

Change-Id: Ice6c2ff11e007c28093d051c118d733c4e757788
2023-08-25 15:57:48 +00:00
Sergiy Markin f0a24b11a0 Shipyard updates
This PS makes the following changes:

- makes sure the airskiff-deploy playbook is using 80Gb partition if
  available
- adds available security updates to docker images

Change-Id: Ibcb8fcddb319c3032483fdb73b55b725ea7737d4
2023-08-24 14:47:04 +00:00
Sergiy Markin 7ea83c25ef Restored ubuntu_bionic image build
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.

Change-Id: I4c4c7186e67c771f8249b2017b2507aac17760b7
2023-05-30 22:39:59 +00:00
Sergiy Markin 9c28c832dd Shipyard timeout issue
This PS  adds default values for chart values and resolves some issues
in python code that utilizes these values:

      validation_connect_timeout: 20
      validation_read_timeout: 300
      deckhand_client_connect_timeout: 20
      deckhand_client_read_timeout: 300
      drydock_client_connect_timeout: 20
      drydock_client_read_timeout: 300

Change-Id: Ic5b1920257859239613a3ce77134e6b05bd7e9dd
2023-05-16 20:37:29 +00:00
Sergiy Markin 296853f159 Uplift airship modules
This PS uplifts airship dependent modules.

Change-Id: Ib17cd12c6b88fbb618af68a2112e315c2cc9c0fd
2023-05-10 13:19:44 +00:00
Sergiy Markin b7e7d25308 Rollback apache-airflow back to 1.10.5
This PS is created to :
- roll back apache-airflow back to 1.10.5
- change default chart values from bionic to ubuntu_focal
- save freshly generated config and policy files samples
- in focal Dockerfile postgres client version is pubped up to v15
- change airflow docker image requirements from limited to fully frozen for shipyard-airflow project
- adjusted requirements-direct.txt for shipyard_airflow
- adjusted requirements-direct.txt for shipyard_client
- regenerated requirements-frozen for both projects
- fixed unit tests after upgrading click module
- gen_all tox profile processing has been moved over to py8 gate because it requires focal zuul node to run
- upgraded airskiff gate playbook to include latest treasuremap patchset with updated airskiff site admada manifests

Change-Id: I47e44f5cfa19b2649697e7cc5a31557a6f4fcfea
2023-05-08 21:58:34 +00:00
Sergiy Markin 154a099b28 Shipyard upgrade for focal
- upgraded Airflow to 1.10.15  -
  https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html
- disabled xenial, bionic and opensuse images build  gates
- added focal image build  gate
- added focal zuul build node
- adjusted Makefile for focal
- added bindep.txt to utilize bindep zuul base role for zuul build node
  pre-setup
- added focal Dockerfile
- implemented freeze requirements.txt approach like in other Airship
  projects
- removed specific requirements.txt for airflow in favor of using
  requirements-frozen.txt from shipyard_airflow project when building
  airflow docker image
- fixed docker image publishing to Quay
- replaces deprecated LOG.warn with new LOG.warning call
- replaced deprecated body attribute in responce wiht responce.text
  attribute
- update of falcon module deprecated .API call - replaced wiht
  falcon.App call
- deprecated routing.create_http_method_map method replaced with
  routing.map_http_methods
- re-formatted code tabulations based on yapf recommendations
- replaced deprecated protocol attribute in Pytest create_environ() with
  http_version attribute
- replaced deprecated app attribute in Pytest create_environ() with
  root_path attribute
- fixed airflow CLI commands to match 1.10.15 version
- updated zuul gates to work on focal nodes and added focal specific
  node setup items by adding appriate ansible tasks and roles
- uplifted Helm to 3.9.4
- uplifted stable HTK commit id
- updated tox.in to with with tox v4
- uplifted dependences references to other Airship projects
- common python dependences were syncronized with other Airship
  projects(Promenade, Deckhand, Armada, Drydock)
- fixed airskiff deployment gate
- fixed genconfig* profiles in shipyard-airflow tox.ini responsible for
  maintanance of policy.yaml.sample and shipyard.conf.sample

Change-Id: I0c85187dc9bacf0849382563dd5ff7e9b2814c59
2023-04-28 20:40:50 +00:00
Ruslan Aliev e3e71f7d16 Bump airship dependencies
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I3bc287af7a89410b7e7e248673a854477360f9ac
2022-09-28 09:51:39 -05:00
Zuul aba1bb1355 Merge "[airflow] Bumping up kubectl version to 1.24.6" 2022-09-26 21:16:03 +00:00
Zuul f4142cb03f Merge "[shipyard] Postgres upgrade to 14.5" 2022-09-26 21:16:02 +00:00
Zuul 34ac308ff2 Merge "[helm-toolkit] Update to latest helm-toolkit" 2022-09-26 21:12:02 +00:00
Markin, Sergiy (sm515x) a3d60cfbc8 [shipyard] Fix pyOpenSSL issue in xenial images
Change-Id: I4c57f92b56a2956497bf6466e23afdfb9af8080f
2022-09-23 16:32:06 +00:00
Markin, Sergiy (sm515x) e79867956b [airflow] Bumping up kubectl version to 1.24.6
Change-Id: I93613f02a4d86e8776c64b6f7756f21bc1cb95ff
2022-09-23 14:25:28 +00:00
Markin, Sergiy (sm515x) ed5775c0ed [shipyard] Postgres upgrade to 14.5
This PS upgrades shipyard postgres to 14.5

Change-Id: I1524cca569015aa7c4068fbaa6cf9b3221821941
2022-09-23 13:14:12 +00:00
Markin, Sergiy (sm515x) 2fb7b5371b [helm-toolkit] Update to latest helm-toolkit
Change-Id: Ia6cc049d2f1526b8e6bc13f98c2867d6d9e39ee7
2022-09-23 01:13:26 +00:00
Markin, Sergiy (sm515x) ad19720444 [zuul] Added a flush node firewall task to Airskiff gates
Airskiff gates are setting up a k8s cluster by minikube, so test node needs a clear firewall rules in order to let calico to get initialized. So a task that flushes firewall rules and  deploy-package role have been added to ansible in gates.

Change-Id: I19f9c8a28b394fa239313bda8b008d79cc048469
2022-09-22 23:18:46 +00:00
Markin, Sergiy (sm515x) d49d4ce52b [openssl] Bump up version of OpenSSL in requirements
OpenSSL has been updated to 22.0.0 in order to image build process.

Change-Id: I8f3a81439c7fea47d5769ed343a20d273052e4bf
2022-09-22 21:56:14 +00:00
Ruslan Aliev 8240608fe5 Loose pyyaml requirements
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I2a3cf7b3af74db3331329931761b5686988f0ac0
2022-07-14 15:12:47 -05:00
Ruslan Aliev 9a8218aa25 Bump airship dependencies
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I61d65874864f233372df72787cdb25a54b2edbc3
2022-07-07 14:14:51 -05:00
Phil Sphicas 7dcada80f1 Uplift Armada client, enable socket keepalives
Connections from airflow-worker can be long-lived (e.g. when applying
large Armada manifests), and are prone to disconnection due to network
inactivity.

This change uplifts the Armada client to enable socket keepalives in
order to avoid this issue. (Promenade is also uplifted to address a
requirements conflict.)

Change-Id: I067a5edbbdff06f45394e772ebb16d5012987b6c
2022-03-21 13:54:38 -07:00
Chris Wedgwood 034b906dd6 [airflow] fix ordering of affinity/psp in charts
Change-Id: I08afee40ac3965adc2bcff7f1a2a21ebb39fb87c
2021-11-03 14:08:25 -05:00
Phil Sphicas aa99869546 Update HTK stable commit (Ingress)
Update helm-toolkit to include updated Ingress templates that use
current API versions (i.e. networking.k8s.io/v1)

https://review.opendev.org/c/openstack/openstack-helm-infra/+/813115

Change-Id: I7878ec1ecfe63e84155e69b0902d5c0612c867da
2021-10-18 12:43:13 -07:00
Phil Sphicas 0b985d60a9 Gate fixes: remove py35, make xenial non-voting
Update Zuul config and tox settings to not run python3.5 jobs, and to
make the xenial image jobs non-voting.

Change-Id: Icc9f71e60e6b48254917ea1d41ce9bc823cce512
2021-10-18 12:43:13 -07:00
Sean Eagan e64f17b91b Remove Helm 2 based Armada integrations
- Tiller status check
- Test cleanup arg removed

Depends-On: https://review.opendev.org/c/airship/armada/+/812047
Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: I77ef3fb8e952ad28132e3476138d34bbb5a6fd3d
2021-10-04 15:03:04 -05:00
Sean Eagan a0da68409a Use helm 3 in chart build
`helm serve` is removed in helm 3 so this moves
to using local `file://` dependencies [0] instead.

[0]: https://helm.sh/docs/chart_best_practices/dependencies/#repository-urls

Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: Ibdea9f1ccda043259620add022643e5c9800259a
2021-10-01 11:48:37 -05:00
Sean Eagan 27b4dc952c Helm 3: Fix Job labels
See the dependency below for details.

Depends-On: https://review.opendev.org/c/openstack/openstack-helm-infra/+/811826
Change-Id: I100a68eb4cf457fba0783e41779f9fdc2c8daf78
2021-10-01 11:21:52 -05:00
anthony.bellino 3a27007e8a (zuul) Fix Shipyard Post Gates
Change-Id: I9f6f633399069f28c47c381faf4230e4f22276e2
2021-09-10 13:13:38 -07:00
Maximilian Weiss 88286bc690 Gate fixes for Shipyard
* Fixed typo for jsonschema version

* Setuptools version forced to <58 to allow usage of use_2to3, which
  some dependencies require

* Fixed scripting error that allowed Airflow installation to silently
  fail

Change-Id: I237801488795df07f2d4de1cde6cd4ec41182e31
2021-09-09 20:23:05 +00:00
Maximilian Weiss ac236ef52a Update HTK stable commit to 0.2.19
Update helm-toolkit stable commit to merge of this change:
https://review.opendev.org/c/openstack/openstack-helm-infra/+/802925

Change-Id: I07e82486c901b558ab2f9619d5663593305c504f
2021-08-17 17:50:35 +00:00
Maximilian Weiss 99e253092a Update helm installation script
Updates the helm installation script to download and install v2.17.0
from get.helm.sh (instead of v2.16.9 from storage.googleapis.com).

Change-Id: Ib08d39cec82c850b2308880f92f268e4cbf8cb66
2021-08-17 17:50:35 +00:00
anthony.bellino 00200ef98d Gate fixes for Shipyard
* Install older version of pip<21.0 for ubuntu_xenial images
* Install setuptools via pip for ubuntu_xenial images
* Pin typing-extensions to 3.7.2 and apache-airflow to 1.10.5
* Move promenade Dependencies under UCP components in requirements.txt
* With apache-airflow=1.10.5, strip ANSI escape sequences
  in test_deployment_group_manager.py
* Update tox.ini to support apache-airflow=1.10.5
* airskiff gate fixes
  - Pin treasuremap to v1.9 branch
  - Pin openstack-helm-infra to master
  - Remove openstack client setup as it's not used

Change-Id: Iee4ce59fdceacb165120a69d11c44e6e47feaea8
2021-06-25 13:15:31 -07:00
Rick Bartra 35e5a7f796 Update pip package versions in preparation of pip 20.3
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].

These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.

Change-Id: Idd9ea0d57b5be063b133036cfc9ebaa69956f4fc
2020-09-30 01:19:51 +00:00
Zuul fb1c281b83 Merge "Accelerate YAML operations with LibYAML" 2020-09-28 17:12:04 +00:00
Zuul 24d301851a Merge "Include LibYAML in container builds" 2020-09-28 17:01:55 +00:00
Zuul 19a11a7766 Merge "Fix airflow quicktest error" 2020-09-28 17:01:54 +00:00
Andrii Ostapenko af63252e8d
Change helm-toolkit dependency version to ">= 0.1.0"
Since we introduced chart version check in gates, requirements are not
satisfied with strict check of 0.1.0

Change-Id: I0b52182baa9a0541b4e79a66e64829f2619e91b5
2020-09-24 19:43:13 -05:00