Commit Graph

126 Commits

Author SHA1 Message Date
Sergiy Markin 6fc09a6713 Kubeadm based Airskiff gate
This PS replaces old airskiff integration gate
with new kubeadm based airskiff integration gate.
The main goal of this gate is to test new Shipyard
and Airflow images and chart together with other
Airship components.

Change-Id: I7a2df36a89f92f720eabbc5783b935aeeebb3afc
2024-03-13 16:17:58 +00:00
Sergiy Markin 20593bc746 Airflow stable 2.8.1
Change-Id: Iac800d9bc6e53a5a60e11c649d05dc663e8dd8e9
2024-02-23 19:43:03 +00:00
Ritchie, Frank (fr801x) 84cd4e524d Update helm toolkit reference
Update helm toolkit reference

Update helm toolkit ref so that ingress will be created with a pathType
of Prefix.

https: //review.opendev.org/c/openstack/openstack-helm-infra/+/905757
Change-Id: I233194b2bdda8b8f909592cd1c1f7cffe135a315
2024-01-23 17:21:23 -05:00
Anselme, Schubert (sa246v) f3ea82daec Deprecating the Ingress Class Annotation
Upgrading htk to version 0.2.55, which deprecates the ingress class
annotation (kubernetes.io/ingress.class) with .spec.ingressClassName

https://review.opendev.org/c/openstack/openstack-helm-infra/+/891720

Change-Id: I392b9ebaebc696ae07a97b0a5d34c1a433b442ae
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
2023-10-05 22:56:53 +00:00
Sergiy Markin 181da83d34 Airflow webserver UI
This PS adds deployment with Airflow webserver UI in viewer mode
protected by base http autorization and exposed via ingress.

Change-Id: I5692eecf5a9af2930f8cce98b7a1e430f26b5a1b
Signed-off-by: Sergiy Markin <smarkin@mirantis.com>
2023-09-25 22:01:00 +00:00
Sergiy Markin 99c2da745a Airflow fix
This PS is mainly fixing SubDAGs timing issues when they
started along with main DAG and not at the time the main
DAG needs them. In Airflow 2.6.2 SubDAGs are deprecated
in favor of TaskGroups. So in this PS all SubDAGS were
replaced with TaskGroups.

Also task level logging config was extended by adding
py-console to obtain logs from tasks like it was
configured in Airflow 1.10.5.

Change-Id: I3f6d3961b1511e3b7cd2f7aab9810d033cfc14a3
2023-09-15 19:14:39 +00:00
Sergiy Markin 1ba0e72628 Shipyard fix
This PS has to fix select tasks by id action.

Change-Id: I6503a48fb43793b2aae91dc7e5549f6bc3818d4a
2023-08-31 19:40:13 +00:00
Sergiy Markin 81066ae98f Airflow stable 2.6.2
This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:

- bionic py36 gates  were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
  airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
  items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
  schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
  new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2

Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
2023-08-30 16:04:47 +00:00
Sergiy Markin f0a24b11a0 Shipyard updates
This PS makes the following changes:

- makes sure the airskiff-deploy playbook is using 80Gb partition if
  available
- adds available security updates to docker images

Change-Id: Ibcb8fcddb319c3032483fdb73b55b725ea7737d4
2023-08-24 14:47:04 +00:00
Sergiy Markin 7ea83c25ef Restored ubuntu_bionic image build
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.

Change-Id: I4c4c7186e67c771f8249b2017b2507aac17760b7
2023-05-30 22:39:59 +00:00
Sergiy Markin 9c28c832dd Shipyard timeout issue
This PS  adds default values for chart values and resolves some issues
in python code that utilizes these values:

      validation_connect_timeout: 20
      validation_read_timeout: 300
      deckhand_client_connect_timeout: 20
      deckhand_client_read_timeout: 300
      drydock_client_connect_timeout: 20
      drydock_client_read_timeout: 300

Change-Id: Ic5b1920257859239613a3ce77134e6b05bd7e9dd
2023-05-16 20:37:29 +00:00
Sergiy Markin b7e7d25308 Rollback apache-airflow back to 1.10.5
This PS is created to :
- roll back apache-airflow back to 1.10.5
- change default chart values from bionic to ubuntu_focal
- save freshly generated config and policy files samples
- in focal Dockerfile postgres client version is pubped up to v15
- change airflow docker image requirements from limited to fully frozen for shipyard-airflow project
- adjusted requirements-direct.txt for shipyard_airflow
- adjusted requirements-direct.txt for shipyard_client
- regenerated requirements-frozen for both projects
- fixed unit tests after upgrading click module
- gen_all tox profile processing has been moved over to py8 gate because it requires focal zuul node to run
- upgraded airskiff gate playbook to include latest treasuremap patchset with updated airskiff site admada manifests

Change-Id: I47e44f5cfa19b2649697e7cc5a31557a6f4fcfea
2023-05-08 21:58:34 +00:00
Sergiy Markin 154a099b28 Shipyard upgrade for focal
- upgraded Airflow to 1.10.15  -
  https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html
- disabled xenial, bionic and opensuse images build  gates
- added focal image build  gate
- added focal zuul build node
- adjusted Makefile for focal
- added bindep.txt to utilize bindep zuul base role for zuul build node
  pre-setup
- added focal Dockerfile
- implemented freeze requirements.txt approach like in other Airship
  projects
- removed specific requirements.txt for airflow in favor of using
  requirements-frozen.txt from shipyard_airflow project when building
  airflow docker image
- fixed docker image publishing to Quay
- replaces deprecated LOG.warn with new LOG.warning call
- replaced deprecated body attribute in responce wiht responce.text
  attribute
- update of falcon module deprecated .API call - replaced wiht
  falcon.App call
- deprecated routing.create_http_method_map method replaced with
  routing.map_http_methods
- re-formatted code tabulations based on yapf recommendations
- replaced deprecated protocol attribute in Pytest create_environ() with
  http_version attribute
- replaced deprecated app attribute in Pytest create_environ() with
  root_path attribute
- fixed airflow CLI commands to match 1.10.15 version
- updated zuul gates to work on focal nodes and added focal specific
  node setup items by adding appriate ansible tasks and roles
- uplifted Helm to 3.9.4
- uplifted stable HTK commit id
- updated tox.in to with with tox v4
- uplifted dependences references to other Airship projects
- common python dependences were syncronized with other Airship
  projects(Promenade, Deckhand, Armada, Drydock)
- fixed airskiff deployment gate
- fixed genconfig* profiles in shipyard-airflow tox.ini responsible for
  maintanance of policy.yaml.sample and shipyard.conf.sample

Change-Id: I0c85187dc9bacf0849382563dd5ff7e9b2814c59
2023-04-28 20:40:50 +00:00
Zuul 34ac308ff2 Merge "[helm-toolkit] Update to latest helm-toolkit" 2022-09-26 21:12:02 +00:00
Markin, Sergiy (sm515x) 2fb7b5371b [helm-toolkit] Update to latest helm-toolkit
Change-Id: Ia6cc049d2f1526b8e6bc13f98c2867d6d9e39ee7
2022-09-23 01:13:26 +00:00
Markin, Sergiy (sm515x) ad19720444 [zuul] Added a flush node firewall task to Airskiff gates
Airskiff gates are setting up a k8s cluster by minikube, so test node needs a clear firewall rules in order to let calico to get initialized. So a task that flushes firewall rules and  deploy-package role have been added to ansible in gates.

Change-Id: I19f9c8a28b394fa239313bda8b008d79cc048469
2022-09-22 23:18:46 +00:00
Phil Sphicas aa99869546 Update HTK stable commit (Ingress)
Update helm-toolkit to include updated Ingress templates that use
current API versions (i.e. networking.k8s.io/v1)

https://review.opendev.org/c/openstack/openstack-helm-infra/+/813115

Change-Id: I7878ec1ecfe63e84155e69b0902d5c0612c867da
2021-10-18 12:43:13 -07:00
Sean Eagan e64f17b91b Remove Helm 2 based Armada integrations
- Tiller status check
- Test cleanup arg removed

Depends-On: https://review.opendev.org/c/airship/armada/+/812047
Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: I77ef3fb8e952ad28132e3476138d34bbb5a6fd3d
2021-10-04 15:03:04 -05:00
Sean Eagan a0da68409a Use helm 3 in chart build
`helm serve` is removed in helm 3 so this moves
to using local `file://` dependencies [0] instead.

[0]: https://helm.sh/docs/chart_best_practices/dependencies/#repository-urls

Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: Ibdea9f1ccda043259620add022643e5c9800259a
2021-10-01 11:48:37 -05:00
Sean Eagan 27b4dc952c Helm 3: Fix Job labels
See the dependency below for details.

Depends-On: https://review.opendev.org/c/openstack/openstack-helm-infra/+/811826
Change-Id: I100a68eb4cf457fba0783e41779f9fdc2c8daf78
2021-10-01 11:21:52 -05:00
anthony.bellino 3a27007e8a (zuul) Fix Shipyard Post Gates
Change-Id: I9f6f633399069f28c47c381faf4230e4f22276e2
2021-09-10 13:13:38 -07:00
Maximilian Weiss ac236ef52a Update HTK stable commit to 0.2.19
Update helm-toolkit stable commit to merge of this change:
https://review.opendev.org/c/openstack/openstack-helm-infra/+/802925

Change-Id: I07e82486c901b558ab2f9619d5663593305c504f
2021-08-17 17:50:35 +00:00
Maximilian Weiss 99e253092a Update helm installation script
Updates the helm installation script to download and install v2.17.0
from get.helm.sh (instead of v2.16.9 from storage.googleapis.com).

Change-Id: Ib08d39cec82c850b2308880f92f268e4cbf8cb66
2021-08-17 17:50:35 +00:00
anthony.bellino 00200ef98d Gate fixes for Shipyard
* Install older version of pip<21.0 for ubuntu_xenial images
* Install setuptools via pip for ubuntu_xenial images
* Pin typing-extensions to 3.7.2 and apache-airflow to 1.10.5
* Move promenade Dependencies under UCP components in requirements.txt
* With apache-airflow=1.10.5, strip ANSI escape sequences
  in test_deployment_group_manager.py
* Update tox.ini to support apache-airflow=1.10.5
* airskiff gate fixes
  - Pin treasuremap to v1.9 branch
  - Pin openstack-helm-infra to master
  - Remove openstack client setup as it's not used

Change-Id: Iee4ce59fdceacb165120a69d11c44e6e47feaea8
2021-06-25 13:15:31 -07:00
Rick Bartra b117fadbdb fix: Docker image build jobs
Update the deb-docker path to fix the docker image build jobs

Change-Id: I31660fcc5358fc67caba39fdaf38b60b13e7eaff
2020-09-22 09:23:18 -05:00
KAVVA, JAGAN MOHAN REDDY (jk330k) 796f2ddcca Move Tiller version to 2.16.9
Update Helm chart for Shipyard to use Tiller version 2.16.9.

Change-Id: Ia61098db57c741f36864db084d39dd9b27d25251
2020-09-16 17:00:54 +00:00
Mahmoudi, Ahmad (am495p) c13c9940dc (fix) Added task to install pip3
Added task to install pip3 addres the zuul issue not finding pip.
The zuul job airship-shipyard-airskiff-deployment failes with:
sudo -H -E pip install 'cmd2<=08.7'
sudo: pip: command not found

Change-Id: Ib43aced2c054ecc38b2ce61377b735543c0a9f9f
2020-06-24 16:06:44 +00:00
Ahmad Mahmoudi fb9cc9a65e (fix) - updated to pip3
Updated to pip to pip3 to address zuul gate issue.
Set the ensure_global_symlinks to true for zuul-jobs to set
symlink for tox path."

Change-Id: I1b0634ef18328bccc4d6929072f53db779d70ef1
2020-06-18 19:51:47 +00:00
Ahmad Mahmoudi df751e302c Upgrade apache-airflow to 1.10.5
Upgraded apache-airflow to 1.10.5 to address the database migration issue
https://issues.apache.org/jira/browse/AIRFLOW-6046.

Change-Id: Ib8678cddb2deb4f5ef944d026e0d2633a326d9cc
2020-03-30 05:21:31 +00:00
Ahmad Mahmoudi 70410cc478 (fix) Address image build issues, bionic
- With bionic image based shipyard docker images, uwsgi crashes
  with segmentation fault, when it tries to load the psycopg2 library,
  causing the api become unreachable on both shipyard docker images.
  This happens because psycopg2 2.7.x and uwsgi binary wheels are built
  with incompatible ssl libraries. This patch upgrades psycopg2 to the
  latest release to address this issue.

- The existing image build script cannot run in a docker or a pod,
  based pipeline because of two reasons:
  - The build script runs a docker (docker-in-docker) and mounts a
    volume.
    In a dind case, volume bind mounts will not work, because the nested
    container will need the host file system's path for the source path.
  - The shipyard service listens to its exposed service port in the
    nested docker network namespace, which is not reachable from the host
    pod/container.
This patch address both of the above issues. It first creates the
container, copies needed config files to the container and then starts
it. Also it execs into the nested docker to access the shipyard services
in a dind (docker-in-dcoker) case.

Change-Id: Ifdfed539babab01608bfaef37001bb79cd3a080d
2020-03-10 03:23:05 +00:00
NarlaSandeepNarlaSaibaba 5aa0cde5f0 Adding default apparmor profile to shipyard components
Change-Id: Idfc103c85bc95c8cd0a48aa0c18a17a4b1d12d3f
2020-02-17 09:52:37 -06:00
Ahmad Mahmoudi 24f6a01e0b Add support for Ubuntu bionic base image
Added support to buid shipyard and airflow images using either a xenial
or Ubuntu bionic base image.
The default base image is set to bionic.

Change-Id: I6ad4d42dede081586b3ccea87a42e250979ac106
2020-02-04 13:38:39 -06:00
Sean Eagan 6051476cd5 Full clone of osh-infra to fix htk pinning
Depth 1 clone does not pull the pinned htk version, so resetting
to that version was failing, leaving master in place.

Change-Id: Ice638d429b7051023a381e97df1334d406903f9e
2019-11-01 10:08:14 -05:00
Roman Gorshunov c65a93a895 Change URLs to opendev.org due to migration
Change URLs from git.openstack.org & github.com to opendev.org due to the
migration; wrap multiple LABELs into one.

Change-Id: I240fa6f746bd1e424e5a2b7fd381903c46059ae5
2019-10-07 18:59:45 +02:00
Ahmad Mahmoudi f42e85d7ce (airflow) - Added no_proxy env variable
Added no_proxy environment variable to be able to reach internal
repos when http_proxy is used.

Change-Id: If23fb644bfc2a6486e9089ad5ca1eb21f81e7df6
2019-09-19 17:56:55 +00:00
Drew Walters b1ae6dee45 CI: Update Airskiff full-site manifest location
The location of the Airskiff site recently changed [0], causing the
Airskiff job to fail. This change updates the "reduce site" playbook to
match the job in treasuremap, thereby avoiding the directory which no
longer exists.

The change also removes the usage of install-packages script, because
it was removed [1].

[0] https://review.opendev.org/674963
[1] https://review.opendev.org/672540

Change-Id: I7b20ff7c50cfa085039e893558df2cf022c4333c
Signed-off-by: Drew Walters <andrew.walters@att.com>
2019-09-13 20:24:13 +00:00
Roman Gorshunov da4426d173 Set default images to Ubuntu Xenial -based images
Since ':master' and ':latest' Shipyard and Airflow images are outdated,
set defaults to Ubuntu Xenial -based images.

Change-Id: I40978747f31c6a8c5cc8689a9768f8c4836ac1a1
2019-07-12 21:21:58 +02:00
Zuul d1d945951c Merge "Get password for shipyard user for special variable if it exists" 2019-06-21 20:53:54 +00:00
Jiří Suchomel d045def452 Get password for shipyard user for special variable if it exists
In case users have OS_PASSWORD set for a different cloud, it may
have sense to use different variable just for shiyard user's password.

Change-Id: I709df849e1109582917453c9fcec8bed565c2814
2019-06-12 15:22:25 +02:00
Michael Beaver 6fa4ce6ee0 Remove required-projects from Airskiff gate
This removes several of the the required-projects from the Airskiff
gate as the scripts for the gate can already clone down the necessary
repositories and the required-projects can overwrite any pins that may
have been included in the Airskiff scripts.

Change-Id: Ia70169bc89b493562e4a0db9d703a46fb2d2f87d
2019-06-08 00:42:08 +00:00
Sean Eagan 912821f22a Move to helm 2.14.0
Change-Id: I6e0c2a849965ac0c8123d72a3be4470dfcaf451f
2019-05-21 13:49:27 -05:00
Arun Kant cd6f154da5 Adding opensuse support in image building of airflow and shipyard
Adding DISTRO parameter for makefile to invoke distribution specific
Dockerfile and build image accordingly.

Also qualifying image job name to distinguish ubuntu vs opensuse
check jobs and passing distro specific variable.
Similar logic for pushing distro specific images.

Added new document section for multiple distro support.

Change-Id: I215f8a107cab8770181c0472e0cc29053b2b978e
2019-05-14 09:52:24 -07:00
Carter, Matthew (mc981n) 0761099337 Validate existence of "deployment-version" during create configdocs
This PS adds funtionality to Shipyard to validate the existence of
the Pegleg-generated "deployment-version" document (Pegleg change id:
I7919b02d70c9797f689cdad85066d3953b978901) when a user runs create
configdocs. This validation only checks the presence of the document
(by name and schema) and does not care about the document's other
contents.
The severity of a failed validation is configurable through the
"validations" config section in shipyard.conf. The default severity
is "Skip", meaning the validation is not ran at all.

Note that with the default configuration of new validation, Shipyard
functionality should be unchanged.

Change-Id: I754617de81f628a24232e890b12b157ba6731c25
2019-05-09 08:33:52 -05:00
Carter, Matthew (mc981n) 64171aacf4 Validate existence of "deployment-version" doc on configdocs commit
This PS adds funtionality to Shipyard to validate the existence of
the Pegleg-generated "deployment-version" document (Pegleg change id:
I7919b02d70c9797f689cdad85066d3953b978901). As implied, this new
validation only checks for the presence of the document (by name and
schema) and currently does not care about any of the document's
contents under "data".
The severity of a failed validation is configurable through the new
"validations" configuration section in shipyard.conf, and is
defaulted to skip the validation altogether. This means that by
default, this patch set does not alter the functionality of Shipyard

Note that with the default configuration of this new validation,
Shipyard functionality should be unchanged.

Change-Id: I5e7269066f769804710a0fd1f2c8d0aece0d3314
2019-05-09 08:25:37 -05:00
Zuul e2a87a1830 Merge "Be configuration driven when referencing document names/schemas" 2019-05-08 18:20:59 +00:00
Carter, Matthew (mc981n) 222b074cb1 Be configuration driven when referencing document names/schemas
Currently, any document name or schema referenced in the Shipyard
code base is a hard-coded string. Often times, these strings are
repeated throughout the code. This patch set adds a new configuration
section to shipyard.conf to define document names and schemas so they
can then be referenced in the Shipyard code via the oslo
configuration object. This functionality will be important for
upcoming Shipyard features which will call for more documents to be
validated as well as some new Shipyard-created docs.

Change-Id: I34ae8cd578bab730d004c3d176e3817b5a45c89e
2019-05-02 10:47:38 -05:00
Zuul ae1ab1014e Merge "tools: add test_release script" 2019-05-01 22:41:57 +00:00
Crank, Daniel (dc6350) 0ab1bf552d Don't log OS_PASSWORD in auth.log
By default, all sudo commands are logged to auth.log with their
full command line. Previously, Shipyard scripts called 'sudo docker'
with -e OS_PASSWORD=foo, resulting in the password value appearing
in auth.log in plaintext. This change adds -E to the sudo command
to pass the user's environment through, and removes the value from
-e OS_PASSWORD which tells Docker to use the environment value
directly. This prevents the password value from being logged.

Change-Id: Ifcf7f6525876144a609ff42be42da57a3f7f6f60
2019-04-26 17:17:06 -05:00
Zuul 8d0b3f671f Merge "CI: Add Airskiff check" 2019-04-26 15:03:50 +00:00
Zuul f395a2efb9 Merge "Return error if execution of the step fails" 2019-04-24 17:48:36 +00:00