Commit Graph

53 Commits

Author SHA1 Message Date
Sergiy Markin 31a3dc6b83 Kubeadm based Airskiff gate
This PS replaces old airskiff integration gate
with new kubeadm based airskiff integration gate.
The main goal of this gate is to test new deckhand
image and chart together with other Airship
components.

Change-Id: I4221b7be64e9a7e07964fa817d42c21a3f172db0
2024-03-13 05:04:00 +00:00
Sergiy Markin c51e574abd Fix deckhand-api dependences
Also this PS bumps up k8s to 1.29 for gates

Change-Id: Ic7f5dcd681875bc35663a53b2a5b052c20cc3f7a
2024-03-02 00:00:47 +00:00
Sergiy Markin 32e9950db4 Airflow stable 2.8.2
This PS bumps up Airflow version to the latest
2.8.2 and also bumps up openstack dependences to
Antelope 2023.1

Change-Id: If7191c8291e31488a8a5f26107981616e14ea12c
2024-02-26 21:00:30 +00:00
Sergiy Markin 5d2765e513 Airflow stable 2.8.1
Change-Id: I479c43de1c7a4a54a77be4a6652754f0d95a8991
2024-02-20 23:07:28 +00:00
Sergiy Markin 8b7de6e492 Use deploy-env role
Change-Id: I50530eabc8506319478d6ed02b7abb5eb1d2dd05
2024-02-20 22:58:42 +00:00
Sergiy Markin f936cfd4ae Remove openstack-helm nodeset
Change-Id: Ife8c57e2b03e1e4d56788d33ce02e30829465f04
2024-02-14 00:47:41 +00:00
Anselme, Schubert (sa246v) e6bcff90b1 Deprecating the Ingress Class Annotation
Upgrading htk to version 0.2.55, which deprecates the ingress class
annotation (kubernetes.io/ingress.class) with .spec.ingressClassName

https://review.opendev.org/c/openstack/openstack-helm-infra/+/891720

Change-Id: I573a926ab6fb07f10c0c4d9020746ba80e6d8dbd
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
2023-10-05 22:56:16 +00:00
Sergiy Markin 37ba567da5 Airflow stable 2.6.2
This PS updates python modules and code to match Airflow 2.6.2:

- bionic py36 gates  were removed
- python code corrected to match new modules versions
- selection of python modules versions was perfrmed based on
  airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- postgresql image updated to 14.8

Change-Id: I65a1b86473ee3e988aae353b59fb5473d75851f9
2023-08-29 17:09:37 +00:00
Sergiy Markin 8d055a0aa9 Deckhand updates
This PS makes the following changes:

- uses deploy-k8s.sh from treasuremap
- makes sure the airskiff-deploy playbook is using 80Gb partition if
  available
- adds available security updates to docker images

Change-Id: I0f330cb15ec32b12703f0bc6620b3f3c797a25bb
2023-08-25 17:57:43 +00:00
Sergiy Markin 06b1631d8a Restored ubuntu_bionic image build
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.

Also the latest osh-infra commit was used with kubernetes 1.27.1 for
integration tests.

Change-Id: Ia2c951f27b96774b553e7c0c7c6809172312f753
2023-05-20 05:27:20 +00:00
Sergiy Markin e4cf0c17df Update airskiff deployment gate
This PS updates armada to use new reliable airskiff site manifests for
testing armada.

Change-Id: Ia71077536f1973b0179d9011fa662f0b74caa6cf
2023-05-09 14:13:02 +00:00
Sergiy Markin ac4edb0c64 [focal] Deckhand project updates
- adjusted .gitignore to keep fresh egg-info and omit build artifacts
- fresh egg-info data is needed for promenade that depends on Deckhand
- restored deckhand-functional-uwsgi-py38 gate
- restored deckhand-integration-uwsgi-py38 gate
- made deckhand-airskiff-deployment gate voting ( treasuremap project
  has been updated)
- removed bionic gates
- updated focal dockerfile
- added more binary deps into bindep.txt
- updated deckhand chart values to latest images - focal and wallaby
- fixed python code to compy with CVE's found by fresh version of bandit
- implemented pip freeze approach
- added tox -e freeze profile to manage it
- requirements-frozen.txt is now main file with requirements
- requirements-direct.txt is the file to control deps
- updated setup.cfg to adjust to newer version of setuptools
- fixed airskiff-deploy gate
- fixed docker-image-build playbook to restore Quay repo image publish
- updated other playbooks to include roles from zuul/base-jobs in order
  to setup build hosts properly
- removed workaround with hardcoded dns resolver ip 10.96.0.10 as it
  became obsolette due to recent fix in openstack-helm-infra
- adjusted tools/whitespace-linter.sh script
- tox.ini has been brought to compliance with tox4 requirements
- replaced str() calls with six.text_type() according to D325 Deckhand specific
  commandment from Hacking.rst
- locked python-barbicanclient version with 5.2.0 because of breaking
  changes in the upper versions

Change-Id: I1cd3c97e83569c4db7e958b3400bdd4b7ea5e668
2023-04-20 19:39:43 +00:00
Wahlstedt, Walter (ww229g) 70aa35a396 update to focal and python 3.8
update dockerfile for python deckhand install
add deckhand version to chart 1.0
add chart version 0.2.0
update all packages to latest in requirements.txt
update zuul jobs for focal and python 3.8
remove zuul job functional-uwsgi-py38 in favor of functional-docker-py38
update tox config
typecast to string in re.sub() function
add stestr to test-requirements.txt
add SQLAlchemy jsonpickle sphinx-rtd-theme stestr to requirements.txt
deprecated function: BarbicanException -> BarbicanClientException
fix mock import using unittest
fix import collections to collections.abc
fix for collections modules for older than python 3.10 versions.
deprecated function: json -> to_json
deprecated function:  werkzeug.contrib.profiler ->
    werkzeug.middleware.profiler
deprecated function: falcon.AIP -> falcon.App
deprecation warning: switch from resp.body to resp.text
rename fixtures to dh_fixtures because there is an imported module
    fixtures
switch from stream.read to bounded_stream.read
deprecated function: falcon process_response needed additional parameter
deprecated function: falcon default_exception_handler changed parameter
    order
move from MagicMock object to falcon test generated object to fix
    incompatability with upgraded Falcon module.
Adjust gabbi tests to fix incompatability with upgraded DeepDiff module
update Makefile to execute ubuntu_focal
update HTK (helmtoolkit)
unpin barbican to pass integration tests
Use helm 3 in chart build.
    `helm serve` is removed in helm 3 so this moves
    to using local `file://` dependencies [0] instead.

Change-Id: I180416f480edea1b8968d80c993b3e1fcc95c08d
2023-02-24 10:51:57 -05:00
Phil Sphicas d47659e062 Make failing Zuul job non-voting
The following job is failing, seemingly due to changes in openstack-helm
and openstack-helm-infra:

    deckhand-integration-docker-py36-ubuntu_bionic

For now, disable the job.

Change-Id: I55e441079a470d6f346d70857655f0a77ddc67c3
2022-01-14 08:30:26 -08:00
Phil Sphicas 7be8e1e6ac Drop Python 3.5, make xenial/opensuse non-voting
Update tox configuration and Zuul config to use Python 3.6, and make the
xenial and opensuse gates non-voting.

Change-Id: Iab98ad6cf269d57acf0598ae431b2f661673b234
2021-10-19 15:20:38 -07:00
Roman Gorshunov b74ba277eb Re-enable all Zuul CI tests
All gates seem to report green status at the moment.

Change-Id: Id13ac60d11f46fa4e521658ffb1e0d36d8c6f22d
2020-03-06 18:07:03 +01:00
Ahmad Mahmoudi e7ba6828a0 (fix) Address uwsgi and other gating issues
- With Ubuntu bionic based deckhand docker image, uswsi crashes with
  segmentation fault when it tries to load psycopg2 library, causing
  the deckhand service become unavailable.
  The root cause of this problem seems to be that uwsgi and psycopg2
  binary wheels are built with different ssl libraries.
  To address this issue, upgrading psycopg2 to the latest release to
  the latest binary release.

- For opensuse 15.1 image build, python3-six was updated to address
  package not found issue.

- Updated gating ansbile playbooks to address non-voting gate failures.

Change-Id: I7be920e16e6114eb2bdbc052a6761f29008baf81
2020-02-28 21:28:56 +00:00
Ahmad Mahmoudi e203c15e94 Add support for Ubuntu bionic base image
- Added ability to build deckhand docker image, using either the
  Ubuntu xenial or the Ubuntu bionic base image.
- Made the bionic base image the default base image for deckhand
  docker image build.

Change-Id: I26657de34a233ee3223a7f93fc667e734ac9140b
2020-02-06 17:42:33 -06:00
Crank, Daniel (dc6350) eadae6d7b1 Fix Deckhand integration test gates
Fix problems with integration test zuul jobs which were made non-voting
by [0] and restore them to voting status. These jobs exercise some
secret-management functions that test integration between Deckhand and
Barbican that doesn't happen in the Airskiff jobs, so they still have
some test value.

- Some integration test scripts were pointing to old versions of
  openstack-helm scripts that are no longer maintained. This PS
  updates those scripts to point to current OSH scripts.

- Uplift openstack-helm-infra pin in helm_tk.sh to a recent
  version

- Add build-images to uwsgi test jobs to ensure airship_deckhand_path
  is set

[0]: https://review.opendev.org/660738

Change-Id: If83dead126307a98b44ad5d8f367a972c48a30d0
2019-10-08 15:15:58 -05:00
Roman Gorshunov 416df18353 Remove Python 2.x support
Drop support of Python 2.x as it is being deprecated.

Updated documentation, tox config, requirements.txt.

Job templates added:
- openstack-python35-jobs: openstack-tox-py35
- openstack-python3-ussuri-jobs: openstack-tox-py36, openstack-tox-py37

Job added:
- openstack-tox-pep8

Job templates removed:
- openstack-python-jobs: openstack-tox-pep8, openstack-tox-py27

Jobs removed:
- deckhand-tox-py27-postgresql

Change-Id: I7783ba841258a913f5ecd9d6f1130d378345ab5a
2019-10-08 12:41:12 +00:00
qingszhao 7031f14e06 Add Python 3 Train unit tests
This is a mechanically generated patch to ensure unit testing is in place
for all of the Tested Runtimes for Train.

See the Train python3-updates goal document for details:
https://governance.openstack.org/tc/goals/train/python3-updates.html

Change-Id: I13689a385bf237681bc96f3cc62fc17864f12659
2019-08-13 02:57:49 +00:00
Michael Beaver ea2fb8b972 Remove required-projects from Airskiff gate
This removes several of the the required-projects from the Airskiff
gate as the scripts for the gate can already clone down the necessary
repositories and the required-projects can overwrite any pins that may
have been included in the Airskiff scripts.

This requires a new nodeset as well, because the Airskiff gate does not
presently work with ubuntu-bionic

Change-Id: Idff16a450af1e3f9fe0ee49496fd7d72df69b595
2019-06-13 18:13:01 +00:00
Arun Kant d8c8e15074 Fixing secret name used for publishing image on quay.io
Fixing incorrect secret name used in previous review to address
image publish feature
https://review.opendev.org/#/c/638301/

Change-Id: I9eadad315db271169303265ce82a9f08008116e1
2019-05-30 13:11:49 -07:00
Arun Kant 8eb74b0509 Adding opensuse image build for deckhand
Adding DISTRO parameter for makefile to invoke distribution specific
Dockerfile and build image accordingly.

Updated logic for existing jobs to have distro specific logic
for building and publishing images.

Added multiple distro specific document in operator section.

Change-Id: I415ab28b06ea17b21e76d28ccb3e284041c8072d
2019-05-30 17:22:57 +00:00
Matt McEuen 2b03b5de00 Move nodeset to bionic
The deckhand gates run osh-infra scripts that now have a dependency
on Ubuntu Bionic.  This updates the nodepool to match.

Change-Id: I433264f7aec7cbe42b6b0a57562c2959d6c82a33
2019-05-30 10:48:05 -05:00
Kaspars Skels 9a2cc80ec0 Encrypt git mirroring ssh_key to specific project
Change-Id: Ibf1a05ea0ffc8b25c1445505c2eb763941346c2f
2019-05-23 12:03:41 -05:00
Zuul 7476519eeb Merge "Fix rtd publishing" 2019-05-23 11:23:29 +00:00
Kaspars Skels 75d5eaf8bb Add Zuul job for mirroring to GitHub
Change-Id: Ia6b9ab3f237e2707e7853090055c5d42da77e76c
2019-05-22 20:17:22 +00:00
Matt McEuen 8152bbe29b Make Deckhand integration tests non-voting
Deckhand integration tests are currently broken, and this makes them
non-voting.  The test harness is superflous now that the (also
non-voting) airskiff job is in place, and it should be considered as
follow-on work whether we should remove the integration tests in favor
of the airskiff integration job, or fix/maintain the current integration
test harness and/or tests themselves.

Change-Id: Ib3ce88693d80de6c9c79c278eed2e312fe3a0b6f
2019-05-22 09:51:22 -05:00
Andreas Jaeger 3f28bc3984 Fix rtd publishing
The trigger URL is [0].

The tooling sets rtd_project_name to "airmada" but the project is
named "airship-xxxx" in RTD. Introduce rtd_project_name and set it to
correct value.

See also [1].

[0] https://readthedocs.org/api/v2/webhook/{{ rtd_project_name }}/{{ rtd_webhook_id }}
[1] https://zuul-ci.org/docs/zuul-jobs/general-roles.html#rolevar-trigger-readthedocs.rtd_project_name

Change-Id: Id30ebbea47222177f2825a9547d1814bae7f6628
2019-05-17 16:04:43 +02:00
Drew Walters a611a326be CI: Add Airskiff check
This commit adds a non-voting Zuul job that deploys Memcached using the
Airship-Treasuremap airskiff site and submitted Deckhand changes.

[0] https://airship-treasuremap.readthedocs.io/en/latest/airskiff.html

Change-Id: I01eef6fb6b880edec2546e16c122002b6f00f224
2019-04-22 20:23:30 +00:00
Drew Walters 080ba9f0ca CI: Update OSH relative paths for OpenDev
On April 19, OpenStack completed the migration from openstack.org to
opendev.org [0]. During this migration, Airship projects moved from the
"openstack" namespace to the "airship" namespace. This commit makes
preliminary updates to gate jobs to account for the migration and fixes
a broken integration gate.

[0] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005011.html

Change-Id: I9257b0adab21bcc7596a5898ac9160bc6d63c225
2019-04-22 20:23:27 +00:00
Drew Walters efb82a5c74 CI: Add chart build jobs
This commit introduces a non-voting job to lint Helm charts against the
latest version of Helm toolkit from OpenStack-Helm Infra and a gate for
a pinned version of Helm toolkit.

Change-Id: Ia98209745641db011469d54220777110b2f6ae3a
2019-04-04 14:49:24 +00:00
Roman Gorshunov ede034ded6 Fix: proper ordering: tagging after build
With this patch, if commit being merged modifies both code and non-code
files, we will first build and push new image, and then will run
tagging job (which will do no harm).

Change-Id: Id9e9e705f9f5f69b969003c444895e1e0676c0dd
2018-11-13 15:54:40 +01:00
Felipe Monteiro fb08d1fd5f chore: Migrate templates from project-config to in-tree
This patch set moves remaining Zuul templates from project-config
to Deckhand to satisfy the following requirement:

https://docs.openstack.org/infra/manual/zuulv3.html#where-should-jobs-and-templates-live

Change-Id: Ia10a8c7fa34ab130a01f834bd22c947b980a22a3
Needed-By: Iaf2e0e1fd79b68ce1ea607f19cff5c40e47b05b1
2018-10-18 05:30:50 +00:00
Zuul 40301cc3ba Merge "Adding image tags on every commit" 2018-10-16 13:43:49 +00:00
Roman Gorshunov 9dcc0ae937 Adding image tags on every commit
Added container image tagging on quay.io with commit id on every merge,
when certain files but not the code are changed (docs, charts, etc.).

Previously we were building updated container images, adding tags and
uploading images onto quay.io only when the code has been changed. The
main reason for not re-building images for every single merged commit
was a long waiting time in Zuul `post` queue for a node (VM)
assignment for the job to run on.

With this change we will have image tag for every merged commit,
including documentation commits, and those tags would be added almost
immediately (within ~1 min), as job runs directly on Zuul and does not
wait in Zuul queue for a node (VM) assignment.

Change-Id: I9692947b74a16de0eb3e8100d30cd4310069abfe
2018-10-11 09:04:21 +02:00
Felipe Monteiro 38ac103f60 add python 3.6 unit test job
This is a mechanically generated patch to add a unit test job running
under Python 3.6 as part of the python3-first goal.

See the python3-first goal document for details:
https://governance.openstack.org/tc/goals/stein/python3-first.html

Change-Id: I4d08c67fb07cc4efeb48b7f81db9f5a01e444575
Story: #2002586
2018-09-12 10:55:34 -06:00
Felipe Monteiro b9aafceda6 Correct docs-on-readthedocs to work with RTD publish
This patch set follows the guide [0] to leverage
the Zuul template [1] for correctly publishing RTD
doc changes via openstackci. Accordingly, all the
old code for doing this has been removed.

[0] http://lists.openstack.org/pipermail/openstack-dev/2018-August/132836.html
[1] https://docs.openstack.org/infra/openstack-zuul-jobs/project-templates.html#project_template-docs-on-readthedocs

Change-Id: I4c1f279d2dfada0396b99c3834294a41619ba474
2018-09-09 12:09:38 -06:00
Felipe Monteiro f962ffa0c0 trivial: Use airship-deckhand-single-node for nodeset in zuul.yaml
This PS simply uses airship-deckhand-single-node instead of the
openstack-helm variety for the nodeset in .zuul.yaml for
consistency.

Change-Id: I0a881c89752dd24a48e918a0ffb4637d4a5e6483
2018-07-11 09:50:38 -04:00
Felipe Monteiro 70e4f24732 Combine integration and airship-deckhand-ubuntu jobs together
To cut down on Zuul nodes, this patchset combines the integration
docker job with the airship-deckhand-ubuntu job together, because
the former essentially subsumes the latter: integration docker
job sets up the host, deploys docker, deploys k8s, generates
the Deckhand chart using Helm, builds the Deckhand image using
Docker, deploys Deckhand, PostgreSQL and other Keystone/Barbican
dependencies using OSH, then finally runs integration tests
against the deployment.

This, then, validates everything that airship-deckhand-ubuntu
did deployment-wise.

Also, this patchset switches the 2 uwsgi jobs over to voting: false
as the Docker jobs are already voting.

Change-Id: I10b15f5a45a32413dc904ee32bdbbb4e9cdf98a2
2018-07-03 02:34:13 -04:00
Felipe Monteiro dec159e4f8 Add integration tests job to .zuul.yaml
This patchset adds integration tests job to .zuul.yaml so that
they can be run against Zuul per each commit.

The number of jobs that are executed in the gate pipeline
have been pruned because only *-docker jobs are of critical
importance. We can safely skip over *-uwsgi jobs in order
to save on Zuul resources.

Change-Id: I98ce49163b2db64cbdf18fe00b82c99fecd4139d
2018-07-03 02:07:19 +00:00
Felipe Monteiro f50abff695 Add a readthedocs publish trigger to .zuul.yaml
This patch set adds a trigger on post to trigger a RTD job to start
building the documentation post-merge.

Change-Id: Icd0ad1670b0d239e601a4456f84cc46e2f28aaf2
2018-06-21 03:56:05 +01:00
Tin Lam d0f1f62b83 chore(gate): consolidate zuul job
Moving the whitespace linting job as part of PEP8 job to reduce the need to
create a VM just to find whitespaces in none *.py files. Python files'
whitespace violation is covered as part of PEP8 job.

Change-Id: I3bc6047c9ab65e8f55f91a949a6c05ef4e38ea16
Signed-off-by: Tin Lam <tin@irrational.io>
2018-06-16 02:52:52 -05:00
Felipe Monteiro 8c546fbab6 Add irrelevant-files to all appropriate .zuul.yaml jobs
This patchset either expands on existing entries in irrelevant-files
or adds irrelevant-files to various jobs to save on resources
in Zuul.

Change-Id: I97c58069dce21a368d4267f15cd656c8d858ecc3
2018-06-11 10:06:00 -04:00
Scott Hussey c0ea5606d1 (zuul) Docker image jobs
- Gate/check jobs to build docker images
- Post jobs to build and publish docker images

Change-Id: Ia1c19689ed5f6c4fee4277f66b512205306ebffa
2018-06-09 00:25:15 +00:00
Felipe Monteiro fa0e26938b Add docs-on-readthedocs to .zuul.yaml templates
This patchset adds docs-on-readthedocs openstack-infra template
to Deckhand's .zuul.yaml templates and makes some documentation
changes to test the template.

.. https://github.com/openstack-infra/zuul-jobs/blob/master/playbooks/python/readthedocs.yaml

Change-Id: I000fc1e3be9dd4cf333fa28d7739747eb9003adc
2018-06-06 03:29:08 +00:00
Felipe Monteiro 0118874a4a Add py27/35 postgresql unit tests to .zuul.yaml
This patchset adds py27/35 postgresql unit tests to .zuul.yaml;
they use the pre-defined openstack-tox-py27|35 template defined
in project-config as the parent.

Change-Id: I572764417d98b6ab902fe2329cfb3b396ac38966
2018-06-01 23:01:58 -04:00
Felipe Monteiro a552bf2a0f Zuul: Integration tests via uwsgi.
This adds a uwsgi integration test gate to .zuul.yaml so that
deploying Deckhand via uwsgi (in a more standalone fashion,
sans containerization) works as intended.

Change-Id: I933f4781cd72e2df309efcb0515441db7ab96895
2018-05-31 17:31:50 +00:00
Felipe Monteiro 3d53d53712 Add uwsgi functional test check to .zuul.yaml
This adds a uwsgi functional test check to .zuul.yaml so that
deploying Deckhand via uwsgi (in a more standalone fashion,
sans containerization) works as intended.

Change-Id: I931ab4d11719daca7665d3a25b00e353c707237e
2018-05-29 20:25:15 +00:00