Commit Graph

735 Commits

Author SHA1 Message Date
Sergiy Markin 31a3dc6b83 Kubeadm based Airskiff gate
This PS replaces old airskiff integration gate
with new kubeadm based airskiff integration gate.
The main goal of this gate is to test new deckhand
image and chart together with other Airship
components.

Change-Id: I4221b7be64e9a7e07964fa817d42c21a3f172db0
2024-03-13 05:04:00 +00:00
Sergiy Markin c51e574abd Fix deckhand-api dependences
Also this PS bumps up k8s to 1.29 for gates

Change-Id: Ic7f5dcd681875bc35663a53b2a5b052c20cc3f7a
2024-03-02 00:00:47 +00:00
Sergiy Markin 32e9950db4 Airflow stable 2.8.2
This PS bumps up Airflow version to the latest
2.8.2 and also bumps up openstack dependences to
Antelope 2023.1

Change-Id: If7191c8291e31488a8a5f26107981616e14ea12c
2024-02-26 21:00:30 +00:00
Sergiy Markin 5d2765e513 Airflow stable 2.8.1
Change-Id: I479c43de1c7a4a54a77be4a6652754f0d95a8991
2024-02-20 23:07:28 +00:00
Sergiy Markin 8b7de6e492 Use deploy-env role
Change-Id: I50530eabc8506319478d6ed02b7abb5eb1d2dd05
2024-02-20 22:58:42 +00:00
Sergiy Markin 5b7da9ebd6 Rollback python deps update
This PS rolls back python deps update done after adding xattr==0.10.1

https: //review.opendev.org/c/airship/deckhand/+/906424
Change-Id: I0d246d3a72555e1fb6c7542a20af207e7dd74ba8
2024-02-15 21:18:13 +00:00
Sergiy Markin f936cfd4ae Remove openstack-helm nodeset
Change-Id: Ife8c57e2b03e1e4d56788d33ce02e30829465f04
2024-02-14 00:47:41 +00:00
Ritchie, Frank (fr801x) 1767178234 Update helm toolkit reference
Update helm toolkit ref so that ingress will be created with a pathType
of Prefix.

https: //review.opendev.org/c/openstack/openstack-helm-infra/+/905757
Change-Id: I261603e3c68337347955996f77b86bd60a2f2af5
2024-01-25 14:57:38 +00:00
Anselme, Schubert (sa246v) e6bcff90b1 Deprecating the Ingress Class Annotation
Upgrading htk to version 0.2.55, which deprecates the ingress class
annotation (kubernetes.io/ingress.class) with .spec.ingressClassName

https://review.opendev.org/c/openstack/openstack-helm-infra/+/891720

Change-Id: I573a926ab6fb07f10c0c4d9020746ba80e6d8dbd
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
2023-10-05 22:56:16 +00:00
Sergiy Markin 37ba567da5 Airflow stable 2.6.2
This PS updates python modules and code to match Airflow 2.6.2:

- bionic py36 gates  were removed
- python code corrected to match new modules versions
- selection of python modules versions was perfrmed based on
  airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- postgresql image updated to 14.8

Change-Id: I65a1b86473ee3e988aae353b59fb5473d75851f9
2023-08-29 17:09:37 +00:00
Sergiy Markin 8d055a0aa9 Deckhand updates
This PS makes the following changes:

- uses deploy-k8s.sh from treasuremap
- makes sure the airskiff-deploy playbook is using 80Gb partition if
  available
- adds available security updates to docker images

Change-Id: I0f330cb15ec32b12703f0bc6620b3f3c797a25bb
2023-08-25 17:57:43 +00:00
Sergiy Markin 06b1631d8a Restored ubuntu_bionic image build
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.

Also the latest osh-infra commit was used with kubernetes 1.27.1 for
integration tests.

Change-Id: Ia2c951f27b96774b553e7c0c7c6809172312f753
2023-05-20 05:27:20 +00:00
Sergiy Markin 3a06b1b604 [focal] Fix requests.body attribute deprecation
This PS updates calls to body attribute of requests module with new text
attribute.

Change-Id: I696d57ed48cf28a06784c94fbdafc2644e94633c
2023-05-11 21:02:16 +00:00
Sergiy Markin e4cf0c17df Update airskiff deployment gate
This PS updates armada to use new reliable airskiff site manifests for
testing armada.

Change-Id: Ia71077536f1973b0179d9011fa662f0b74caa6cf
2023-05-09 14:13:02 +00:00
Sergiy Markin 03f6932e16 Deckhand updates
This PS delivers the following updates:
- fixed sample config and policy files generation in tox
- rolled back chart version incremention back to 0.2.0

Change-Id: I509030319a724b18bb21f45f7ede7c07ab18e894
2023-04-28 22:18:29 +00:00
Sergiy Markin 13c5199f18 Removing egg-info folder
Change-Id: Id80a5208b08d695e2ca582d4ad113b9cd72b3fb5
2023-04-26 22:23:30 +00:00
Sergiy Markin dc3445030a Sync requirements with shipyard
This PS adjusts list of dependences needed to get syncronized with Shipyard project.

Change-Id: I9235702aa34b9f0bf6634229a24e5f1847ac9a7d
2023-04-25 13:11:36 +00:00
Sergiy Markin ac4edb0c64 [focal] Deckhand project updates
- adjusted .gitignore to keep fresh egg-info and omit build artifacts
- fresh egg-info data is needed for promenade that depends on Deckhand
- restored deckhand-functional-uwsgi-py38 gate
- restored deckhand-integration-uwsgi-py38 gate
- made deckhand-airskiff-deployment gate voting ( treasuremap project
  has been updated)
- removed bionic gates
- updated focal dockerfile
- added more binary deps into bindep.txt
- updated deckhand chart values to latest images - focal and wallaby
- fixed python code to compy with CVE's found by fresh version of bandit
- implemented pip freeze approach
- added tox -e freeze profile to manage it
- requirements-frozen.txt is now main file with requirements
- requirements-direct.txt is the file to control deps
- updated setup.cfg to adjust to newer version of setuptools
- fixed airskiff-deploy gate
- fixed docker-image-build playbook to restore Quay repo image publish
- updated other playbooks to include roles from zuul/base-jobs in order
  to setup build hosts properly
- removed workaround with hardcoded dns resolver ip 10.96.0.10 as it
  became obsolette due to recent fix in openstack-helm-infra
- adjusted tools/whitespace-linter.sh script
- tox.ini has been brought to compliance with tox4 requirements
- replaced str() calls with six.text_type() according to D325 Deckhand specific
  commandment from Hacking.rst
- locked python-barbicanclient version with 5.2.0 because of breaking
  changes in the upper versions

Change-Id: I1cd3c97e83569c4db7e958b3400bdd4b7ea5e668
2023-04-20 19:39:43 +00:00
Wahlstedt, Walter (ww229g) 70aa35a396 update to focal and python 3.8
update dockerfile for python deckhand install
add deckhand version to chart 1.0
add chart version 0.2.0
update all packages to latest in requirements.txt
update zuul jobs for focal and python 3.8
remove zuul job functional-uwsgi-py38 in favor of functional-docker-py38
update tox config
typecast to string in re.sub() function
add stestr to test-requirements.txt
add SQLAlchemy jsonpickle sphinx-rtd-theme stestr to requirements.txt
deprecated function: BarbicanException -> BarbicanClientException
fix mock import using unittest
fix import collections to collections.abc
fix for collections modules for older than python 3.10 versions.
deprecated function: json -> to_json
deprecated function:  werkzeug.contrib.profiler ->
    werkzeug.middleware.profiler
deprecated function: falcon.AIP -> falcon.App
deprecation warning: switch from resp.body to resp.text
rename fixtures to dh_fixtures because there is an imported module
    fixtures
switch from stream.read to bounded_stream.read
deprecated function: falcon process_response needed additional parameter
deprecated function: falcon default_exception_handler changed parameter
    order
move from MagicMock object to falcon test generated object to fix
    incompatability with upgraded Falcon module.
Adjust gabbi tests to fix incompatability with upgraded DeepDiff module
update Makefile to execute ubuntu_focal
update HTK (helmtoolkit)
unpin barbican to pass integration tests
Use helm 3 in chart build.
    `helm serve` is removed in helm 3 so this moves
    to using local `file://` dependencies [0] instead.

Change-Id: I180416f480edea1b8968d80c993b3e1fcc95c08d
2023-02-24 10:51:57 -05:00
Phil Sphicas 5cd799cc5d Allow source substring extraction
When performing substitutions, there are occasions when the source value
does not exactly match the format required by the destination document
(e.g. the values.yaml structure of an Armada chart).

This change provides the ability extract a substring of the source
value, and substitute that into the destination document.

Two optional fields are added to `src` under `metadata.substitutions`:

  * `pattern`: a regular expression, with optional capture groups
  * `match_group`: the number of the desired capture group

The canonical use case is a chart that requires an image with the repo
name and tag in separate fields, while the substitution source has the
full image path as a single value.

For example, assuming that the source document "software-versions" has:

    data:
      images:
        hello: docker.io/library/hello-world:latest

Then the following set of substitutions would put the repo and tag in
the applicable values in the destination document:

    metadata:
      substitutions:
        - src:
            schema: pegleg/SoftwareVersions/v1
            name: software-versions
            path: .images.hello
            pattern: '^(.*):(.*)'
            match_group: 1
          dest:
            path: .values.images.hello.repo
        - src:
            schema: pegleg/SoftwareVersions/v1
            name: software-versions
            path: .images.hello
            pattern: '^(.*):(.*)'
            match_group: 2
          dest:
            path: .values.images.hello.tag
    data:
      values:
        images:
          hello:
            repo:  # docker.io/library/hello-world
            tag:   # latest

Change-Id: I2fcb0d2b8e2fe3d85479ac2bad0b7b90f434eb77
2022-01-18 13:04:25 -08:00
Phil Sphicas d47659e062 Make failing Zuul job non-voting
The following job is failing, seemingly due to changes in openstack-helm
and openstack-helm-infra:

    deckhand-integration-docker-py36-ubuntu_bionic

For now, disable the job.

Change-Id: I55e441079a470d6f346d70857655f0a77ddc67c3
2022-01-14 08:30:26 -08:00
Phil Sphicas 3f4458690d Update HTK stable commit (Ingress)
Update helm-toolkit to include updated Ingress templates that use
current API versions (i.e. networking.k8s.io/v1)

https://review.opendev.org/c/openstack/openstack-helm-infra/+/813115

Change-Id: I5c4453388418b08a8c8f13fcad574d3afef60e5d
2021-10-19 15:20:38 -07:00
Phil Sphicas 7be8e1e6ac Drop Python 3.5, make xenial/opensuse non-voting
Update tox configuration and Zuul config to use Python 3.6, and make the
xenial and opensuse gates non-voting.

Change-Id: Iab98ad6cf269d57acf0598ae431b2f661673b234
2021-10-19 15:20:38 -07:00
Sean Eagan 4ba85d2432 Helm 3: Fix Job labels
See the dependency below for details.

Depends-On: https://review.opendev.org/c/openstack/openstack-helm-infra/+/811826
Change-Id: I9f0d9d99b798a41f8d8ac841e3b00e7d8af40ff3
2021-10-01 11:28:22 -05:00
anthony.bellino 28b2bd0a14 (zuul) Fix Deckhand post jobs
Change-Id: I63c7814ca7a6f7013c194038d14a4721373e9717
2021-09-09 12:09:05 -07:00
Phil Sphicas d7d3cbde2b Revert jsonschema to 3.2.0
Change-Id: I5dbd56e40592cd7d3da71889ee17fff44cfee6ef
2021-09-08 05:09:32 +00:00
anthony.bellino 1328f5b568 Deckhand gate fix
Unpin python3-six for Opensuse image build.

Update helm-toolkit stable commit to merge of this change:
https://review.opendev.org/#/c/803654/

Update the helm installation script to download and install v2.17.0

Fix integration tests by pinning Barbican to stable commit.

Pinn jsonschema to 4.0.0a2 to fix tox tests

Change-Id: I2badd0e2f6c934098f0c9f5ef7e52354756c12e0
2021-08-30 07:53:24 -07:00
anthony.bellino d5babeaa86 Gate fixes
* Pin openstacksdk to stein release

* Leap 15.3 changes for opensuse image build

* airskiff gate fixes
  - Pin treasuremap to v1.9 branch
  - Pin openstack-helm-infra to master
  - Remove openstack client setup as it's not used

Change-Id: I139e5b291cde22590f0b9ad5fbbecc5a4f023c33
2021-05-18 08:05:23 -07:00
Rick Bartra 1f0c011a17 Update pip package versions in preparation of pip 20.3
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].

These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.

[0] https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020

Change-Id: Id75acea82ddf5d915a8b8805e076dac49cab800f
2020-09-28 12:21:20 -04:00
Phil Sphicas 42fe1b85cb Accelerate YAML operations with LibYAML
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.

https://pypi.org/project/pylibyaml/

Change-Id: Iebcc50b5db87518b3b7e0fac124c712afd06da2b
2020-09-25 04:54:48 +00:00
Phil Sphicas 9c8069d26a Include LibYAML in container builds
Updates Dockerfiles to build the LibYAML library, which can provide much
faster YAML parsing and emitting than the native Python library.

https://pyyaml.org/wiki/LibYAML

Change-Id: I1c6f41a72c7d32e810cf64f572dc2a1cc6a1e710
2020-09-25 04:54:48 +00:00
Phil Sphicas 0325f82d55 Sort package list in Dockerfiles
Sort the list of apt/zypper packages in Dockerfiles to make it obvious
where to insert new ones.

Change-Id: Ib1d890dbea88c009c067ffd8bc9cc9175979ed32
2020-09-25 04:54:48 +00:00
Andrii Ostapenko 1b396f220a Change helm-toolkit dependency version to ">= 0.1.0"
Since we introduced chart version check in gates, requirements are not
satisfied with strict check of 0.1.0

Change-Id: I547a7f0e6106fee2f560b62671e1eceb312e5c4e
2020-09-25 03:35:37 +00:00
Andrii Ostapenko 9d07671866
Fix pep8 gate running on py3.8
* Bumps hacking to 2.0.0 and addresses corresponding issues
* Bumps bandit to 1.6.2

Change-Id: I9d540acbd40700874b574687bc1fee4a9b5c0cb2
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-09-24 22:35:22 -05:00
Zuul 177fb31d1a Merge "Update HTK stable commit" 2020-08-11 01:53:20 +00:00
Ahmad Mahmoudi 0545625da9 Scaling deckhand uwsgi workers
Updated obsolete uwsgi default configuration parameters for better
performance.
Increased number of worker threads to increase performance.
Uplifted uwsgi to the latest for bug fixes since 2018.

For more info please see:
https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html

Change-Id: Ifedb9c6279e64be86deb6ec375810c5ecf97958a
2020-08-05 22:05:57 +00:00
Phil Sphicas 80a69e7c2b Update HTK stable commit
Update helm-toolkit stable commit to merge of this change:
https://review.opendev.org/#/c/734702/

Change-Id: I7796a4c76209d032b46d46c260b5c3a27f1aa11b
2020-07-28 16:32:11 +00:00
Zuul 033b278fd0 Merge "Add configmap-hash annotations for deckhand" 2020-07-17 16:10:30 +00:00
DODDA, PRATEEK REDDY 3a3657b6e5 Add configmap-hash annotations for deckhand
Adds configmap-hash annotations to the job-db-init and job-db-sync
for configmap-bin and configmap-etc.

These annotations ensure that if configmaps change, the pods
are redeployed according to their upgrade strategy.

Change-Id: I8ff282d8279c934590d5308e9c26efaf65685e2b
2020-07-06 10:41:28 -05:00
KHIYANI, RAHUL (rk0850) 2f9e0d7601 Implement helm-toolkit snippet to deckhand pods/containers
This updates the deckhand chart to include the pod
security context on the pod template.

This also adds the container security context to set
readOnlyRootFilesystem flag to true

Change-Id: I9bfd889b163e280cf17c4e7b49974a077e889f2f
2020-07-02 09:32:18 -05:00
Zuul 13fd8bfd1f Merge "Enabling Apparmor profile to deckhand init containers" 2020-06-29 19:00:30 +00:00
DODDA, PRATEEK b0ec40f033 [FIX] Image build checks missing setuptools
Use pip3 in event system has both pip2 and pip3 installed. 
Use apt to install setuptools for Ansible's consumption.

Change-Id: I6929ecb0cce2ec8ac70e9261acb9f87dc7031153
Co-authored-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-06-29 14:16:45 +00:00
DODDA, PRATEEK 897f596185 Enabling Apparmor profile to deckhand init containers
Remove OSH Authors copyright

The current copyright refers to a non-existent group
"openstack helm authors" with often out-of-date references that
are confusing when adding a new file to the repo.

This change removes all references to this copyright by the
non-existent group and any blank lines underneath.

Change-Id: Ib0b21b33d8bf91ea6da4c2421cc81355cf2b23b1
2020-06-26 11:08:56 -05:00
Kumar, Nishant (nk613n) a2606e75b1 Remove unused code for policy validation as feature not implemented
Policy validation in Deckhand was not implemented completely. Refer link
below:
https://airshipit.readthedocs.io/projects/deckhand/en/latest/users/validation.html#policy-validations

This PS removes some of the code related to the feature which was being
used in a code path when a set of documents are uploaded to Deckhand.
In standard Airship deployments the number of documents could be quite
high and this leads to significant delay (more than 300seconds in some
cases). As there are no plans to implement the policy validation feature,
it makes sense to remove it from code path which could cause delay and
sometimes timeouts while uploading documents.

This has been tested on a Baremetal lab: GF and BF.

Change-Id: I2ff3f40a7fe37bed5a589fab00d829db726604fe
2020-05-14 00:34:42 +00:00
Zuul 9201c9376f Merge "Re-enable all Zuul CI tests" 2020-05-05 17:48:25 +00:00
Roman Gorshunov b74ba277eb Re-enable all Zuul CI tests
All gates seem to report green status at the moment.

Change-Id: Id13ac60d11f46fa4e521658ffb1e0d36d8c6f22d
2020-03-06 18:07:03 +01:00
Zuul a03e9b427f Merge "Fix deckhand-integration-uwsgi-py35 tests" 2020-03-04 18:37:05 +00:00
Drew Walters 2e4a01302b Add SECURITY.md
All Airship projects are moving to GitHub issues. This change adds a
GitHub security policy that links to the official Airship vulnerability
management process [0]. When users on GitHub click "New Issue" on this
GitHub repository, they will see an option to report a security
vulnerability, which will direct them to our official policy.

[0] https://airship-docs.readthedocs.io/en/latest/security/vulnerabilities.html

Change-Id: Iaf060dd0085c21f0c4f18f100e3e053b5ceedbed
Signed-off-by: Drew Walters <andrew.walters@att.com>
2020-03-02 16:32:42 +00:00
Roman Gorshunov 5241d1d6ea Fix deckhand-integration-uwsgi-py35 tests
During Gabbi tests server returns one of

  application/json
  application/json; charset=UTF-8

in a Content-Type HTTP header, depending on which test is being run.
This might be related to different pip/pip3 versions and dependencies
installed being used during standalone vs. containerized tests.

This patch allows for both returned header's values to be accepted as
valid as a remediate solution until versions of packages and pip/pip3
usage is unified.

Change-Id: Ifb8f2d68e3474946b3df154cb016cc18cfc95d23
2020-03-01 19:47:29 +01:00
Ahmad Mahmoudi e7ba6828a0 (fix) Address uwsgi and other gating issues
- With Ubuntu bionic based deckhand docker image, uswsi crashes with
  segmentation fault when it tries to load psycopg2 library, causing
  the deckhand service become unavailable.
  The root cause of this problem seems to be that uwsgi and psycopg2
  binary wheels are built with different ssl libraries.
  To address this issue, upgrading psycopg2 to the latest release to
  the latest binary release.

- For opensuse 15.1 image build, python3-six was updated to address
  package not found issue.

- Updated gating ansbile playbooks to address non-voting gate failures.

Change-Id: I7be920e16e6114eb2bdbc052a6761f29008baf81
2020-02-28 21:28:56 +00:00