Commit Graph

372 Commits

Author SHA1 Message Date
Sergiy Markin 37ba567da5 Airflow stable 2.6.2
This PS updates python modules and code to match Airflow 2.6.2:

- bionic py36 gates  were removed
- python code corrected to match new modules versions
- selection of python modules versions was perfrmed based on
  airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- postgresql image updated to 14.8

Change-Id: I65a1b86473ee3e988aae353b59fb5473d75851f9
2023-08-29 17:09:37 +00:00
Sergiy Markin 3a06b1b604 [focal] Fix requests.body attribute deprecation
This PS updates calls to body attribute of requests module with new text
attribute.

Change-Id: I696d57ed48cf28a06784c94fbdafc2644e94633c
2023-05-11 21:02:16 +00:00
Sergiy Markin ac4edb0c64 [focal] Deckhand project updates
- adjusted .gitignore to keep fresh egg-info and omit build artifacts
- fresh egg-info data is needed for promenade that depends on Deckhand
- restored deckhand-functional-uwsgi-py38 gate
- restored deckhand-integration-uwsgi-py38 gate
- made deckhand-airskiff-deployment gate voting ( treasuremap project
  has been updated)
- removed bionic gates
- updated focal dockerfile
- added more binary deps into bindep.txt
- updated deckhand chart values to latest images - focal and wallaby
- fixed python code to compy with CVE's found by fresh version of bandit
- implemented pip freeze approach
- added tox -e freeze profile to manage it
- requirements-frozen.txt is now main file with requirements
- requirements-direct.txt is the file to control deps
- updated setup.cfg to adjust to newer version of setuptools
- fixed airskiff-deploy gate
- fixed docker-image-build playbook to restore Quay repo image publish
- updated other playbooks to include roles from zuul/base-jobs in order
  to setup build hosts properly
- removed workaround with hardcoded dns resolver ip 10.96.0.10 as it
  became obsolette due to recent fix in openstack-helm-infra
- adjusted tools/whitespace-linter.sh script
- tox.ini has been brought to compliance with tox4 requirements
- replaced str() calls with six.text_type() according to D325 Deckhand specific
  commandment from Hacking.rst
- locked python-barbicanclient version with 5.2.0 because of breaking
  changes in the upper versions

Change-Id: I1cd3c97e83569c4db7e958b3400bdd4b7ea5e668
2023-04-20 19:39:43 +00:00
Wahlstedt, Walter (ww229g) 70aa35a396 update to focal and python 3.8
update dockerfile for python deckhand install
add deckhand version to chart 1.0
add chart version 0.2.0
update all packages to latest in requirements.txt
update zuul jobs for focal and python 3.8
remove zuul job functional-uwsgi-py38 in favor of functional-docker-py38
update tox config
typecast to string in re.sub() function
add stestr to test-requirements.txt
add SQLAlchemy jsonpickle sphinx-rtd-theme stestr to requirements.txt
deprecated function: BarbicanException -> BarbicanClientException
fix mock import using unittest
fix import collections to collections.abc
fix for collections modules for older than python 3.10 versions.
deprecated function: json -> to_json
deprecated function:  werkzeug.contrib.profiler ->
    werkzeug.middleware.profiler
deprecated function: falcon.AIP -> falcon.App
deprecation warning: switch from resp.body to resp.text
rename fixtures to dh_fixtures because there is an imported module
    fixtures
switch from stream.read to bounded_stream.read
deprecated function: falcon process_response needed additional parameter
deprecated function: falcon default_exception_handler changed parameter
    order
move from MagicMock object to falcon test generated object to fix
    incompatability with upgraded Falcon module.
Adjust gabbi tests to fix incompatability with upgraded DeepDiff module
update Makefile to execute ubuntu_focal
update HTK (helmtoolkit)
unpin barbican to pass integration tests
Use helm 3 in chart build.
    `helm serve` is removed in helm 3 so this moves
    to using local `file://` dependencies [0] instead.

Change-Id: I180416f480edea1b8968d80c993b3e1fcc95c08d
2023-02-24 10:51:57 -05:00
Phil Sphicas 5cd799cc5d Allow source substring extraction
When performing substitutions, there are occasions when the source value
does not exactly match the format required by the destination document
(e.g. the values.yaml structure of an Armada chart).

This change provides the ability extract a substring of the source
value, and substitute that into the destination document.

Two optional fields are added to `src` under `metadata.substitutions`:

  * `pattern`: a regular expression, with optional capture groups
  * `match_group`: the number of the desired capture group

The canonical use case is a chart that requires an image with the repo
name and tag in separate fields, while the substitution source has the
full image path as a single value.

For example, assuming that the source document "software-versions" has:

    data:
      images:
        hello: docker.io/library/hello-world:latest

Then the following set of substitutions would put the repo and tag in
the applicable values in the destination document:

    metadata:
      substitutions:
        - src:
            schema: pegleg/SoftwareVersions/v1
            name: software-versions
            path: .images.hello
            pattern: '^(.*):(.*)'
            match_group: 1
          dest:
            path: .values.images.hello.repo
        - src:
            schema: pegleg/SoftwareVersions/v1
            name: software-versions
            path: .images.hello
            pattern: '^(.*):(.*)'
            match_group: 2
          dest:
            path: .values.images.hello.tag
    data:
      values:
        images:
          hello:
            repo:  # docker.io/library/hello-world
            tag:   # latest

Change-Id: I2fcb0d2b8e2fe3d85479ac2bad0b7b90f434eb77
2022-01-18 13:04:25 -08:00
Rick Bartra 1f0c011a17 Update pip package versions in preparation of pip 20.3
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].

These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.

[0] https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020

Change-Id: Id75acea82ddf5d915a8b8805e076dac49cab800f
2020-09-28 12:21:20 -04:00
Phil Sphicas 42fe1b85cb Accelerate YAML operations with LibYAML
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.

https://pypi.org/project/pylibyaml/

Change-Id: Iebcc50b5db87518b3b7e0fac124c712afd06da2b
2020-09-25 04:54:48 +00:00
Andrii Ostapenko 9d07671866
Fix pep8 gate running on py3.8
* Bumps hacking to 2.0.0 and addresses corresponding issues
* Bumps bandit to 1.6.2

Change-Id: I9d540acbd40700874b574687bc1fee4a9b5c0cb2
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-09-24 22:35:22 -05:00
DODDA, PRATEEK b0ec40f033 [FIX] Image build checks missing setuptools
Use pip3 in event system has both pip2 and pip3 installed. 
Use apt to install setuptools for Ansible's consumption.

Change-Id: I6929ecb0cce2ec8ac70e9261acb9f87dc7031153
Co-authored-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-06-29 14:16:45 +00:00
Kumar, Nishant (nk613n) a2606e75b1 Remove unused code for policy validation as feature not implemented
Policy validation in Deckhand was not implemented completely. Refer link
below:
https://airshipit.readthedocs.io/projects/deckhand/en/latest/users/validation.html#policy-validations

This PS removes some of the code related to the feature which was being
used in a code path when a set of documents are uploaded to Deckhand.
In standard Airship deployments the number of documents could be quite
high and this leads to significant delay (more than 300seconds in some
cases). As there are no plans to implement the policy validation feature,
it makes sense to remove it from code path which could cause delay and
sometimes timeouts while uploading documents.

This has been tested on a Baremetal lab: GF and BF.

Change-Id: I2ff3f40a7fe37bed5a589fab00d829db726604fe
2020-05-14 00:34:42 +00:00
Roman Gorshunov 5241d1d6ea Fix deckhand-integration-uwsgi-py35 tests
During Gabbi tests server returns one of

  application/json
  application/json; charset=UTF-8

in a Content-Type HTTP header, depending on which test is being run.
This might be related to different pip/pip3 versions and dependencies
installed being used during standalone vs. containerized tests.

This patch allows for both returned header's values to be accepted as
valid as a remediate solution until versions of packages and pip/pip3
usage is unified.

Change-Id: Ifb8f2d68e3474946b3df154cb016cc18cfc95d23
2020-03-01 19:47:29 +01:00
Phil Sphicas 4ccb4368ce Barbican driver simplification
Under some circumstances, the payloads retrieved from Barbican do not
match what was stored. This primarily affects surrounding whitespace[0],
but the implications for passphrases are significant, and even for PEM
encoded data, a difference in whitespace in a configmap is enough to
trigger a chart upgrade.

In general, the effort to align Deckhand document types with Barbican
secret types adds complexity without tangible benefit. Barbican does no
enforcement of the contents of the data, and if it did, that could lead
to further incompatibilities.

This change uses the 'opaque' secret type for all secret document types.
Before storage (or caching), the payload is serialized using `repr`, and
base64 encoded. Upon retrieval, the payload is base64 decoded and parsed
back into an object with `ast.literal_eval`.

[0]: https://storyboard.openstack.org/#!/story/2007017

Change-Id: I9c2f3427f52a87aad718f95160cf688db35e1b83
2020-01-24 22:26:29 +00:00
Roman Gorshunov 416df18353 Remove Python 2.x support
Drop support of Python 2.x as it is being deprecated.

Updated documentation, tox config, requirements.txt.

Job templates added:
- openstack-python35-jobs: openstack-tox-py35
- openstack-python3-ussuri-jobs: openstack-tox-py36, openstack-tox-py37

Job added:
- openstack-tox-pep8

Job templates removed:
- openstack-python-jobs: openstack-tox-pep8, openstack-tox-py27

Jobs removed:
- deckhand-tox-py27-postgresql

Change-Id: I7783ba841258a913f5ecd9d6f1130d378345ab5a
2019-10-08 12:41:12 +00:00
Doug Aaser 2786769de5 Fix encrypted doc rendering
This patchset fixes a bug where Deckhand was failing to perform
substitution and layering on document sets where all the documents had a
storagePolicy of encrypted. Deckhand would attempt to substitute from an
encrypted source document, but when that document marked as encrypted,
it fails because the source doc had been redacted. The behavior now goes
as follows:

- Resolve Barbican references before layering and substitution have been
  performed so that the prior two operations don't attempt to operate on a
  Barbican reference
- After substitution, redact the destination document if it is marked as
  encrypted
- Now, after substition, we can redact the rest of the documents and
  substitutions

Change-Id: I725775d554c9eed2692fc6203c416a7119646680
2019-10-04 16:33:46 +00:00
Carter, Matt (mc981n) 460eb7fb6c Add retries to Barbican secret create
Occasionally when Deckhand is creating secrets in Barbican, Barbican
encounters an error in which a subsequent attempt at creating the
secret would succeed. This patch set adds logic to the Deckhand
Barbican driver to retry secret creates a configurable number of
times to work around this Barbican issue.

Change-Id: I52293195dd708255508949723d89117ce2e32b71
2019-10-04 11:25:50 -05:00
Sean Eagan 7d2092b100 Fix v2 schema support
Fix v2 schema support [0] and add functional test.

[0]: https://review.opendev.org/#/c/666659/

Change-Id: I6a1be7e4f557fe9fd24d02416675df6e757ba4f1
2019-08-13 11:50:28 -05:00
Sean Eagan 48c774c648 Support v2 schema versions
Previously only v1 were supported, but Armada recently added [0]
v2 schemas, so this is needed to support those.

[0]: https://airship-armada.readthedocs.io/en/latest/operations/documents/v2/index.html

Change-Id: Ib460205c1e9ae7e7360308b8b51f162221c63982
2019-06-20 14:05:40 -05:00
Crank, Daniel (dc6350) 661350777b Log client-id in UCP API endpoints
Adds functionality to read context marker and end-user
from request headers and log that information where
available, to aid in tracing transactions that span
multiple Airship components.

Change-Id: I35c9e56f84f29420c4f3c081453cb81aa892fa7d
2019-04-05 09:55:09 +00:00
Felipe Monteiro d5462d0c70 schema: Fix metadata schema patterns
This patch set fixes the schema pattern enforced by metadata_document
and metadata_control. Currently, both allow a schema with either
pattern:

- ^metadata/Control/v\d+$
- ^metadata/Document/v\d+$

However, the metadata_control schema should only allow the former
and the metadata_document schema should only allow the latter.

Change-Id: Ic1b88a7158755818002de4c88cdf2d7b716f656d
2019-01-18 21:52:04 +00:00
Zuul bf98dfdbbc Merge "Revision diffing issue with revision rollback." 2018-11-27 18:56:49 +00:00
Smruti Soumitra Khuntia 8fc98631b9 Revision diffing issue with revision rollback.
* Fix for diffing issue after rollback in
conjunction with created and deleted buckets.
* Changed rollback function to check against the full set of documents
for a revision instead of just the documents at that particular revision
* Created a document_delete function to encapsulate document deletion
* Added additional test case to check that a rollback to
something other than 0 deletes the created buckets in between

Co-Authored-By: Michael Beaver <michaelbeaver64@gmail.com>
Change-Id: I0d57e67d68def1f15255a8c89290e8c70deedc03
2018-11-21 11:32:16 -06:00
Zuul cc7e00970d Merge "[FIX] Secrets substitution issue" 2018-11-15 23:47:23 +00:00
Ahmad Mahmoudi e095137766 [FIX] Secrets substitution issue
Fixed issue with secrets substitution, which have more than
one substitution destinations.

Change-Id: I6c0e9719cacc7cf4189b57379e9ebf6a8d3a4fd8
2018-11-13 09:24:02 -06:00
Zuul c240af18d7 Merge "fix wrong spelling" 2018-11-13 15:06:38 +00:00
wangqiangbj d9bec709c0 fix wrong spelling
Change-Id: I468da7d9053f526b1d014927d27b118f65ebb4c2
2018-11-13 20:29:44 +08:00
zhouxinyong 9b4518f3e1 omit the twice occured words in layering-with-replacement-single-bucket.yaml
Change-Id: I50099748bce590bba82fa98036850ef6df4e918d
2018-11-13 09:51:01 +08:00
Zuul 2f596fd627 Merge "Fix logging when "Duplicate document exists" error occurs" 2018-11-06 15:51:50 +00:00
Zuul 947810ada5 Merge "docs: Add config documentation to operator's section" 2018-11-05 18:05:49 +00:00
Evgeny L ec8bad3bf4 Fix logging when "Duplicate document exists" error occurs
Currently validation fails with "KeyError: 'schema'",
which makes it hard to determine a root cause of error.

Change-Id: Ifd40faf485578cc0a133e17650f8df6758a6c8ae
2018-11-01 13:45:04 +00:00
Felipe Monteiro b03a4522cb fix: Use schema instead of metadata.schema for replacement check
Recently added replacement check incorrectly uses metadata.schema
and metadata.name to key on the document -- but it should be schema
and metadata.name, the combination of which uniquely defines a
document.

Change-Id: I6cd1679ad41be38cb78d65ce2763e60f7da390d2
2018-10-31 15:02:28 -04:00
Rick Bartra 60e82b7bd6 Validate additional 'metadata.replacement' scenarios
This patch set adds additional documentation and unit tests
to validate further replacement scenarios.

In particular this commit adds an additional document check that
looks for documents exisitng in different layers that contain the
same name and same schema without any of them having `replacement: true`

Change-Id: I7c033d32a6755f36e609789a748cbc6d4af06bc2
2018-10-30 10:23:14 -04:00
Rick Bartra 88fe773cd7 Fix document is_control method
The document.py `is_control` method incorrectly checks if a document
is a Control document. Per the documentation [0], Control documents
have `metadata.schema` of `metadata/Control/v1`. This commit updates
the `is_control` method to correctly check for Control documents.

[0] 1d4cc81dfa/doc/source/users/document-types.rst (control-documents)

Change-Id: I60ca8f31a61987b4e756784fce0f5a751639ae9e
2018-10-30 09:59:38 -04:00
Felipe Monteiro b34156ae7b docs: Add config documentation to operator's section
This PS adds configuration documentation that includes
a literalinclude of the config file as well as some
information on each of the cache config options as
these are important on performance.

Change-Id: I3b06012b8843b7bfbd46307f81397172a41d3675
2018-10-29 18:21:38 +00:00
Zuul 464d2c0ea5 Merge "Validate bucket diffing works with revision rollback" 2018-10-29 17:26:38 +00:00
Zuul eb178e1d7f Merge "refactor: Move replacement checks into separate module" 2018-10-29 17:26:37 +00:00
Zuul 56e606bf4b Merge "fix: Redact secondhand substitutions of sensitive data" 2018-10-29 17:13:25 +00:00
Zuul 475655ac5a Merge "fix: Correct .data path layering edge case" 2018-10-29 14:31:11 +00:00
Felipe Monteiro 9d91a072cd docs: Use sphinx-apidoc library for autodoc compatibility
This package is used for generation autodoc documentation
automatically which can be linked to by Deckhand
documentation from other places. This is to make autodoc
generation work in RTD.

More info: https://pypi.org/project/sphinxcontrib-apidoc/

Change-Id: I43aac82728e5935a5a2626f2fd29d7a7188d19f9
2018-10-27 22:52:39 +01:00
Felipe Monteiro 47ade1f0da fix: Redact secondhand substitutions of sensitive data
This patch set ensures that documents that substitute data from
encrypted document sources are themselves redacted, assuming that
cleartext-secrets=true. Note that this redaction fix only applies
to the substitution dest/src paths. The data section is already
being correctly redacted for secondhand sources.

Change-Id: I6ce16a109628259b2cc8132cd9db63261b5dbace
2018-10-25 09:39:50 -04:00
anthony.bellino 7defe473d2 Redact rendered Documents
- Uses the rendered-documents endpoint
- Adds a query parameter ?cleartext-secrets
- Adds unit tests, updates integration tests

Change-Id: I02423b9bf7456008d707b3cd91edc4fc281fa5fc
2018-10-24 22:42:25 -04:00
Zuul 7d697012fc Merge "Redacts Raw Documents" 2018-10-22 15:24:20 +00:00
Felipe Monteiro 24d86ea749 refactor: Move replacement checks into separate module
This patch set refactors replacement validation checks
in Deckhand's layering module into a separate module for
better code organization.

Change-Id: If973148ac8220b96f61128b8a7266e6fd57e76b9
2018-10-20 17:08:16 -04:00
Aaron Sheffield 349e5600df Redacts Raw Documents
- If a document has a storage policy of encrypted
 - Redacts (sha256) the data section.
 - Redacts (sha256) the substition paths.
- Uses the same /documents endpoint, adds a new query parameter
  ?cleartext-secrets=true to show the non-redacted values.

Change-Id: I42808901b97c667a1148c00fbb7717a0847c9981
2018-10-19 23:56:12 -05:00
Zuul a991513eff Merge "fix: Add validation logic to check for duplicate documents in engine" 2018-10-19 21:35:47 +00:00
Felipe Monteiro 035841416b Validate bucket diffing works with revision rollback
Adds a unit test to validate following scenario:

1) create revision 1 with document
2) create revision 2 with no documents
3) rollback to revision 1 (creating revision 3)

Validate that diffing works for rolled-back revision.
All cases above use same bucket.

Also refactors some test logic for neatness.

Change-Id: I71bf7d34e8aae3ad5abb3c53b05cb96a7038ddc2
2018-10-18 19:07:42 +01:00
Felipe Monteiro b80df59d11 fix: Address small issues with revision rollback controller
1. There is no exception called `InvalidRollback` in Deckhand (it
was removed a while back). Instead, the only exception that
db_api.revision_rollback raises is RevisionNotFound from
the revision_get call internally.

So catch that instead from the controller.

2. The default value of parameters is `str` so when revision_id
of '0' is passed to the db module for processing, it skips over
the check for `if revision_id == 0` as revision_id is a str,
not int. So this leverages builtin int converter logic in
falcon [0] but requires uplifting the version of falcon to
at least 1.3.0 to make use of it [1].

[0] https://falcon.readthedocs.io/en/stable/api/routing.html#field-converters
[1] https://falcon.readthedocs.io/en/1.3.0/api/routing.html#field-converters

Change-Id: I068cd9e9b6818a5d51501f2718ee2d40d556c094
2018-10-18 09:45:16 -04:00
Zuul bd9e7c7e96 Merge "optimization: Skip post-validation for rendered document cache hit" 2018-10-16 16:30:32 +00:00
Prateek Dodda 2c4c5a9c63 fix: Add validation logic to check for duplicate documents in engine
This patch set adds validation logic to document_validation.py (in
Deckhand's engine module) so that components that rely on
Deckhand's engine for document rendering (such as Promenade
or Pegleg) can fail fast when they provide Deckhand with a
duplicate document. Must pass pre_validate=True to layering
module which currently is the case for Promenade, et. al.

Before this change, Deckand only supported this logic at
the DB level (requiring service instantion); this is now no longer
the case.

Change-Id: I6d1c8214775aa0f3b5efb1049972cf847f74585b
2018-10-11 22:33:01 +00:00
Felipe Monteiro 2ea808cae2 fix: Correct .data path layering edge case
This patch set corrects logic for an edge case in layering where
the action `path` is set to `.data`. In this case this means
that the root of the data section should be used, i.e. '.'
or '$.'. The previous adjustment was incorrect: .data was being
changed to empty string ''. This fixes that logic to change to
'.'.

Change-Id: Id6cf0d4d65020220c540eb162a33055035336cde
2018-10-07 15:10:09 -04:00
Felipe Monteiro 88e1c12b23 Add explicit start/end to Deckhand response middleware
Pegleg has linting rules dedicated to checking for explicit starts
and so on, so it makes sense that Deckhand just adds this in for
every response as it is a nice feature that better delineates
starting and endpoints points for individual YAML documents.

Change-Id: I6324cfa268ddf250a9c78cb663e7015a171bbc19
Related-Change: https://review.openstack.org/#/c/604123
2018-10-05 18:07:45 +00:00