This PS updates python modules and code to match Airflow 2.6.2:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was perfrmed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- postgresql image updated to 14.8
Change-Id: I65a1b86473ee3e988aae353b59fb5473d75851f9
- adjusted .gitignore to keep fresh egg-info and omit build artifacts
- fresh egg-info data is needed for promenade that depends on Deckhand
- restored deckhand-functional-uwsgi-py38 gate
- restored deckhand-integration-uwsgi-py38 gate
- made deckhand-airskiff-deployment gate voting ( treasuremap project
has been updated)
- removed bionic gates
- updated focal dockerfile
- added more binary deps into bindep.txt
- updated deckhand chart values to latest images - focal and wallaby
- fixed python code to compy with CVE's found by fresh version of bandit
- implemented pip freeze approach
- added tox -e freeze profile to manage it
- requirements-frozen.txt is now main file with requirements
- requirements-direct.txt is the file to control deps
- updated setup.cfg to adjust to newer version of setuptools
- fixed airskiff-deploy gate
- fixed docker-image-build playbook to restore Quay repo image publish
- updated other playbooks to include roles from zuul/base-jobs in order
to setup build hosts properly
- removed workaround with hardcoded dns resolver ip 10.96.0.10 as it
became obsolette due to recent fix in openstack-helm-infra
- adjusted tools/whitespace-linter.sh script
- tox.ini has been brought to compliance with tox4 requirements
- replaced str() calls with six.text_type() according to D325 Deckhand specific
commandment from Hacking.rst
- locked python-barbicanclient version with 5.2.0 because of breaking
changes in the upper versions
Change-Id: I1cd3c97e83569c4db7e958b3400bdd4b7ea5e668
update dockerfile for python deckhand install
add deckhand version to chart 1.0
add chart version 0.2.0
update all packages to latest in requirements.txt
update zuul jobs for focal and python 3.8
remove zuul job functional-uwsgi-py38 in favor of functional-docker-py38
update tox config
typecast to string in re.sub() function
add stestr to test-requirements.txt
add SQLAlchemy jsonpickle sphinx-rtd-theme stestr to requirements.txt
deprecated function: BarbicanException -> BarbicanClientException
fix mock import using unittest
fix import collections to collections.abc
fix for collections modules for older than python 3.10 versions.
deprecated function: json -> to_json
deprecated function: werkzeug.contrib.profiler ->
werkzeug.middleware.profiler
deprecated function: falcon.AIP -> falcon.App
deprecation warning: switch from resp.body to resp.text
rename fixtures to dh_fixtures because there is an imported module
fixtures
switch from stream.read to bounded_stream.read
deprecated function: falcon process_response needed additional parameter
deprecated function: falcon default_exception_handler changed parameter
order
move from MagicMock object to falcon test generated object to fix
incompatability with upgraded Falcon module.
Adjust gabbi tests to fix incompatability with upgraded DeepDiff module
update Makefile to execute ubuntu_focal
update HTK (helmtoolkit)
unpin barbican to pass integration tests
Use helm 3 in chart build.
`helm serve` is removed in helm 3 so this moves
to using local `file://` dependencies [0] instead.
Change-Id: I180416f480edea1b8968d80c993b3e1fcc95c08d
When performing substitutions, there are occasions when the source value
does not exactly match the format required by the destination document
(e.g. the values.yaml structure of an Armada chart).
This change provides the ability extract a substring of the source
value, and substitute that into the destination document.
Two optional fields are added to `src` under `metadata.substitutions`:
* `pattern`: a regular expression, with optional capture groups
* `match_group`: the number of the desired capture group
The canonical use case is a chart that requires an image with the repo
name and tag in separate fields, while the substitution source has the
full image path as a single value.
For example, assuming that the source document "software-versions" has:
data:
images:
hello: docker.io/library/hello-world:latest
Then the following set of substitutions would put the repo and tag in
the applicable values in the destination document:
metadata:
substitutions:
- src:
schema: pegleg/SoftwareVersions/v1
name: software-versions
path: .images.hello
pattern: '^(.*):(.*)'
match_group: 1
dest:
path: .values.images.hello.repo
- src:
schema: pegleg/SoftwareVersions/v1
name: software-versions
path: .images.hello
pattern: '^(.*):(.*)'
match_group: 2
dest:
path: .values.images.hello.tag
data:
values:
images:
hello:
repo: # docker.io/library/hello-world
tag: # latest
Change-Id: I2fcb0d2b8e2fe3d85479ac2bad0b7b90f434eb77
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].
These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.
[0] https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020
Change-Id: Id75acea82ddf5d915a8b8805e076dac49cab800f
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.
https://pypi.org/project/pylibyaml/
Change-Id: Iebcc50b5db87518b3b7e0fac124c712afd06da2b
Use pip3 in event system has both pip2 and pip3 installed.
Use apt to install setuptools for Ansible's consumption.
Change-Id: I6929ecb0cce2ec8ac70e9261acb9f87dc7031153
Co-authored-by: Alexander Hughes <Alexander.Hughes@pm.me>
Policy validation in Deckhand was not implemented completely. Refer link
below:
https://airshipit.readthedocs.io/projects/deckhand/en/latest/users/validation.html#policy-validations
This PS removes some of the code related to the feature which was being
used in a code path when a set of documents are uploaded to Deckhand.
In standard Airship deployments the number of documents could be quite
high and this leads to significant delay (more than 300seconds in some
cases). As there are no plans to implement the policy validation feature,
it makes sense to remove it from code path which could cause delay and
sometimes timeouts while uploading documents.
This has been tested on a Baremetal lab: GF and BF.
Change-Id: I2ff3f40a7fe37bed5a589fab00d829db726604fe
During Gabbi tests server returns one of
application/json
application/json; charset=UTF-8
in a Content-Type HTTP header, depending on which test is being run.
This might be related to different pip/pip3 versions and dependencies
installed being used during standalone vs. containerized tests.
This patch allows for both returned header's values to be accepted as
valid as a remediate solution until versions of packages and pip/pip3
usage is unified.
Change-Id: Ifb8f2d68e3474946b3df154cb016cc18cfc95d23
Under some circumstances, the payloads retrieved from Barbican do not
match what was stored. This primarily affects surrounding whitespace[0],
but the implications for passphrases are significant, and even for PEM
encoded data, a difference in whitespace in a configmap is enough to
trigger a chart upgrade.
In general, the effort to align Deckhand document types with Barbican
secret types adds complexity without tangible benefit. Barbican does no
enforcement of the contents of the data, and if it did, that could lead
to further incompatibilities.
This change uses the 'opaque' secret type for all secret document types.
Before storage (or caching), the payload is serialized using `repr`, and
base64 encoded. Upon retrieval, the payload is base64 decoded and parsed
back into an object with `ast.literal_eval`.
[0]: https://storyboard.openstack.org/#!/story/2007017
Change-Id: I9c2f3427f52a87aad718f95160cf688db35e1b83
This patchset fixes a bug where Deckhand was failing to perform
substitution and layering on document sets where all the documents had a
storagePolicy of encrypted. Deckhand would attempt to substitute from an
encrypted source document, but when that document marked as encrypted,
it fails because the source doc had been redacted. The behavior now goes
as follows:
- Resolve Barbican references before layering and substitution have been
performed so that the prior two operations don't attempt to operate on a
Barbican reference
- After substitution, redact the destination document if it is marked as
encrypted
- Now, after substition, we can redact the rest of the documents and
substitutions
Change-Id: I725775d554c9eed2692fc6203c416a7119646680
Occasionally when Deckhand is creating secrets in Barbican, Barbican
encounters an error in which a subsequent attempt at creating the
secret would succeed. This patch set adds logic to the Deckhand
Barbican driver to retry secret creates a configurable number of
times to work around this Barbican issue.
Change-Id: I52293195dd708255508949723d89117ce2e32b71
Adds functionality to read context marker and end-user
from request headers and log that information where
available, to aid in tracing transactions that span
multiple Airship components.
Change-Id: I35c9e56f84f29420c4f3c081453cb81aa892fa7d
This patch set fixes the schema pattern enforced by metadata_document
and metadata_control. Currently, both allow a schema with either
pattern:
- ^metadata/Control/v\d+$
- ^metadata/Document/v\d+$
However, the metadata_control schema should only allow the former
and the metadata_document schema should only allow the latter.
Change-Id: Ic1b88a7158755818002de4c88cdf2d7b716f656d
* Fix for diffing issue after rollback in
conjunction with created and deleted buckets.
* Changed rollback function to check against the full set of documents
for a revision instead of just the documents at that particular revision
* Created a document_delete function to encapsulate document deletion
* Added additional test case to check that a rollback to
something other than 0 deletes the created buckets in between
Co-Authored-By: Michael Beaver <michaelbeaver64@gmail.com>
Change-Id: I0d57e67d68def1f15255a8c89290e8c70deedc03
Currently validation fails with "KeyError: 'schema'",
which makes it hard to determine a root cause of error.
Change-Id: Ifd40faf485578cc0a133e17650f8df6758a6c8ae
Recently added replacement check incorrectly uses metadata.schema
and metadata.name to key on the document -- but it should be schema
and metadata.name, the combination of which uniquely defines a
document.
Change-Id: I6cd1679ad41be38cb78d65ce2763e60f7da390d2
This patch set adds additional documentation and unit tests
to validate further replacement scenarios.
In particular this commit adds an additional document check that
looks for documents exisitng in different layers that contain the
same name and same schema without any of them having `replacement: true`
Change-Id: I7c033d32a6755f36e609789a748cbc6d4af06bc2
The document.py `is_control` method incorrectly checks if a document
is a Control document. Per the documentation [0], Control documents
have `metadata.schema` of `metadata/Control/v1`. This commit updates
the `is_control` method to correctly check for Control documents.
[0] 1d4cc81dfa/doc/source/users/document-types.rst (control-documents)
Change-Id: I60ca8f31a61987b4e756784fce0f5a751639ae9e
This PS adds configuration documentation that includes
a literalinclude of the config file as well as some
information on each of the cache config options as
these are important on performance.
Change-Id: I3b06012b8843b7bfbd46307f81397172a41d3675
This package is used for generation autodoc documentation
automatically which can be linked to by Deckhand
documentation from other places. This is to make autodoc
generation work in RTD.
More info: https://pypi.org/project/sphinxcontrib-apidoc/
Change-Id: I43aac82728e5935a5a2626f2fd29d7a7188d19f9
This patch set ensures that documents that substitute data from
encrypted document sources are themselves redacted, assuming that
cleartext-secrets=true. Note that this redaction fix only applies
to the substitution dest/src paths. The data section is already
being correctly redacted for secondhand sources.
Change-Id: I6ce16a109628259b2cc8132cd9db63261b5dbace
This patch set refactors replacement validation checks
in Deckhand's layering module into a separate module for
better code organization.
Change-Id: If973148ac8220b96f61128b8a7266e6fd57e76b9
- If a document has a storage policy of encrypted
- Redacts (sha256) the data section.
- Redacts (sha256) the substition paths.
- Uses the same /documents endpoint, adds a new query parameter
?cleartext-secrets=true to show the non-redacted values.
Change-Id: I42808901b97c667a1148c00fbb7717a0847c9981
Adds a unit test to validate following scenario:
1) create revision 1 with document
2) create revision 2 with no documents
3) rollback to revision 1 (creating revision 3)
Validate that diffing works for rolled-back revision.
All cases above use same bucket.
Also refactors some test logic for neatness.
Change-Id: I71bf7d34e8aae3ad5abb3c53b05cb96a7038ddc2
1. There is no exception called `InvalidRollback` in Deckhand (it
was removed a while back). Instead, the only exception that
db_api.revision_rollback raises is RevisionNotFound from
the revision_get call internally.
So catch that instead from the controller.
2. The default value of parameters is `str` so when revision_id
of '0' is passed to the db module for processing, it skips over
the check for `if revision_id == 0` as revision_id is a str,
not int. So this leverages builtin int converter logic in
falcon [0] but requires uplifting the version of falcon to
at least 1.3.0 to make use of it [1].
[0] https://falcon.readthedocs.io/en/stable/api/routing.html#field-converters
[1] https://falcon.readthedocs.io/en/1.3.0/api/routing.html#field-converters
Change-Id: I068cd9e9b6818a5d51501f2718ee2d40d556c094
This patch set adds validation logic to document_validation.py (in
Deckhand's engine module) so that components that rely on
Deckhand's engine for document rendering (such as Promenade
or Pegleg) can fail fast when they provide Deckhand with a
duplicate document. Must pass pre_validate=True to layering
module which currently is the case for Promenade, et. al.
Before this change, Deckand only supported this logic at
the DB level (requiring service instantion); this is now no longer
the case.
Change-Id: I6d1c8214775aa0f3b5efb1049972cf847f74585b
This patch set corrects logic for an edge case in layering where
the action `path` is set to `.data`. In this case this means
that the root of the data section should be used, i.e. '.'
or '$.'. The previous adjustment was incorrect: .data was being
changed to empty string ''. This fixes that logic to change to
'.'.
Change-Id: Id6cf0d4d65020220c540eb162a33055035336cde
Pegleg has linting rules dedicated to checking for explicit starts
and so on, so it makes sense that Deckhand just adds this in for
every response as it is a nice feature that better delineates
starting and endpoints points for individual YAML documents.
Change-Id: I6324cfa268ddf250a9c78cb663e7015a171bbc19
Related-Change: https://review.openstack.org/#/c/604123