update dockerfile for python deckhand install
add deckhand version to chart 1.0
add chart version 0.2.0
update all packages to latest in requirements.txt
update zuul jobs for focal and python 3.8
remove zuul job functional-uwsgi-py38 in favor of functional-docker-py38
update tox config
typecast to string in re.sub() function
add stestr to test-requirements.txt
add SQLAlchemy jsonpickle sphinx-rtd-theme stestr to requirements.txt
deprecated function: BarbicanException -> BarbicanClientException
fix mock import using unittest
fix import collections to collections.abc
fix for collections modules for older than python 3.10 versions.
deprecated function: json -> to_json
deprecated function: werkzeug.contrib.profiler ->
werkzeug.middleware.profiler
deprecated function: falcon.AIP -> falcon.App
deprecation warning: switch from resp.body to resp.text
rename fixtures to dh_fixtures because there is an imported module
fixtures
switch from stream.read to bounded_stream.read
deprecated function: falcon process_response needed additional parameter
deprecated function: falcon default_exception_handler changed parameter
order
move from MagicMock object to falcon test generated object to fix
incompatability with upgraded Falcon module.
Adjust gabbi tests to fix incompatability with upgraded DeepDiff module
update Makefile to execute ubuntu_focal
update HTK (helmtoolkit)
unpin barbican to pass integration tests
Use helm 3 in chart build.
`helm serve` is removed in helm 3 so this moves
to using local `file://` dependencies [0] instead.
Change-Id: I180416f480edea1b8968d80c993b3e1fcc95c08d
When performing substitutions, there are occasions when the source value
does not exactly match the format required by the destination document
(e.g. the values.yaml structure of an Armada chart).
This change provides the ability extract a substring of the source
value, and substitute that into the destination document.
Two optional fields are added to `src` under `metadata.substitutions`:
* `pattern`: a regular expression, with optional capture groups
* `match_group`: the number of the desired capture group
The canonical use case is a chart that requires an image with the repo
name and tag in separate fields, while the substitution source has the
full image path as a single value.
For example, assuming that the source document "software-versions" has:
data:
images:
hello: docker.io/library/hello-world:latest
Then the following set of substitutions would put the repo and tag in
the applicable values in the destination document:
metadata:
substitutions:
- src:
schema: pegleg/SoftwareVersions/v1
name: software-versions
path: .images.hello
pattern: '^(.*):(.*)'
match_group: 1
dest:
path: .values.images.hello.repo
- src:
schema: pegleg/SoftwareVersions/v1
name: software-versions
path: .images.hello
pattern: '^(.*):(.*)'
match_group: 2
dest:
path: .values.images.hello.tag
data:
values:
images:
hello:
repo: # docker.io/library/hello-world
tag: # latest
Change-Id: I2fcb0d2b8e2fe3d85479ac2bad0b7b90f434eb77
Under some circumstances, the payloads retrieved from Barbican do not
match what was stored. This primarily affects surrounding whitespace[0],
but the implications for passphrases are significant, and even for PEM
encoded data, a difference in whitespace in a configmap is enough to
trigger a chart upgrade.
In general, the effort to align Deckhand document types with Barbican
secret types adds complexity without tangible benefit. Barbican does no
enforcement of the contents of the data, and if it did, that could lead
to further incompatibilities.
This change uses the 'opaque' secret type for all secret document types.
Before storage (or caching), the payload is serialized using `repr`, and
base64 encoded. Upon retrieval, the payload is base64 decoded and parsed
back into an object with `ast.literal_eval`.
[0]: https://storyboard.openstack.org/#!/story/2007017
Change-Id: I9c2f3427f52a87aad718f95160cf688db35e1b83
This patch set ensures that documents that substitute data from
encrypted document sources are themselves redacted, assuming that
cleartext-secrets=true. Note that this redaction fix only applies
to the substitution dest/src paths. The data section is already
being correctly redacted for secondhand sources.
Change-Id: I6ce16a109628259b2cc8132cd9db63261b5dbace
Adds a unit test to validate following scenario:
1) create revision 1 with document
2) create revision 2 with no documents
3) rollback to revision 1 (creating revision 3)
Validate that diffing works for rolled-back revision.
All cases above use same bucket.
Also refactors some test logic for neatness.
Change-Id: I71bf7d34e8aae3ad5abb3c53b05cb96a7038ddc2
This patchset adds Barbican validation/assertions to integration
tests by querying the Barbican API server where appropriate
and validating that the expected data is returned in order
to sanity-check the integration scenarios further.
Change-Id: If5d30712b289f09ac9712ee205673be4150cda16
This patchset moves retrieval of encrypted documents to the
Deckhand controller so that components like Pegleg and
Promenade can consume the Deckhand engine offline without
running into Barbican errors.
Components can pass in `encryption_sources` to Deckhand's
rendering module which Deckhand will now use instead to resolve
secret references.
`encryption_sources` is a dictionary that maps the reference
contained in the destination document's data section to the
actual unecrypted data. If encrypting data with Barbican, the
reference will be a Barbican secret reference.
Change-Id: I1a457d3bd37101d73a28882845c2ce74ac09fdf4
This PS adds an integration test scenario for validating that
encrypting a generic document type and using it as a substitution
source during document rendering works.
Deckhand will now submit all generic documents to be encrypted
to Barbican with a 'secret_type' of 'passphrase'. No encoding
is provided Deckhand-side (i.e. base64) because encoding is
deprecated in Barbican since it lead to strange behavior;
Barbican will figure out what to encode the payload as
automatically. For more information, see [0] and [1].
In addition, this PS handles 2 edge cases around secret
payloads that are rejected by Barbican if not handled
correctly by Deckhand: empty payloads and non-string
type payloads [2]. For the first case Deckhand forcibly
changes the document to cleartext because there is no
point in encrypting a document with an empty payload.
For the second case Deckhand sets overrides any
previously set secret_type to 'opaque' and encodes
the payload to base64 -- when it goes to render
the secret it decodes the payload also using base64.
Integration tests have been added to handle both edge
cases described above.
[0] https://bugs.launchpad.net/python-barbicanclient/+bug/1419166
[1] 49505b9aec/barbicanclient/v1/secrets.py (L252)
[2] 49505b9aec/barbicanclient/v1/secrets.py (L297)
Change-Id: I1964aa84ad07b6f310b39974f078b84a1dc84983
This patchset adds a regression unit test for validating that index >= 10
works with substitution. This handles indices [10] and [11] together,
validating multiple cases:
- that index == 10 works
- that index > 10 works
- that high-level indicies can work in a row
Change-Id: I3c018d96ed43bf88417d9d30bdc3cb0ffa09225c
This is to fix a use case where a subpath like 'filter:authtoken' in
a JSON path like ".values.conf.paste.'filter:authtoken'.password" fails
because the resulting Python object that is created in memory
for substitution constructs a dict key like "'filter:authtoken'" resulting
in Deckhand failing to index into the dict as the key needs to be
stripped of start and end quotes that are only necessary for achieving
valid YAML syntax.
A unit test is added for regression.
Change-Id: I19974efc977b0cdc5793e649fa068d1a3bd7339e
This is to change passing the secret URI instead of the secret
UUID to barbican's get secret endpoint from which the secret
itself can be extracted.
While the API [0] expects a UUID the CLI instead expects a URI
and the latter extracts the UUID from the URI automatically [1].
API ref:
GET /v1/secrets/{uuid}
Headers:
Accept: application/json
X-Auth-Token: {token}
(or X-Project-Id: {project_id})
CLI ref:
$ barbican help secret get
usage: barbican secret get [-h] [-f {shell,table,value}] [-c COLUMN]
[--max-width <integer>] [--prefix PREFIX]
[--decrypt] [--payload]
[--payload_content_type PAYLOAD_CONTENT_TYPE]
URI
Retrieve a secret by providing its URI.
Finally, this adds logic for ensuring that all encrypted data is retrieved
and injected back into the raw documents with Barbican references, during
document rendering. Currently, this process is only performed for
documents with substitutions, but should also be carried out for encrypted
documents themselves.
[0] https://docs.openstack.org/barbican/latest/api/reference/secrets.html#get-v1-secrets-uuid
[1] https://docs.openstack.org/python-barbicanclient/latest/reference/index.html#barbicanclient.v1.secrets.SecretManager.get
Change-Id: I1717592b7acdedb66353c25fb5dcda2d5330196b
This PS introduces a new exception SubstitutionSourceSecretNotFound
which is raised when a src.path referenced under the substitutions
of a destination document isn't found in the data section of
the corresponding source document if fail_on_missing_sub_src
is True -- else a warning is logged.
Change-Id: If2b08f443cde765a1dbfaf7bac6b549591e59148
This updates the unique constraint for Document model from
schema/metadata.name to schema/metadata.name/layer which is
a pre-requisite for document replacement implementation.
The remainder fo the changes are taken of in child PS
(particulary those related to the layering module):
https://review.gerrithub.io/#/c/403888/
Change-Id: Icc4f4960b3a3951f649c7886dbe0bce77341a9f7
This is to fix secrets_manager.SecretsManager.get method which
is passing in the secret reference to Barbican directly for
GET /secrets/{uuid} [0] causing Barbican to raise a
ValueError exception when it attempts to validate that
{secret_uuid} is in fact a UUID.
The fix is to extract the secret_uuid from the secret_ref returned
by Barbican before querying the GET /secrets/{uuid} API.
[0] https://docs.openstack.org/barbican/latest/api/reference/secrets.html#get-v1-secrets-uuid
Change-Id: I4db317e3ba12b4268df5b84b79be8da1da5ac2ba
This PS fixes Deckhand currently wrongly substituting the secret
reference Barbican returns into documents, rather than the secret
payload itself.
Closes #19
Change-Id: I1d4eed85ed336e83a777b4343f37b10c91038342
Also, this accounts for the the resp.to_dict() call not absorbing
the secret_ref key.
This commit also adds type mapping for missing secret types to
their barbican equivalents.
Change-Id: Idd4895fd441443a3dc41a3358edf6bd3648be5c1
This PS solves the following issues for which only minor changes
were needed:
1) Using copy.copy() around the substitution value passed in
to jsonpath_replace so as to avoid updating the substitution
entry referentially which was happening before this change,
causing future substitutions using that entry to fail or
behaving unexpectedly.
2) Supporting non-string substitution values when a substitution
pattern is provided: before this change, this was failing
because calling re.sub() and passing in a non-string
value causes an error to be thrown.
3) Adding better logging and error handling to
deckhand.utils.jsonpath_replace to assist with debugging.
Unit tests are included for some of the scenarios above.
Change-Id: I8562d43a717f477e3297504c1522331b3a993f88
This PS resolves two issues, both related to not dynamically creating
missing arrays in the destination document, when the substitution
destination path indicates that an array should be created.
The first issue is that the array itself isn't created. For example,
if substitutions.dest.path = '.foo.bar[0].baz' then the data
section of the destination document should render like this:
data:
foo:
bar:
- baz: <some-value>
The second issue this PS resolves is not nuking the destination
document data if JSON path creation failed (for whatever reason).
This means that if jsonpath_replace in utils returns None,
then a warning is instead logged, with no substitution taking
place for a specific substitution in a document.
Change-Id: I87d0bb606b74fc1e05669da639ab22ec7bf55b25
This is to make Deckhand layering more performant. Layering is
currently the main bottleneck in the rendered-documents endpoint.
The bottleneck is specifically related to calculating document
children in the layering module. The runtime was O(N^2) but
has been decreased to ~O(N) resulting in much faster performance
overall. Using local testing against the lab deployment YAML,
runtime for layering is decreased to 15 seconds or so, down
from 55 seconds, which is roughly 4 times faster. This
performance shouldn't increase by much given even larger
YAMLs due to the linear-time performance change.
Change-Id: Ib5f7fd08a38d05ae79d18227f8aafc25bd13f7ca
This PS moves DB calls out of the engine module and into the
appropriate controllers so that a "production" set of documents
can leverage Deckhand layering and substitution after importing
the Deckhand engine module directly. These operations will be
carried out offline meaning that DB calls are not possible.
Unit tests were refactored to work with the changes.
Some testing documentation was also updated.
Closes 16
Change-Id: I6e0757746cd949985d57102d1c85acfbbed86078
This PS fixes a bug related to Deckhand only using "secret"
document types to be used as substitution sources; the substitution
logic should be made generic, because it shouldn't just apply to
secrets.
This entailed removing the "is_secret" database column from the
Document table as it's no longer needed and dropping it from a DB
query made to find the source document for substitution in the
secrets_manager module.
This PS also increased resiliency via exception handling and some
edge cases surrounding substitution.
Finally, unit tests and functional tests were added to validate
substitition using a generic document as the source.
Change-Id: I2c4b49b2eb55473c56b8253a456803e793b0b0b0
Unusual documents are documents with different data
types for the data field. The data types include:
object, array, string and integer.
This PS makes necessary ORM model and schema
changes needed to support the different data types.
The ORM data type for the data column has been changed
to JSONB for PostgreSQL. Thus, DH now only supports
PostgreSQL. As a result, the tox jobs have been updated
to only use postgre.
Change-Id: I53694d56bef71adacb5eb79162678be73acb4ad8
Currently, the rendered-documents endpoint returns only documents
that require substitution, rather than all concrete documents, as
specified in the requirements (DECKHAND-65).
This PS adds a filter to the endpoint so that only concrete documents
are returned. Also, all concrete documents are returned, not just
the ones that require substitution.
Included in this PS:
- logic changes described above
- unit test to verify the above logic
Change-Id: Ib552b084bb00b6e180bba973be420449a292fb05
This PS implements documentation substitution and
the rendered-documents endpoint. Each time the
rendered-documents is queried, the documents for
the reqeust revision_id dynamically undergo
secret substitution.
All functional tests related to secret substitution
have been unskipped.
Deckhand currently does not real testing for
verifying that secret substitution works
for encrypted documents. This will only happen
when integration testing is added to Deckhand to
test its interaction with Keystone and Barbican.
Included in this PS:
- basic implementation for secret substitution
- introduction of jsonpath_ng for searching for and
updating jsonpaths in documents
- rendered-documents endpoint
- unit tests
- all relevant functional tests unskipped
- additional bucket controller tests include RBAC
tests and framework testing RBAC via unit tests
Change-Id: I86f269a5b616b518e5f742a4005891412226fe2a
This commit adds a DocumentSecret model to the DB for
storing secrets directly in Deckhand as well as references
to secrets stored in Barbican if the encryption type
for the secret is encrypted.
This commit also adds a new class called SecretsManager
for managing the lifecycle of secrets from a higher level.
This commit also adds Postgres compliance. So now all
the DB models should work with Postgres.
Also includes unit tests.
Change-Id: Id7c4be8de2e70735f42b1f6710139d553ab4bea2
Wahlstedt, Walter (ww229g)