When performing substitutions, there are occasions when the source value
does not exactly match the format required by the destination document
(e.g. the values.yaml structure of an Armada chart).
This change provides the ability extract a substring of the source
value, and substitute that into the destination document.
Two optional fields are added to `src` under `metadata.substitutions`:
* `pattern`: a regular expression, with optional capture groups
* `match_group`: the number of the desired capture group
The canonical use case is a chart that requires an image with the repo
name and tag in separate fields, while the substitution source has the
full image path as a single value.
For example, assuming that the source document "software-versions" has:
data:
images:
hello: docker.io/library/hello-world:latest
Then the following set of substitutions would put the repo and tag in
the applicable values in the destination document:
metadata:
substitutions:
- src:
schema: pegleg/SoftwareVersions/v1
name: software-versions
path: .images.hello
pattern: '^(.*):(.*)'
match_group: 1
dest:
path: .values.images.hello.repo
- src:
schema: pegleg/SoftwareVersions/v1
name: software-versions
path: .images.hello
pattern: '^(.*):(.*)'
match_group: 2
dest:
path: .values.images.hello.tag
data:
values:
images:
hello:
repo: # docker.io/library/hello-world
tag: # latest
Change-Id: I2fcb0d2b8e2fe3d85479ac2bad0b7b90f434eb77
This patchset fixes a bug where Deckhand was failing to perform
substitution and layering on document sets where all the documents had a
storagePolicy of encrypted. Deckhand would attempt to substitute from an
encrypted source document, but when that document marked as encrypted,
it fails because the source doc had been redacted. The behavior now goes
as follows:
- Resolve Barbican references before layering and substitution have been
performed so that the prior two operations don't attempt to operate on a
Barbican reference
- After substitution, redact the destination document if it is marked as
encrypted
- Now, after substition, we can redact the rest of the documents and
substitutions
Change-Id: I725775d554c9eed2692fc6203c416a7119646680
This package is used for generation autodoc documentation
automatically which can be linked to by Deckhand
documentation from other places. This is to make autodoc
generation work in RTD.
More info: https://pypi.org/project/sphinxcontrib-apidoc/
Change-Id: I43aac82728e5935a5a2626f2fd29d7a7188d19f9
This patch set ensures that documents that substitute data from
encrypted document sources are themselves redacted, assuming that
cleartext-secrets=true. Note that this redaction fix only applies
to the substitution dest/src paths. The data section is already
being correctly redacted for secondhand sources.
Change-Id: I6ce16a109628259b2cc8132cd9db63261b5dbace
Patterns may now be replaced recursively. This can be
achieved by using specifying a ``pattern`` value and
``recurse`` (with a required ``depth`` argument).
Example:
substitutions:
- dest:
path: .
pattern: REGEX
recurse:
depth: -1
src:
schema: deckhand/Passphrase/v1
name: example-password
path: .
NOTE:
Recursive selection of patterns will only consider
matching patterns. Non-matching patterns will be ignored.
Thus, even if recursion can "pass over" non-matching patterns,
they will be silently ignored.
This is useful for reducing the number of substitution
stanzas that are required for the purposes of performing
all the string pattern replacements that are required.
Best practice is to limit the scope of the recursion
as much as possible: e.g. avoid passing in "$" as the
``jsonpath``, but rather a JSON path that lives closer
to the nested strings in question.
Included in this patch set:
* recursive implementation for pattern replacement
* unit tests for most scenarios and edge cases
* docstring updates
* documentation updates
* schema updates for validation
Change-Id: I85048349097ed696667fae80f1180808d264bbcf
This patch set removes the deprecated `substitution_sources`
kwarg from DocumentLayering class. The necessary changes to
Pegleg and Promenade have been made here:
https://review.openstack.org/#/q/topic:substitution_sources+(status:open+OR+status:merged)
So this can be safely dropped.
Also note that this corrects a previously incorrect docstring
in secrets_manager: the substitution_sources kwarg there was
tagged as DEPRECATED but this is only true for the layering
module which currently serves as the front door into the
Deckhand engine.
Change-Id: Ia4872f237a70c9b0c322710bb1ac7db1079357bf
This patch set uses concurrent.futures.ThreadPoolExecutor
[0] to retrieve multiple Barbican secrets concurrently.
This is because currently it is only possible to retrieve
1 secret payload from Barbican at a time -- for revisions
with several dozen secrets it is therefore too costly
to serially perform these API requests.
A new configuration option is added to the [barbican]
group called `max_workers` which specifies the number
of threads to use. The default value is 10. Note that:
"If max_workers is None or not given, it will default
to the number of processors on the machine, multiplied by 5"
[0] so the default is 10 for 2 * 5 which is overly
conservative if anything.
If any error occurs during any of the requests a 500
is raised with appropriate details.
[0] https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor
Change-Id: I76a5bb6c345054e160c14bdf9fb7087e3a746a5e
This patchset adds logic to delete all secret references from Barbican
when deleting all Deckhand revisions -- because this action effectively
purges the Deckhand database. Consequently, to avoid stale secrets
lingering about in Barbican, this action should clean up those
secret references too.
Change-Id: I88782404bd7f6d953cae343f21231b2943eaf7d2
This patchset uses yaml.add_representer for DocumentDict
which enables yaml.safe_load/safe_load_all to correctly
serialize the DocumentDict object without a recursive
routine.
This also completely removes the usage of jsonpath_parse
from DocumentDict as jsonpath-ng is a rather expensive
library to call continuously; and even though Deckhand
does some caching to alleviate this, it is simply better
to avoid it altogether in a wrapper that is used everywhere
across the engine module, which does all the heavy processing.
This also reduces the amount of wrapping using DocumentDict
because the better way to do this in the DB module is to
have a helper function retrieve the data from the DB and
immediately wrap it in a DocumentDict if applicable;
this is left as an exercise for later.
Change-Id: I715ff7e314cf0ec0d34c17f3378514d235dfb377
This patchset moves retrieval of encrypted documents to the
Deckhand controller so that components like Pegleg and
Promenade can consume the Deckhand engine offline without
running into Barbican errors.
Components can pass in `encryption_sources` to Deckhand's
rendering module which Deckhand will now use instead to resolve
secret references.
`encryption_sources` is a dictionary that maps the reference
contained in the destination document's data section to the
actual unecrypted data. If encrypting data with Barbican, the
reference will be a Barbican secret reference.
Change-Id: I1a457d3bd37101d73a28882845c2ce74ac09fdf4
This patchset fixes an edge case with respect to updating
substitution sources after a document has been rendered vis-a-vis
replacement.
Substitution sources only use schema/name which doesn't uniquely
identify replacement documents. Thus, an additional check is
required in `update_substitution_sources` to ensure that the
parent-replacement document doesn't overwrite data for the
child-replacement document.
Note that Deckhand prioritizes the child-replacement to be rendered
immediately after the parent-replacement document, meaning that the
child-replacement document will be the one who correctly updates
the substitution sources (which don't include parent-replacement
documents). Afterward, all other documents that reference the
parent-replacement should get the correct data.
Unit and functional tests have been added for a multi-layer
replacement scenario which regression-test the bug.
Change-Id: Ie6d921d98b7aa87e35a7aa5256cc7da2c0a256b0
This PS adds an integration test scenario for validating that
encrypting a generic document type and using it as a substitution
source during document rendering works.
Deckhand will now submit all generic documents to be encrypted
to Barbican with a 'secret_type' of 'passphrase'. No encoding
is provided Deckhand-side (i.e. base64) because encoding is
deprecated in Barbican since it lead to strange behavior;
Barbican will figure out what to encode the payload as
automatically. For more information, see [0] and [1].
In addition, this PS handles 2 edge cases around secret
payloads that are rejected by Barbican if not handled
correctly by Deckhand: empty payloads and non-string
type payloads [2]. For the first case Deckhand forcibly
changes the document to cleartext because there is no
point in encrypting a document with an empty payload.
For the second case Deckhand sets overrides any
previously set secret_type to 'opaque' and encodes
the payload to base64 -- when it goes to render
the secret it decodes the payload also using base64.
Integration tests have been added to handle both edge
cases described above.
[0] https://bugs.launchpad.net/python-barbicanclient/+bug/1419166
[1] 49505b9aec/barbicanclient/v1/secrets.py (L252)
[2] 49505b9aec/barbicanclient/v1/secrets.py (L297)
Change-Id: I1964aa84ad07b6f310b39974f078b84a1dc84983
This patchset fixes the error message format of a string that
isn't doing correct string substitution. This corrects that.
Change-Id: I2e16e231c7bf1633b41ab30a7f910e9bfb4e416e
This patch set removes few pep8/flake8 ignored rules and implemented
the fix in the code to address those rules.
Change-Id: I2e613acd760818a6e18288d284f6224c38c4353a
Signed-off-by: Tin Lam <tin@irrational.io>
Fix failing pep8 errors which were never being flagged but now are,
possibly due to changes in flake8 rules. This patchset corrects
the following errors:
./deckhand/engine/layering.py:567:21: W503 line break before binary operator
./deckhand/engine/secrets_manager.py:406:33: W503 line break before binary operator
./deckhand/engine/utils.py:33:17: W503 line break before binary operator
./deckhand/common/utils.py:292:17: W503 line break before binary operator
Change-Id: Ic26aecb6b8049e138a826af9953f45298e817795
This patch set accomplishes 2 things:
1) Fixes an issue in Deckhand caused by improperly using the wrong
Barbican secret type for the PrivateKey Deckhand document type.
2) Tests all Deckhand secret types against Barbican via integration
testing.
The current error being raised is this: "Provided object does not match
schema 'Secret': u'privatekey' is not one of ['symmetric', 'passphrase',
'private', 'public', 'certificate', 'opaque']."
Change-Id: I8231c87782902850fe0632d0346c567c7481e95f
This is to fix a use case where a subpath like 'filter:authtoken' in
a JSON path like ".values.conf.paste.'filter:authtoken'.password" fails
because the resulting Python object that is created in memory
for substitution constructs a dict key like "'filter:authtoken'" resulting
in Deckhand failing to index into the dict as the key needs to be
stripped of start and end quotes that are only necessary for achieving
valid YAML syntax.
A unit test is added for regression.
Change-Id: I19974efc977b0cdc5793e649fa068d1a3bd7339e
This is to change passing the secret URI instead of the secret
UUID to barbican's get secret endpoint from which the secret
itself can be extracted.
While the API [0] expects a UUID the CLI instead expects a URI
and the latter extracts the UUID from the URI automatically [1].
API ref:
GET /v1/secrets/{uuid}
Headers:
Accept: application/json
X-Auth-Token: {token}
(or X-Project-Id: {project_id})
CLI ref:
$ barbican help secret get
usage: barbican secret get [-h] [-f {shell,table,value}] [-c COLUMN]
[--max-width <integer>] [--prefix PREFIX]
[--decrypt] [--payload]
[--payload_content_type PAYLOAD_CONTENT_TYPE]
URI
Retrieve a secret by providing its URI.
Finally, this adds logic for ensuring that all encrypted data is retrieved
and injected back into the raw documents with Barbican references, during
document rendering. Currently, this process is only performed for
documents with substitutions, but should also be carried out for encrypted
documents themselves.
[0] https://docs.openstack.org/barbican/latest/api/reference/secrets.html#get-v1-secrets-uuid
[1] https://docs.openstack.org/python-barbicanclient/latest/reference/index.html#barbicanclient.v1.secrets.SecretManager.get
Change-Id: I1717592b7acdedb66353c25fb5dcda2d5330196b
It is currently surprising for users when null is getting substituted
into documents when there is no actual source data to grab.
Silent "None" substitution sometimes results in surprising symptoms in
complex configurations.
Depends-On: If2b08f443cde765a1dbfaf7bac6b549591e59148
Change-Id: I253dc1d10f9493b9611fb6abd86810c6d57afbf6
This PS introduces a new exception SubstitutionSourceSecretNotFound
which is raised when a src.path referenced under the substitutions
of a destination document isn't found in the data section of
the corresponding source document if fail_on_missing_sub_src
is True -- else a warning is logged.
Change-Id: If2b08f443cde765a1dbfaf7bac6b549591e59148
This adds support for document replacement to the
Deckhand engine _only_ for the following scenarios
_only_:
* generic case (a replaces b, returns a only)
* substitution case (a replaces b, c substitutes from a instead)
TODO:
* layering case (a replaces b, c layers with a instead)
* Modify Document unique constraint to work with
(schema, name, layer) throughout all of Deckhand
(including controllers, database models, and anywhere
else as needed)
Change-Id: Ie2cea2a49ba3b9ebc42706fbe1060d94db2e5daa
This updates the unique constraint for Document model from
schema/metadata.name to schema/metadata.name/layer which is
a pre-requisite for document replacement implementation.
The remainder fo the changes are taken of in child PS
(particulary those related to the layering module):
https://review.gerrithub.io/#/c/403888/
Change-Id: Icc4f4960b3a3951f649c7886dbe0bce77341a9f7
This is to fix secrets_manager.SecretsManager.get method which
is passing in the secret reference to Barbican directly for
GET /secrets/{uuid} [0] causing Barbican to raise a
ValueError exception when it attempts to validate that
{secret_uuid} is in fact a UUID.
The fix is to extract the secret_uuid from the secret_ref returned
by Barbican before querying the GET /secrets/{uuid} API.
[0] https://docs.openstack.org/barbican/latest/api/reference/secrets.html#get-v1-secrets-uuid
Change-Id: I4db317e3ba12b4268df5b84b79be8da1da5ac2ba
This is to fix the condition in secrets_manager used to determine
whether the substitution is secret. It currently checks whether the
potential secret reference contains the substring
of 'key-manager/v1/secrets' but the environment-agnostic way
of doing this is to check whether the secret reference contains
the barbican endpoint registered under CONF.barbican.api_endpoint.
Change-Id: I633021571255c8393e19ec60a614ede981a86d9f
This is to add better logging to assist with debugging 500
Internal Server Errors that manifest from either internal
bugs in Deckhand or document typos/errors ingested by Deckhand
during document substitution.
Change-Id: I5f4c1cb07bea8e6546d08a858d4f83a24d75ef5a
This sanitizes any potential secrets contained in
'message' section of validation output if the document
has substitions (implying that a secret may have been
substituted into it) or if the document itself was
encrypted, implying that the document already contains
a secret.
Change-Id: I394eb8c4e6002f896ecdaa14d2be1e5f948e5048
This PS fixes Deckhand currently wrongly substituting the secret
reference Barbican returns into documents, rather than the secret
payload itself.
Closes #19
Change-Id: I1d4eed85ed336e83a777b4343f37b10c91038342
This PS resolves a bug related to the _substitution_sources in
secrets_manager.SecretsSubstitution not getting updated with
the most recently updated layering data.
Currently, the DocumentLayering class, during initialization, passes
the list of substitution sources to the SecretsSubstitution class
as an optimization. During layering, documents that are substitution
sources can have their data updated -- and if their data is not
updated then that implies that a substitution source's data is
stale -- causing future substitutions using that substitution source
data to use stale data.
The solution is to introduce a new method called
`update_substitution_sources` which updates a specific substitution
source entry with the most update-to-date layered data following
every single layering update, such that all substitution sources
should always have the most up-to-date data.
Change-Id: Idc375cfdf17375d3c401342dff259bdcd1718941
This PS introduces a flag to only warn on missing substitution
sources because right now Promenade is failing on that. However,
a PS will also have to be added to Promenade to set the new flag
-- `fail_on_missing_sub_src` -- to False during genesis.
Change-Id: I462f721c41e23d2e5e3e698c0bd452b6764d51eb
This PS causes layering module to fail fast on malformed
``metadata.substitutions`` entry in a document by performing
built-in schema validation when validate=True is passed to
the DocumentLayering constructor. This kwarg is useful for
when the layering module is called directly -- i.e. by
Promenade or Pegleg. (The Deckhand server already performs
document pre-validation during document ingestion so there
is no need for documents stored inside Deckhand to be
re-validated again for rendered-documents endpoint.)
Next, a new exception was added -- SubstitutionSourceNotFound
-- which is raised when a substitution document is referenced
by another document but isn't found.
Finally, the previous exception raised by the
secrets_manager module has been renamed to UnknownSubstitutionError
which now raises a 500 instead of a 400 as this exception will
most likely be due to an internal server error of some kind.
Unit tests were added and documentation changes were made.
Change-Id: Idfd91a52ef9ffd8f9b1c06c6b84c3405acab6f16
Currently, Deckhand fails to perform substitution given
a substitution dependency chain for a group of documents.
That is, if A depends on B and C depends on B for substitution
then substitution will fail. Deckhand, at present, can only
perform substitution if A depends on B and C depends on B
and D depends on B, for example: In this case, the dependency
chain is no bigger than 2.
However, for cases where the substitution dependency chain
is larger than 2, then the dependency is no longer trivial. This
is because the substitution dependencies form a DAG that must
be topologically sorted in order to derive the correct order of
substitution. Once the documents are correctly sorted according
to this scheme, then substitution can be carried out as usual.
This PS makes the aforementioned changes to Deckhand's layering
module to make substitution work for non-cyclical dependencies:
A DAG is used to topologically sort the documents according to
their substitution dependency chain. A unit and functional
test has been added to verify the code.
If a cycle is detected, a critical error is thrown. Unit tests
have been added to validate this case as well.
Change-Id: Iaca3963f44aec6c897ad9fd690ce314a3a4d97a2
Also, this accounts for the the resp.to_dict() call not absorbing
the secret_ref key.
This commit also adds type mapping for missing secret types to
their barbican equivalents.
Change-Id: Idd4895fd441443a3dc41a3358edf6bd3648be5c1
Recently the Deckhand Validations API was updated to
return a more nuanced validation error message in
https://review.gerrithub.io/#/c/396251/
This PS adds functional tests to validate the new
format as well as to validate that secrets are
sanitized from the validation error output for
security reasons.
Change-Id: Ic92f2ccd03946a8897aeb2c2a506b3c7eb89836b
This PS solves the following issues for which only minor changes
were needed:
1) Using copy.copy() around the substitution value passed in
to jsonpath_replace so as to avoid updating the substitution
entry referentially which was happening before this change,
causing future substitutions using that entry to fail or
behaving unexpectedly.
2) Supporting non-string substitution values when a substitution
pattern is provided: before this change, this was failing
because calling re.sub() and passing in a non-string
value causes an error to be thrown.
3) Adding better logging and error handling to
deckhand.utils.jsonpath_replace to assist with debugging.
Unit tests are included for some of the scenarios above.
Change-Id: I8562d43a717f477e3297504c1522331b3a993f88
This PS resolves two issues, both related to not dynamically creating
missing arrays in the destination document, when the substitution
destination path indicates that an array should be created.
The first issue is that the array itself isn't created. For example,
if substitutions.dest.path = '.foo.bar[0].baz' then the data
section of the destination document should render like this:
data:
foo:
bar:
- baz: <some-value>
The second issue this PS resolves is not nuking the destination
document data if JSON path creation failed (for whatever reason).
This means that if jsonpath_replace in utils returns None,
then a warning is instead logged, with no substitution taking
place for a specific substitution in a document.
Change-Id: I87d0bb606b74fc1e05669da639ab22ec7bf55b25
This is to make Deckhand layering more performant. Layering is
currently the main bottleneck in the rendered-documents endpoint.
The bottleneck is specifically related to calculating document
children in the layering module. The runtime was O(N^2) but
has been decreased to ~O(N) resulting in much faster performance
overall. Using local testing against the lab deployment YAML,
runtime for layering is decreased to 15 seconds or so, down
from 55 seconds, which is roughly 4 times faster. This
performance shouldn't increase by much given even larger
YAMLs due to the linear-time performance change.
Change-Id: Ib5f7fd08a38d05ae79d18227f8aafc25bd13f7ca
Currently the format of the validation error messages returned
by the Deckhand Validation API is lacking. The new response is a
dictionary with the following keys:
* validation_schema: The schema body that was used to validate the
document.
* schema_path: The JSON path in the schema where the failure originated.
* name: The document name.
* schema: The document schema.
* path: The JSON path in the document where the failure originated.
* error_section: The "section" in the document above which the error
originated (i.e. the dict in which ``path`` is found).
* message: The error message returned by the ``jsonschema`` validator.
This PS updates the document validation module and associated unit tests
to return and verify the above format.
Change-Id: I9ef1c36db85233cbfb866dea786228ef1416468c
This PS adds multiple log statements to the secrets manager
module to assist with debugging and also uses a dictionary keyed
with (schema, name) => document to quickly retrieve substitution
sources; rather than doing a linear-time lookup per iteration,
a constant-time lookup is used instead, with only one linear-time
initialization done during the constructor to initialize the
substitution source dictionary.
Change-Id: I209430c48312be621fdc3787009346d3e9c12ac6
This PS simplifies the DocumentWrapper class by changing the way
it is designed. The purpose of the class was to make it easier
to retrieve nested dictionary attributes from a document. The class
previously inherited from `object` meaning that the object could not
directly be treated as a dictionary, complicating usage of the class.
With this change, the class now inherits from a `dict` meaning that
it can be manipulated the same way a dictionary can, while still
able to return nested dictionary attributes without having to worry
about exceptions getting thrown.
Each property implemented by `DocumentWrapper` uses jsonpath_parse
implements in `deckhand.utils` to retrieve nested attributes or
else self.get() to retrieve first-level dictionary attributes.
Change-Id: I1d73a79aa4c3117be31aab978c20258c1052ad6d
This PS moves DB calls out of the engine module and into the
appropriate controllers so that a "production" set of documents
can leverage Deckhand layering and substitution after importing
the Deckhand engine module directly. These operations will be
carried out offline meaning that DB calls are not possible.
Unit tests were refactored to work with the changes.
Some testing documentation was also updated.
Closes 16
Change-Id: I6e0757746cd949985d57102d1c85acfbbed86078
This PS fixes a bug related to Deckhand only using "secret"
document types to be used as substitution sources; the substitution
logic should be made generic, because it shouldn't just apply to
secrets.
This entailed removing the "is_secret" database column from the
Document table as it's no longer needed and dropping it from a DB
query made to find the source document for substitution in the
secrets_manager module.
This PS also increased resiliency via exception handling and some
edge cases surrounding substitution.
Finally, unit tests and functional tests were added to validate
substitition using a generic document as the source.
Change-Id: I2c4b49b2eb55473c56b8253a456803e793b0b0b0
Unusual documents are documents with different data
types for the data field. The data types include:
object, array, string and integer.
This PS makes necessary ORM model and schema
changes needed to support the different data types.
The ORM data type for the data column has been changed
to JSONB for PostgreSQL. Thus, DH now only supports
PostgreSQL. As a result, the tox jobs have been updated
to only use postgre.
Change-Id: I53694d56bef71adacb5eb79162678be73acb4ad8
This PS integrates layering functionality with rendered
documents API endpoint. No new functionality was really
added; instead, a lot of code was refactored to make
layering work alongside substitution.
The following changes have been made:
- Moved document filtering functionality to deckhand.utils
because rendered documents must be filtered twice: once
to retrieve all documents necessary for rendering from
the DB and again by the controller to filter out documents
the user doesn't want returned
- Additional LOG statements in the layering module
- Additional layering unit tests
- Additional functional tests
- Removal of some stricter validations around layering:
if a parent document is not found for a document,
an error is no longer returned, as not all documents
need to have a parent (that is, not all documents
need to be rendered together, though this might need
to be expanded on later: what if a document has a
`parentSelector` but no parent is found?)
Change-Id: I6c66ed824fba0216ba868a6101a72cfe3bdda181
A document with a distinct metadata.name/schema can be re-created
in a different bucket after it has been deleted in its original
bucket. This is also true for layering policies.
This PS fixes the above problem. It also updates validation
documentation to be clearer.
Change-Id: If232f6ca613611995674f9d0149d5f4e5d155040
Phil Sphicas