- adjusted .gitignore to keep fresh egg-info and omit build artifacts
- fresh egg-info data is needed for promenade that depends on Deckhand
- restored deckhand-functional-uwsgi-py38 gate
- restored deckhand-integration-uwsgi-py38 gate
- made deckhand-airskiff-deployment gate voting ( treasuremap project
has been updated)
- removed bionic gates
- updated focal dockerfile
- added more binary deps into bindep.txt
- updated deckhand chart values to latest images - focal and wallaby
- fixed python code to compy with CVE's found by fresh version of bandit
- implemented pip freeze approach
- added tox -e freeze profile to manage it
- requirements-frozen.txt is now main file with requirements
- requirements-direct.txt is the file to control deps
- updated setup.cfg to adjust to newer version of setuptools
- fixed airskiff-deploy gate
- fixed docker-image-build playbook to restore Quay repo image publish
- updated other playbooks to include roles from zuul/base-jobs in order
to setup build hosts properly
- removed workaround with hardcoded dns resolver ip 10.96.0.10 as it
became obsolette due to recent fix in openstack-helm-infra
- adjusted tools/whitespace-linter.sh script
- tox.ini has been brought to compliance with tox4 requirements
- replaced str() calls with six.text_type() according to D325 Deckhand specific
commandment from Hacking.rst
- locked python-barbicanclient version with 5.2.0 because of breaking
changes in the upper versions
Change-Id: I1cd3c97e83569c4db7e958b3400bdd4b7ea5e668
This patchset fixes a bug where Deckhand was failing to perform
substitution and layering on document sets where all the documents had a
storagePolicy of encrypted. Deckhand would attempt to substitute from an
encrypted source document, but when that document marked as encrypted,
it fails because the source doc had been redacted. The behavior now goes
as follows:
- Resolve Barbican references before layering and substitution have been
performed so that the prior two operations don't attempt to operate on a
Barbican reference
- After substitution, redact the destination document if it is marked as
encrypted
- Now, after substition, we can redact the rest of the documents and
substitutions
Change-Id: I725775d554c9eed2692fc6203c416a7119646680
Currently validation fails with "KeyError: 'schema'",
which makes it hard to determine a root cause of error.
Change-Id: Ifd40faf485578cc0a133e17650f8df6758a6c8ae
This patch set adds additional documentation and unit tests
to validate further replacement scenarios.
In particular this commit adds an additional document check that
looks for documents exisitng in different layers that contain the
same name and same schema without any of them having `replacement: true`
Change-Id: I7c033d32a6755f36e609789a748cbc6d4af06bc2
This patch set ensures that documents that substitute data from
encrypted document sources are themselves redacted, assuming that
cleartext-secrets=true. Note that this redaction fix only applies
to the substitution dest/src paths. The data section is already
being correctly redacted for secondhand sources.
Change-Id: I6ce16a109628259b2cc8132cd9db63261b5dbace
This patch set refactors replacement validation checks
in Deckhand's layering module into a separate module for
better code organization.
Change-Id: If973148ac8220b96f61128b8a7266e6fd57e76b9
This patch set corrects logic for an edge case in layering where
the action `path` is set to `.data`. In this case this means
that the root of the data section should be used, i.e. '.'
or '$.'. The previous adjustment was incorrect: .data was being
changed to empty string ''. This fixes that logic to change to
'.'.
Change-Id: Id6cf0d4d65020220c540eb162a33055035336cde
This implements a rendered documents cache which is keyed by
revision IDs. This means that repeatedly trying to re-render
documents for the same revision ID will be much, much faster as
the results will be cached.
Change-Id: Ie92f55a9234d038683ba1fcad76710d968ed67ab
This patch set removes the deprecated `substitution_sources`
kwarg from DocumentLayering class. The necessary changes to
Pegleg and Promenade have been made here:
https://review.openstack.org/#/q/topic:substitution_sources+(status:open+OR+status:merged)
So this can be safely dropped.
Also note that this corrects a previously incorrect docstring
in secrets_manager: the substitution_sources kwarg there was
tagged as DEPRECATED but this is only true for the layering
module which currently serves as the front door into the
Deckhand engine.
Change-Id: Ia4872f237a70c9b0c322710bb1ac7db1079357bf
This patchset uses yaml.add_representer for DocumentDict
which enables yaml.safe_load/safe_load_all to correctly
serialize the DocumentDict object without a recursive
routine.
This also completely removes the usage of jsonpath_parse
from DocumentDict as jsonpath-ng is a rather expensive
library to call continuously; and even though Deckhand
does some caching to alleviate this, it is simply better
to avoid it altogether in a wrapper that is used everywhere
across the engine module, which does all the heavy processing.
This also reduces the amount of wrapping using DocumentDict
because the better way to do this in the DB module is to
have a helper function retrieve the data from the DB and
immediately wrap it in a DocumentDict if applicable;
this is left as an exercise for later.
Change-Id: I715ff7e314cf0ec0d34c17f3378514d235dfb377
This patchset moves retrieval of encrypted documents to the
Deckhand controller so that components like Pegleg and
Promenade can consume the Deckhand engine offline without
running into Barbican errors.
Components can pass in `encryption_sources` to Deckhand's
rendering module which Deckhand will now use instead to resolve
secret references.
`encryption_sources` is a dictionary that maps the reference
contained in the destination document's data section to the
actual unecrypted data. If encrypting data with Barbican, the
reference will be a Barbican secret reference.
Change-Id: I1a457d3bd37101d73a28882845c2ce74ac09fdf4
This patchset adds orientation='reverse' when looking for
substitution cycles in the layering module in order to work
with newer versions of networkx which throw:
networkx.exception.NetworkXUnfeasible: Graph contains a cycle.
This corrects the issue.
Change-Id: I854a10ed524daf45bef9ad856938ffedb3e76baf
This patchset adds support for layering between primitive types
of data. It is not possible to merge together a string with
an integer or a boolean with a dictionary -- however, merge
actions can still be resolved in such cases in order to get layering
to work:
merge => child replaces parent data
replace => child replaces parent data
delete => empty dictionary (in effect)
The only path that is possible in such cases is "." as
a primitive doesn't have a JSON path. Else missing document key
exceptions are raised.
Change-Id: I9da557bf9444c37b3e59672f3a93f49fdf1e4a02
This patchset fixes an edge case with respect to updating
substitution sources after a document has been rendered vis-a-vis
replacement.
Substitution sources only use schema/name which doesn't uniquely
identify replacement documents. Thus, an additional check is
required in `update_substitution_sources` to ensure that the
parent-replacement document doesn't overwrite data for the
child-replacement document.
Note that Deckhand prioritizes the child-replacement to be rendered
immediately after the parent-replacement document, meaning that the
child-replacement document will be the one who correctly updates
the substitution sources (which don't include parent-replacement
documents). Afterward, all other documents that reference the
parent-replacement should get the correct data.
Unit and functional tests have been added for a multi-layer
replacement scenario which regression-test the bug.
Change-Id: Ie6d921d98b7aa87e35a7aa5256cc7da2c0a256b0
This patch set removes few pep8/flake8 ignored rules and implemented
the fix in the code to address those rules.
Change-Id: I2e613acd760818a6e18288d284f6224c38c4353a
Signed-off-by: Tin Lam <tin@irrational.io>
Currently it doesn't seem document replacement works
exactly as expected: The parent-replacement document
can receive layering and substitution data prior to
being replaced. Currently, Deckhand does not account
for this scenario.
A child-replacement depends on its parent-replacement
the same way any child depends on its parent: so that the
child layers with its parent only after the parent has
received all layering and substitution data. But other
documents that depend on the parent-replacement actually
depend on the child-replacement instead as the
child-replacement replaces its parent. So the dependency
chain is: PR -> CR -> anything that layers with PR.
A unit and functional test have been added for regression.
Co-Authored-By: Felipe Monteiro <felipe.monteiro@att.com>
Change-Id: I353393f416aa6e441d84add9ebedcd152944d7e8
Fix failing pep8 errors which were never being flagged but now are,
possibly due to changes in flake8 rules. This patchset corrects
the following errors:
./deckhand/engine/layering.py:567:21: W503 line break before binary operator
./deckhand/engine/secrets_manager.py:406:33: W503 line break before binary operator
./deckhand/engine/utils.py:33:17: W503 line break before binary operator
./deckhand/common/utils.py:292:17: W503 line break before binary operator
Change-Id: Ic26aecb6b8049e138a826af9953f45298e817795
This PS makes Deckhand raise an exception formatted including
the list ValidationMessage-formatted error messages following
any validation error. This adheres to the format specified
under [0].
To accomplish this, logic was added to raise an exception with
a status code corresponding to the `code` attribute for each
DeckhandException subclass. This means it is no longer necessary
to raise a specific falcon exception as the process has been
automated.
In addition, the 'reason' key in the UCP error exception message
is now populated if specified for any DeckhandException instance.
The same is true for 'error_list'.
TODO (in a follow up):
* Allow 'info_list' to specified for any DeckhandException
instance.
* Pass the 'reason' and 'error_list' and etc. arguments to
all instances of DeckhandException that are raised.
[0] https://github.com/att-comdev/ucp-integration/blob/master/docs/source/api-conventions.rst#output-structure
Change-Id: I0cc2909f515ace762be805288981224fc5098c9c
This is to change passing the secret URI instead of the secret
UUID to barbican's get secret endpoint from which the secret
itself can be extracted.
While the API [0] expects a UUID the CLI instead expects a URI
and the latter extracts the UUID from the URI automatically [1].
API ref:
GET /v1/secrets/{uuid}
Headers:
Accept: application/json
X-Auth-Token: {token}
(or X-Project-Id: {project_id})
CLI ref:
$ barbican help secret get
usage: barbican secret get [-h] [-f {shell,table,value}] [-c COLUMN]
[--max-width <integer>] [--prefix PREFIX]
[--decrypt] [--payload]
[--payload_content_type PAYLOAD_CONTENT_TYPE]
URI
Retrieve a secret by providing its URI.
Finally, this adds logic for ensuring that all encrypted data is retrieved
and injected back into the raw documents with Barbican references, during
document rendering. Currently, this process is only performed for
documents with substitutions, but should also be carried out for encrypted
documents themselves.
[0] https://docs.openstack.org/barbican/latest/api/reference/secrets.html#get-v1-secrets-uuid
[1] https://docs.openstack.org/python-barbicanclient/latest/reference/index.html#barbicanclient.v1.secrets.SecretManager.get
Change-Id: I1717592b7acdedb66353c25fb5dcda2d5330196b
This PS integrates document replacement with document layering. The case
works something like this:
GIVEN:
- Parent A
- Child B
- Child C
WHEN:
- Child B is a replacement for A
THEN:
- B must layer with A, then C must layer with B, rather than A,
as B replaces A.
This is the most basic scenario and there are certainly far more
intricate ones, involving interplay with substitution as well.
To implement this new functionality, relatively minor coding changes
were made, mostly in whether to consider a document's parent or its
parent's replacement while layering, as well as determining the
dependency chain for document sorting.
Unit tests surrounding replacement have been moved into their own files
and a scenario has been added for the case described above. In addition
the same case is tested via a functional test scenario.
The unit tests have been "hardened" to run the layering scenarios twice:
once by passing in the documents in their original order, an order which
is usually written for human maintainability (i.e. B depends on A, so
make the order A followed by B). However, in reality the order of the
documents will be randomized, so every layering unit test is also
run a second time with the documents in reverse order to better ensure
that the dependency chain is resolved correctly.
Change-Id: Ieb058267f3a46b78e899922b6bc5fd726ed15a1b
This is to log out all document data following any layering action
failure. This consists of two stages:
1) Scrubbing all primitives contained in the data sections of both
the child and parent being layered together.
2) Logging scrubbed-out data sections for both documents, in addition
to their names, schemas, and the layering action itself.
This will hopefully provide DEs with enough information about why
a layering action may have failed to apply while at the same time
preventing any secret data from being logged out.
Change-Id: I3fedd259bba7b930c7969e9c30d1fffef5bf77bd
This adds support for document replacement to the
Deckhand engine _only_ for the following scenarios
_only_:
* generic case (a replaces b, returns a only)
* substitution case (a replaces b, c substitutes from a instead)
TODO:
* layering case (a replaces b, c layers with a instead)
* Modify Document unique constraint to work with
(schema, name, layer) throughout all of Deckhand
(including controllers, database models, and anywhere
else as needed)
Change-Id: Ie2cea2a49ba3b9ebc42706fbe1060d94db2e5daa
This updates the unique constraint for Document model from
schema/metadata.name to schema/metadata.name/layer which is
a pre-requisite for document replacement implementation.
The remainder fo the changes are taken of in child PS
(particulary those related to the layering module):
https://review.gerrithub.io/#/c/403888/
Change-Id: Icc4f4960b3a3951f649c7886dbe0bce77341a9f7
Deprecate substitution_sources from layering module because we
can just use the concrete documents as all the substitution
sources to simplify things.
Change-Id: Ibd8dff50402508417457655c367ebc9b6f28d70a
This is to skip layering for control documents (those whose
metadata.schema starts with "deckhand/Control") as these documents
consist of ValidationPolicy or LayeringPolicy documents -- and
it would be both nonsensical and scary to try to layer
these types of documents.
Documentation for this will be updated during a larger overhaul
effort to improve Deckhand's documentation.
Change-Id: Ia785e54c4e26a4158b6bdc89da8b96b4455f7b39
Currently, if a document has a parent but no layering actions,
the document immediately inherents its parents' data, which is a
bug. Instead, the child document should only layer with its
parent's data and then update its own data if it has at least
one layering action.
In addition, the base_schema.yaml under `deckhand.schemas`
has been updated to require that actions be required and
contain at least 1 layering action when parentSelector
is provided and that parentSelector be required when
actions is provided and that at least one key-value
pair be provided. (Empty actions array or empty
parentSelector object is meaningless and should be
disallowed/discouraged.)
This means that actions and parentSelector must always
both be provided (though providing neither is also
legal because layering is optional).
Unit tests have been added to verify the schema updates.
Change-Id: I77d54e2b216efc54b466f94d82ee8d36ca169c26
This is to remove document data printout from the MissingDocumentKey
exception message which could expose sensitive data if it is caught
and logged by other services, for example. Instead, the child
and parent documents' schema and name are printed, in addition to
the action object in which the path that could not be resolved
in either parent or document is contained.
Change-Id: I07f43e57527d05e98e98e5f80567b97dd2a762f9
This PS makes updates to layering merge actions to allow for
numeric indices to work. jsonpath_* utility methods are used
instead for merging and replacing. Added unit tests to verify
that layering scenarios for replace, merge and delete work
with numeric indices.
Change-Id: Id6d592231cb90144bb5857bee48cecf9f6478692
This is to fix abstract parent documents that have registered
substitutions but are not propagating those substitutions
down to concrete children that rely on that substituted data
(whereas it is irrelevant to the end user whether the abstract
parent receives those substitutions or not).
After this change abstract documents will always undergo
substitution such that their concrete children documents should
inherit those changes.
A unit test has been added for regression.
Change-Id: I73ed8d4caa6923492cc5ff042ecc57fbfeeb7c1c
This PS will add layering of documents in addition to existing
substitution to create topological order or Document Graph.
Render the documents based on sorted topological order.
Add test case from https://review.gerrithub.io/#/c/401465/
Change-Id: I06db302aacb9d366d109fa93d81e83eff7fefd4e
This is intended to lessen the number of LOG statements being
dumped out. The log statement regarding missing document
parent will only be issued if the document has a parentSelector
and yet no parent for it could be found. Also, the log level
is changed to DEBUG so that it's only issued if CONF.debug is
True.
Change-Id: I32d3a43604b5fce564f6af5579a0b173625533a1
Use YAML formatting for built-in Deckhand schemas
used for validations to align with other UCP services.
The second most important intention behind this PS
is to allow pre_validate flag to cascade correctly
between the layering and document_validation modules.
If pre_validate is true, then:
* the base_schema validates ALL documents
* ALL built-in schemas validate the appropriate
document given a schema match
* NO externally registered DataSchema documents
are used for validation
Else (if pre_validate is false):
* the base_schema validates ALL documents
* ALL built-in schemas validate the appropriate
document given a schema match
* ALL externally registered DataSchema documents
are used for validation given a schema match
A more minor change is setting pre_validate flags in
all modules to True for consistency. The idea is to
facilitate the way other projects that import Deckhand
in directly interface with Deckhand.
Change-Id: I859f61989ec15bede1c104b86625d116064f056d
This PS resolves a bug related to the _substitution_sources in
secrets_manager.SecretsSubstitution not getting updated with
the most recently updated layering data.
Currently, the DocumentLayering class, during initialization, passes
the list of substitution sources to the SecretsSubstitution class
as an optimization. During layering, documents that are substitution
sources can have their data updated -- and if their data is not
updated then that implies that a substitution source's data is
stale -- causing future substitutions using that substitution source
data to use stale data.
The solution is to introduce a new method called
`update_substitution_sources` which updates a specific substitution
source entry with the most update-to-date layered data following
every single layering update, such that all substitution sources
should always have the most up-to-date data.
Change-Id: Idc375cfdf17375d3c401342dff259bdcd1718941
This PS introduces a flag to only warn on missing substitution
sources because right now Promenade is failing on that. However,
a PS will also have to be added to Promenade to set the new flag
-- `fail_on_missing_sub_src` -- to False during genesis.
Change-Id: I462f721c41e23d2e5e3e698c0bd452b6764d51eb
This PS causes layering module to fail fast on malformed
``metadata.substitutions`` entry in a document by performing
built-in schema validation when validate=True is passed to
the DocumentLayering constructor. This kwarg is useful for
when the layering module is called directly -- i.e. by
Promenade or Pegleg. (The Deckhand server already performs
document pre-validation during document ingestion so there
is no need for documents stored inside Deckhand to be
re-validated again for rendered-documents endpoint.)
Next, a new exception was added -- SubstitutionSourceNotFound
-- which is raised when a substitution document is referenced
by another document but isn't found.
Finally, the previous exception raised by the
secrets_manager module has been renamed to UnknownSubstitutionError
which now raises a 500 instead of a 400 as this exception will
most likely be due to an internal server error of some kind.
Unit tests were added and documentation changes were made.
Change-Id: Idfd91a52ef9ffd8f9b1c06c6b84c3405acab6f16
The document layer that is invalid should be included in
the exception called InvalidDocumentLayer for obvious reasons.
Change-Id: Ie7fcecc96bc7667530959af34ec146b4e4a47303
Currently, Deckhand fails to perform substitution given
a substitution dependency chain for a group of documents.
That is, if A depends on B and C depends on B for substitution
then substitution will fail. Deckhand, at present, can only
perform substitution if A depends on B and C depends on B
and D depends on B, for example: In this case, the dependency
chain is no bigger than 2.
However, for cases where the substitution dependency chain
is larger than 2, then the dependency is no longer trivial. This
is because the substitution dependencies form a DAG that must
be topologically sorted in order to derive the correct order of
substitution. Once the documents are correctly sorted according
to this scheme, then substitution can be carried out as usual.
This PS makes the aforementioned changes to Deckhand's layering
module to make substitution work for non-cyclical dependencies:
A DAG is used to topologically sort the documents according to
their substitution dependency chain. A unit and functional
test has been added to verify the code.
If a cycle is detected, a critical error is thrown. Unit tests
have been added to validate this case as well.
Change-Id: Iaca3963f44aec6c897ad9fd690ce314a3a4d97a2
Pegleg uses Deckhand to invoke the layering module
directly which returned all documents which were only later
filtered to only return concrete in the controller. However,
the Deckhand layering module must itself only return concrete
so that Promenade, when it calls the module directly, only
receives concrete documents.
Abstract documents should only be processed internally by DH,
never returning them to user.
This PS makes necessary changes to layering module and unit tests.
Functional tests, as they call the controller itself, didn't need
to be refactored.
Change-Id: Ib5a49c5d31124133a10b646f55100628aa442512
This PS makes layering work for grandparents, not just parents. This
means that given a document with layer N, then not only can a parent
document in layer N+1 be used, but so can a grandparent in layer N+2.
Note that the document in layer N+1 will be preferred over the one
in N+2.
To provide a concrete example, given layers 'global', 'region'
and 'site', a document with layer 'site' can be layered with
a grandparent in layer 'global' if a document in layer
'region' isn't found. Alternatively, if the document with
layer 'region' is found then it will be used as the parent
document.
This PS refactors algorithm for determining the parent document,
tweaks the layering algorithm itself to reference the
correct parent document, and adds unit tests to confirm the
two scenarios described above.
Change-Id: I3779c7ae030bbad44b2d5ddfa5105d1a073ba670
This PS updates the layering module so that a child document with
a compound parentSelector like: [{'a': 'b'}, {'c': 'd'}] can select
a parent with the exact same labels. Further, this works if
child's parentSelector is a sub-subset of parent's labels.
Adds unit tests for positive and negative test cases.
Change-Id: I7c4aac583365d90803eda77b82763decd78cfdcf
This is to make Deckhand layering more performant. Layering is
currently the main bottleneck in the rendered-documents endpoint.
The bottleneck is specifically related to calculating document
children in the layering module. The runtime was O(N^2) but
has been decreased to ~O(N) resulting in much faster performance
overall. Using local testing against the lab deployment YAML,
runtime for layering is decreased to 15 seconds or so, down
from 55 seconds, which is roughly 4 times faster. This
performance shouldn't increase by much given even larger
YAMLs due to the linear-time performance change.
Change-Id: Ib5f7fd08a38d05ae79d18227f8aafc25bd13f7ca
This PS adds multiple log statements to the secrets manager
module to assist with debugging and also uses a dictionary keyed
with (schema, name) => document to quickly retrieve substitution
sources; rather than doing a linear-time lookup per iteration,
a constant-time lookup is used instead, with only one linear-time
initialization done during the constructor to initialize the
substitution source dictionary.
Change-Id: I209430c48312be621fdc3787009346d3e9c12ac6
This PS introduces functional tests for layering and substitution
scenarios working in tandem. Only unit tests currently offer this type
of coverage.
This PS is a follow-up to https://review.gerrithub.io/#/c/395610/
which adds functional tests that test success paths for basic
substitutions including the following edge cases:
* Substitution/layering works with top layer empty
* Substitution/layering works with multiple top layers empty
* Substitution/layering works with intermediate layer empty
These tests are more robust than the ones introduced by
https://review.gerrithub.io/#/c/395388/ while offering
the same exact coverage so the tests in that patch have
been removed.
This PS also fixes a subtle bug related to layering which was
causing the layering + substitution scenarios above to fail.
Change-Id: I9303a93f4d73f7fdca664b861bc5ab06f4162b79
Sergiy Markin