This patchset fixes a bug where Deckhand was failing to perform
substitution and layering on document sets where all the documents had a
storagePolicy of encrypted. Deckhand would attempt to substitute from an
encrypted source document, but when that document marked as encrypted,
it fails because the source doc had been redacted. The behavior now goes
as follows:
- Resolve Barbican references before layering and substitution have been
performed so that the prior two operations don't attempt to operate on a
Barbican reference
- After substitution, redact the destination document if it is marked as
encrypted
- Now, after substition, we can redact the rest of the documents and
substitutions
Change-Id: I725775d554c9eed2692fc6203c416a7119646680
- If a document has a storage policy of encrypted
- Redacts (sha256) the data section.
- Redacts (sha256) the substition paths.
- Uses the same /documents endpoint, adds a new query parameter
?cleartext-secrets=true to show the non-redacted values.
Change-Id: I42808901b97c667a1148c00fbb7717a0847c9981
This patch set adds logic to optimize performance around rendered
documents caching further by implementing logic to skip over
post-rendering validation when cache hits occur. This works because:
* if the result set isn't cached, then post-validation is performed;
if the post-validation fails the just-cached result set is
invalidated
* if the result set is cached, then post-validation for that
result set has already been performed successfully, so it
can be safely skipped over
It was discovered via profiler anaylsis that document validation
accounts for an appreciable amount of run time, so optimizing
around this makes a noticeable difference.
Change-Id: I18344ea750cf1028a377028f80ecdd931d9672c0
GET /revisions/{{revision_id}}/deepdiff/{{comparison_revision_id}}
- Added deepdiff api for generating diff between
two rendered documents.
- Deep diffing for data and metadata
- Refactor diff functions
- Client update
- Added unit testcases
- Added funtional testcases
- Doc update
Change-Id: Ib60fa60a3b33e9125a1595a999272ca595721b38
This patch set cleans up jsonpath_replace method to increase
its readability. Currently, there is some weird legacy code in
there which doesn't make much sense and is most likely dead code.
In fact, before this change, there was no unit test for validating
that `MissingDocumentPattern` is raised correctly.
The `jsonpath_replace` is refactored to only call the inner
private function _do_replace once, after performing data
expansion (populating the data dictionary with nonexistent
nested keys).
Unit tests have been added to validate the exception above.
Change-Id: I1c18c4f8c79c1b9d3124747f8aa04743f27434eb
This implements a rendered documents cache which is keyed by
revision IDs. This means that repeatedly trying to re-render
documents for the same revision ID will be much, much faster as
the results will be cached.
Change-Id: Ie92f55a9234d038683ba1fcad76710d968ed67ab
This patch set uses concurrent.futures.ThreadPoolExecutor
[0] to retrieve multiple Barbican secrets concurrently.
This is because currently it is only possible to retrieve
1 secret payload from Barbican at a time -- for revisions
with several dozen secrets it is therefore too costly
to serially perform these API requests.
A new configuration option is added to the [barbican]
group called `max_workers` which specifies the number
of threads to use. The default value is 10. Note that:
"If max_workers is None or not given, it will default
to the number of processors on the machine, multiplied by 5"
[0] so the default is 10 for 2 * 5 which is overly
conservative if anything.
If any error occurs during any of the requests a 500
is raised with appropriate details.
[0] https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor
Change-Id: I76a5bb6c345054e160c14bdf9fb7087e3a746a5e
This patchset uses yaml.add_representer for DocumentDict
which enables yaml.safe_load/safe_load_all to correctly
serialize the DocumentDict object without a recursive
routine.
This also completely removes the usage of jsonpath_parse
from DocumentDict as jsonpath-ng is a rather expensive
library to call continuously; and even though Deckhand
does some caching to alleviate this, it is simply better
to avoid it altogether in a wrapper that is used everywhere
across the engine module, which does all the heavy processing.
This also reduces the amount of wrapping using DocumentDict
because the better way to do this in the DB module is to
have a helper function retrieve the data from the DB and
immediately wrap it in a DocumentDict if applicable;
this is left as an exercise for later.
Change-Id: I715ff7e314cf0ec0d34c17f3378514d235dfb377
This patchset moves retrieval of encrypted documents to the
Deckhand controller so that components like Pegleg and
Promenade can consume the Deckhand engine offline without
running into Barbican errors.
Components can pass in `encryption_sources` to Deckhand's
rendering module which Deckhand will now use instead to resolve
secret references.
`encryption_sources` is a dictionary that maps the reference
contained in the destination document's data section to the
actual unecrypted data. If encrypting data with Barbican, the
reference will be a Barbican secret reference.
Change-Id: I1a457d3bd37101d73a28882845c2ce74ac09fdf4
- Treat internal Deckhand schemas equivalent to other
service schemas
- Remove validating sections other than `data` outside of
base schema
- Create schemas for metadata sections metadata/Control/v1 and
metadata/Document/v1
- Use a single validator and let that validator check for document
structure (validate against the base schema and metadata)
and for post-validation also validate against service schemas
Change-Id: I5f9b9a3cfa1692a69b5982a6424edd65bdfed0ef
This PS adds a limit query filter parameter to allow users to limit
the number of documents returned by revision documents as well
as rendered documents.
Change-Id: Ic15dc59cd21d82be552fa7b9885754bde47724a0
This PS makes Deckhand raise an exception formatted including
the list ValidationMessage-formatted error messages following
any validation error. This adheres to the format specified
under [0].
To accomplish this, logic was added to raise an exception with
a status code corresponding to the `code` attribute for each
DeckhandException subclass. This means it is no longer necessary
to raise a specific falcon exception as the process has been
automated.
In addition, the 'reason' key in the UCP error exception message
is now populated if specified for any DeckhandException instance.
The same is true for 'error_list'.
TODO (in a follow up):
* Allow 'info_list' to specified for any DeckhandException
instance.
* Pass the 'reason' and 'error_list' and etc. arguments to
all instances of DeckhandException that are raised.
[0] https://github.com/att-comdev/ucp-integration/blob/master/docs/source/api-conventions.rst#output-structure
Change-Id: I0cc2909f515ace762be805288981224fc5098c9c
It is currently surprising for users when null is getting substituted
into documents when there is no actual source data to grab.
Silent "None" substitution sometimes results in surprising symptoms in
complex configurations.
Depends-On: If2b08f443cde765a1dbfaf7bac6b549591e59148
Change-Id: I253dc1d10f9493b9611fb6abd86810c6d57afbf6
This adds support for document replacement to the
Deckhand engine _only_ for the following scenarios
_only_:
* generic case (a replaces b, returns a only)
* substitution case (a replaces b, c substitutes from a instead)
TODO:
* layering case (a replaces b, c layers with a instead)
* Modify Document unique constraint to work with
(schema, name, layer) throughout all of Deckhand
(including controllers, database models, and anywhere
else as needed)
Change-Id: Ie2cea2a49ba3b9ebc42706fbe1060d94db2e5daa
This updates the unique constraint for Document model from
schema/metadata.name to schema/metadata.name/layer which is
a pre-requisite for document replacement implementation.
The remainder fo the changes are taken of in child PS
(particulary those related to the layering module):
https://review.gerrithub.io/#/c/403888/
Change-Id: Icc4f4960b3a3951f649c7886dbe0bce77341a9f7
This is to fix secrets_manager.SecretsManager.get method which
is passing in the secret reference to Barbican directly for
GET /secrets/{uuid} [0] causing Barbican to raise a
ValueError exception when it attempts to validate that
{secret_uuid} is in fact a UUID.
The fix is to extract the secret_uuid from the secret_ref returned
by Barbican before querying the GET /secrets/{uuid} API.
[0] https://docs.openstack.org/barbican/latest/api/reference/secrets.html#get-v1-secrets-uuid
Change-Id: I4db317e3ba12b4268df5b84b79be8da1da5ac2ba
Deprecate substitution_sources from layering module because we
can just use the concrete documents as all the substitution
sources to simplify things.
Change-Id: Ibd8dff50402508417457655c367ebc9b6f28d70a
This adds a functional test for chained substitution.
For this test to pass a subtle bug had to be fixed in
the code in DocumentDict wrapper. It was possible for
instances of this class to return a NEW object when
`self.data` was invoked causing
`substitution_src.data.update(data)` to fail when
attempting to update a substitution source with updated
layered data because -- if data={} at that moment in time --
the code in DocumentDict would return a NEW object {}.
This was because `self.get('data') or {}` would return {}
on the RHS instead of self['data'] because self['data']
itself would be {} which would evaluate to false in terms
of truthiness resulting in the wrong object ref being returned...
essentially causing the update to fail. This has been
fixed.
Change-Id: I23ad0010e1d7df73e8e1a1456ba21b3e611bb0dd
This is a self-evident fix, but errors.errors.UnknownSubstitutionError
is used instead of errors.UnknownSubstitutionError which is a
syntax mistake causing an AttributeError to get thrown; this
PS fixes that.
Change-Id: I0c988280fda8c68be63add0c581ebadb918c055a
Use YAML formatting for built-in Deckhand schemas
used for validations to align with other UCP services.
The second most important intention behind this PS
is to allow pre_validate flag to cascade correctly
between the layering and document_validation modules.
If pre_validate is true, then:
* the base_schema validates ALL documents
* ALL built-in schemas validate the appropriate
document given a schema match
* NO externally registered DataSchema documents
are used for validation
Else (if pre_validate is false):
* the base_schema validates ALL documents
* ALL built-in schemas validate the appropriate
document given a schema match
* ALL externally registered DataSchema documents
are used for validation given a schema match
A more minor change is setting pre_validate flags in
all modules to True for consistency. The idea is to
facilitate the way other projects that import Deckhand
in directly interface with Deckhand.
Change-Id: I859f61989ec15bede1c104b86625d116064f056d
This PS causes layering module to fail fast on malformed
``metadata.substitutions`` entry in a document by performing
built-in schema validation when validate=True is passed to
the DocumentLayering constructor. This kwarg is useful for
when the layering module is called directly -- i.e. by
Promenade or Pegleg. (The Deckhand server already performs
document pre-validation during document ingestion so there
is no need for documents stored inside Deckhand to be
re-validated again for rendered-documents endpoint.)
Next, a new exception was added -- SubstitutionSourceNotFound
-- which is raised when a substitution document is referenced
by another document but isn't found.
Finally, the previous exception raised by the
secrets_manager module has been renamed to UnknownSubstitutionError
which now raises a 500 instead of a 400 as this exception will
most likely be due to an internal server error of some kind.
Unit tests were added and documentation changes were made.
Change-Id: Idfd91a52ef9ffd8f9b1c06c6b84c3405acab6f16
Pegleg uses Deckhand to invoke the layering module
directly which returned all documents which were only later
filtered to only return concrete in the controller. However,
the Deckhand layering module must itself only return concrete
so that Promenade, when it calls the module directly, only
receives concrete documents.
Abstract documents should only be processed internally by DH,
never returning them to user.
This PS makes necessary changes to layering module and unit tests.
Functional tests, as they call the controller itself, didn't need
to be refactored.
Change-Id: Ib5a49c5d31124133a10b646f55100628aa442512
Currently Pegleg uses a lot of raw documents that are missing
properties at first because those properties are only included
in the documents only after they undergo substitution (are rendered).
This means that when these raw documents are PRE-validated against
registered DataSchemas a lot of noise is created.
However, after the documents are rendered (undergo substitution)
then they should be POST-validated against the registered DataSchemas.
This PS makes the changes necessary to make pre-validation ignore
validation against registered DataSchemas but makes post-validation
raise all validation errors while validating against all built-in
and registered schemas.
Necessary changes were made to tests to make them pass with the
new changes. A follow up will be needed to do better testing
for pre-validation vs. post-validation but the functional test
scenario in schema-validation-success.yaml should test both
scenarios.
Change-Id: I5c139fa528639d43fc45eda067a9ea807fe26c61
This PS makes layering work for grandparents, not just parents. This
means that given a document with layer N, then not only can a parent
document in layer N+1 be used, but so can a grandparent in layer N+2.
Note that the document in layer N+1 will be preferred over the one
in N+2.
To provide a concrete example, given layers 'global', 'region'
and 'site', a document with layer 'site' can be layered with
a grandparent in layer 'global' if a document in layer
'region' isn't found. Alternatively, if the document with
layer 'region' is found then it will be used as the parent
document.
This PS refactors algorithm for determining the parent document,
tweaks the layering algorithm itself to reference the
correct parent document, and adds unit tests to confirm the
two scenarios described above.
Change-Id: I3779c7ae030bbad44b2d5ddfa5105d1a073ba670
This PS solves the following issues for which only minor changes
were needed:
1) Using copy.copy() around the substitution value passed in
to jsonpath_replace so as to avoid updating the substitution
entry referentially which was happening before this change,
causing future substitutions using that entry to fail or
behaving unexpectedly.
2) Supporting non-string substitution values when a substitution
pattern is provided: before this change, this was failing
because calling re.sub() and passing in a non-string
value causes an error to be thrown.
3) Adding better logging and error handling to
deckhand.utils.jsonpath_replace to assist with debugging.
Unit tests are included for some of the scenarios above.
Change-Id: I8562d43a717f477e3297504c1522331b3a993f88
This PS moves DB calls out of the engine module and into the
appropriate controllers so that a "production" set of documents
can leverage Deckhand layering and substitution after importing
the Deckhand engine module directly. These operations will be
carried out offline meaning that DB calls are not possible.
Unit tests were refactored to work with the changes.
Some testing documentation was also updated.
Closes 16
Change-Id: I6e0757746cd949985d57102d1c85acfbbed86078
This PS rewrites the document_validation module in
Deckhand to achieve the following goals:
* better validation resiliency
* add support for different document schema versions
* better support for DataSchema validation
* separation of concerns by splitting up validations
into separate classes
* support for validating documents that rely on
a DataSchema passed in via the same payload
* support for generating multiple validation errors
rather than returning after the first one found
* increase testing validations for unit/functional
tests
Better validation resiliency is achieved through more
robust exception handling. For example, it is possible
for a ``DataSchema`` to be 100% valid from the POV of
built-in schema validation, but if the "data" section
itself is utterly invalid, then an exception will be
raised -- such an exception is treated as a critical
failure.
Better generation of error messages is achieved by
creation more validation error message results.
DataSchema validation was previously wonky. A DataSchema
had to first be created in 1 revision before it could be
referenced by a batch of documents in sequential revisions.
Now, a DataSchema can be created in the same (or previous)
revision as documents that rely on it and used to validate
said documents.
Finally, the module was heavily rewritten so that more
nuanced validations can be built by inheriting from
``BaseValidator`` so as to allow for easier code
readability and maintainability.
Change-Id: Ie75742b984b7ad392cb41decc203d42842050c80
This PS adds currently lacking validation around ensuring that the
right documents are pooled together for rendering. The validation
checks that documents from older revisions are unused, and that
only documents from the latest revision corresponding to each
bucket are used for rendering.
Change-Id: I9494c8d7055aac815c5baf0b15c7b1743c8ff259
This PS implements sorting and filtering for rendered-documents
endpoint, adds additional validations for sorting, filtering
and other layering scenarios, and updates rendered-documents
and buckets documentation.
Layering scenarios added:
- Updating the LayeringPolicy with 2 layers in the layerOrder
(down from 3) such that the site document should have its
parent document recomputed as the global document.
- A deletion action layering scenario (DH currently only has
merge, replace scenarios in its funcitonal test suite.)
Documentation updated:
- clarify the access levels for buckets, which has been a
source of confusion.
- update api-ref documentation for rendered-documents
Change-Id: Idb9b42351dfbdf75a19282c8478065e7564cfc26
This PS fixes a bug related to Deckhand only using "secret"
document types to be used as substitution sources; the substitution
logic should be made generic, because it shouldn't just apply to
secrets.
This entailed removing the "is_secret" database column from the
Document table as it's no longer needed and dropping it from a DB
query made to find the source document for substitution in the
secrets_manager module.
This PS also increased resiliency via exception handling and some
edge cases surrounding substitution.
Finally, unit tests and functional tests were added to validate
substitition using a generic document as the source.
Change-Id: I2c4b49b2eb55473c56b8253a456803e793b0b0b0
This PS simplifies the layering interface, adds additional
exception handling around document layering in the rendered
documents controller, and removes an unused exception related
to document layering.
Change-Id: I0491ae32a43fe4a6a01c8da530528d8573f91a64
This PS integrates layering functionality with rendered
documents API endpoint. No new functionality was really
added; instead, a lot of code was refactored to make
layering work alongside substitution.
The following changes have been made:
- Moved document filtering functionality to deckhand.utils
because rendered documents must be filtered twice: once
to retrieve all documents necessary for rendering from
the DB and again by the controller to filter out documents
the user doesn't want returned
- Additional LOG statements in the layering module
- Additional layering unit tests
- Additional functional tests
- Removal of some stricter validations around layering:
if a parent document is not found for a document,
an error is no longer returned, as not all documents
need to have a parent (that is, not all documents
need to be rendered together, though this might need
to be expanded on later: what if a document has a
`parentSelector` but no parent is found?)
Change-Id: I6c66ed824fba0216ba868a6101a72cfe3bdda181
This PS implements the sort filter, allowing (for now)
the GET /revisions and GET /revision/{revision_id}/documents
endpoints to be sorted as per the API documentation in
Deckhand [0].
An additional filter has also been added to the 2 aforementioned
endpoints as well -- order -- which determines the order in
which sorted results are returned: "asc" for ascending
order and "desc" for descending order.
[0] http://deckhand.readthedocs.io/en/latest/api_ref.html#get-revisions-revision-id-documents
Change-Id: Ifb9e15b8379b0a28889a14c331d81d9a4147f1d4
This PS adds a filter for deleted=False to rendered documents endpoint
so that previously deleted documents are excluded from current
revision.
Change-Id: Id45786d680f2ada8c2a27d2b44a677eb6b14921d
This PS implements schema validation for fully rendered documents.
Failed validation when calling GET /revisions/{revision_id}/rendered-documents
results in a 500 Internal Server Error being raised.
Included in this PS:
- Post-rendering validation logic in the appropriate controller
- Unit tests
- Documentation update
Change-Id: I000043ba797b223be6e141bf851d9b2999fc3140
Currently, the rendered-documents endpoint returns only documents
that require substitution, rather than all concrete documents, as
specified in the requirements (DECKHAND-65).
This PS adds a filter to the endpoint so that only concrete documents
are returned. Also, all concrete documents are returned, not just
the ones that require substitution.
Included in this PS:
- logic changes described above
- unit test to verify the above logic
Change-Id: Ib552b084bb00b6e180bba973be420449a292fb05
To increase resiliency, add Deckhand error handling hooks to
format unknown errors into something more useful for debugging.
Also override exception formatting to be consistent with UCP
error formatting standard.
Most of this logic is borrowed from Shipyard for consistency.
Also includes basic unit tests to validate error formatting.
Change-Id: If7f8c3bf6b6ada7697611a0bef7bf8f635fc0b7f
This PS manually removes logic for adding application/x-yaml
content type inside controllers and moves it into a better
layer of abstraction: middleware.
This PS also fixes a bug with using recently removed function
to_yaml_body from the VersionsResource and adds unit tests
for testing YAMLTranslator middleware.
Change-Id: I4388eb212efee7b300f242eebbc20e22b766fd7d
This PS integrates Deckhand with keystone auth so
that Deckhand can check whether a keystone token is
authenticated (by way of keystonemiddleware)
before proceeding with any requests.
The architecture for this PS is borrowed from [0]
which successfully integrates keystone authentication
with the falcon web application framework. However,
additional Deckhand-specific changes were made for
tests to pass.
The following changes have been made:
- add paste deploy configuration file which adds
keystonemiddleware integration to Deckhand; this
makes it trivial for keystonemiddleware to determine
whether a token in the X-Auth-Token header is authenticated
- use paste.deploy to create a web app
- update unit tests for testing controllers
- update functional test script to ignore keystone authentication
because functional tests don't currently support keystone
integration
[0] https://github.com/stannum-l/nautilus
Change-Id: I6eeeb4a4d9ab1f1cc8fb338e5cc21136ab4d5684
This PS implements documentation substitution and
the rendered-documents endpoint. Each time the
rendered-documents is queried, the documents for
the reqeust revision_id dynamically undergo
secret substitution.
All functional tests related to secret substitution
have been unskipped.
Deckhand currently does not real testing for
verifying that secret substitution works
for encrypted documents. This will only happen
when integration testing is added to Deckhand to
test its interaction with Keystone and Barbican.
Included in this PS:
- basic implementation for secret substitution
- introduction of jsonpath_ng for searching for and
updating jsonpaths in documents
- rendered-documents endpoint
- unit tests
- all relevant functional tests unskipped
- additional bucket controller tests include RBAC
tests and framework testing RBAC via unit tests
Change-Id: I86f269a5b616b518e5f742a4005891412226fe2a
This PS implements oslo.policy integration in Deckhand.
The policy.py file implements 2 types of functions for
performing policy enforcement in Deckhand: authorize,
which is a decorator that is used directly around
falcon on_HTTP_VERB methods that raises a 403 immediately
if policy enforcement fails; and conditional_authorize,
to be used inside controller code conditionally.
For example, since Deckhand has two types of documents
with respect to security -- encrypted and cleartext
documents -- policy enforcement is conditioned on the
type of the documents' metadata.storagePolicy.
Included in this PS:
- policy framework implementation
- policy in code and policy documentation for all
Deckhand policies
- modification of functional test script to override
default admin-only policies with custom policy file
dynamically created using lax permissions
- bug fix for filtering out deleted documents (and
its predecessors in previous revisions) for
PUT /revisions/{revision_id}/documents
- policy documentation
- basic unit tests for policy enforcement framework
- allow functional tests to be filtered via regex
Due to the size of this PS, functional tests related to
policy enforcement will be done in a follow up.
Change-Id: If418129f9b401091e098c0bd6c7336b8a5cd2359
This PS adds support for filtering revisions and
revision documents documents by any legal filter
(those enumerated in the design document).
Deckhand now supports the following filter arguments:
* schema
* metadata.name
* metadata.label
* metadata.layeringDefinition.abstract
* metadata.layeringDefinition.layer
* status.bucket
Deckhand now supports the following filter arguments for filtering
revisions:
* tag
Deckhand now supports multiple filters, e.g.:
* ?metdata.layeringDefinition.layer=site&metadata.name=foo
Deckhand now supports repeated filters, e.g.:
* ?metadata.label=foo=bar&metadata.label=baz=qux
The following has yet to be implemented will be done in a future
follow-up PS:
- support sorting by specific keywords as well
- support query limit and offset filters
Change-Id: I8558481e075715fe7fab98140094d37782a986d9
This commit implements logic to realize bucket deletion. This
commit also adds logic for raising an exception when trying
to create the same (document.schema, document.metadata.name)
in a different bucket than the one it was originally created in.
Included in this commit:
- Implementation of document deletion logic.
- Documents are always saved, even if they have been deleted
or remain unchanged between revisions. This makes it easier
to compute the diff between revisions.
- Associated unit tests.
- Unskip all remaining functional tests for
'document-crud-success-single-bucket.yaml`
- Raise a 409 exception when trying to create the same
(document.schema, document.metadata.name) in a different
bucket.
- Unskip functional tests for
'document-crud-error-bucket-conflict.yaml'
Change-Id: I6693bbb918cb672de315a66bb087de547df302d1
This commit adds an additional attribute called `tags` to each
Revision DB model. This allows Revisions to be tagged with whatever
arbitrary tag/tag data a service chooses to identify a revision by.
This commit:
- creates a new DB model called `RevisionTag`
- adds the following endpoints:
* POST /api/v1.0/revisions/{revision_id}/tags/{tag} (create a tag)
* GET /api/v1.0/revisions/tags/{tag} (show tag details)
* GET /api/v1.0/revisions/{revision_id}/tags (list revision tags)
* DELETE /api/v1.0/revisions/{revision_id}/tags/{tag} (delete a tag)
* DELETE /api/v1.0/revisions/{revision_id}/tags (delete all tags)
- adds appropriate unit test coverage for the changes
- adds functional testing for each API endpoint
Change-Id: I49a7155ef5aa274c3a85ff6f8b85951f155a4b92
Sergiy Markin