- adjusted .gitignore to keep fresh egg-info and omit build artifacts
- fresh egg-info data is needed for promenade that depends on Deckhand
- restored deckhand-functional-uwsgi-py38 gate
- restored deckhand-integration-uwsgi-py38 gate
- made deckhand-airskiff-deployment gate voting ( treasuremap project
has been updated)
- removed bionic gates
- updated focal dockerfile
- added more binary deps into bindep.txt
- updated deckhand chart values to latest images - focal and wallaby
- fixed python code to compy with CVE's found by fresh version of bandit
- implemented pip freeze approach
- added tox -e freeze profile to manage it
- requirements-frozen.txt is now main file with requirements
- requirements-direct.txt is the file to control deps
- updated setup.cfg to adjust to newer version of setuptools
- fixed airskiff-deploy gate
- fixed docker-image-build playbook to restore Quay repo image publish
- updated other playbooks to include roles from zuul/base-jobs in order
to setup build hosts properly
- removed workaround with hardcoded dns resolver ip 10.96.0.10 as it
became obsolette due to recent fix in openstack-helm-infra
- adjusted tools/whitespace-linter.sh script
- tox.ini has been brought to compliance with tox4 requirements
- replaced str() calls with six.text_type() according to D325 Deckhand specific
commandment from Hacking.rst
- locked python-barbicanclient version with 5.2.0 because of breaking
changes in the upper versions
Change-Id: I1cd3c97e83569c4db7e958b3400bdd4b7ea5e668
update dockerfile for python deckhand install
add deckhand version to chart 1.0
add chart version 0.2.0
update all packages to latest in requirements.txt
update zuul jobs for focal and python 3.8
remove zuul job functional-uwsgi-py38 in favor of functional-docker-py38
update tox config
typecast to string in re.sub() function
add stestr to test-requirements.txt
add SQLAlchemy jsonpickle sphinx-rtd-theme stestr to requirements.txt
deprecated function: BarbicanException -> BarbicanClientException
fix mock import using unittest
fix import collections to collections.abc
fix for collections modules for older than python 3.10 versions.
deprecated function: json -> to_json
deprecated function: werkzeug.contrib.profiler ->
werkzeug.middleware.profiler
deprecated function: falcon.AIP -> falcon.App
deprecation warning: switch from resp.body to resp.text
rename fixtures to dh_fixtures because there is an imported module
fixtures
switch from stream.read to bounded_stream.read
deprecated function: falcon process_response needed additional parameter
deprecated function: falcon default_exception_handler changed parameter
order
move from MagicMock object to falcon test generated object to fix
incompatability with upgraded Falcon module.
Adjust gabbi tests to fix incompatability with upgraded DeepDiff module
update Makefile to execute ubuntu_focal
update HTK (helmtoolkit)
unpin barbican to pass integration tests
Use helm 3 in chart build.
`helm serve` is removed in helm 3 so this moves
to using local `file://` dependencies [0] instead.
Change-Id: I180416f480edea1b8968d80c993b3e1fcc95c08d
The document.py `is_control` method incorrectly checks if a document
is a Control document. Per the documentation [0], Control documents
have `metadata.schema` of `metadata/Control/v1`. This commit updates
the `is_control` method to correctly check for Control documents.
[0] 1d4cc81dfa/doc/source/users/document-types.rst (control-documents)
Change-Id: I60ca8f31a61987b4e756784fce0f5a751639ae9e
This patchset uses yaml.add_representer for DocumentDict
which enables yaml.safe_load/safe_load_all to correctly
serialize the DocumentDict object without a recursive
routine.
This also completely removes the usage of jsonpath_parse
from DocumentDict as jsonpath-ng is a rather expensive
library to call continuously; and even though Deckhand
does some caching to alleviate this, it is simply better
to avoid it altogether in a wrapper that is used everywhere
across the engine module, which does all the heavy processing.
This also reduces the amount of wrapping using DocumentDict
because the better way to do this in the DB module is to
have a helper function retrieve the data from the DB and
immediately wrap it in a DocumentDict if applicable;
this is left as an exercise for later.
Change-Id: I715ff7e314cf0ec0d34c17f3378514d235dfb377
This patch set removes few pep8/flake8 ignored rules and implemented
the fix in the code to address those rules.
Change-Id: I2e613acd760818a6e18288d284f6224c38c4353a
Signed-off-by: Tin Lam <tin@irrational.io>
This PS makes Deckhand raise an exception formatted including
the list ValidationMessage-formatted error messages following
any validation error. This adheres to the format specified
under [0].
To accomplish this, logic was added to raise an exception with
a status code corresponding to the `code` attribute for each
DeckhandException subclass. This means it is no longer necessary
to raise a specific falcon exception as the process has been
automated.
In addition, the 'reason' key in the UCP error exception message
is now populated if specified for any DeckhandException instance.
The same is true for 'error_list'.
TODO (in a follow up):
* Allow 'info_list' to specified for any DeckhandException
instance.
* Pass the 'reason' and 'error_list' and etc. arguments to
all instances of DeckhandException that are raised.
[0] https://github.com/att-comdev/ucp-integration/blob/master/docs/source/api-conventions.rst#output-structure
Change-Id: I0cc2909f515ace762be805288981224fc5098c9c
This PS integrates document replacement with document layering. The case
works something like this:
GIVEN:
- Parent A
- Child B
- Child C
WHEN:
- Child B is a replacement for A
THEN:
- B must layer with A, then C must layer with B, rather than A,
as B replaces A.
This is the most basic scenario and there are certainly far more
intricate ones, involving interplay with substitution as well.
To implement this new functionality, relatively minor coding changes
were made, mostly in whether to consider a document's parent or its
parent's replacement while layering, as well as determining the
dependency chain for document sorting.
Unit tests surrounding replacement have been moved into their own files
and a scenario has been added for the case described above. In addition
the same case is tested via a functional test scenario.
The unit tests have been "hardened" to run the layering scenarios twice:
once by passing in the documents in their original order, an order which
is usually written for human maintainability (i.e. B depends on A, so
make the order A followed by B). However, in reality the order of the
documents will be randomized, so every layering unit test is also
run a second time with the documents in reverse order to better ensure
that the dependency chain is resolved correctly.
Change-Id: Ieb058267f3a46b78e899922b6bc5fd726ed15a1b
This is to log out all document data following any layering action
failure. This consists of two stages:
1) Scrubbing all primitives contained in the data sections of both
the child and parent being layered together.
2) Logging scrubbed-out data sections for both documents, in addition
to their names, schemas, and the layering action itself.
This will hopefully provide DEs with enough information about why
a layering action may have failed to apply while at the same time
preventing any secret data from being logged out.
Change-Id: I3fedd259bba7b930c7969e9c30d1fffef5bf77bd
This adds support for document replacement to the
Deckhand engine _only_ for the following scenarios
_only_:
* generic case (a replaces b, returns a only)
* substitution case (a replaces b, c substitutes from a instead)
TODO:
* layering case (a replaces b, c layers with a instead)
* Modify Document unique constraint to work with
(schema, name, layer) throughout all of Deckhand
(including controllers, database models, and anywhere
else as needed)
Change-Id: Ie2cea2a49ba3b9ebc42706fbe1060d94db2e5daa
This updates the unique constraint for Document model from
schema/metadata.name to schema/metadata.name/layer which is
a pre-requisite for document replacement implementation.
The remainder fo the changes are taken of in child PS
(particulary those related to the layering module):
https://review.gerrithub.io/#/c/403888/
Change-Id: Icc4f4960b3a3951f649c7886dbe0bce77341a9f7
Currently, if a document has a parent but no layering actions,
the document immediately inherents its parents' data, which is a
bug. Instead, the child document should only layer with its
parent's data and then update its own data if it has at least
one layering action.
In addition, the base_schema.yaml under `deckhand.schemas`
has been updated to require that actions be required and
contain at least 1 layering action when parentSelector
is provided and that parentSelector be required when
actions is provided and that at least one key-value
pair be provided. (Empty actions array or empty
parentSelector object is meaningless and should be
disallowed/discouraged.)
This means that actions and parentSelector must always
both be provided (though providing neither is also
legal because layering is optional).
Unit tests have been added to verify the schema updates.
Change-Id: I77d54e2b216efc54b466f94d82ee8d36ca169c26
This is intended to lessen the number of LOG statements being
dumped out. The log statement regarding missing document
parent will only be issued if the document has a parentSelector
and yet no parent for it could be found. Also, the log level
is changed to DEBUG so that it's only issued if CONF.debug is
True.
Change-Id: I32d3a43604b5fce564f6af5579a0b173625533a1
Use YAML formatting for built-in Deckhand schemas
used for validations to align with other UCP services.
The second most important intention behind this PS
is to allow pre_validate flag to cascade correctly
between the layering and document_validation modules.
If pre_validate is true, then:
* the base_schema validates ALL documents
* ALL built-in schemas validate the appropriate
document given a schema match
* NO externally registered DataSchema documents
are used for validation
Else (if pre_validate is false):
* the base_schema validates ALL documents
* ALL built-in schemas validate the appropriate
document given a schema match
* ALL externally registered DataSchema documents
are used for validation given a schema match
A more minor change is setting pre_validate flags in
all modules to True for consistency. The idea is to
facilitate the way other projects that import Deckhand
in directly interface with Deckhand.
Change-Id: I859f61989ec15bede1c104b86625d116064f056d
This PS causes layering module to fail fast on malformed
``metadata.substitutions`` entry in a document by performing
built-in schema validation when validate=True is passed to
the DocumentLayering constructor. This kwarg is useful for
when the layering module is called directly -- i.e. by
Promenade or Pegleg. (The Deckhand server already performs
document pre-validation during document ingestion so there
is no need for documents stored inside Deckhand to be
re-validated again for rendered-documents endpoint.)
Next, a new exception was added -- SubstitutionSourceNotFound
-- which is raised when a substitution document is referenced
by another document but isn't found.
Finally, the previous exception raised by the
secrets_manager module has been renamed to UnknownSubstitutionError
which now raises a 500 instead of a 400 as this exception will
most likely be due to an internal server error of some kind.
Unit tests were added and documentation changes were made.
Change-Id: Idfd91a52ef9ffd8f9b1c06c6b84c3405acab6f16
Currently, Deckhand fails to perform substitution given
a substitution dependency chain for a group of documents.
That is, if A depends on B and C depends on B for substitution
then substitution will fail. Deckhand, at present, can only
perform substitution if A depends on B and C depends on B
and D depends on B, for example: In this case, the dependency
chain is no bigger than 2.
However, for cases where the substitution dependency chain
is larger than 2, then the dependency is no longer trivial. This
is because the substitution dependencies form a DAG that must
be topologically sorted in order to derive the correct order of
substitution. Once the documents are correctly sorted according
to this scheme, then substitution can be carried out as usual.
This PS makes the aforementioned changes to Deckhand's layering
module to make substitution work for non-cyclical dependencies:
A DAG is used to topologically sort the documents according to
their substitution dependency chain. A unit and functional
test has been added to verify the code.
If a cycle is detected, a critical error is thrown. Unit tests
have been added to validate this case as well.
Change-Id: Iaca3963f44aec6c897ad9fd690ce314a3a4d97a2
Pegleg uses Deckhand to invoke the layering module
directly which returned all documents which were only later
filtered to only return concrete in the controller. However,
the Deckhand layering module must itself only return concrete
so that Promenade, when it calls the module directly, only
receives concrete documents.
Abstract documents should only be processed internally by DH,
never returning them to user.
This PS makes necessary changes to layering module and unit tests.
Functional tests, as they call the controller itself, didn't need
to be refactored.
Change-Id: Ib5a49c5d31124133a10b646f55100628aa442512
This PS makes layering work for grandparents, not just parents. This
means that given a document with layer N, then not only can a parent
document in layer N+1 be used, but so can a grandparent in layer N+2.
Note that the document in layer N+1 will be preferred over the one
in N+2.
To provide a concrete example, given layers 'global', 'region'
and 'site', a document with layer 'site' can be layered with
a grandparent in layer 'global' if a document in layer
'region' isn't found. Alternatively, if the document with
layer 'region' is found then it will be used as the parent
document.
This PS refactors algorithm for determining the parent document,
tweaks the layering algorithm itself to reference the
correct parent document, and adds unit tests to confirm the
two scenarios described above.
Change-Id: I3779c7ae030bbad44b2d5ddfa5105d1a073ba670
This PS updates the layering module so that a child document with
a compound parentSelector like: [{'a': 'b'}, {'c': 'd'}] can select
a parent with the exact same labels. Further, this works if
child's parentSelector is a sub-subset of parent's labels.
Adds unit tests for positive and negative test cases.
Change-Id: I7c4aac583365d90803eda77b82763decd78cfdcf
This is to make Deckhand layering more performant. Layering is
currently the main bottleneck in the rendered-documents endpoint.
The bottleneck is specifically related to calculating document
children in the layering module. The runtime was O(N^2) but
has been decreased to ~O(N) resulting in much faster performance
overall. Using local testing against the lab deployment YAML,
runtime for layering is decreased to 15 seconds or so, down
from 55 seconds, which is roughly 4 times faster. This
performance shouldn't increase by much given even larger
YAMLs due to the linear-time performance change.
Change-Id: Ib5f7fd08a38d05ae79d18227f8aafc25bd13f7ca
This PS handles yet another annoying edge case with layering:
What if multiple layers in the layerOrder defined in the
LayeringPolicy are empty?
https://review.gerrithub.io/#/c/395388/ fixed substitution
for a layerOrder edge case like:
- region (empty)
- site (document needs substitution)
But that PS does not handle the following case:
- empty layer
- empty layer
- site (document needs substitution)
Or even:
- empty layer
- empty layer
- global (document needs substitution)
- empty layer
- site (document needs substitution)
Change-Id: Ia64ce627454289b18739964bdc488c0713c6b39a
This PS simplifies the DocumentWrapper class by changing the way
it is designed. The purpose of the class was to make it easier
to retrieve nested dictionary attributes from a document. The class
previously inherited from `object` meaning that the object could not
directly be treated as a dictionary, complicating usage of the class.
With this change, the class now inherits from a `dict` meaning that
it can be manipulated the same way a dictionary can, while still
able to return nested dictionary attributes without having to worry
about exceptions getting thrown.
Each property implemented by `DocumentWrapper` uses jsonpath_parse
implements in `deckhand.utils` to retrieve nested attributes or
else self.get() to retrieve first-level dictionary attributes.
Change-Id: I1d73a79aa4c3117be31aab978c20258c1052ad6d
This PS simplifies the layering interface, adds additional
exception handling around document layering in the rendered
documents controller, and removes an unused exception related
to document layering.
Change-Id: I0491ae32a43fe4a6a01c8da530528d8573f91a64
This PS integrates layering functionality with rendered
documents API endpoint. No new functionality was really
added; instead, a lot of code was refactored to make
layering work alongside substitution.
The following changes have been made:
- Moved document filtering functionality to deckhand.utils
because rendered documents must be filtered twice: once
to retrieve all documents necessary for rendering from
the DB and again by the controller to filter out documents
the user doesn't want returned
- Additional LOG statements in the layering module
- Additional layering unit tests
- Additional functional tests
- Removal of some stricter validations around layering:
if a parent document is not found for a document,
an error is no longer returned, as not all documents
need to have a parent (that is, not all documents
need to be rendered together, though this might need
to be expanded on later: what if a document has a
`parentSelector` but no parent is found?)
Change-Id: I6c66ed824fba0216ba868a6101a72cfe3bdda181
This commit constitutes 1 of 2 monolithic ports from Github.
This commit implements the foundation for document layering
or merging. Included in this commit:
- Algorithm for layering documents with the same schema
- Dozens of positive test cases
- About a dozen negative test cases
- Factory for dynamically creating testing documents for
layering
Change-Id: I580bb69a341910b21be8610a416c691c54f7b946
Sergiy Markin