- adjusted .gitignore to keep fresh egg-info and omit build artifacts
- fresh egg-info data is needed for promenade that depends on Deckhand
- restored deckhand-functional-uwsgi-py38 gate
- restored deckhand-integration-uwsgi-py38 gate
- made deckhand-airskiff-deployment gate voting ( treasuremap project
has been updated)
- removed bionic gates
- updated focal dockerfile
- added more binary deps into bindep.txt
- updated deckhand chart values to latest images - focal and wallaby
- fixed python code to compy with CVE's found by fresh version of bandit
- implemented pip freeze approach
- added tox -e freeze profile to manage it
- requirements-frozen.txt is now main file with requirements
- requirements-direct.txt is the file to control deps
- updated setup.cfg to adjust to newer version of setuptools
- fixed airskiff-deploy gate
- fixed docker-image-build playbook to restore Quay repo image publish
- updated other playbooks to include roles from zuul/base-jobs in order
to setup build hosts properly
- removed workaround with hardcoded dns resolver ip 10.96.0.10 as it
became obsolette due to recent fix in openstack-helm-infra
- adjusted tools/whitespace-linter.sh script
- tox.ini has been brought to compliance with tox4 requirements
- replaced str() calls with six.text_type() according to D325 Deckhand specific
commandment from Hacking.rst
- locked python-barbicanclient version with 5.2.0 because of breaking
changes in the upper versions
Change-Id: I1cd3c97e83569c4db7e958b3400bdd4b7ea5e668
This patch set fixes the schema pattern enforced by metadata_document
and metadata_control. Currently, both allow a schema with either
pattern:
- ^metadata/Control/v\d+$
- ^metadata/Document/v\d+$
However, the metadata_control schema should only allow the former
and the metadata_document schema should only allow the latter.
Change-Id: Ic1b88a7158755818002de4c88cdf2d7b716f656d
This package is used for generation autodoc documentation
automatically which can be linked to by Deckhand
documentation from other places. This is to make autodoc
generation work in RTD.
More info: https://pypi.org/project/sphinxcontrib-apidoc/
Change-Id: I43aac82728e5935a5a2626f2fd29d7a7188d19f9
This patch set adds validation logic to document_validation.py (in
Deckhand's engine module) so that components that rely on
Deckhand's engine for document rendering (such as Promenade
or Pegleg) can fail fast when they provide Deckhand with a
duplicate document. Must pass pre_validate=True to layering
module which currently is the case for Promenade, et. al.
Before this change, Deckand only supported this logic at
the DB level (requiring service instantion); this is now no longer
the case.
Change-Id: I6d1c8214775aa0f3b5efb1049972cf847f74585b
1) UCP -> Airship
2) readthedocs.org -> readthedocs.io (there is redirect)
3) http -> https
4) attcomdev -> airshipit (repo on quay.io)
5) att-comdev -> openstack/airship-* (repo on github/openstack git)
6) many URLs have been verified and adjusted to be current
7) no need for 'en/latest/' path in URL of the RTD
8) added more info to some setup.cfg and setup.py files
9) ucp-integration docs are now in airship-in-a-bottle
10) various other minor fixes
Change-Id: I12b2fa8fbec37a483a0ad50382e08f51ed97533a
This patchset uses yaml.add_representer for DocumentDict
which enables yaml.safe_load/safe_load_all to correctly
serialize the DocumentDict object without a recursive
routine.
This also completely removes the usage of jsonpath_parse
from DocumentDict as jsonpath-ng is a rather expensive
library to call continuously; and even though Deckhand
does some caching to alleviate this, it is simply better
to avoid it altogether in a wrapper that is used everywhere
across the engine module, which does all the heavy processing.
This also reduces the amount of wrapping using DocumentDict
because the better way to do this in the DB module is to
have a helper function retrieve the data from the DB and
immediately wrap it in a DocumentDict if applicable;
this is left as an exercise for later.
Change-Id: I715ff7e314cf0ec0d34c17f3378514d235dfb377
- Treat internal Deckhand schemas equivalent to other
service schemas
- Remove validating sections other than `data` outside of
base schema
- Create schemas for metadata sections metadata/Control/v1 and
metadata/Document/v1
- Use a single validator and let that validator check for document
structure (validate against the base schema and metadata)
and for post-validation also validate against service schemas
Change-Id: I5f9b9a3cfa1692a69b5982a6424edd65bdfed0ef
This patchset adds a dict() cast as a workaround the fact
that PyYAML 4.1 recently changed yaml.dump to yaml.safe_dump,
compelling developers to use yaml.danger_dump to achieve
the previous behavior of yaml.dump [0].
However, yaml.danger_dump should not be used and this
technically corrects antecedent use of yaml.dump by
introducing a recursive function that ensures the
dictionary prior to being dumped is compatible
with yaml.safe_dump.
Such a function is needed because yaml.safe_dump
rejects serialization of Deckhand's DocumentDict
dictionary wrapper helper -- even though it is
a subclass of a dict. Thus, the recursive
function simply casts each instance of DocumentDict
into a dictionary.
[0] https://stackoverflow.com/questions/51053903/new-pyyaml-version-breaks-on-most-custom-python-objects-representererror
Change-Id: I67966b45e0865864bd5e6bb4578548769fc13eeb
This patch set removes few pep8/flake8 ignored rules and implemented
the fix in the code to address those rules.
Change-Id: I2e613acd760818a6e18288d284f6224c38c4353a
Signed-off-by: Tin Lam <tin@irrational.io>
This PS makes Deckhand raise an exception formatted including
the list ValidationMessage-formatted error messages following
any validation error. This adheres to the format specified
under [0].
To accomplish this, logic was added to raise an exception with
a status code corresponding to the `code` attribute for each
DeckhandException subclass. This means it is no longer necessary
to raise a specific falcon exception as the process has been
automated.
In addition, the 'reason' key in the UCP error exception message
is now populated if specified for any DeckhandException instance.
The same is true for 'error_list'.
TODO (in a follow up):
* Allow 'info_list' to specified for any DeckhandException
instance.
* Pass the 'reason' and 'error_list' and etc. arguments to
all instances of DeckhandException that are raised.
[0] https://github.com/att-comdev/ucp-integration/blob/master/docs/source/api-conventions.rst#output-structure
Change-Id: I0cc2909f515ace762be805288981224fc5098c9c
This updates the unique constraint for Document model from
schema/metadata.name to schema/metadata.name/layer which is
a pre-requisite for document replacement implementation.
The remainder fo the changes are taken of in child PS
(particulary those related to the layering module):
https://review.gerrithub.io/#/c/403888/
Change-Id: Icc4f4960b3a3951f649c7886dbe0bce77341a9f7
This sanitizes any potential secrets contained in
'message' section of validation output if the document
has substitions (implying that a secret may have been
substituted into it) or if the document itself was
encrypted, implying that the document already contains
a secret.
Change-Id: I394eb8c4e6002f896ecdaa14d2be1e5f948e5048
This PS removes microversions from document versions because
no services use microversions and microversioning for documents
isn't supported in Deckhand.
Change-Id: I3635d15513a2c7b8154ec6be4d0b8577e7d4ce3d
Use YAML formatting for built-in Deckhand schemas
used for validations to align with other UCP services.
The second most important intention behind this PS
is to allow pre_validate flag to cascade correctly
between the layering and document_validation modules.
If pre_validate is true, then:
* the base_schema validates ALL documents
* ALL built-in schemas validate the appropriate
document given a schema match
* NO externally registered DataSchema documents
are used for validation
Else (if pre_validate is false):
* the base_schema validates ALL documents
* ALL built-in schemas validate the appropriate
document given a schema match
* ALL externally registered DataSchema documents
are used for validation given a schema match
A more minor change is setting pre_validate flags in
all modules to True for consistency. The idea is to
facilitate the way other projects that import Deckhand
in directly interface with Deckhand.
Change-Id: I859f61989ec15bede1c104b86625d116064f056d
Currently Pegleg uses a lot of raw documents that are missing
properties at first because those properties are only included
in the documents only after they undergo substitution (are rendered).
This means that when these raw documents are PRE-validated against
registered DataSchemas a lot of noise is created.
However, after the documents are rendered (undergo substitution)
then they should be POST-validated against the registered DataSchemas.
This PS makes the changes necessary to make pre-validation ignore
validation against registered DataSchemas but makes post-validation
raise all validation errors while validating against all built-in
and registered schemas.
Necessary changes were made to tests to make them pass with the
new changes. A follow up will be needed to do better testing
for pre-validation vs. post-validation but the functional test
scenario in schema-validation-success.yaml should test both
scenarios.
Change-Id: I5c139fa528639d43fc45eda067a9ea807fe26c61
Recently the Deckhand Validations API was updated to
return a more nuanced validation error message in
https://review.gerrithub.io/#/c/396251/
This PS adds functional tests to validate the new
format as well as to validate that secrets are
sanitized from the validation error output for
security reasons.
Change-Id: Ic92f2ccd03946a8897aeb2c2a506b3c7eb89836b
Currently the format of the validation error messages returned
by the Deckhand Validation API is lacking. The new response is a
dictionary with the following keys:
* validation_schema: The schema body that was used to validate the
document.
* schema_path: The JSON path in the schema where the failure originated.
* name: The document name.
* schema: The document schema.
* path: The JSON path in the document where the failure originated.
* error_section: The "section" in the document above which the error
originated (i.e. the dict in which ``path`` is found).
* message: The error message returned by the ``jsonschema`` validator.
This PS updates the document validation module and associated unit tests
to return and verify the above format.
Change-Id: I9ef1c36db85233cbfb866dea786228ef1416468c
This PS simplifies the DocumentWrapper class by changing the way
it is designed. The purpose of the class was to make it easier
to retrieve nested dictionary attributes from a document. The class
previously inherited from `object` meaning that the object could not
directly be treated as a dictionary, complicating usage of the class.
With this change, the class now inherits from a `dict` meaning that
it can be manipulated the same way a dictionary can, while still
able to return nested dictionary attributes without having to worry
about exceptions getting thrown.
Each property implemented by `DocumentWrapper` uses jsonpath_parse
implements in `deckhand.utils` to retrieve nested attributes or
else self.get() to retrieve first-level dictionary attributes.
Change-Id: I1d73a79aa4c3117be31aab978c20258c1052ad6d
This PS moves DB calls out of the engine module and into the
appropriate controllers so that a "production" set of documents
can leverage Deckhand layering and substitution after importing
the Deckhand engine module directly. These operations will be
carried out offline meaning that DB calls are not possible.
Unit tests were refactored to work with the changes.
Some testing documentation was also updated.
Closes 16
Change-Id: I6e0757746cd949985d57102d1c85acfbbed86078
This PS rewrites the document_validation module in
Deckhand to achieve the following goals:
* better validation resiliency
* add support for different document schema versions
* better support for DataSchema validation
* separation of concerns by splitting up validations
into separate classes
* support for validating documents that rely on
a DataSchema passed in via the same payload
* support for generating multiple validation errors
rather than returning after the first one found
* increase testing validations for unit/functional
tests
Better validation resiliency is achieved through more
robust exception handling. For example, it is possible
for a ``DataSchema`` to be 100% valid from the POV of
built-in schema validation, but if the "data" section
itself is utterly invalid, then an exception will be
raised -- such an exception is treated as a critical
failure.
Better generation of error messages is achieved by
creation more validation error message results.
DataSchema validation was previously wonky. A DataSchema
had to first be created in 1 revision before it could be
referenced by a batch of documents in sequential revisions.
Now, a DataSchema can be created in the same (or previous)
revision as documents that rely on it and used to validate
said documents.
Finally, the module was heavily rewritten so that more
nuanced validations can be built by inheriting from
``BaseValidator`` so as to allow for easier code
readability and maintainability.
Change-Id: Ie75742b984b7ad392cb41decc203d42842050c80
This PS allows Deckhand to be able to run functional tests with
multiple workers. To achieve that, a document validation bug
was fixed: undeleted data schemas from all previous revisions are
considered. (The test schema-validation-success_add_invalid_document
was failing sporadically because of data race conditions with
only considering data schemas from the last revision with multiple
workers.)
The number of workers for running functional tests via uwsgi
has been increased to the number of CPU cores available on
the server to consistently validate concurrency.
Change-Id: I12589c2ed10495a1eb30757b6bacc5370503d0f4
This PS adds functional tests for built-in or "owned"
Deckhand documents. This includes 4 new document types
for which schemas were created:
- CertificateAuthority
- CertificateAuthority
- PublicKey
- PrivateKey
These new types are required by Promenade.
Finally, a bug was fixed which was causing secrets to
return as {'secret': "original secret payload"} which has
been fixed to return as simply "original secret payload".
Change-Id: Ifb4d41f5f4ac96b3103210853ad763d766ace93e
This PS implements the Deckhand API client library
which is based off the python-novaclient code base.
The client library includes managers for all the
Deckhand APIs.
The following features have been implemented:
* Framework for API client library
* Manager for each Deckhand API (buckets, revisions, etc.)
* API client library documentation
Tests will be added in a follow-up (once Deckhand functional
tests use Keystone).
Change-Id: I829a030738f42dc7ddec623d881a99ed97d04520
This PS implements schema validation for fully rendered documents.
Failed validation when calling GET /revisions/{revision_id}/rendered-documents
results in a 500 Internal Server Error being raised.
Included in this PS:
- Post-rendering validation logic in the appropriate controller
- Unit tests
- Documentation update
Change-Id: I000043ba797b223be6e141bf851d9b2999fc3140
A document with a distinct metadata.name/schema can be re-created
in a different bucket after it has been deleted in its original
bucket. This is also true for layering policies.
This PS fixes the above problem. It also updates validation
documentation to be clearer.
Change-Id: If232f6ca613611995674f9d0149d5f4e5d155040
This PS prevents the same DataSchema from being used more than once
for validation. Otherwise the list of errors generated will be
duplicated.
Change-Id: I4eb1d33cdbe084ddea522b2c9ea91d507de4ca23
The Validations API has been introduced to Deckhand, allowing users
to register new validation results in Deckhand, as well as query
the API for validation results for a revision. The validation results
include a list of errors that occurred during document validation.
All functional tests related to the API are now passing.
The following endpoints have been implemented:
* /api/v1.0/revisions/{revision_id}/validations
* /api/v1.0/revisions/{revision_id}/validations/{validation_name}
* /api/v1.0/revisions/{revision_id}/validations/{validation_name}/entries
* /api/v1.0/revisions/{revision_id}/validations/{validation_name}/entries/{entry_id}
Some back-end refactoring was needed to implement this API. In
particular:
- Added a new Validation sqlalchemy DB model
- Introduced DataSchema handling to the engine.document_validation
module so that registered schema validations can be used
- Changed the way the result of the 'deckhand-schema-validation' internal
validation is generated: it is now the amalgamation of all the
internal and registered schema validations executed
- Introduced rawquery generation so that raw SQL queries can be used to
get results from DB
Fixed following bug:
- UniqueConstraint is now used to correctly generate unique constraints
for sqlalchemy models that are supposed to be combinations of columns
Change-Id: I53c79a6544f44ef8beab2600ddc8a3ea91ada903
This commit fixes flake8 errors and fixes a minor bug related to
a schema version being v1 rather than v1.0.
OpenStack hacking rules are used to pin down flake8 to sane
standards using [0].
[0] 06e676c461/test-requirements.txt (L5)
Change-Id: Ib236df6f5ec9505c0e635f0faa9877d3397a2e55
This commit constitutes 1 of 2 monolithic ports from Github.
The following major changes have been made:
- Created schemas for validating different types of documents
(control and document schemas), including:
* certificate key
* certificate
* data schema
* document
* layering policy
* passphrase
* validation policy
- Implemented pre-validation logic which validates that each
type of document conforms to the correct schema specifications
- Implemented views for APIs -- this allows views to change the
DB data to conform with API specifications
- Implemented relevant unit tests
- Implement functional testing foundation
Change-Id: I83582cc26ffef91fbe95d2f5f437f82d6fef6aa9
This commit adds the documents API and adds logic for performing
pre-validation schema checking wherever applicable in the
documents API.
The following endpoints in the documents API have been implemented:
- POST /documents
Sergiy Markin