- adjusted .gitignore to keep fresh egg-info and omit build artifacts
- fresh egg-info data is needed for promenade that depends on Deckhand
- restored deckhand-functional-uwsgi-py38 gate
- restored deckhand-integration-uwsgi-py38 gate
- made deckhand-airskiff-deployment gate voting ( treasuremap project
has been updated)
- removed bionic gates
- updated focal dockerfile
- added more binary deps into bindep.txt
- updated deckhand chart values to latest images - focal and wallaby
- fixed python code to compy with CVE's found by fresh version of bandit
- implemented pip freeze approach
- added tox -e freeze profile to manage it
- requirements-frozen.txt is now main file with requirements
- requirements-direct.txt is the file to control deps
- updated setup.cfg to adjust to newer version of setuptools
- fixed airskiff-deploy gate
- fixed docker-image-build playbook to restore Quay repo image publish
- updated other playbooks to include roles from zuul/base-jobs in order
to setup build hosts properly
- removed workaround with hardcoded dns resolver ip 10.96.0.10 as it
became obsolette due to recent fix in openstack-helm-infra
- adjusted tools/whitespace-linter.sh script
- tox.ini has been brought to compliance with tox4 requirements
- replaced str() calls with six.text_type() according to D325 Deckhand specific
commandment from Hacking.rst
- locked python-barbicanclient version with 5.2.0 because of breaking
changes in the upper versions
Change-Id: I1cd3c97e83569c4db7e958b3400bdd4b7ea5e668
update dockerfile for python deckhand install
add deckhand version to chart 1.0
add chart version 0.2.0
update all packages to latest in requirements.txt
update zuul jobs for focal and python 3.8
remove zuul job functional-uwsgi-py38 in favor of functional-docker-py38
update tox config
typecast to string in re.sub() function
add stestr to test-requirements.txt
add SQLAlchemy jsonpickle sphinx-rtd-theme stestr to requirements.txt
deprecated function: BarbicanException -> BarbicanClientException
fix mock import using unittest
fix import collections to collections.abc
fix for collections modules for older than python 3.10 versions.
deprecated function: json -> to_json
deprecated function: werkzeug.contrib.profiler ->
werkzeug.middleware.profiler
deprecated function: falcon.AIP -> falcon.App
deprecation warning: switch from resp.body to resp.text
rename fixtures to dh_fixtures because there is an imported module
fixtures
switch from stream.read to bounded_stream.read
deprecated function: falcon process_response needed additional parameter
deprecated function: falcon default_exception_handler changed parameter
order
move from MagicMock object to falcon test generated object to fix
incompatability with upgraded Falcon module.
Adjust gabbi tests to fix incompatability with upgraded DeepDiff module
update Makefile to execute ubuntu_focal
update HTK (helmtoolkit)
unpin barbican to pass integration tests
Use helm 3 in chart build.
`helm serve` is removed in helm 3 so this moves
to using local `file://` dependencies [0] instead.
Change-Id: I180416f480edea1b8968d80c993b3e1fcc95c08d
This patchset fixes a bug where Deckhand was failing to perform
substitution and layering on document sets where all the documents had a
storagePolicy of encrypted. Deckhand would attempt to substitute from an
encrypted source document, but when that document marked as encrypted,
it fails because the source doc had been redacted. The behavior now goes
as follows:
- Resolve Barbican references before layering and substitution have been
performed so that the prior two operations don't attempt to operate on a
Barbican reference
- After substitution, redact the destination document if it is marked as
encrypted
- Now, after substition, we can redact the rest of the documents and
substitutions
Change-Id: I725775d554c9eed2692fc6203c416a7119646680
This patch set ensures that documents that substitute data from
encrypted document sources are themselves redacted, assuming that
cleartext-secrets=true. Note that this redaction fix only applies
to the substitution dest/src paths. The data section is already
being correctly redacted for secondhand sources.
Change-Id: I6ce16a109628259b2cc8132cd9db63261b5dbace
This PS makes Deckhand raise an exception formatted including
the list ValidationMessage-formatted error messages following
any validation error. This adheres to the format specified
under [0].
To accomplish this, logic was added to raise an exception with
a status code corresponding to the `code` attribute for each
DeckhandException subclass. This means it is no longer necessary
to raise a specific falcon exception as the process has been
automated.
In addition, the 'reason' key in the UCP error exception message
is now populated if specified for any DeckhandException instance.
The same is true for 'error_list'.
TODO (in a follow up):
* Allow 'info_list' to specified for any DeckhandException
instance.
* Pass the 'reason' and 'error_list' and etc. arguments to
all instances of DeckhandException that are raised.
[0] https://github.com/att-comdev/ucp-integration/blob/master/docs/source/api-conventions.rst#output-structure
Change-Id: I0cc2909f515ace762be805288981224fc5098c9c
This is to add a helper function to base controller resource class
for detecting whether a list of documents has any empty (None)
or non-object entries in it (since all documents should be
objects), resulting in a 400 Bad Request getting raised.
This is to prevent the following stacktrace from occurring:
File "/usr/local/lib/python3.5/dist-packages/falcon/api.py", line 244, in __call__
responder(req, resp, **params)
File "./deckhand/policy.py", line 104, in handler
return func(*args, **kwargs)
File "./deckhand/control/buckets.py", line 58, in on_put
documents, data_schemas, pre_validate=True)
File "./deckhand/engine/document_validation.py", line 387, in __init__
raw_document[prop] = document.get(prop)
AttributeError: 'NoneType' object has no attribute 'get'
Change-Id: I76fc9b0d7662358f8b26b5bddf1187e92d1554de
Currently, if a document has a parent but no layering actions,
the document immediately inherents its parents' data, which is a
bug. Instead, the child document should only layer with its
parent's data and then update its own data if it has at least
one layering action.
In addition, the base_schema.yaml under `deckhand.schemas`
has been updated to require that actions be required and
contain at least 1 layering action when parentSelector
is provided and that parentSelector be required when
actions is provided and that at least one key-value
pair be provided. (Empty actions array or empty
parentSelector object is meaningless and should be
disallowed/discouraged.)
This means that actions and parentSelector must always
both be provided (though providing neither is also
legal because layering is optional).
Unit tests have been added to verify the schema updates.
Change-Id: I77d54e2b216efc54b466f94d82ee8d36ca169c26
This PS rewrites the document_validation module in
Deckhand to achieve the following goals:
* better validation resiliency
* add support for different document schema versions
* better support for DataSchema validation
* separation of concerns by splitting up validations
into separate classes
* support for validating documents that rely on
a DataSchema passed in via the same payload
* support for generating multiple validation errors
rather than returning after the first one found
* increase testing validations for unit/functional
tests
Better validation resiliency is achieved through more
robust exception handling. For example, it is possible
for a ``DataSchema`` to be 100% valid from the POV of
built-in schema validation, but if the "data" section
itself is utterly invalid, then an exception will be
raised -- such an exception is treated as a critical
failure.
Better generation of error messages is achieved by
creation more validation error message results.
DataSchema validation was previously wonky. A DataSchema
had to first be created in 1 revision before it could be
referenced by a batch of documents in sequential revisions.
Now, a DataSchema can be created in the same (or previous)
revision as documents that rely on it and used to validate
said documents.
Finally, the module was heavily rewritten so that more
nuanced validations can be built by inheriting from
``BaseValidator`` so as to allow for easier code
readability and maintainability.
Change-Id: Ie75742b984b7ad392cb41decc203d42842050c80
This PS adds currently lacking validation around ensuring that the
right documents are pooled together for rendering. The validation
checks that documents from older revisions are unused, and that
only documents from the latest revision corresponding to each
bucket are used for rendering.
Change-Id: I9494c8d7055aac815c5baf0b15c7b1743c8ff259
This PS implements sorting and filtering for rendered-documents
endpoint, adds additional validations for sorting, filtering
and other layering scenarios, and updates rendered-documents
and buckets documentation.
Layering scenarios added:
- Updating the LayeringPolicy with 2 layers in the layerOrder
(down from 3) such that the site document should have its
parent document recomputed as the global document.
- A deletion action layering scenario (DH currently only has
merge, replace scenarios in its funcitonal test suite.)
Documentation updated:
- clarify the access levels for buckets, which has been a
source of confusion.
- update api-ref documentation for rendered-documents
Change-Id: Idb9b42351dfbdf75a19282c8478065e7564cfc26
Unusual documents are documents with different data
types for the data field. The data types include:
object, array, string and integer.
This PS makes necessary ORM model and schema
changes needed to support the different data types.
The ORM data type for the data column has been changed
to JSONB for PostgreSQL. Thus, DH now only supports
PostgreSQL. As a result, the tox jobs have been updated
to only use postgre.
Change-Id: I53694d56bef71adacb5eb79162678be73acb4ad8
This PS integrates layering functionality with rendered
documents API endpoint. No new functionality was really
added; instead, a lot of code was refactored to make
layering work alongside substitution.
The following changes have been made:
- Moved document filtering functionality to deckhand.utils
because rendered documents must be filtered twice: once
to retrieve all documents necessary for rendering from
the DB and again by the controller to filter out documents
the user doesn't want returned
- Additional LOG statements in the layering module
- Additional layering unit tests
- Additional functional tests
- Removal of some stricter validations around layering:
if a parent document is not found for a document,
an error is no longer returned, as not all documents
need to have a parent (that is, not all documents
need to be rendered together, though this might need
to be expanded on later: what if a document has a
`parentSelector` but no parent is found?)
Change-Id: I6c66ed824fba0216ba868a6101a72cfe3bdda181
This PS adds a filter for deleted=False to rendered documents endpoint
so that previously deleted documents are excluded from current
revision.
Change-Id: Id45786d680f2ada8c2a27d2b44a677eb6b14921d
This PS implements schema validation for fully rendered documents.
Failed validation when calling GET /revisions/{revision_id}/rendered-documents
results in a 500 Internal Server Error being raised.
Included in this PS:
- Post-rendering validation logic in the appropriate controller
- Unit tests
- Documentation update
Change-Id: I000043ba797b223be6e141bf851d9b2999fc3140
Currently, the rendered-documents endpoint returns only documents
that require substitution, rather than all concrete documents, as
specified in the requirements (DECKHAND-65).
This PS adds a filter to the endpoint so that only concrete documents
are returned. Also, all concrete documents are returned, not just
the ones that require substitution.
Included in this PS:
- logic changes described above
- unit test to verify the above logic
Change-Id: Ib552b084bb00b6e180bba973be420449a292fb05
The framework for being able to do RBAC unit testing
in Deckhand was added here:
#I86f269a5b616b518e5f742a4005891412226fe2a
https://review.gerrithub.io/#/c/381205/
This PS expands on that foundation by implementing
negative RBAC tests for the remainder of the Deckhand
APIs. Negative testing means attempting to call APIs
with insufficient permissions and expecting 403s or
empty response bodies, depending on whether the
policy enforcement is critical or conditionally
applied.
Also fixes a minor bug related to returning a deleted
document for the endpoint PUT /api/v1.0/bucket/{bucket_name}/documents
Change-Id: I7ae50f300c1c877c3c162a032611a380f8948065
Sergiy Markin