- adjusted .gitignore to keep fresh egg-info and omit build artifacts
- fresh egg-info data is needed for promenade that depends on Deckhand
- restored deckhand-functional-uwsgi-py38 gate
- restored deckhand-integration-uwsgi-py38 gate
- made deckhand-airskiff-deployment gate voting ( treasuremap project
has been updated)
- removed bionic gates
- updated focal dockerfile
- added more binary deps into bindep.txt
- updated deckhand chart values to latest images - focal and wallaby
- fixed python code to compy with CVE's found by fresh version of bandit
- implemented pip freeze approach
- added tox -e freeze profile to manage it
- requirements-frozen.txt is now main file with requirements
- requirements-direct.txt is the file to control deps
- updated setup.cfg to adjust to newer version of setuptools
- fixed airskiff-deploy gate
- fixed docker-image-build playbook to restore Quay repo image publish
- updated other playbooks to include roles from zuul/base-jobs in order
to setup build hosts properly
- removed workaround with hardcoded dns resolver ip 10.96.0.10 as it
became obsolette due to recent fix in openstack-helm-infra
- adjusted tools/whitespace-linter.sh script
- tox.ini has been brought to compliance with tox4 requirements
- replaced str() calls with six.text_type() according to D325 Deckhand specific
commandment from Hacking.rst
- locked python-barbicanclient version with 5.2.0 because of breaking
changes in the upper versions
Change-Id: I1cd3c97e83569c4db7e958b3400bdd4b7ea5e668
This patchset fixes a bug where Deckhand was failing to perform
substitution and layering on document sets where all the documents had a
storagePolicy of encrypted. Deckhand would attempt to substitute from an
encrypted source document, but when that document marked as encrypted,
it fails because the source doc had been redacted. The behavior now goes
as follows:
- Resolve Barbican references before layering and substitution have been
performed so that the prior two operations don't attempt to operate on a
Barbican reference
- After substitution, redact the destination document if it is marked as
encrypted
- Now, after substition, we can redact the rest of the documents and
substitutions
Change-Id: I725775d554c9eed2692fc6203c416a7119646680
This patch set ensures that documents that substitute data from
encrypted document sources are themselves redacted, assuming that
cleartext-secrets=true. Note that this redaction fix only applies
to the substitution dest/src paths. The data section is already
being correctly redacted for secondhand sources.
Change-Id: I6ce16a109628259b2cc8132cd9db63261b5dbace
This patch set adds logic to optimize performance around rendered
documents caching further by implementing logic to skip over
post-rendering validation when cache hits occur. This works because:
* if the result set isn't cached, then post-validation is performed;
if the post-validation fails the just-cached result set is
invalidated
* if the result set is cached, then post-validation for that
result set has already been performed successfully, so it
can be safely skipped over
It was discovered via profiler anaylsis that document validation
accounts for an appreciable amount of run time, so optimizing
around this makes a noticeable difference.
Change-Id: I18344ea750cf1028a377028f80ecdd931d9672c0
GET /revisions/{{revision_id}}/deepdiff/{{comparison_revision_id}}
- Added deepdiff api for generating diff between
two rendered documents.
- Deep diffing for data and metadata
- Refactor diff functions
- Client update
- Added unit testcases
- Added funtional testcases
- Doc update
Change-Id: Ib60fa60a3b33e9125a1595a999272ca595721b38
This patchset implements caching lookup and reverse-lookup
functions to allow for much faster retrieval of encrypted data
from Barbican, which doesn't currently support batched
requests in its Secrets API.
This behavior is necessary since Deckhand has to potentially
retrieve and store up to dozens of secrets per request. Note
that data for both lookup functions are invalidated
together, as they are tied to the same cache.
This change implements caching around arguably the most
expensive operation in Deckhand: encryption. By
caching encryption itself, the performance of rendering
documents thereby increases in a meaningful way, without
having to implement much logic to pull it off.
A follow up patch set here: https://review.openstack.org/#/c/585842
focuses on caching rendered documents themselves.
Change-Id: I0d330690a3c5e899b763ddcaa00d356007aa23fb
When DELETE /revisions is called all revisions are deleted from
the DB, which means that all cached documents for each revision
should be cleared from memory. This patch set addresses this
issue. This is part of a set of patch sets aimed at fixing edge
cases like this so rendered documents are always returned correctly.
In the same vein, invalidation is performed for Barbican secrets,
for the same endpoint above.
Change-Id: I0334b638d6ce7569e8771e960dd283b1a85a5a6b
This PS adds an integration test scenario for validating that
encrypting a generic document type and using it as a substitution
source during document rendering works.
Deckhand will now submit all generic documents to be encrypted
to Barbican with a 'secret_type' of 'passphrase'. No encoding
is provided Deckhand-side (i.e. base64) because encoding is
deprecated in Barbican since it lead to strange behavior;
Barbican will figure out what to encode the payload as
automatically. For more information, see [0] and [1].
In addition, this PS handles 2 edge cases around secret
payloads that are rejected by Barbican if not handled
correctly by Deckhand: empty payloads and non-string
type payloads [2]. For the first case Deckhand forcibly
changes the document to cleartext because there is no
point in encrypting a document with an empty payload.
For the second case Deckhand sets overrides any
previously set secret_type to 'opaque' and encodes
the payload to base64 -- when it goes to render
the secret it decodes the payload also using base64.
Integration tests have been added to handle both edge
cases described above.
[0] https://bugs.launchpad.net/python-barbicanclient/+bug/1419166
[1] 49505b9aec/barbicanclient/v1/secrets.py (L252)
[2] 49505b9aec/barbicanclient/v1/secrets.py (L297)
Change-Id: I1964aa84ad07b6f310b39974f078b84a1dc84983
This PS adds a limit query filter parameter to allow users to limit
the number of documents returned by revision documents as well
as rendered documents.
Change-Id: Ic15dc59cd21d82be552fa7b9885754bde47724a0
This PS implements the sort filter, allowing (for now)
the GET /revisions and GET /revision/{revision_id}/documents
endpoints to be sorted as per the API documentation in
Deckhand [0].
An additional filter has also been added to the 2 aforementioned
endpoints as well -- order -- which determines the order in
which sorted results are returned: "asc" for ascending
order and "desc" for descending order.
[0] http://deckhand.readthedocs.io/en/latest/api_ref.html#get-revisions-revision-id-documents
Change-Id: Ifb9e15b8379b0a28889a14c331d81d9a4147f1d4
This PS adds support for filtering revisions and
revision documents documents by any legal filter
(those enumerated in the design document).
Deckhand now supports the following filter arguments:
* schema
* metadata.name
* metadata.label
* metadata.layeringDefinition.abstract
* metadata.layeringDefinition.layer
* status.bucket
Deckhand now supports the following filter arguments for filtering
revisions:
* tag
Deckhand now supports multiple filters, e.g.:
* ?metdata.layeringDefinition.layer=site&metadata.name=foo
Deckhand now supports repeated filters, e.g.:
* ?metadata.label=foo=bar&metadata.label=baz=qux
The following has yet to be implemented will be done in a future
follow-up PS:
- support sorting by specific keywords as well
- support query limit and offset filters
Change-Id: I8558481e075715fe7fab98140094d37782a986d9
This commit adds a DocumentSecret model to the DB for
storing secrets directly in Deckhand as well as references
to secrets stored in Barbican if the encryption type
for the secret is encrypted.
This commit also adds a new class called SecretsManager
for managing the lifecycle of secrets from a higher level.
This commit also adds Postgres compliance. So now all
the DB models should work with Postgres.
Also includes unit tests.
Change-Id: Id7c4be8de2e70735f42b1f6710139d553ab4bea2
This commit adds endpoints for:
* DELETE /revisions
* PUT /bucket/{{bucket_name}}/revisions
Included in this commit:
* Initial DB code for buckets
* Initial API code for API buckets
* Refactored unit tests to work with buckets
* Passing *some* functional tests for:
- revision-crud-success-single-bucket (*all*)
- document-crud-success-single-bucket (*some*)
* Corrected document view for list and corrected
loads in MultidocJsonpaths for test_gabbi to not
fix up the response body
Change-Id: Idf941591d24804b77441ab84259f8b7063c88a33
This commit fixes flake8 errors and fixes a minor bug related to
a schema version being v1 rather than v1.0.
OpenStack hacking rules are used to pin down flake8 to sane
standards using [0].
[0] 06e676c461/test-requirements.txt (L5)
Change-Id: Ib236df6f5ec9505c0e635f0faa9877d3397a2e55
Sergiy Markin