update dockerfile for python deckhand install
add deckhand version to chart 1.0
add chart version 0.2.0
update all packages to latest in requirements.txt
update zuul jobs for focal and python 3.8
remove zuul job functional-uwsgi-py38 in favor of functional-docker-py38
update tox config
typecast to string in re.sub() function
add stestr to test-requirements.txt
add SQLAlchemy jsonpickle sphinx-rtd-theme stestr to requirements.txt
deprecated function: BarbicanException -> BarbicanClientException
fix mock import using unittest
fix import collections to collections.abc
fix for collections modules for older than python 3.10 versions.
deprecated function: json -> to_json
deprecated function: werkzeug.contrib.profiler ->
werkzeug.middleware.profiler
deprecated function: falcon.AIP -> falcon.App
deprecation warning: switch from resp.body to resp.text
rename fixtures to dh_fixtures because there is an imported module
fixtures
switch from stream.read to bounded_stream.read
deprecated function: falcon process_response needed additional parameter
deprecated function: falcon default_exception_handler changed parameter
order
move from MagicMock object to falcon test generated object to fix
incompatability with upgraded Falcon module.
Adjust gabbi tests to fix incompatability with upgraded DeepDiff module
update Makefile to execute ubuntu_focal
update HTK (helmtoolkit)
unpin barbican to pass integration tests
Use helm 3 in chart build.
`helm serve` is removed in helm 3 so this moves
to using local `file://` dependencies [0] instead.
Change-Id: I180416f480edea1b8968d80c993b3e1fcc95c08d
Adds a unit test to validate following scenario:
1) create revision 1 with document
2) create revision 2 with no documents
3) rollback to revision 1 (creating revision 3)
Validate that diffing works for rolled-back revision.
All cases above use same bucket.
Also refactors some test logic for neatness.
Change-Id: I71bf7d34e8aae3ad5abb3c53b05cb96a7038ddc2
This patch set employs a simple pattern to ensure that
all database objects created within the scope of
documents_create (which corresponds to the PUT
/api/v1.0/buckets/{bucket_name/documents endpoint)
fall underneath the same session transaction, such that
any exception raised during that transaction results in
all in-flight database objects getting rolled back.
This fixes an issue where a revision could be created
(and automatically committed) even if documents failed
to be created immediately afterward (due to data
conflict issues, for example), leading to a junk revision
existing in the database. Now, the revision will no
longer be created in the above scenario.
This is achieved by using with session.begin() and
placing all database operations underneath that transaction.
Nested helper functions such as bucket_get_or_create
(which is called from within documents_create) no longer
uses its own session.begin() because there is no
need to create a subtransaction: it still falls underneath
the parent transaction as all we care about is idempotence
with respect to document/revision/bucket DB object creation.
A unit test is added to validate the correct behavior to
avoid regression in behavior.
Change-Id: Ifd19b1404a7f932cf4e045ca47acf364ce992c11
This patch set removes few pep8/flake8 ignored rules and implemented
the fix in the code to address those rules.
Change-Id: I2e613acd760818a6e18288d284f6224c38c4353a
Signed-off-by: Tin Lam <tin@irrational.io>
This updates the unique constraint for Document model from
schema/metadata.name to schema/metadata.name/layer which is
a pre-requisite for document replacement implementation.
The remainder fo the changes are taken of in child PS
(particulary those related to the layering module):
https://review.gerrithub.io/#/c/403888/
Change-Id: Icc4f4960b3a3951f649c7886dbe0bce77341a9f7
This PS fixes a bug related to Deckhand only using "secret"
document types to be used as substitution sources; the substitution
logic should be made generic, because it shouldn't just apply to
secrets.
This entailed removing the "is_secret" database column from the
Document table as it's no longer needed and dropping it from a DB
query made to find the source document for substitution in the
secrets_manager module.
This PS also increased resiliency via exception handling and some
edge cases surrounding substitution.
Finally, unit tests and functional tests were added to validate
substitition using a generic document as the source.
Change-Id: I2c4b49b2eb55473c56b8253a456803e793b0b0b0
Unusual documents are documents with different data
types for the data field. The data types include:
object, array, string and integer.
This PS makes necessary ORM model and schema
changes needed to support the different data types.
The ORM data type for the data column has been changed
to JSONB for PostgreSQL. Thus, DH now only supports
PostgreSQL. As a result, the tox jobs have been updated
to only use postgre.
Change-Id: I53694d56bef71adacb5eb79162678be73acb4ad8
This PS integrates layering functionality with rendered
documents API endpoint. No new functionality was really
added; instead, a lot of code was refactored to make
layering work alongside substitution.
The following changes have been made:
- Moved document filtering functionality to deckhand.utils
because rendered documents must be filtered twice: once
to retrieve all documents necessary for rendering from
the DB and again by the controller to filter out documents
the user doesn't want returned
- Additional LOG statements in the layering module
- Additional layering unit tests
- Additional functional tests
- Removal of some stricter validations around layering:
if a parent document is not found for a document,
an error is no longer returned, as not all documents
need to have a parent (that is, not all documents
need to be rendered together, though this might need
to be expanded on later: what if a document has a
`parentSelector` but no parent is found?)
Change-Id: I6c66ed824fba0216ba868a6101a72cfe3bdda181
This PS implements the sort filter, allowing (for now)
the GET /revisions and GET /revision/{revision_id}/documents
endpoints to be sorted as per the API documentation in
Deckhand [0].
An additional filter has also been added to the 2 aforementioned
endpoints as well -- order -- which determines the order in
which sorted results are returned: "asc" for ascending
order and "desc" for descending order.
[0] http://deckhand.readthedocs.io/en/latest/api_ref.html#get-revisions-revision-id-documents
Change-Id: Ifb9e15b8379b0a28889a14c331d81d9a4147f1d4
The framework for being able to do RBAC unit testing
in Deckhand was added here:
#I86f269a5b616b518e5f742a4005891412226fe2a
https://review.gerrithub.io/#/c/381205/
This PS expands on that foundation by implementing
negative RBAC tests for the remainder of the Deckhand
APIs. Negative testing means attempting to call APIs
with insufficient permissions and expecting 403s or
empty response bodies, depending on whether the
policy enforcement is critical or conditionally
applied.
Also fixes a minor bug related to returning a deleted
document for the endpoint PUT /api/v1.0/bucket/{bucket_name}/documents
Change-Id: I7ae50f300c1c877c3c162a032611a380f8948065
This commit implements revision diffing API and the
required back-end logic needed to realize it.
Included in this commit:
- implementation of revision diffing
- unskip all revision diff functional tests
- add additional functional tests for revision diffing
- relevant unit tests
- document comparison is performed using hashing as opposed
to more inefficient, direct comparisons
Change-Id: I0419ee9b8cf3fca6fe75818615d2338dc00b1003
This commit implements logic to realize bucket deletion. This
commit also adds logic for raising an exception when trying
to create the same (document.schema, document.metadata.name)
in a different bucket than the one it was originally created in.
Included in this commit:
- Implementation of document deletion logic.
- Documents are always saved, even if they have been deleted
or remain unchanged between revisions. This makes it easier
to compute the diff between revisions.
- Associated unit tests.
- Unskip all remaining functional tests for
'document-crud-success-single-bucket.yaml`
- Raise a 409 exception when trying to create the same
(document.schema, document.metadata.name) in a different
bucket.
- Unskip functional tests for
'document-crud-error-bucket-conflict.yaml'
Change-Id: I6693bbb918cb672de315a66bb087de547df302d1
This commit adds a DocumentSecret model to the DB for
storing secrets directly in Deckhand as well as references
to secrets stored in Barbican if the encryption type
for the secret is encrypted.
This commit also adds a new class called SecretsManager
for managing the lifecycle of secrets from a higher level.
This commit also adds Postgres compliance. So now all
the DB models should work with Postgres.
Also includes unit tests.
Change-Id: Id7c4be8de2e70735f42b1f6710139d553ab4bea2
This commit adds endpoints for:
* DELETE /revisions
* PUT /bucket/{{bucket_name}}/revisions
Included in this commit:
* Initial DB code for buckets
* Initial API code for API buckets
* Refactored unit tests to work with buckets
* Passing *some* functional tests for:
- revision-crud-success-single-bucket (*all*)
- document-crud-success-single-bucket (*some*)
* Corrected document view for list and corrected
loads in MultidocJsonpaths for test_gabbi to not
fix up the response body
Change-Id: Idf941591d24804b77441ab84259f8b7063c88a33
This commit constitutes 1 of 2 monolithic ports from Github.
The following major changes have been made:
- Created schemas for validating different types of documents
(control and document schemas), including:
* certificate key
* certificate
* data schema
* document
* layering policy
* passphrase
* validation policy
- Implemented pre-validation logic which validates that each
type of document conforms to the correct schema specifications
- Implemented views for APIs -- this allows views to change the
DB data to conform with API specifications
- Implemented relevant unit tests
- Implement functional testing foundation
Change-Id: I83582cc26ffef91fbe95d2f5f437f82d6fef6aa9
This commit makes the following changes:
* removes unncessary code (timeutils, oslo_utils.timeutils can
be used instead)
* oslo_db.types.JsonEncodedDict can be used instead of a custom
JSONEncodedDict (forces Deckhand to save an actual dict in the
DB as well)
* oslo_db.types.JsonEncodedList used for new `results` Column
in Revisions table
Wahlstedt, Walter (ww229g)