Adds a unit test to validate following scenario:
1) create revision 1 with document
2) create revision 2 with no documents
3) rollback to revision 1 (creating revision 3)
Validate that diffing works for rolled-back revision.
All cases above use same bucket.
Also refactors some test logic for neatness.
Change-Id: I71bf7d34e8aae3ad5abb3c53b05cb96a7038ddc2
This patchset adds Barbican validation/assertions to integration
tests by querying the Barbican API server where appropriate
and validating that the expected data is returned in order
to sanity-check the integration scenarios further.
Change-Id: If5d30712b289f09ac9712ee205673be4150cda16
This updates the unique constraint for Document model from
schema/metadata.name to schema/metadata.name/layer which is
a pre-requisite for document replacement implementation.
The remainder fo the changes are taken of in child PS
(particulary those related to the layering module):
https://review.gerrithub.io/#/c/403888/
Change-Id: Icc4f4960b3a3951f649c7886dbe0bce77341a9f7
This PS adds currently lacking validation around ensuring that the
right documents are pooled together for rendering. The validation
checks that documents from older revisions are unused, and that
only documents from the latest revision corresponding to each
bucket are used for rendering.
Change-Id: I9494c8d7055aac815c5baf0b15c7b1743c8ff259
The Validations API has been introduced to Deckhand, allowing users
to register new validation results in Deckhand, as well as query
the API for validation results for a revision. The validation results
include a list of errors that occurred during document validation.
All functional tests related to the API are now passing.
The following endpoints have been implemented:
* /api/v1.0/revisions/{revision_id}/validations
* /api/v1.0/revisions/{revision_id}/validations/{validation_name}
* /api/v1.0/revisions/{revision_id}/validations/{validation_name}/entries
* /api/v1.0/revisions/{revision_id}/validations/{validation_name}/entries/{entry_id}
Some back-end refactoring was needed to implement this API. In
particular:
- Added a new Validation sqlalchemy DB model
- Introduced DataSchema handling to the engine.document_validation
module so that registered schema validations can be used
- Changed the way the result of the 'deckhand-schema-validation' internal
validation is generated: it is now the amalgamation of all the
internal and registered schema validations executed
- Introduced rawquery generation so that raw SQL queries can be used to
get results from DB
Fixed following bug:
- UniqueConstraint is now used to correctly generate unique constraints
for sqlalchemy models that are supposed to be combinations of columns
Change-Id: I53c79a6544f44ef8beab2600ddc8a3ea91ada903
This PS implements documentation substitution and
the rendered-documents endpoint. Each time the
rendered-documents is queried, the documents for
the reqeust revision_id dynamically undergo
secret substitution.
All functional tests related to secret substitution
have been unskipped.
Deckhand currently does not real testing for
verifying that secret substitution works
for encrypted documents. This will only happen
when integration testing is added to Deckhand to
test its interaction with Keystone and Barbican.
Included in this PS:
- basic implementation for secret substitution
- introduction of jsonpath_ng for searching for and
updating jsonpaths in documents
- rendered-documents endpoint
- unit tests
- all relevant functional tests unskipped
- additional bucket controller tests include RBAC
tests and framework testing RBAC via unit tests
Change-Id: I86f269a5b616b518e5f742a4005891412226fe2a
This PS implements oslo.policy integration in Deckhand.
The policy.py file implements 2 types of functions for
performing policy enforcement in Deckhand: authorize,
which is a decorator that is used directly around
falcon on_HTTP_VERB methods that raises a 403 immediately
if policy enforcement fails; and conditional_authorize,
to be used inside controller code conditionally.
For example, since Deckhand has two types of documents
with respect to security -- encrypted and cleartext
documents -- policy enforcement is conditioned on the
type of the documents' metadata.storagePolicy.
Included in this PS:
- policy framework implementation
- policy in code and policy documentation for all
Deckhand policies
- modification of functional test script to override
default admin-only policies with custom policy file
dynamically created using lax permissions
- bug fix for filtering out deleted documents (and
its predecessors in previous revisions) for
PUT /revisions/{revision_id}/documents
- policy documentation
- basic unit tests for policy enforcement framework
- allow functional tests to be filtered via regex
Due to the size of this PS, functional tests related to
policy enforcement will be done in a follow up.
Change-Id: If418129f9b401091e098c0bd6c7336b8a5cd2359
Unskip some pep8 rules that aren't unreasonably annoying:
E121 - continuation line under-indented for hanging indent
E122 - continuation line missing indentation or outdented
E123 - closing bracket does not match indentation of opening bracket’s line
E124 - closing bracket does not match visual indentation
E125 - continuation line with same indent as next logical line
E126 - continuation line over-indented for hanging indent
E251 - unexpected spaces around keyword / parameter equals
Change-Id: Idf2640fc2d10715a687c46c3e853122ce38109ee
This PS revamps document hashing. Instead of relying on Python's
built-in hash function to hash the contents of a document (i.e.
metadata and data values), sha256 from hashlib is used instead,
mostly for security purposes.
Further, new parameters have been added to the document DB model:
data_hash and metadata_hash, and the old value hash has been
dropped. The data type for storing the hashes has been changed
to String from BigInt.
Finally, testing documentation was added.
Change-Id: I428ddcbce1007ea990ca0df1aa630072a050c722
This commit implements the revision rollback API, allowing
users to rollback to a previous revision, whereby a new
revision is created.
An exception (400 Bad Request) is raised if the revision being
rolled back to is identical to the latest revision or if no
changes exist between the latest revision and the one being rolled
back to.
Included in this commit:
- API endpoint for revision rollback.
- Back-end logic for rolling back to a previous revision.
The associated documents are also re-recreated. The
revision_id assigned to each document depends on whether
it has changed between the latest revision and the one
being rolled back to: if changed, the new revision_id
is assigned, else the original one, to maintain the
correct revision history.
- Associated unit tests.
- Unskip all associated functional tests.
Change-Id: I5c120a92e106544f7f8a4266fc386fb60622d6b3
This commit implements revision diffing API and the
required back-end logic needed to realize it.
Included in this commit:
- implementation of revision diffing
- unskip all revision diff functional tests
- add additional functional tests for revision diffing
- relevant unit tests
- document comparison is performed using hashing as opposed
to more inefficient, direct comparisons
Change-Id: I0419ee9b8cf3fca6fe75818615d2338dc00b1003
Currently, Deckhand is not fully compatible with postgresql as
it uses sqlite for all of its testing, including functional testing.
Since postgresql will be used in prod, Deckhand obviously must
support it, in addition to sqlite, needed for unit testing.
This commit alters the functional testing script to use postgresql
as well as makes necessary back-end changes to support postgresql.
Included in this commit:
- alter tools/functional-tests.sh so that it uses postgresql
as the db connection
- modifies primary key for Bucket DB model to be an Integer rather
than a String
- updates foreign key to point to new primary key
- updates necessary integration logic so that the bucket name
is still known by the Document DB model and returned in
appropriate response bodies
Change-Id: I7bc806fb18f7b47c13978dcd806d422a573a06b3
Implement update logic for document buckets. Relevant functional
tests for a single bucket now pass.
Also removed validation policy DB model to simplify things.
Change-Id: Icd27cae1d2212a24604eb879518d0bd6974cdc9e
This commit adds an additional attribute called `tags` to each
Revision DB model. This allows Revisions to be tagged with whatever
arbitrary tag/tag data a service chooses to identify a revision by.
This commit:
- creates a new DB model called `RevisionTag`
- adds the following endpoints:
* POST /api/v1.0/revisions/{revision_id}/tags/{tag} (create a tag)
* GET /api/v1.0/revisions/tags/{tag} (show tag details)
* GET /api/v1.0/revisions/{revision_id}/tags (list revision tags)
* DELETE /api/v1.0/revisions/{revision_id}/tags/{tag} (delete a tag)
* DELETE /api/v1.0/revisions/{revision_id}/tags (delete all tags)
- adds appropriate unit test coverage for the changes
- adds functional testing for each API endpoint
Change-Id: I49a7155ef5aa274c3a85ff6f8b85951f155a4b92
This commit adds endpoints for:
* DELETE /revisions
* PUT /bucket/{{bucket_name}}/revisions
Included in this commit:
* Initial DB code for buckets
* Initial API code for API buckets
* Refactored unit tests to work with buckets
* Passing *some* functional tests for:
- revision-crud-success-single-bucket (*all*)
- document-crud-success-single-bucket (*some*)
* Corrected document view for list and corrected
loads in MultidocJsonpaths for test_gabbi to not
fix up the response body
Change-Id: Idf941591d24804b77441ab84259f8b7063c88a33
This commit adds a ViewBuilder for document creation (the only
API endpoint for documents with an associated view). The view
returns the following body format:
---
revision_id: <created_revision_id>
documents:
- <created_doc_id_1>
- <created_doc_id_2>
...
This commit also includes unit tests.
Change-Id: Ic23124bebc861c55881ab1e0096d0418bdbe9bd5
This commit fixes flake8 errors and fixes a minor bug related to
a schema version being v1 rather than v1.0.
OpenStack hacking rules are used to pin down flake8 to sane
standards using [0].
[0] 06e676c461/test-requirements.txt (L5)
Change-Id: Ib236df6f5ec9505c0e635f0faa9877d3397a2e55
This commit constitutes 1 of 2 monolithic ports from Github.
The following major changes have been made:
- Created schemas for validating different types of documents
(control and document schemas), including:
* certificate key
* certificate
* data schema
* document
* layering policy
* passphrase
* validation policy
- Implemented pre-validation logic which validates that each
type of document conforms to the correct schema specifications
- Implemented views for APIs -- this allows views to change the
DB data to conform with API specifications
- Implemented relevant unit tests
- Implement functional testing foundation
Change-Id: I83582cc26ffef91fbe95d2f5f437f82d6fef6aa9
Felipe Monteiro