Commit Graph

76 Commits

Author SHA1 Message Date
Sergiy Markin ac4edb0c64 [focal] Deckhand project updates
- adjusted .gitignore to keep fresh egg-info and omit build artifacts
- fresh egg-info data is needed for promenade that depends on Deckhand
- restored deckhand-functional-uwsgi-py38 gate
- restored deckhand-integration-uwsgi-py38 gate
- made deckhand-airskiff-deployment gate voting ( treasuremap project
  has been updated)
- removed bionic gates
- updated focal dockerfile
- added more binary deps into bindep.txt
- updated deckhand chart values to latest images - focal and wallaby
- fixed python code to compy with CVE's found by fresh version of bandit
- implemented pip freeze approach
- added tox -e freeze profile to manage it
- requirements-frozen.txt is now main file with requirements
- requirements-direct.txt is the file to control deps
- updated setup.cfg to adjust to newer version of setuptools
- fixed airskiff-deploy gate
- fixed docker-image-build playbook to restore Quay repo image publish
- updated other playbooks to include roles from zuul/base-jobs in order
  to setup build hosts properly
- removed workaround with hardcoded dns resolver ip 10.96.0.10 as it
  became obsolette due to recent fix in openstack-helm-infra
- adjusted tools/whitespace-linter.sh script
- tox.ini has been brought to compliance with tox4 requirements
- replaced str() calls with six.text_type() according to D325 Deckhand specific
  commandment from Hacking.rst
- locked python-barbicanclient version with 5.2.0 because of breaking
  changes in the upper versions

Change-Id: I1cd3c97e83569c4db7e958b3400bdd4b7ea5e668
2023-04-20 19:39:43 +00:00
Smruti Soumitra Khuntia 8fc98631b9 Revision diffing issue with revision rollback.
* Fix for diffing issue after rollback in
conjunction with created and deleted buckets.
* Changed rollback function to check against the full set of documents
for a revision instead of just the documents at that particular revision
* Created a document_delete function to encapsulate document deletion
* Added additional test case to check that a rollback to
something other than 0 deletes the created buckets in between

Co-Authored-By: Michael Beaver <michaelbeaver64@gmail.com>
Change-Id: I0d57e67d68def1f15255a8c89290e8c70deedc03
2018-11-21 11:32:16 -06:00
pallav 9345035522 Adding api for revisions deep diffing
GET /revisions/{{revision_id}}/deepdiff/{{comparison_revision_id}}

 - Added deepdiff api for generating diff between
   two rendered documents.
 - Deep diffing for data and metadata
 - Refactor diff functions
 - Client update
 - Added unit testcases
 - Added funtional testcases
 - Doc update

Change-Id: Ib60fa60a3b33e9125a1595a999272ca595721b38
2018-09-17 17:01:34 +05:30
Felipe Monteiro a342b5e7a2 Fix: Transaction rollback following DB creation error
This patch set employs a simple pattern to ensure that
all database objects created within the scope of
documents_create (which corresponds to the PUT
/api/v1.0/buckets/{bucket_name/documents endpoint)
fall underneath the same session transaction, such that
any exception raised during that transaction results in
all in-flight database objects getting rolled back.

This fixes an issue where a revision could be created
(and automatically committed) even if documents failed
to be created immediately afterward (due to data
conflict issues, for example), leading to a junk revision
existing in the database. Now, the revision will no
longer be created in the above scenario.

This is achieved by using with session.begin() and
placing all database operations underneath that transaction.
Nested helper functions such as bucket_get_or_create
(which is called from within documents_create) no longer
uses its own session.begin() because there is no
need to create a subtransaction: it still falls underneath
the parent transaction as all we care about is idempotence
with respect to document/revision/bucket DB object creation.

A unit test is added to validate the correct behavior to
avoid regression in behavior.

Change-Id: Ifd19b1404a7f932cf4e045ca47acf364ce992c11
2018-09-10 05:13:40 +00:00
Zuul 580d617db8 Merge "Support rolling back to revision 0" 2018-08-24 17:31:31 +00:00
Felipe Monteiro ca1997ec5d Support rolling back to revision 0
This patch set allows revision rollback functionality to work
with revision 0 in order to allow Deckhand to create a blank
revision (without any documents), without having to delete every
revision in the database (which is currently done here: [1] for
example). This is less than ideal as revision history is lost.
Instead, Deckhand should just support rolling back to revision 0
which simulates a blank slate (though in reality the previous
revision history is retained).

[1] 18ed6674d2/src/bin/shipyard_airflow/shipyard_airflow/control/helpers/configdocs_helper.py (L159)

Change-Id: Ic94b25ca15f88ba492e08c907d4a330aeecca810
2018-08-23 23:23:55 +01:00
melissaml 7290c91889 Remove the duplicated word
Change-Id: Ifdd74e418c7387e39fd0d8e667e86ccf2ed889fe
2018-08-22 10:57:02 +08:00
pallav 5979f7f93a Fix typo in revision_diff function
Change-Id: I8fc875825d8e66c3bb9238858de0112ced0538ed
2018-08-14 13:22:45 +05:30
Felipe Monteiro 1c394a2bdc Delete secret references from Barbican when deleting all revisions
This patchset adds logic to delete all secret references from Barbican
when deleting all Deckhand revisions -- because this action effectively
purges the Deckhand database. Consequently, to avoid stale secrets
lingering about in Barbican, this action should clean up those
secret references too.

Change-Id: I88782404bd7f6d953cae343f21231b2943eaf7d2
2018-07-22 15:53:31 +00:00
Felipe Monteiro cd2d3020ec refactor: Use yaml.add_representer to reduce complexity
This patchset uses yaml.add_representer for DocumentDict
which enables yaml.safe_load/safe_load_all to correctly
serialize the DocumentDict object without a recursive
routine.

This also completely removes the usage of jsonpath_parse
from DocumentDict as jsonpath-ng is a rather expensive
library to call continuously; and even though Deckhand
does some caching to alleviate this, it is simply better
to avoid it altogether in a wrapper that is used everywhere
across the engine module, which does all the heavy processing.

This also reduces the amount of wrapping using DocumentDict
because the better way to do this in the DB module is to
have a helper function retrieve the data from the DB and
immediately wrap it in a DocumentDict if applicable;
this is left as an exercise for later.

Change-Id: I715ff7e314cf0ec0d34c17f3378514d235dfb377
2018-07-10 19:23:52 +01:00
Felipe Monteiro 039f9830da Move retrieval of encrypted documents to Deckhand controller
This patchset moves retrieval of encrypted documents to the
Deckhand controller so that components like Pegleg and
Promenade can consume the Deckhand engine offline without
running into Barbican errors.

Components can pass in `encryption_sources` to Deckhand's
rendering module which Deckhand will now use instead to resolve
secret references.

`encryption_sources` is a dictionary that maps the reference
contained in the destination document's data section to the
actual unecrypted data. If encrypting data with Barbican, the
reference will be a Barbican secret reference.

Change-Id: I1a457d3bd37101d73a28882845c2ce74ac09fdf4
2018-07-08 23:16:26 +00:00
Felipe Monteiro 84ab5c5096 [test] Add integration test scenario for encrypting generic type
This PS adds an integration test scenario for validating that
encrypting a generic document type and using it as a substitution
source during document rendering works.

Deckhand will now submit all generic documents to be encrypted
to Barbican with a 'secret_type' of 'passphrase'. No encoding
is provided Deckhand-side (i.e. base64) because encoding is
deprecated in Barbican since it lead to strange behavior;
Barbican will figure out what to encode the payload as
automatically. For more information, see [0] and [1].

In addition, this PS handles 2 edge cases around secret
payloads that are rejected by Barbican if not handled
correctly by Deckhand: empty payloads and non-string
type payloads [2]. For the first case Deckhand forcibly
changes the document to cleartext because there is no
point in encrypting a document with an empty payload.
For the second case Deckhand sets overrides any
previously set secret_type to 'opaque' and encodes
the payload to base64 -- when it goes to render
the secret it decodes the payload also using base64.

Integration tests have been added to handle both edge
cases described above.

[0] https://bugs.launchpad.net/python-barbicanclient/+bug/1419166
[1] 49505b9aec/barbicanclient/v1/secrets.py (L252)
[2] 49505b9aec/barbicanclient/v1/secrets.py (L297)

Change-Id: I1964aa84ad07b6f310b39974f078b84a1dc84983
2018-06-16 15:11:20 -04:00
Tin Lam 33e2203f5e style(pep8): remove identation ignores
This patch set removes few pep8/flake8 ignored rules and implemented
the fix in the code to address those rules.

Change-Id: I2e613acd760818a6e18288d284f6224c38c4353a
Signed-off-by: Tin Lam <tin@irrational.io>
2018-06-01 22:08:42 +00:00
Felipe Monteiro d02e1bcf53 [feature] Endpoint for listing revision validations with details
This patch set adds a new endpoint to the Validations API which allows
for listing all validations for a given revision with details.
The response body for GET /api/v1.0/{revision_id}/validations/detail
looks like:

  ---
  count: 1
  next: null
  prev: null
  results:
    - name: promenade-site-validation
      url: https://deckhand/api/v1.0/revisions/4/validations/promenade-site-validation/entries/0
      status: failure
      createdAt: 2017-07-16T02:03Z
      expiresAfter: null
      expiresAt: null
      errors:
        - documents:
            - schema: promenade/Node/v1
              name: node-document-name
            - schema: promenade/Masters/v1
              name: kubernetes-masters
          message: Node has master role, but not included in cluster masters list.

Note that the Validations API in general is currently missing fields
like url (as well as next and prev references) which will be included
in a follow up.

This will enable Shipyard to avoid performing a quadratic number
of API look ups when querying Deckhand's Validations API: [0].

The policy enforced for this endpoint is deckhand:list_validations.

APIImpact
DocImpact

[0] 06b5e82ea8/shipyard_airflow/control/configdocs/deckhand_client.py (L265)

Change-Id: I827e5f47bffb23fa16ee5c8a705058034633baed
2018-04-29 01:37:38 +00:00
Bryan Strassner 5f1fbbee3c [396582] Add alembic support to Deckhand
Updates Deckhand to use alembic to manage database upgrades.
Moves from creating tables at startup of Deckhand to the
db-sync job.

Change-Id: I6f4cb237fadc46fbee81d1c33096f48a720f589f
2018-04-06 23:30:16 -04:00
Felipe Monteiro 74528a518d Document replacement: Layering dependency integration
This PS integrates document replacement with document layering. The case
works something like this:

  GIVEN:
  - Parent A
  - Child B
  - Child C

  WHEN:
  - Child B is a replacement for A

  THEN:
  - B must layer with A, then C must layer with B, rather than A,
    as B replaces A.

This is the most basic scenario and there are certainly far more
intricate ones, involving interplay with substitution as well.

To implement this new functionality, relatively minor coding changes
were made, mostly in whether to consider a document's parent or its
parent's replacement while layering, as well as determining the
dependency chain for document sorting.

Unit tests surrounding replacement have been moved into their own files
and a scenario has been added for the case described above. In addition
the same case is tested via a functional test scenario.

The unit tests have been "hardened" to run the layering scenarios twice:
once by passing in the documents in their original order, an order which
is usually written for human maintainability (i.e. B depends on A, so
make the order A followed by B). However, in reality the order of the
documents will be randomized, so every layering unit test is also
run a second time with the documents in reverse order to better ensure
that the dependency chain is resolved correctly.

Change-Id: Ieb058267f3a46b78e899922b6bc5fd726ed15a1b
2018-04-04 10:58:28 -04:00
Felipe Monteiro 4a586e370e Remove unused functions from DB module
This is a trivial PS that removes 2 unused functions from the
DB module.

Change-Id: I5c84d0b9d7b4c7e64a3608dcc5cc5262e0b9424a
2018-04-03 14:42:48 -04:00
Felipe Monteiro 1264e5af6c Document replacement: Update Document unique constraint
This updates the unique constraint for Document model from
schema/metadata.name to schema/metadata.name/layer which is
a pre-requisite for document replacement implementation.

The remainder fo the changes are taken of in child PS
(particulary those related to the layering module):
https://review.gerrithub.io/#/c/403888/

Change-Id: Icc4f4960b3a3951f649c7886dbe0bce77341a9f7
2018-03-28 17:08:03 -04:00
Felipe Monteiro 91de02be34 Fix secret_uuid used to query Barbican's Secrets API
This is to fix secrets_manager.SecretsManager.get method which
is passing in the secret reference to Barbican directly for
GET /secrets/{uuid} [0] causing Barbican to raise a
ValueError exception when it attempts to validate that
{secret_uuid} is in fact a UUID.

The fix is to extract the secret_uuid from the secret_ref returned
by Barbican before querying the GET /secrets/{uuid} API.

[0] https://docs.openstack.org/barbican/latest/api/reference/secrets.html#get-v1-secrets-uuid

Change-Id: I4db317e3ba12b4268df5b84b79be8da1da5ac2ba
2018-03-26 21:30:57 +01:00
Felipe Monteiro cce6ddaf6e Fix uniqueness not being enforced at DB level for documents
UniqueConstraint is currently implemented incorrectly in terms
of syntax in Deckhand's Document DB model. This PS fixes that.
Now UniqueConstraint should be enforcing document uniqueness
at DB level such that an error is thrown for duplicate documents
(with same metadata.name and schema).

Closes #17

Change-Id: I7d66457f471ec48b5766733046977117b509d592
2018-03-22 05:57:19 -04:00
Felipe Monteiro d82d0cfaf7 ValidationPolicy integration with Validations API
This PS integrates ValidationPolicy logic with the
Deckhand Validations API.

Support for multiple ValidationPolicy documents is
included.

If a ValidationPolicy is found, then the validations
contained therein are used to determine whether
a revision is successful or not. For example,
if a VP contains 'promenade-schema-validation' then
DH will return success if the externally registered
validation result for that validation is success.
However, if the result was never registered in DH
then the returned result is 'failure'.

In addition, if "extra" validations are registered
(that is validations not present in any VP) then
they are effectively ignored. An error message is
added with enough details to indicate why the validation
is ignored.

This PS adds unit tests to verify the correct behavior
for the above scenarios.

Functional tests and documentation changes will be added
in a follow up once design is ironed out.

Change-Id: I44c657974589ea3563e0a23ad667894329048b46
2018-03-12 12:41:06 -04:00
Felipe Monteiro b0c2f1c4e2 Allow unit tests to be run against in-memory sqlite
Recently JSONB replaced a back-end agnostic data type
for the "data" column in the Document model. This
made it necessary to drop support for running Deckhand
unit tests with any other database store.

However, this arragenement is undesirable as a user
shouldn't need to have postgresql installed just to
kick off unit tests.

So, this PS re-adds support for running unit tests
via an in-memory sqlite database.

To run unit tests with sqlite:

    tox -e py35

Unit tests still run against postgresql via:

    tox -e py35-postgresql

Both jobs are executed in CICD already.

This PS also updates the remaining DB columns to use JSONB if
postgresql is enabled; else fallback columns are used for testing
with sqlite. This is a necessary change to make the column data
types consistent.

Change-Id: I951f2f04fd013d635bb7653a238ff1eb3725b5e1
2018-02-12 22:10:29 -05:00
Felipe Monteiro 18704ff74d Optimize runtime for excluding deleted documents
Optimize runtime for excluding deleted documents in Deckhand's
database layer. Currently the runtime is O(N^2): For each
document, check to see if it is deleted, then iterate over
every other document and delete it if its creation date
is earlier than the current document's deletion date
and their schema and metadata.name are the same (in other
words delete every document from an earlier revision that
was deleted in a more current one if it's the same document).

The runtime was changed to O(NlogN).

Change-Id: I4aa4e1429014731751288861735c705e6b6c6ed4
2018-02-06 15:22:08 -05:00
Felipe Monteiro 02d4af2b16 Allow same tag to be created for multiple revisions
This PS fixes a unique constraint for revision tags, allowing a
tag to be associated with only one revision. It also adds an
additional try/catch block when attempting to update an existing
revision (although it is mostly superfluous).

Closes 14

Change-Id: I83cfacb25f71a33641054c5a26e5b9a1c9907658
2018-01-18 16:15:37 -04:00
Felipe Monteiro 2620913499 Validate correct documents used for rendering.
This PS adds currently lacking validation around ensuring that the
right documents are pooled together for rendering. The validation
checks that documents from older revisions are unused, and that
only documents from the latest revision corresponding to each
bucket are used for rendering.

Change-Id: I9494c8d7055aac815c5baf0b15c7b1743c8ff259
2018-01-15 14:36:10 -05:00
Felipe Monteiro 4b70927bb2 Fix: Allow generic documents to be used as substitution sources.
This PS fixes a bug related to Deckhand only using "secret"
document types to be used as substitution sources; the substitution
logic should be made generic, because it shouldn't just apply to
secrets.

This entailed removing the "is_secret" database column from the
Document table as it's no longer needed and dropping it from a DB
query made to find the source document for substitution in the
secrets_manager module.

This PS also increased resiliency via exception handling and some
edge cases surrounding substitution.

Finally, unit tests and functional tests were added to validate
substitition using a generic document as the source.

Change-Id: I2c4b49b2eb55473c56b8253a456803e793b0b0b0
2018-01-12 14:06:44 -04:00
Mark Burnett 69db7f81fa Test: add unusual documents to functional testing
Unusual documents are documents with different data
types for the data field. The data types include:
object, array, string and integer.

This PS makes necessary ORM model and schema
changes needed to support the different data types.

The ORM data type for the data column has been changed
to JSONB for PostgreSQL. Thus, DH now only supports
PostgreSQL. As a result, the tox jobs have been updated
to only use postgre.

Change-Id: I53694d56bef71adacb5eb79162678be73acb4ad8
2018-01-10 23:33:01 -04:00
Felipe Monteiro 1505c76388 DECKHAND-89: Integrate layering with rendered documents
This PS integrates layering functionality with rendered
documents API endpoint. No new functionality was really
added; instead, a lot of code was refactored to make
layering work alongside substitution.

The following changes have been made:
  - Moved document filtering functionality to deckhand.utils
    because rendered documents must be filtered twice: once
    to retrieve all documents necessary for rendering from
    the DB and again by the controller to filter out documents
    the user doesn't want returned
  - Additional LOG statements in the layering module
  - Additional layering unit tests
  - Additional functional tests
  - Removal of some stricter validations around layering:
    if a parent document is not found for a document,
    an error is no longer returned, as not all documents
    need to have a parent (that is, not all documents
    need to be rendered together, though this might need
    to be expanded on later: what if a document has a
    `parentSelector` but no parent is found?)

Change-Id: I6c66ed824fba0216ba868a6101a72cfe3bdda181
2017-12-15 11:07:41 -05:00
Felipe Monteiro b47f421abf DECKHAND-87: Deckhand API client library
This PS implements the Deckhand API client library
which is based off the python-novaclient code base.
The client library includes managers for all the
Deckhand APIs.

The following features have been implemented:
  * Framework for API client library
  * Manager for each Deckhand API (buckets, revisions, etc.)
  * API client library documentation

Tests will be added in a follow-up (once Deckhand functional
tests use Keystone).

Change-Id: I829a030738f42dc7ddec623d881a99ed97d04520
2017-12-13 20:56:23 +00:00
Felipe Monteiro 14f1b7a0e8 Always rollback to the target revision
This PS removes some additional validations around not being
able to rollback to a target revision:
  - if the target revision == current revision
  - if the target revision is effectively equivalent to the current
    revision (in terms of constituent documents)

Change-Id: I92f8f9557f96b6a27f0dcef4f3138d542e5aa915
2017-11-30 16:32:18 +00:00
Felipe Monteiro fb15186b44 Support filtering by schema namespace
This PS adds support for filtering by schema namespace. According
to docs [0], "schema=promenade [should] select all kind and version
schemas owned by promenade, or schema=promenade/Node [should] select
all versions", for example.

This PS also adds bucket.status filtering support to rendered-documents
endpoint which should also be supported according to docs [1].

[0] http://deckhand.readthedocs.io/en/latest/api_ref.html#get-revisions-revision-id-documents
[1] http://deckhand.readthedocs.io/en/latest/api_ref.html#get-revisions-revision-id-rendered-documents

Change-Id: I43f4c0158096a23b934f4446bfbeaaf3bd7365e4
2017-11-29 23:49:21 -05:00
Felipe Monteiro 16c7ec196f Implement sort filter
This PS implements the sort filter, allowing (for now)
the GET /revisions and GET /revision/{revision_id}/documents
endpoints to be sorted as per the API documentation in
Deckhand [0].

An additional filter has also been added to the 2 aforementioned
endpoints as well -- order -- which determines the order in
which sorted results are returned: "asc" for ascending
order and "desc" for descending order.

[0] http://deckhand.readthedocs.io/en/latest/api_ref.html#get-revisions-revision-id-documents

Change-Id: Ifb9e15b8379b0a28889a14c331d81d9a4147f1d4
2017-11-29 20:53:21 +00:00
Felipe Monteiro d941b09ef0 Reset primary key back to 1 after deleting all revisions
This PS resets the primary key for each table in the
database back to 1 after deleting all revisions
(i.e. DELETE /revisions) which effectively purges all
the data in the database.

Change-Id: Id64e01ea85d7098e261a56cf9eacdcce716b1f48
2017-11-25 05:40:58 +00:00
Felipe Monteiro ce799bd758 Fix corner case for document re-creation in different bucket
A document with a distinct metadata.name/schema can be re-created
in a different bucket after it has been deleted in its original
bucket. This is also true for layering policies.

This PS fixes the above problem. It also updates validation
documentation to be clearer.

Change-Id: If232f6ca613611995674f9d0149d5f4e5d155040
2017-11-06 23:24:01 +00:00
Felipe Monteiro 52a9632e40 Prevent same DataSchema from being used more than once for validation.
This PS prevents the same DataSchema from being used more than once
for validation. Otherwise the list of errors generated will be
duplicated.

Change-Id: I4eb1d33cdbe084ddea522b2c9ea91d507de4ca23
2017-11-01 09:54:53 -04:00
Felipe Monteiro 55b13dc4eb Only allow one LayeringPolicy to exist in the system.
This PS enforces the design requirement that says that only 1
layering policy can exist in the system at once. Attempting to
create another layering policy with a different name is a 409 error.
The existing layering policy can be updated by passing in a document
with the same `metadata.name` and `schema` as the existing one.

Closes-Bug: https://github.com/att-comdev/deckhand/issues/12
Change-Id: I7cad2d600c931c8701c3faaf2967be782984528b
2017-10-26 17:38:24 -04:00
Felipe Monteiro 8aec0390f8 DECKHAND-80: Validations API Implementation
The Validations API has been introduced to Deckhand, allowing users
to register new validation results in Deckhand, as well as query
the API for validation results for a revision. The validation results
include a list of errors that occurred during document validation.

All functional tests related to the API are now passing.

The following endpoints have been implemented:

   * /api/v1.0/revisions/{revision_id}/validations
   * /api/v1.0/revisions/{revision_id}/validations/{validation_name}
   * /api/v1.0/revisions/{revision_id}/validations/{validation_name}/entries
   * /api/v1.0/revisions/{revision_id}/validations/{validation_name}/entries/{entry_id}

Some back-end refactoring was needed to implement this API. In
particular:

  - Added a new Validation sqlalchemy DB model
  - Introduced DataSchema handling to the engine.document_validation
    module so that registered schema validations can be used
  - Changed the way the result of the 'deckhand-schema-validation' internal
    validation is generated: it is now the amalgamation of all the
    internal and registered schema validations executed
  - Introduced rawquery generation so that raw SQL queries can be used to
    get results from DB

Fixed following bug:

  - UniqueConstraint is now used to correctly generate unique constraints
    for sqlalchemy models that are supposed to be combinations of columns

Change-Id: I53c79a6544f44ef8beab2600ddc8a3ea91ada903
2017-10-21 23:09:16 -04:00
Felipe Monteiro d2d2312af9 DECKHAND-66: Document substitution implementation
This PS implements documentation substitution and
the rendered-documents endpoint. Each time the
rendered-documents is queried, the documents for
the reqeust revision_id dynamically undergo
secret substitution.

All functional tests related to secret substitution
have been unskipped.

Deckhand currently does not real testing for
verifying that secret substitution works
for encrypted documents. This will only happen
when integration testing is added to Deckhand to
test its interaction with Keystone and Barbican.

Included in this PS:
  - basic implementation for secret substitution
  - introduction of jsonpath_ng for searching for and
    updating jsonpaths in documents
  - rendered-documents endpoint
  - unit tests
  - all relevant functional tests unskipped
  - additional bucket controller tests include RBAC
    tests and framework testing RBAC via unit tests

Change-Id: I86f269a5b616b518e5f742a4005891412226fe2a
2017-10-13 15:16:27 -04:00
Felipe Monteiro 582dee6fb9 DECKHAND-61: oslo.policy integration
This PS implements oslo.policy integration in Deckhand.
The policy.py file implements 2 types of functions for
performing policy enforcement in Deckhand: authorize,
which is a decorator that is used directly around
falcon on_HTTP_VERB methods that raises a 403 immediately
if policy enforcement fails; and conditional_authorize,
to be used inside controller code conditionally.

For example, since Deckhand has two types of documents
with respect to security -- encrypted and cleartext
documents -- policy enforcement is conditioned on the
type of the documents' metadata.storagePolicy.

Included in this PS:
  - policy framework implementation
  - policy in code and policy documentation for all
    Deckhand policies
  - modification of functional test script to override
    default admin-only policies with custom policy file
    dynamically created using lax permissions
  - bug fix for filtering out deleted documents (and
    its predecessors in previous revisions) for
    PUT /revisions/{revision_id}/documents
  - policy documentation
  - basic unit tests for policy enforcement framework
  - allow functional tests to be filtered via regex

Due to the size of this PS, functional tests related to
policy enforcement will be done in a follow up.

Change-Id: If418129f9b401091e098c0bd6c7336b8a5cd2359
2017-10-07 18:43:28 +01:00
Felipe Monteiro 3e62ace8ed Support filtering revision (documents) by any legal filter
This PS adds support for filtering revisions and
revision documents documents by any legal filter
(those enumerated in the design document).

Deckhand now supports the following filter arguments:
  * schema
  * metadata.name
  * metadata.label
  * metadata.layeringDefinition.abstract
  * metadata.layeringDefinition.layer
  * status.bucket

Deckhand now supports the following filter arguments for filtering
revisions:
  * tag

Deckhand now supports multiple filters, e.g.:
  * ?metdata.layeringDefinition.layer=site&metadata.name=foo

Deckhand now supports repeated filters, e.g.:
  * ?metadata.label=foo=bar&metadata.label=baz=qux

The following has yet to be implemented will be done in a future
follow-up PS:
  - support sorting by specific keywords as well
  - support query limit and offset filters

Change-Id: I8558481e075715fe7fab98140094d37782a986d9
2017-10-06 16:48:45 -04:00
Tin Lam 47b8d4d035 Fix bandit [B101:assert_used]
This patch set refactors the usage of assert in the python code.
The current implementation does not emit assert statement when optimization
is requested at compile time (with command line option -O).  Thus, it
may not be behaving the way one assumes.

Change-Id: If2f2cac86eeae1de5974971b50a2342d8458e97c
2017-10-03 00:19:58 -05:00
Felipe Monteiro 8bf4f7407d Revamp document hashing
This PS revamps document hashing. Instead of relying on Python's
built-in hash function to hash the contents of a document (i.e.
metadata and data values), sha256 from hashlib is used instead,
mostly for security purposes.

Further, new parameters have been added to the document DB model:
data_hash and metadata_hash, and the old value hash has been
dropped. The data type for storing the hashes has been changed
to String from BigInt.

Finally, testing documentation was added.

Change-Id: I428ddcbce1007ea990ca0df1aa630072a050c722
2017-10-02 18:09:13 +01:00
Felipe Monteiro 81b3e42013 Revision rollback API.
This commit implements the revision rollback API, allowing
users to rollback to a previous revision, whereby a new
revision is created.

An exception (400 Bad Request) is raised if the revision being
rolled back to is identical to the latest revision or if no
changes exist between the latest revision and the one being rolled
back to.

Included in this commit:

  - API endpoint for revision rollback.
  - Back-end logic for rolling back to a previous revision.
    The associated documents are also re-recreated. The
    revision_id assigned to each document depends on whether
    it has changed between the latest revision and the one
    being rolled back to: if changed, the new revision_id
    is assigned, else the original one, to maintain the
    correct revision history.
  - Associated unit tests.
  - Unskip all associated functional tests.

Change-Id: I5c120a92e106544f7f8a4266fc386fb60622d6b3
2017-09-25 22:20:47 +01:00
Felipe Monteiro ef4f65037d Revision diffing API.
This commit implements revision diffing API and the
required back-end logic needed to realize it.

Included in this commit:

  - implementation of revision diffing
  - unskip all revision diff functional tests
  - add additional functional tests for revision diffing
  - relevant unit tests
  - document comparison is performed using hashing as opposed
    to more inefficient, direct comparisons

Change-Id: I0419ee9b8cf3fca6fe75818615d2338dc00b1003
2017-09-25 21:58:30 +01:00
Felipe Monteiro af0bfd813d Deckhand postgresql compatibility.
Currently, Deckhand is not fully compatible with postgresql as
it uses sqlite for all of its testing, including functional testing.
Since postgresql will be used in prod, Deckhand obviously must
support it, in addition to sqlite, needed for unit testing.

This commit alters the functional testing script to use postgresql
as well as makes necessary back-end changes to support postgresql.

Included in this commit:

  - alter tools/functional-tests.sh so that it uses postgresql
    as the db connection
  - modifies primary key for Bucket DB model to be an Integer rather
    than a String
  - updates foreign key to point to new primary key
  - updates necessary integration logic so that the bucket name
    is still known by the Document DB model and returned in
    appropriate response bodies

Change-Id: I7bc806fb18f7b47c13978dcd806d422a573a06b3
2017-09-22 19:28:47 +01:00
Felipe Monteiro e32a5a9319 Bucket deletion implementation
This commit implements logic to realize bucket deletion. This
commit also adds logic for raising an exception when trying
to create the same (document.schema, document.metadata.name)
in a different bucket than the one it was originally created in.

Included in this commit:

  - Implementation of document deletion logic.
  - Documents are always saved, even if they have been deleted
    or remain unchanged between revisions. This makes it easier
    to compute the diff between revisions.
  - Associated unit tests.
  - Unskip all remaining functional tests for
    'document-crud-success-single-bucket.yaml`
  - Raise a 409 exception when trying to create the same
    (document.schema, document.metadata.name) in a different
    bucket.
  - Unskip functional tests for
    'document-crud-error-bucket-conflict.yaml'

Change-Id: I6693bbb918cb672de315a66bb087de547df302d1
2017-09-18 23:03:38 +01:00
Felipe Monteiro c9cdd7514c [feat] DECKHAND-38: Secrets DB model and secrets manager.
This commit adds a DocumentSecret model to the DB for
storing secrets directly in Deckhand as well as references
to secrets stored in Barbican if the encryption type
for the secret is encrypted.

This commit also adds a new class called SecretsManager
for managing the lifecycle of secrets from a higher level.

This commit also adds Postgres compliance. So now all
the DB models should work with Postgres.

Also includes unit tests.

Change-Id: Id7c4be8de2e70735f42b1f6710139d553ab4bea2
2017-09-11 12:39:38 -04:00
Felipe Monteiro 5dfcc600ac Unskip all multi-doc CRUD functional tests
Change-Id: Ibda6028ea26290c6a2eee5b8f183f9b2d11e616e
2017-09-08 20:02:16 +01:00
Felipe Monteiro 7e5bb07541 Document buckets - update logic
Implement update logic for document buckets. Relevant functional
tests for a single bucket now pass.

Also removed validation policy DB model to simplify things.

Change-Id: Icd27cae1d2212a24604eb879518d0bd6974cdc9e
2017-09-07 20:48:57 +01:00
Felipe Monteiro 7b0a69b39a [feat] DECKHAND-36 Revision tagging API
This commit adds an additional attribute called `tags` to each
Revision DB model. This allows Revisions to be tagged with whatever
arbitrary tag/tag data a service chooses to identify a revision by.

This commit:
  - creates a new DB model called `RevisionTag`
  - adds the following endpoints:
     * POST /api/v1.0/revisions/{revision_id}/tags/{tag} (create a tag)
     * GET /api/v1.0/revisions/tags/{tag} (show tag details)
     * GET /api/v1.0/revisions/{revision_id}/tags (list revision tags)
     * DELETE /api/v1.0/revisions/{revision_id}/tags/{tag} (delete a tag)
     * DELETE /api/v1.0/revisions/{revision_id}/tags (delete all tags)
  - adds appropriate unit test coverage for the changes
  - adds functional testing for each API endpoint

Change-Id: I49a7155ef5aa274c3a85ff6f8b85951f155a4b92
2017-08-29 15:41:20 +01:00