Commit Graph

50 Commits

Author SHA1 Message Date
Wahlstedt, Walter (ww229g) 70aa35a396 update to focal and python 3.8
update dockerfile for python deckhand install
add deckhand version to chart 1.0
add chart version 0.2.0
update all packages to latest in requirements.txt
update zuul jobs for focal and python 3.8
remove zuul job functional-uwsgi-py38 in favor of functional-docker-py38
update tox config
typecast to string in re.sub() function
add stestr to test-requirements.txt
add SQLAlchemy jsonpickle sphinx-rtd-theme stestr to requirements.txt
deprecated function: BarbicanException -> BarbicanClientException
fix mock import using unittest
fix import collections to collections.abc
fix for collections modules for older than python 3.10 versions.
deprecated function: json -> to_json
deprecated function:  werkzeug.contrib.profiler ->
    werkzeug.middleware.profiler
deprecated function: falcon.AIP -> falcon.App
deprecation warning: switch from resp.body to resp.text
rename fixtures to dh_fixtures because there is an imported module
    fixtures
switch from stream.read to bounded_stream.read
deprecated function: falcon process_response needed additional parameter
deprecated function: falcon default_exception_handler changed parameter
    order
move from MagicMock object to falcon test generated object to fix
    incompatability with upgraded Falcon module.
Adjust gabbi tests to fix incompatability with upgraded DeepDiff module
update Makefile to execute ubuntu_focal
update HTK (helmtoolkit)
unpin barbican to pass integration tests
Use helm 3 in chart build.
    `helm serve` is removed in helm 3 so this moves
    to using local `file://` dependencies [0] instead.

Change-Id: I180416f480edea1b8968d80c993b3e1fcc95c08d
2023-02-24 10:51:57 -05:00
Felipe Monteiro b03a4522cb fix: Use schema instead of metadata.schema for replacement check
Recently added replacement check incorrectly uses metadata.schema
and metadata.name to key on the document -- but it should be schema
and metadata.name, the combination of which uniquely defines a
document.

Change-Id: I6cd1679ad41be38cb78d65ce2763e60f7da390d2
2018-10-31 15:02:28 -04:00
Roman Gorshunov d41e5a44ca Fix: various documentation and URL fixes
1) UCP -> Airship
2) readthedocs.org -> readthedocs.io (there is redirect)
3) http -> https
4) attcomdev -> airshipit (repo on quay.io)
5) att-comdev -> openstack/airship-* (repo on github/openstack git)
6) many URLs have been verified and adjusted to be current
7) no need for 'en/latest/' path in URL of the RTD
8) added more info to some setup.cfg and setup.py files
9) ucp-integration docs are now in airship-in-a-bottle
10) various other minor fixes

Change-Id: I12b2fa8fbec37a483a0ad50382e08f51ed97533a
2018-09-25 11:44:57 +02:00
pallav 9345035522 Adding api for revisions deep diffing
GET /revisions/{{revision_id}}/deepdiff/{{comparison_revision_id}}

 - Added deepdiff api for generating diff between
   two rendered documents.
 - Deep diffing for data and metadata
 - Refactor diff functions
 - Client update
 - Added unit testcases
 - Added funtional testcases
 - Doc update

Change-Id: Ib60fa60a3b33e9125a1595a999272ca595721b38
2018-09-17 17:01:34 +05:30
Felipe Monteiro cd2d3020ec refactor: Use yaml.add_representer to reduce complexity
This patchset uses yaml.add_representer for DocumentDict
which enables yaml.safe_load/safe_load_all to correctly
serialize the DocumentDict object without a recursive
routine.

This also completely removes the usage of jsonpath_parse
from DocumentDict as jsonpath-ng is a rather expensive
library to call continuously; and even though Deckhand
does some caching to alleviate this, it is simply better
to avoid it altogether in a wrapper that is used everywhere
across the engine module, which does all the heavy processing.

This also reduces the amount of wrapping using DocumentDict
because the better way to do this in the DB module is to
have a helper function retrieve the data from the DB and
immediately wrap it in a DocumentDict if applicable;
this is left as an exercise for later.

Change-Id: I715ff7e314cf0ec0d34c17f3378514d235dfb377
2018-07-10 19:23:52 +01:00
Felipe Monteiro 039f9830da Move retrieval of encrypted documents to Deckhand controller
This patchset moves retrieval of encrypted documents to the
Deckhand controller so that components like Pegleg and
Promenade can consume the Deckhand engine offline without
running into Barbican errors.

Components can pass in `encryption_sources` to Deckhand's
rendering module which Deckhand will now use instead to resolve
secret references.

`encryption_sources` is a dictionary that maps the reference
contained in the destination document's data section to the
actual unecrypted data. If encrypting data with Barbican, the
reference will be a Barbican secret reference.

Change-Id: I1a457d3bd37101d73a28882845c2ce74ac09fdf4
2018-07-08 23:16:26 +00:00
Felipe Monteiro 807990a099 Fix gate following strange PyYAML 4.1 behavior
This patchset adds a dict() cast as a workaround the fact
that PyYAML 4.1 recently changed yaml.dump to yaml.safe_dump,
compelling developers to use yaml.danger_dump to achieve
the previous behavior of yaml.dump [0].

However, yaml.danger_dump should not be used and this
technically corrects antecedent use of yaml.dump by
introducing a recursive function that ensures the
dictionary prior to being dumped is compatible
with yaml.safe_dump.

Such a function is needed because yaml.safe_dump
rejects serialization of Deckhand's DocumentDict
dictionary wrapper helper -- even though it is
a subclass of a dict. Thus, the recursive
function simply casts each instance of DocumentDict
into a dictionary.

[0] https://stackoverflow.com/questions/51053903/new-pyyaml-version-breaks-on-most-custom-python-objects-representererror

Change-Id: I67966b45e0865864bd5e6bb4578548769fc13eeb
2018-06-27 19:29:05 +01:00
Felipe Monteiro e65710bf1a Make Deckhand validation exceptions adhere to UCP standard
This PS makes Deckhand raise an exception formatted including
the list ValidationMessage-formatted error messages following
any validation error. This adheres to the format specified
under [0].

To accomplish this, logic was added to raise an exception with
a status code corresponding to the `code` attribute for each
DeckhandException subclass. This means it is no longer necessary
to raise a specific falcon exception as the process has been
automated.

In addition, the 'reason' key in the UCP error exception message
is now populated if specified for any DeckhandException instance.
The same is true for 'error_list'.

TODO (in a follow up):

  * Allow 'info_list' to specified for any DeckhandException
    instance.
  * Pass the 'reason' and 'error_list' and etc. arguments to
    all instances of DeckhandException that are raised.

[0] https://github.com/att-comdev/ucp-integration/blob/master/docs/source/api-conventions.rst#output-structure

Change-Id: I0cc2909f515ace762be805288981224fc5098c9c
2018-04-26 18:51:08 +00:00
Felipe Monteiro a3d32c3459 Raise exception on unfound secret in source document
This PS introduces a new exception SubstitutionSourceSecretNotFound
which is raised when a src.path referenced under the substitutions
of a destination document isn't found in the data section of
the corresponding source document if fail_on_missing_sub_src
is True -- else a warning is logged.

Change-Id: If2b08f443cde765a1dbfaf7bac6b549591e59148
2018-04-09 17:04:58 +01:00
Felipe Monteiro a5f75722dc Log all document data following any layering action failure
This is to log out all document data following any layering action
failure. This consists of two stages:

1) Scrubbing all primitives contained in the data sections of both
   the child and parent being layered together.
2) Logging scrubbed-out data sections for both documents, in addition
   to their names, schemas, and the layering action itself.

This will hopefully provide DEs with enough information about why
a layering action may have failed to apply while at the same time
preventing any secret data from being logged out.

Change-Id: I3fedd259bba7b930c7969e9c30d1fffef5bf77bd
2018-03-29 10:10:03 -04:00
Felipe Monteiro 4799acdbcc Engine implementation for document replacement
This adds support for document replacement to the
Deckhand engine _only_ for the following scenarios
_only_:

  * generic case (a replaces b, returns a only)
  * substitution case (a replaces b, c substitutes from a instead)

TODO:

  * layering case (a replaces b, c layers with a instead)
  * Modify Document unique constraint to work with
    (schema, name, layer) throughout all of Deckhand
    (including controllers, database models, and anywhere
     else as needed)

Change-Id: Ie2cea2a49ba3b9ebc42706fbe1060d94db2e5daa
2018-03-28 17:09:09 -04:00
Felipe Monteiro 1264e5af6c Document replacement: Update Document unique constraint
This updates the unique constraint for Document model from
schema/metadata.name to schema/metadata.name/layer which is
a pre-requisite for document replacement implementation.

The remainder fo the changes are taken of in child PS
(particulary those related to the layering module):
https://review.gerrithub.io/#/c/403888/

Change-Id: Icc4f4960b3a3951f649c7886dbe0bce77341a9f7
2018-03-28 17:08:03 -04:00
Felipe Monteiro cce6ddaf6e Fix uniqueness not being enforced at DB level for documents
UniqueConstraint is currently implemented incorrectly in terms
of syntax in Deckhand's Document DB model. This PS fixes that.
Now UniqueConstraint should be enforcing document uniqueness
at DB level such that an error is thrown for duplicate documents
(with same metadata.name and schema).

Closes #17

Change-Id: I7d66457f471ec48b5766733046977117b509d592
2018-03-22 05:57:19 -04:00
Felipe Monteiro e90c0aedf8 Security fix: Remove document data printout from exception message
This is to remove document data printout from the MissingDocumentKey
exception message which could expose sensitive data if it is caught
and logged by other services, for example. Instead, the child
and parent documents' schema and name are printed, in addition to
the action object in which the path that could not be resolved
in either parent or document is contained.

Change-Id: I07f43e57527d05e98e98e5f80567b97dd2a762f9
2018-03-12 20:42:31 +00:00
Felipe Monteiro 65c459d1f9 Improve secrets_manager logging after 500 Internal Server Error
This is to add better logging to assist with debugging 500
Internal Server Errors that manifest from either internal
bugs in Deckhand or document typos/errors ingested by Deckhand
during document substitution.

Change-Id: I5f4c1cb07bea8e6546d08a858d4f83a24d75ef5a
2018-03-10 16:39:48 -05:00
Felipe Monteiro e0fc59e89b Deckhand schemas as YAML files
Use YAML formatting for built-in Deckhand schemas
used for validations to align with other UCP services.

The second most important intention behind this PS
is to allow pre_validate flag to cascade correctly
between the layering and document_validation modules.

If pre_validate is true, then:
  * the base_schema validates ALL documents
  * ALL built-in schemas validate the appropriate
    document given a schema match
  * NO externally registered DataSchema documents
    are used for validation

Else (if pre_validate is false):
  * the base_schema validates ALL documents
  * ALL built-in schemas validate the appropriate
    document given a schema match
  * ALL externally registered DataSchema documents
    are used for validation given a schema match

A more minor change is setting pre_validate flags in
all modules to True for consistency. The idea is to
facilitate the way other projects that import Deckhand
in directly interface with Deckhand.

Change-Id: I859f61989ec15bede1c104b86625d116064f056d
2018-02-27 11:16:30 -05:00
Felipe Monteiro f0cc8b6c1a [TrivialFix] Fix BarbicanException error propagation
Fix BarbicanException error propagation in Deckhand by ensuring
that the exception details are passed around via str(e) rather
than e.message as the Barbican exceptions [0] don't have additional
attributes like falcon.

Example stack trace:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/falcon/api.py", line 244, in __call__
    responder(req, resp, **params)
  File "./deckhand/policy.py", line 104, in handler
    return func(*args, **kwargs)
  File "./deckhand/control/buckets.py", line 70, in on_put
    self._prepare_secret_documents(documents)
  File "./deckhand/control/buckets.py", line 83, in _prepare_secret_documents
    secret_data = self.secrets_mgr.create(document)
  File "./deckhand/engine/secrets_manager.py", line 74, in create
    resp = self.barbican_driver.create_secret(**kwargs)
  File "./deckhand/barbican/driver.py", line 39, in create_secret
    LOG.exception(e.message)
AttributeError: 'HTTPServerError' object has no attribute 'message'

[0] https://github.com/openstack/python-barbicanclient/blob/master/barbicanclient/exceptions.py

Change-Id: I58410c3729fe4fd066227311589e5ee5d30dc171
2018-02-24 17:01:46 -05:00
Felipe Monteiro b81cebb012 Fail fast on bad substitution input during layering
This PS causes layering module to fail fast on malformed
``metadata.substitutions`` entry in a document by performing
built-in schema validation when validate=True is passed to
the DocumentLayering constructor. This kwarg is useful for
when the layering module is called directly -- i.e. by
Promenade or Pegleg. (The Deckhand server already performs
document pre-validation during document ingestion so there
is no need for documents stored inside Deckhand to be
re-validated again for rendered-documents endpoint.)

Next, a new exception was added -- SubstitutionSourceNotFound
-- which is raised when a substitution document is referenced
by another document but isn't found.

Finally, the previous exception raised by the
secrets_manager module has been renamed to UnknownSubstitutionError
which now raises a 500 instead of a 400 as this exception will
most likely be due to an internal server error of some kind.

Unit tests were added and documentation changes were made.

Change-Id: Idfd91a52ef9ffd8f9b1c06c6b84c3405acab6f16
2018-02-20 05:46:55 -05:00
Felipe Monteiro 99ab93727b [Trivial Fix] Add document layer to error message output
The document layer that is invalid should be included in
the exception called InvalidDocumentLayer for obvious reasons.

Change-Id: Ie7fcecc96bc7667530959af34ec146b4e4a47303
2018-02-14 22:59:27 -05:00
Felipe Monteiro e4abca1cd7 Use DAG to resolve substitution dependency chain
Currently, Deckhand fails to perform substitution given
a substitution dependency chain for a group of documents.
That is, if A depends on B and C depends on B for substitution
then substitution will fail. Deckhand, at present, can only
perform substitution if A depends on B and C depends on B
and D depends on B, for example: In this case, the dependency
chain is no bigger than 2.

However, for cases where the substitution dependency chain
is larger than 2, then the dependency is no longer trivial. This
is because the substitution dependencies form a DAG that must
be topologically sorted in order to derive the correct order of
substitution. Once the documents are correctly sorted according
to this scheme, then substitution can be carried out as usual.

This PS makes the aforementioned changes to Deckhand's layering
module to make substitution work for non-cyclical dependencies:
A DAG is used to topologically sort the documents according to
their substitution dependency chain. A unit and functional
test has been added to verify the code.

If a cycle is detected, a critical error is thrown. Unit tests
have been added to validate this case as well.

Change-Id: Iaca3963f44aec6c897ad9fd690ce314a3a4d97a2
2018-02-12 20:54:54 +00:00
Felipe Monteiro 9f7ecc0582 Make layering work for grandparents not just parents
This PS makes layering work for grandparents, not just parents. This
means that given a document with layer N, then not only can a parent
document in layer N+1 be used, but so can a grandparent in layer N+2.
Note that the document in layer N+1 will be preferred over the one
in N+2.

To provide a concrete example, given layers 'global', 'region'
and 'site', a document with layer 'site' can be layered with
a grandparent in layer 'global' if a document in layer
'region' isn't found. Alternatively, if the document with
layer 'region' is found then it will be used as the parent
document.

This PS refactors algorithm for determining the parent document,
tweaks the layering algorithm itself to reference the
correct parent document, and adds unit tests to confirm the
two scenarios described above.

Change-Id: I3779c7ae030bbad44b2d5ddfa5105d1a073ba670
2018-02-10 17:30:16 -05:00
Krysta 62cd76dc8f Documentation for Exceptions
Adds documentation for Deckhand Exceptions

Change-Id: I1722c07404b2eb12682709e3864dd54a9120f200
2018-02-09 15:33:30 -05:00
Felipe Monteiro 36f752bb93 Fix various substitution issues
This PS solves the following issues for which only minor changes
were needed:

1) Using copy.copy() around the substitution value passed in
   to jsonpath_replace so as to avoid updating the substitution
   entry referentially which was happening before this change,
   causing future substitutions using that entry to fail or
   behaving unexpectedly.
2) Supporting non-string substitution values when a substitution
   pattern is provided: before this change, this was failing
   because calling re.sub() and passing in a non-string
   value causes an error to be thrown.
3) Adding better logging and error handling to
   deckhand.utils.jsonpath_replace to assist with debugging.

Unit tests are included for some of the scenarios above.

Change-Id: I8562d43a717f477e3297504c1522331b3a993f88
2018-02-05 20:15:50 -05:00
Felipe Monteiro 60539353aa Layering edge case: Apply substiutions to parentless document
This PS udpates the layering module so that parentless documents
still undergo substitution if it is concrete and requires substitutions.

Added a unit test to validate the scenario.

Closes 15

Change-Id: Ie131e7a598a5419d2309288234574543c87e4a72
2018-01-19 16:25:06 +00:00
Felipe Monteiro 453927facf Improve document validation module.
This PS rewrites the document_validation module in
Deckhand to achieve the following goals:

  * better validation resiliency
  * add support for different document schema versions
  * better support for DataSchema validation
  * separation of concerns by splitting up validations
    into separate classes
  * support for validating documents that rely on
    a DataSchema passed in via the same payload
  * support for generating multiple validation errors
    rather than returning after the first one found
  * increase testing validations for unit/functional
    tests

Better validation resiliency is achieved through more
robust exception handling. For example, it is possible
for a ``DataSchema`` to be 100% valid from the POV of
built-in schema validation, but if the "data" section
itself is utterly invalid, then an exception will be
raised -- such an exception is treated as a critical
failure.

Better generation of error messages is achieved by
creation more validation error message results.

DataSchema validation was previously wonky. A DataSchema
had to first be created in 1 revision before it could be
referenced by a batch of documents in sequential revisions.
Now, a DataSchema can be created in the same (or previous)
revision as documents that rely on it and used to validate
said documents.

Finally, the module was heavily rewritten so that more
nuanced validations can be built by inheriting from
``BaseValidator`` so as to allow for easier code
readability and maintainability.

Change-Id: Ie75742b984b7ad392cb41decc203d42842050c80
2018-01-15 16:51:52 -05:00
Felipe Monteiro 4b70927bb2 Fix: Allow generic documents to be used as substitution sources.
This PS fixes a bug related to Deckhand only using "secret"
document types to be used as substitution sources; the substitution
logic should be made generic, because it shouldn't just apply to
secrets.

This entailed removing the "is_secret" database column from the
Document table as it's no longer needed and dropping it from a DB
query made to find the source document for substitution in the
secrets_manager module.

This PS also increased resiliency via exception handling and some
edge cases surrounding substitution.

Finally, unit tests and functional tests were added to validate
substitition using a generic document as the source.

Change-Id: I2c4b49b2eb55473c56b8253a456803e793b0b0b0
2018-01-12 14:06:44 -04:00
Felipe Monteiro 8b428743ee Simplify document layering interface
This PS simplifies the layering interface, adds additional
exception handling around document layering in the rendered
documents controller, and removes an unused exception related
to document layering.

Change-Id: I0491ae32a43fe4a6a01c8da530528d8573f91a64
2018-01-05 03:29:30 +00:00
Felipe Monteiro 1505c76388 DECKHAND-89: Integrate layering with rendered documents
This PS integrates layering functionality with rendered
documents API endpoint. No new functionality was really
added; instead, a lot of code was refactored to make
layering work alongside substitution.

The following changes have been made:
  - Moved document filtering functionality to deckhand.utils
    because rendered documents must be filtered twice: once
    to retrieve all documents necessary for rendering from
    the DB and again by the controller to filter out documents
    the user doesn't want returned
  - Additional LOG statements in the layering module
  - Additional layering unit tests
  - Additional functional tests
  - Removal of some stricter validations around layering:
    if a parent document is not found for a document,
    an error is no longer returned, as not all documents
    need to have a parent (that is, not all documents
    need to be rendered together, though this might need
    to be expanded on later: what if a document has a
    `parentSelector` but no parent is found?)

Change-Id: I6c66ed824fba0216ba868a6101a72cfe3bdda181
2017-12-15 11:07:41 -05:00
Felipe Monteiro b47f421abf DECKHAND-87: Deckhand API client library
This PS implements the Deckhand API client library
which is based off the python-novaclient code base.
The client library includes managers for all the
Deckhand APIs.

The following features have been implemented:
  * Framework for API client library
  * Manager for each Deckhand API (buckets, revisions, etc.)
  * API client library documentation

Tests will be added in a follow-up (once Deckhand functional
tests use Keystone).

Change-Id: I829a030738f42dc7ddec623d881a99ed97d04520
2017-12-13 20:56:23 +00:00
Felipe Monteiro 14f1b7a0e8 Always rollback to the target revision
This PS removes some additional validations around not being
able to rollback to a target revision:
  - if the target revision == current revision
  - if the target revision is effectively equivalent to the current
    revision (in terms of constituent documents)

Change-Id: I92f8f9557f96b6a27f0dcef4f3138d542e5aa915
2017-11-30 16:32:18 +00:00
Felipe Monteiro 52a9632e40 Prevent same DataSchema from being used more than once for validation.
This PS prevents the same DataSchema from being used more than once
for validation. Otherwise the list of errors generated will be
duplicated.

Change-Id: I4eb1d33cdbe084ddea522b2c9ea91d507de4ca23
2017-11-01 09:54:53 -04:00
Felipe Monteiro 55b13dc4eb Only allow one LayeringPolicy to exist in the system.
This PS enforces the design requirement that says that only 1
layering policy can exist in the system at once. Attempting to
create another layering policy with a different name is a 409 error.
The existing layering policy can be updated by passing in a document
with the same `metadata.name` and `schema` as the existing one.

Closes-Bug: https://github.com/att-comdev/deckhand/issues/12
Change-Id: I7cad2d600c931c8701c3faaf2967be782984528b
2017-10-26 17:38:24 -04:00
Felipe Monteiro 514338c3bf Add expected errors decorator for more resiliency
To increase resiliency, add Deckhand error handling hooks to
format unknown errors into something more useful for debugging.
Also override exception formatting to be consistent with UCP
error formatting standard.

Most of this logic is borrowed from Shipyard for consistency.

Also includes basic unit tests to validate error formatting.

Change-Id: If7f8c3bf6b6ada7697611a0bef7bf8f635fc0b7f
2017-10-25 18:49:49 +01:00
Felipe Monteiro 8aec0390f8 DECKHAND-80: Validations API Implementation
The Validations API has been introduced to Deckhand, allowing users
to register new validation results in Deckhand, as well as query
the API for validation results for a revision. The validation results
include a list of errors that occurred during document validation.

All functional tests related to the API are now passing.

The following endpoints have been implemented:

   * /api/v1.0/revisions/{revision_id}/validations
   * /api/v1.0/revisions/{revision_id}/validations/{validation_name}
   * /api/v1.0/revisions/{revision_id}/validations/{validation_name}/entries
   * /api/v1.0/revisions/{revision_id}/validations/{validation_name}/entries/{entry_id}

Some back-end refactoring was needed to implement this API. In
particular:

  - Added a new Validation sqlalchemy DB model
  - Introduced DataSchema handling to the engine.document_validation
    module so that registered schema validations can be used
  - Changed the way the result of the 'deckhand-schema-validation' internal
    validation is generated: it is now the amalgamation of all the
    internal and registered schema validations executed
  - Introduced rawquery generation so that raw SQL queries can be used to
    get results from DB

Fixed following bug:

  - UniqueConstraint is now used to correctly generate unique constraints
    for sqlalchemy models that are supposed to be combinations of columns

Change-Id: I53c79a6544f44ef8beab2600ddc8a3ea91ada903
2017-10-21 23:09:16 -04:00
Felipe Monteiro d2d2312af9 DECKHAND-66: Document substitution implementation
This PS implements documentation substitution and
the rendered-documents endpoint. Each time the
rendered-documents is queried, the documents for
the reqeust revision_id dynamically undergo
secret substitution.

All functional tests related to secret substitution
have been unskipped.

Deckhand currently does not real testing for
verifying that secret substitution works
for encrypted documents. This will only happen
when integration testing is added to Deckhand to
test its interaction with Keystone and Barbican.

Included in this PS:
  - basic implementation for secret substitution
  - introduction of jsonpath_ng for searching for and
    updating jsonpaths in documents
  - rendered-documents endpoint
  - unit tests
  - all relevant functional tests unskipped
  - additional bucket controller tests include RBAC
    tests and framework testing RBAC via unit tests

Change-Id: I86f269a5b616b518e5f742a4005891412226fe2a
2017-10-13 15:16:27 -04:00
Felipe Monteiro 582dee6fb9 DECKHAND-61: oslo.policy integration
This PS implements oslo.policy integration in Deckhand.
The policy.py file implements 2 types of functions for
performing policy enforcement in Deckhand: authorize,
which is a decorator that is used directly around
falcon on_HTTP_VERB methods that raises a 403 immediately
if policy enforcement fails; and conditional_authorize,
to be used inside controller code conditionally.

For example, since Deckhand has two types of documents
with respect to security -- encrypted and cleartext
documents -- policy enforcement is conditioned on the
type of the documents' metadata.storagePolicy.

Included in this PS:
  - policy framework implementation
  - policy in code and policy documentation for all
    Deckhand policies
  - modification of functional test script to override
    default admin-only policies with custom policy file
    dynamically created using lax permissions
  - bug fix for filtering out deleted documents (and
    its predecessors in previous revisions) for
    PUT /revisions/{revision_id}/documents
  - policy documentation
  - basic unit tests for policy enforcement framework
  - allow functional tests to be filtered via regex

Due to the size of this PS, functional tests related to
policy enforcement will be done in a follow up.

Change-Id: If418129f9b401091e098c0bd6c7336b8a5cd2359
2017-10-07 18:43:28 +01:00
Felipe Monteiro 81b3e42013 Revision rollback API.
This commit implements the revision rollback API, allowing
users to rollback to a previous revision, whereby a new
revision is created.

An exception (400 Bad Request) is raised if the revision being
rolled back to is identical to the latest revision or if no
changes exist between the latest revision and the one being rolled
back to.

Included in this commit:

  - API endpoint for revision rollback.
  - Back-end logic for rolling back to a previous revision.
    The associated documents are also re-recreated. The
    revision_id assigned to each document depends on whether
    it has changed between the latest revision and the one
    being rolled back to: if changed, the new revision_id
    is assigned, else the original one, to maintain the
    correct revision history.
  - Associated unit tests.
  - Unskip all associated functional tests.

Change-Id: I5c120a92e106544f7f8a4266fc386fb60622d6b3
2017-09-25 22:20:47 +01:00
Felipe Monteiro e32a5a9319 Bucket deletion implementation
This commit implements logic to realize bucket deletion. This
commit also adds logic for raising an exception when trying
to create the same (document.schema, document.metadata.name)
in a different bucket than the one it was originally created in.

Included in this commit:

  - Implementation of document deletion logic.
  - Documents are always saved, even if they have been deleted
    or remain unchanged between revisions. This makes it easier
    to compute the diff between revisions.
  - Associated unit tests.
  - Unskip all remaining functional tests for
    'document-crud-success-single-bucket.yaml`
  - Raise a 409 exception when trying to create the same
    (document.schema, document.metadata.name) in a different
    bucket.
  - Unskip functional tests for
    'document-crud-error-bucket-conflict.yaml'

Change-Id: I6693bbb918cb672de315a66bb087de547df302d1
2017-09-18 23:03:38 +01:00
Felipe Monteiro c9cdd7514c [feat] DECKHAND-38: Secrets DB model and secrets manager.
This commit adds a DocumentSecret model to the DB for
storing secrets directly in Deckhand as well as references
to secrets stored in Barbican if the encryption type
for the secret is encrypted.

This commit also adds a new class called SecretsManager
for managing the lifecycle of secrets from a higher level.

This commit also adds Postgres compliance. So now all
the DB models should work with Postgres.

Also includes unit tests.

Change-Id: Id7c4be8de2e70735f42b1f6710139d553ab4bea2
2017-09-11 12:39:38 -04:00
Felipe Monteiro 7b0a69b39a [feat] DECKHAND-36 Revision tagging API
This commit adds an additional attribute called `tags` to each
Revision DB model. This allows Revisions to be tagged with whatever
arbitrary tag/tag data a service chooses to identify a revision by.

This commit:
  - creates a new DB model called `RevisionTag`
  - adds the following endpoints:
     * POST /api/v1.0/revisions/{revision_id}/tags/{tag} (create a tag)
     * GET /api/v1.0/revisions/tags/{tag} (show tag details)
     * GET /api/v1.0/revisions/{revision_id}/tags (list revision tags)
     * DELETE /api/v1.0/revisions/{revision_id}/tags/{tag} (delete a tag)
     * DELETE /api/v1.0/revisions/{revision_id}/tags (delete all tags)
  - adds appropriate unit test coverage for the changes
  - adds functional testing for each API endpoint

Change-Id: I49a7155ef5aa274c3a85ff6f8b85951f155a4b92
2017-08-29 15:41:20 +01:00
Felipe Monteiro c19309f347 Initial implementation of buckets
This commit adds endpoints for:

  * DELETE /revisions
  * PUT /bucket/{{bucket_name}}/revisions

Included in this commit:

  * Initial DB code for buckets
  * Initial API code for API buckets
  * Refactored unit tests to work with buckets
  * Passing *some* functional tests for:
    - revision-crud-success-single-bucket (*all*)
    - document-crud-success-single-bucket (*some*)
  * Corrected document view for list and corrected
    loads in MultidocJsonpaths for test_gabbi to not
    fix up the response body

Change-Id: Idf941591d24804b77441ab84259f8b7063c88a33
2017-08-24 20:58:26 +01:00
Felipe Monteiro 1bc0c9818e [feat] DECKHAND-13: Document layering (merge) logic
This commit constitutes 1 of 2 monolithic ports from Github.

This commit implements the foundation for document layering
or merging. Included in this commit:

  - Algorithm for layering documents with the same schema
  - Dozens of positive test cases
  - About a dozen negative test cases
  - Factory for dynamically creating testing documents for
    layering

Change-Id: I580bb69a341910b21be8610a416c691c54f7b946
2017-08-15 21:50:09 +01:00
Felipe Monteiro e1446bb9e1 [feat] DECKHAND-28: Document pre-validation logic and API integration
This commit constitutes 1 of 2 monolithic ports from Github.
The following major changes have been made:

  - Created schemas for validating different types of documents
    (control and document schemas), including:
    * certificate key
    * certificate
    * data schema
    * document
    * layering policy
    * passphrase
    * validation policy
  - Implemented pre-validation logic which validates that each
    type of document conforms to the correct schema specifications
  - Implemented views for APIs -- this allows views to change the
    DB data to conform with API specifications
  - Implemented relevant unit tests
  - Implement functional testing foundation

Change-Id: I83582cc26ffef91fbe95d2f5f437f82d6fef6aa9
2017-08-08 18:52:44 +01:00
Felipe Monteiro 8e43f91751 Finish retrieving documents by revision_id, including with filters. 2017-07-30 04:24:33 +01:00
Felipe Monteiro a0df0c459d Skip validation for abstract documents & add unit tests. 2017-07-26 14:26:05 +01:00
Felipe Monteiro 55e1360cbf Merge branch 'master' into documents-api 2017-07-21 12:11:40 -04:00
Felipe Monteiro 8cf795b7a1 Cleaned up some logic, added exception handling to document creation. 2017-07-20 15:21:11 +01:00
Felipe Monteiro 695ef09f72 Initial engine framework
This commit adds the initial engine framework for Deckhand. Included
is the logic for parsing YAML files as well as validating them and
doing forward substitution as specified by the YAML file.

This commit also includes unit tests for the framework changes.
2017-07-17 20:46:49 +01:00
Felipe Monteiro a7d1f0e2f4 Initial engine framework
This commit adds the initial engine framework for Deckhand. Included
is the logic for parsing YAML files as well as validating them and
doing forward substitution as specified by the YAML file.

This commit also includes unit tests for the framework changes.
2017-07-10 19:47:24 +01:00
Felipe Monteiro 00bb92561d DECKHAND-2: Design core Deckhand API framework
This commit implements the core Deckhand API framework.
It does not implement any real API routes. The core
framework is modeled after Drydock's [0].

This commit specifically:

  - implements the core API framework which uses falcon
  - implements errors.py for preliminary errors
  - implements base resource class from which other API
    resources will inherit to build out the API itself
  - implements base API router
  - implements entry-point for kicking off deckhand
  - updates base README.rst with instructions on
  - running and installing -- similar to Drydock's
  - implements dummy API resource for secrets, to
    be fleshed out further in a follow-up commit

[0] https://github.com/att-comdev/drydock
2017-06-27 19:26:51 +01:00