This PS updates python modules and code to match Airflow 2.6.2:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was perfrmed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- postgresql image updated to 14.8
Change-Id: I65a1b86473ee3e988aae353b59fb5473d75851f9
This PS delivers the following updates:
- fixed sample config and policy files generation in tox
- rolled back chart version incremention back to 0.2.0
Change-Id: I509030319a724b18bb21f45f7ede7c07ab18e894
- If a document has a storage policy of encrypted
- Redacts (sha256) the data section.
- Redacts (sha256) the substition paths.
- Uses the same /documents endpoint, adds a new query parameter
?cleartext-secrets=true to show the non-redacted values.
Change-Id: I42808901b97c667a1148c00fbb7717a0847c9981
This patch set pins oslo.messaging to the last stable version
to fix the gate [0]. 9.0.0 is incompatible with current
requirements.
[0] https://pypi.org/project/oslo.messaging/#history
Change-Id: I6cf3e9f33ac3efcbf48675273899af38fef86e91
GET /revisions/{{revision_id}}/deepdiff/{{comparison_revision_id}}
- Added deepdiff api for generating diff between
two rendered documents.
- Deep diffing for data and metadata
- Refactor diff functions
- Client update
- Added unit testcases
- Added funtional testcases
- Doc update
Change-Id: Ib60fa60a3b33e9125a1595a999272ca595721b38
This patchset adds missing Keystone options to registration
of config, including timeout.
This fix is similar to a recent Shipyard patchset [0] which
aimed to solve the same issue but expands on it by following
Nova patterns here: [1].
This also updates the sample config file which now includes
the timeout option.
[0] https://review.openstack.org/#/c/572563/
[1] http://git.openstack.org/cgit/openstack/nova/tree/nova/conf/cinder.py
Change-Id: I4e52170e48dba7064c69f42f4cbbb7e54eae11a9
This PS adds noauth middleware to bypass keystone authentication
which will occur when Deckhand's server is executed in development
mode. Development mode is enabled by setting development_mode as True
in etc/deckhand/deckhand.conf.sample.
The logic is similar to Drydock's here: [0].
[0] 1c78477e95/drydock_provisioner/util.py (L43)
Co-Authored-By: Luna Das <luna.das@imaginea.com>
Co-Authored-By: Felipe Monteiro <felipe.monteiro@att.com>
Change-Id: I677d3d92768e0aa1a550772700403e0f028b0c59
This PS:
* adds a trap to clean up OSH which is deployed in the
course of integration tests. It appears as though node cleanup
in Jenkins is hanging so this is to try to ameliorate that
* creates a deckhand.conf.test to be used by functional and
integration tests instead of writing it out dynamically [0]
* updates logging.conf.sample to dump logs to stdout/stderr
by default as this is amenable to containers
* makes test_gabbi.py common between functional and integration
tests to avoid unnecessary code duplication
[0] review comments in https://review.gerrithub.io/#/c/att-comdev/deckhand/+/407638/
Change-Id: I762fb0bde5f75effcde56316d92bd57b30026995
According to guides like [0] RESTful resource name should
use pluralization for nouns, for the sake of consistency.
As such, this PS renames PUT bucket/{bucket_name}/documents
to PUT buckets/{bucket_name}/documents
Required renames have also been made to the following UCP
projects:
- Shipyard -> https://review.gerrithub.io/#/c/384121/
[0] https://restfulapi.net/resource-naming/
Change-Id: Icb01e29db0b8b12e66895135baa9be525ba7d72b
The Validations API has been introduced to Deckhand, allowing users
to register new validation results in Deckhand, as well as query
the API for validation results for a revision. The validation results
include a list of errors that occurred during document validation.
All functional tests related to the API are now passing.
The following endpoints have been implemented:
* /api/v1.0/revisions/{revision_id}/validations
* /api/v1.0/revisions/{revision_id}/validations/{validation_name}
* /api/v1.0/revisions/{revision_id}/validations/{validation_name}/entries
* /api/v1.0/revisions/{revision_id}/validations/{validation_name}/entries/{entry_id}
Some back-end refactoring was needed to implement this API. In
particular:
- Added a new Validation sqlalchemy DB model
- Introduced DataSchema handling to the engine.document_validation
module so that registered schema validations can be used
- Changed the way the result of the 'deckhand-schema-validation' internal
validation is generated: it is now the amalgamation of all the
internal and registered schema validations executed
- Introduced rawquery generation so that raw SQL queries can be used to
get results from DB
Fixed following bug:
- UniqueConstraint is now used to correctly generate unique constraints
for sqlalchemy models that are supposed to be combinations of columns
Change-Id: I53c79a6544f44ef8beab2600ddc8a3ea91ada903
This PS integrates Deckhand with keystone auth so
that Deckhand can check whether a keystone token is
authenticated (by way of keystonemiddleware)
before proceeding with any requests.
The architecture for this PS is borrowed from [0]
which successfully integrates keystone authentication
with the falcon web application framework. However,
additional Deckhand-specific changes were made for
tests to pass.
The following changes have been made:
- add paste deploy configuration file which adds
keystonemiddleware integration to Deckhand; this
makes it trivial for keystonemiddleware to determine
whether a token in the X-Auth-Token header is authenticated
- use paste.deploy to create a web app
- update unit tests for testing controllers
- update functional test script to ignore keystone authentication
because functional tests don't currently support keystone
integration
[0] https://github.com/stannum-l/nautilus
Change-Id: I6eeeb4a4d9ab1f1cc8fb338e5cc21136ab4d5684
This PS implements documentation substitution and
the rendered-documents endpoint. Each time the
rendered-documents is queried, the documents for
the reqeust revision_id dynamically undergo
secret substitution.
All functional tests related to secret substitution
have been unskipped.
Deckhand currently does not real testing for
verifying that secret substitution works
for encrypted documents. This will only happen
when integration testing is added to Deckhand to
test its interaction with Keystone and Barbican.
Included in this PS:
- basic implementation for secret substitution
- introduction of jsonpath_ng for searching for and
updating jsonpaths in documents
- rendered-documents endpoint
- unit tests
- all relevant functional tests unskipped
- additional bucket controller tests include RBAC
tests and framework testing RBAC via unit tests
Change-Id: I86f269a5b616b518e5f742a4005891412226fe2a
This PS implements oslo.policy integration in Deckhand.
The policy.py file implements 2 types of functions for
performing policy enforcement in Deckhand: authorize,
which is a decorator that is used directly around
falcon on_HTTP_VERB methods that raises a 403 immediately
if policy enforcement fails; and conditional_authorize,
to be used inside controller code conditionally.
For example, since Deckhand has two types of documents
with respect to security -- encrypted and cleartext
documents -- policy enforcement is conditioned on the
type of the documents' metadata.storagePolicy.
Included in this PS:
- policy framework implementation
- policy in code and policy documentation for all
Deckhand policies
- modification of functional test script to override
default admin-only policies with custom policy file
dynamically created using lax permissions
- bug fix for filtering out deleted documents (and
its predecessors in previous revisions) for
PUT /revisions/{revision_id}/documents
- policy documentation
- basic unit tests for policy enforcement framework
- allow functional tests to be filtered via regex
Due to the size of this PS, functional tests related to
policy enforcement will be done in a follow up.
Change-Id: If418129f9b401091e098c0bd6c7336b8a5cd2359
The following deployment logic should be included to get logging
to work correctly:
1) tox -egenconfig
- Store the output in /etc/deckhand/deckhand.conf for example
2) Copy logging.conf.sample in etc folder to /etc/deckhand/logging.conf
3) Set the following options in under [DEFAULT] in
/etc/deckhand/deckhand.conf:
- log_config_append = /etc/deckhand/logging.conf
- log_file = deckhand.log
- log_dir = <path/to/deckhand/dir>
- debug = true (optionally)
Change-Id: I8e8ebd041e801a5eef0f10b1bbc76ce95aecbf55
This commit adds oslo.config options for keystone auth and
updates Deckhand's request context to use oslo_context for
facilitating integration with keystone auth options.
Change-Id: Ifd170e1a192402a970f8538f0c06bf017fe77f88
This commit adds the following namespaces to deckhand's
config-generator.conf file used by oslo-config-generator
to generate deckhand's conf file automatically:
* oslo.db
* oslo.db.concurrency
* oslo.log
This will automatically populate the generate conf file with
needed oslo config options.
* DECKHAND-11: Add oslo.config integration to Deckhand
This commit adds oslo.config integration to Deckhand. It also
creates a lot of preliminary files/configuration settings
needed to run tox as well as lint and oslo-config-generator
jobs.
* Remove sample config file.
Sergiy Markin