update dockerfile for python deckhand install
add deckhand version to chart 1.0
add chart version 0.2.0
update all packages to latest in requirements.txt
update zuul jobs for focal and python 3.8
remove zuul job functional-uwsgi-py38 in favor of functional-docker-py38
update tox config
typecast to string in re.sub() function
add stestr to test-requirements.txt
add SQLAlchemy jsonpickle sphinx-rtd-theme stestr to requirements.txt
deprecated function: BarbicanException -> BarbicanClientException
fix mock import using unittest
fix import collections to collections.abc
fix for collections modules for older than python 3.10 versions.
deprecated function: json -> to_json
deprecated function: werkzeug.contrib.profiler ->
werkzeug.middleware.profiler
deprecated function: falcon.AIP -> falcon.App
deprecation warning: switch from resp.body to resp.text
rename fixtures to dh_fixtures because there is an imported module
fixtures
switch from stream.read to bounded_stream.read
deprecated function: falcon process_response needed additional parameter
deprecated function: falcon default_exception_handler changed parameter
order
move from MagicMock object to falcon test generated object to fix
incompatability with upgraded Falcon module.
Adjust gabbi tests to fix incompatability with upgraded DeepDiff module
update Makefile to execute ubuntu_focal
update HTK (helmtoolkit)
unpin barbican to pass integration tests
Use helm 3 in chart build.
`helm serve` is removed in helm 3 so this moves
to using local `file://` dependencies [0] instead.
Change-Id: I180416f480edea1b8968d80c993b3e1fcc95c08d
Adds a unit test to validate following scenario:
1) create revision 1 with document
2) create revision 2 with no documents
3) rollback to revision 1 (creating revision 3)
Validate that diffing works for rolled-back revision.
All cases above use same bucket.
Also refactors some test logic for neatness.
Change-Id: I71bf7d34e8aae3ad5abb3c53b05cb96a7038ddc2
This implements a rendered documents cache which is keyed by
revision IDs. This means that repeatedly trying to re-render
documents for the same revision ID will be much, much faster as
the results will be cached.
Change-Id: Ie92f55a9234d038683ba1fcad76710d968ed67ab
This PS adds noauth middleware to bypass keystone authentication
which will occur when Deckhand's server is executed in development
mode. Development mode is enabled by setting development_mode as True
in etc/deckhand/deckhand.conf.sample.
The logic is similar to Drydock's here: [0].
[0] 1c78477e95/drydock_provisioner/util.py (L43)
Co-Authored-By: Luna Das <luna.das@imaginea.com>
Co-Authored-By: Felipe Monteiro <felipe.monteiro@att.com>
Change-Id: I677d3d92768e0aa1a550772700403e0f028b0c59
Updates Deckhand to use alembic to manage database upgrades.
Moves from creating tables at startup of Deckhand to the
db-sync job.
Change-Id: I6f4cb237fadc46fbee81d1c33096f48a720f589f
It's possible while running tests via testtools (python -m testtools.run)
that the file being imported in won't have registered all CONF opts
leading to oslo.conf errors getting thrown. This fixes that.
Change-Id: Ie21c89943bf4494ad9e0fd05e25f78139ad18377
This is to fix the condition in secrets_manager used to determine
whether the substitution is secret. It currently checks whether the
potential secret reference contains the substring
of 'key-manager/v1/secrets' but the environment-agnostic way
of doing this is to check whether the secret reference contains
the barbican endpoint registered under CONF.barbican.api_endpoint.
Change-Id: I633021571255c8393e19ec60a614ede981a86d9f
Recently JSONB replaced a back-end agnostic data type
for the "data" column in the Document model. This
made it necessary to drop support for running Deckhand
unit tests with any other database store.
However, this arragenement is undesirable as a user
shouldn't need to have postgresql installed just to
kick off unit tests.
So, this PS re-adds support for running unit tests
via an in-memory sqlite database.
To run unit tests with sqlite:
tox -e py35
Unit tests still run against postgresql via:
tox -e py35-postgresql
Both jobs are executed in CICD already.
This PS also updates the remaining DB columns to use JSONB if
postgresql is enabled; else fallback columns are used for testing
with sqlite. This is a necessary change to make the column data
types consistent.
Change-Id: I951f2f04fd013d635bb7653a238ff1eb3725b5e1
This PS rewrites the document_validation module in
Deckhand to achieve the following goals:
* better validation resiliency
* add support for different document schema versions
* better support for DataSchema validation
* separation of concerns by splitting up validations
into separate classes
* support for validating documents that rely on
a DataSchema passed in via the same payload
* support for generating multiple validation errors
rather than returning after the first one found
* increase testing validations for unit/functional
tests
Better validation resiliency is achieved through more
robust exception handling. For example, it is possible
for a ``DataSchema`` to be 100% valid from the POV of
built-in schema validation, but if the "data" section
itself is utterly invalid, then an exception will be
raised -- such an exception is treated as a critical
failure.
Better generation of error messages is achieved by
creation more validation error message results.
DataSchema validation was previously wonky. A DataSchema
had to first be created in 1 revision before it could be
referenced by a batch of documents in sequential revisions.
Now, a DataSchema can be created in the same (or previous)
revision as documents that rely on it and used to validate
said documents.
Finally, the module was heavily rewritten so that more
nuanced validations can be built by inheriting from
``BaseValidator`` so as to allow for easier code
readability and maintainability.
Change-Id: Ie75742b984b7ad392cb41decc203d42842050c80
This reverts https://review.gerrithub.io/#/c/393980/ which was
a temporary workaround to unblock the Deckhand gate. pifpaf should
be used to run unit tests as having to install Docker just to kick
off unit tests is excessive.
However, the unit-tests.sh script is maintained in tools/ directory
as a fallback.
Change-Id: I24a10d4b3ea00006004f27d0086719fb0bf86dd9
This PS unblocks the gate by replacing pifpaf to run postgresql
for unit tests with docker, as a workaround. This is because
"pifpaf run postgresql" is failing with pifpaf not being able
to find the command "pifpaf run". Steps to reproduce:
python3 -m virtualenv -p python3 /tmp/venv
source /tmp/venv/bin/activate
pip install -U pip wheel devpi-client setuptools
pip install pifpaf
$pifpaf run postgresql
>> pifpaf: 'run' is not a pifpaf command. See 'pifpaf --help'.
>> Did you mean one of these?
help
The unit test script for spinning up the docker postgresql container
and then running unit tests is very similar to the pre-existing
script for running functional tests located in tools/ directory.
Change-Id: Ib0f414ff58007037ac12161876dcd7a10e91f48c
This PS removes some additional validations around not being
able to rollback to a target revision:
- if the target revision == current revision
- if the target revision is effectively equivalent to the current
revision (in terms of constituent documents)
Change-Id: I92f8f9557f96b6a27f0dcef4f3138d542e5aa915
This PS integrates Deckhand with keystone auth so
that Deckhand can check whether a keystone token is
authenticated (by way of keystonemiddleware)
before proceeding with any requests.
The architecture for this PS is borrowed from [0]
which successfully integrates keystone authentication
with the falcon web application framework. However,
additional Deckhand-specific changes were made for
tests to pass.
The following changes have been made:
- add paste deploy configuration file which adds
keystonemiddleware integration to Deckhand; this
makes it trivial for keystonemiddleware to determine
whether a token in the X-Auth-Token header is authenticated
- use paste.deploy to create a web app
- update unit tests for testing controllers
- update functional test script to ignore keystone authentication
because functional tests don't currently support keystone
integration
[0] https://github.com/stannum-l/nautilus
Change-Id: I6eeeb4a4d9ab1f1cc8fb338e5cc21136ab4d5684
This PS revamps document hashing. Instead of relying on Python's
built-in hash function to hash the contents of a document (i.e.
metadata and data values), sha256 from hashlib is used instead,
mostly for security purposes.
Further, new parameters have been added to the document DB model:
data_hash and metadata_hash, and the old value hash has been
dropped. The data type for storing the hashes has been changed
to String from BigInt.
Finally, testing documentation was added.
Change-Id: I428ddcbce1007ea990ca0df1aa630072a050c722
This commit adds a DocumentSecret model to the DB for
storing secrets directly in Deckhand as well as references
to secrets stored in Barbican if the encryption type
for the secret is encrypted.
This commit also adds a new class called SecretsManager
for managing the lifecycle of secrets from a higher level.
This commit also adds Postgres compliance. So now all
the DB models should work with Postgres.
Also includes unit tests.
Change-Id: Id7c4be8de2e70735f42b1f6710139d553ab4bea2
This commit adds an additional attribute called `tags` to each
Revision DB model. This allows Revisions to be tagged with whatever
arbitrary tag/tag data a service chooses to identify a revision by.
This commit:
- creates a new DB model called `RevisionTag`
- adds the following endpoints:
* POST /api/v1.0/revisions/{revision_id}/tags/{tag} (create a tag)
* GET /api/v1.0/revisions/tags/{tag} (show tag details)
* GET /api/v1.0/revisions/{revision_id}/tags (list revision tags)
* DELETE /api/v1.0/revisions/{revision_id}/tags/{tag} (delete a tag)
* DELETE /api/v1.0/revisions/{revision_id}/tags (delete all tags)
- adds appropriate unit test coverage for the changes
- adds functional testing for each API endpoint
Change-Id: I49a7155ef5aa274c3a85ff6f8b85951f155a4b92
This commit fixes flake8 errors and fixes a minor bug related to
a schema version being v1 rather than v1.0.
OpenStack hacking rules are used to pin down flake8 to sane
standards using [0].
[0] 06e676c461/test-requirements.txt (L5)
Change-Id: Ib236df6f5ec9505c0e635f0faa9877d3397a2e55
Wahlstedt, Walter (ww229g)