This PS updates python modules and code to match Airflow 2.6.2:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was perfrmed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- postgresql image updated to 14.8
Change-Id: I65a1b86473ee3e988aae353b59fb5473d75851f9
Occasionally when Deckhand is creating secrets in Barbican, Barbican
encounters an error in which a subsequent attempt at creating the
secret would succeed. This patch set adds logic to the Deckhand
Barbican driver to retry secret creates a configurable number of
times to work around this Barbican issue.
Change-Id: I52293195dd708255508949723d89117ce2e32b71
This PS adds configuration documentation that includes
a literalinclude of the config file as well as some
information on each of the cache config options as
these are important on performance.
Change-Id: I3b06012b8843b7bfbd46307f81397172a41d3675
This patchset implements caching lookup and reverse-lookup
functions to allow for much faster retrieval of encrypted data
from Barbican, which doesn't currently support batched
requests in its Secrets API.
This behavior is necessary since Deckhand has to potentially
retrieve and store up to dozens of secrets per request. Note
that data for both lookup functions are invalidated
together, as they are tied to the same cache.
This change implements caching around arguably the most
expensive operation in Deckhand: encryption. By
caching encryption itself, the performance of rendering
documents thereby increases in a meaningful way, without
having to implement much logic to pull it off.
A follow up patch set here: https://review.openstack.org/#/c/585842
focuses on caching rendered documents themselves.
Change-Id: I0d330690a3c5e899b763ddcaa00d356007aa23fb
This implements a rendered documents cache which is keyed by
revision IDs. This means that repeatedly trying to re-render
documents for the same revision ID will be much, much faster as
the results will be cached.
Change-Id: Ie92f55a9234d038683ba1fcad76710d968ed67ab
This patch set uses concurrent.futures.ThreadPoolExecutor
[0] to retrieve multiple Barbican secrets concurrently.
This is because currently it is only possible to retrieve
1 secret payload from Barbican at a time -- for revisions
with several dozen secrets it is therefore too costly
to serially perform these API requests.
A new configuration option is added to the [barbican]
group called `max_workers` which specifies the number
of threads to use. The default value is 10. Note that:
"If max_workers is None or not given, it will default
to the number of processors on the machine, multiplied by 5"
[0] so the default is 10 for 2 * 5 which is overly
conservative if anything.
If any error occurs during any of the requests a 500
is raised with appropriate details.
[0] https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor
Change-Id: I76a5bb6c345054e160c14bdf9fb7087e3a746a5e
This patchset uses beaker (used by Promenade and Drydock) to
achieve better caching around jsonpath-ng wrapper functions
(jsonpath_replace and jsonpath_parse).
Change-Id: Ifae24775b4741ade7673dc91776c35d2de5b9065
This patchset adds missing Keystone options to registration
of config, including timeout.
This fix is similar to a recent Shipyard patchset [0] which
aimed to solve the same issue but expands on it by following
Nova patterns here: [1].
This also updates the sample config file which now includes
the timeout option.
[0] https://review.openstack.org/#/c/572563/
[1] http://git.openstack.org/cgit/openstack/nova/tree/nova/conf/cinder.py
Change-Id: I4e52170e48dba7064c69f42f4cbbb7e54eae11a9
This patch set removes few pep8/flake8 ignored rules and implemented
the fix in the code to address those rules.
Change-Id: I2e613acd760818a6e18288d284f6224c38c4353a
Signed-off-by: Tin Lam <tin@irrational.io>
This PS adds noauth middleware to bypass keystone authentication
which will occur when Deckhand's server is executed in development
mode. Development mode is enabled by setting development_mode as True
in etc/deckhand/deckhand.conf.sample.
The logic is similar to Drydock's here: [0].
[0] 1c78477e95/drydock_provisioner/util.py (L43)
Co-Authored-By: Luna Das <luna.das@imaginea.com>
Co-Authored-By: Felipe Monteiro <felipe.monteiro@att.com>
Change-Id: I677d3d92768e0aa1a550772700403e0f028b0c59
This is to change passing the secret URI instead of the secret
UUID to barbican's get secret endpoint from which the secret
itself can be extracted.
While the API [0] expects a UUID the CLI instead expects a URI
and the latter extracts the UUID from the URI automatically [1].
API ref:
GET /v1/secrets/{uuid}
Headers:
Accept: application/json
X-Auth-Token: {token}
(or X-Project-Id: {project_id})
CLI ref:
$ barbican help secret get
usage: barbican secret get [-h] [-f {shell,table,value}] [-c COLUMN]
[--max-width <integer>] [--prefix PREFIX]
[--decrypt] [--payload]
[--payload_content_type PAYLOAD_CONTENT_TYPE]
URI
Retrieve a secret by providing its URI.
Finally, this adds logic for ensuring that all encrypted data is retrieved
and injected back into the raw documents with Barbican references, during
document rendering. Currently, this process is only performed for
documents with substitutions, but should also be carried out for encrypted
documents themselves.
[0] https://docs.openstack.org/barbican/latest/api/reference/secrets.html#get-v1-secrets-uuid
[1] https://docs.openstack.org/python-barbicanclient/latest/reference/index.html#barbicanclient.v1.secrets.SecretManager.get
Change-Id: I1717592b7acdedb66353c25fb5dcda2d5330196b
This is to fix secrets_manager.SecretsManager.get method which
is passing in the secret reference to Barbican directly for
GET /secrets/{uuid} [0] causing Barbican to raise a
ValueError exception when it attempts to validate that
{secret_uuid} is in fact a UUID.
The fix is to extract the secret_uuid from the secret_ref returned
by Barbican before querying the GET /secrets/{uuid} API.
[0] https://docs.openstack.org/barbican/latest/api/reference/secrets.html#get-v1-secrets-uuid
Change-Id: I4db317e3ba12b4268df5b84b79be8da1da5ac2ba
This is to fix the condition in secrets_manager used to determine
whether the substitution is secret. It currently checks whether the
potential secret reference contains the substring
of 'key-manager/v1/secrets' but the environment-agnostic way
of doing this is to check whether the secret reference contains
the barbican endpoint registered under CONF.barbican.api_endpoint.
Change-Id: I633021571255c8393e19ec60a614ede981a86d9f
- Seeing issues with a lot of Drydock
requests timing out and it seems to be a
downstream issue with pulling Deckhand
docs
- Add jsonpath cacheing as the jsonpath-ng
parser was consuming 54s of the total 56s
runtime of a rendered-documents GET call.
With caching, the call is taking closer to 2s.
- All add a .dockerignore file to make image
building a little faster
Change-Id: I6ef84ffd946dcf2713b4f7570b985156deb1d697
This PS integrates Deckhand with keystone auth so
that Deckhand can check whether a keystone token is
authenticated (by way of keystonemiddleware)
before proceeding with any requests.
The architecture for this PS is borrowed from [0]
which successfully integrates keystone authentication
with the falcon web application framework. However,
additional Deckhand-specific changes were made for
tests to pass.
The following changes have been made:
- add paste deploy configuration file which adds
keystonemiddleware integration to Deckhand; this
makes it trivial for keystonemiddleware to determine
whether a token in the X-Auth-Token header is authenticated
- use paste.deploy to create a web app
- update unit tests for testing controllers
- update functional test script to ignore keystone authentication
because functional tests don't currently support keystone
integration
[0] https://github.com/stannum-l/nautilus
Change-Id: I6eeeb4a4d9ab1f1cc8fb338e5cc21136ab4d5684
This PS implements oslo.policy integration in Deckhand.
The policy.py file implements 2 types of functions for
performing policy enforcement in Deckhand: authorize,
which is a decorator that is used directly around
falcon on_HTTP_VERB methods that raises a 403 immediately
if policy enforcement fails; and conditional_authorize,
to be used inside controller code conditionally.
For example, since Deckhand has two types of documents
with respect to security -- encrypted and cleartext
documents -- policy enforcement is conditioned on the
type of the documents' metadata.storagePolicy.
Included in this PS:
- policy framework implementation
- policy in code and policy documentation for all
Deckhand policies
- modification of functional test script to override
default admin-only policies with custom policy file
dynamically created using lax permissions
- bug fix for filtering out deleted documents (and
its predecessors in previous revisions) for
PUT /revisions/{revision_id}/documents
- policy documentation
- basic unit tests for policy enforcement framework
- allow functional tests to be filtered via regex
Due to the size of this PS, functional tests related to
policy enforcement will be done in a follow up.
Change-Id: If418129f9b401091e098c0bd6c7336b8a5cd2359
This commit adds oslo.config options for keystone auth and
updates Deckhand's request context to use oslo_context for
facilitating integration with keystone auth options.
Change-Id: Ifd170e1a192402a970f8538f0c06bf017fe77f88
This commit adds the following namespaces to deckhand's
config-generator.conf file used by oslo-config-generator
to generate deckhand's conf file automatically:
* oslo.db
* oslo.db.concurrency
* oslo.log
This will automatically populate the generate conf file with
needed oslo config options.
* DECKHAND-11: Add oslo.config integration to Deckhand
This commit adds oslo.config integration to Deckhand. It also
creates a lot of preliminary files/configuration settings
needed to run tox as well as lint and oslo-config-generator
jobs.
* Remove sample config file.
Sergiy Markin