This PS bumps up Airflow version to the latest
2.8.2 and also bumps up openstack dependences to
Antelope 2023.1
Change-Id: If7191c8291e31488a8a5f26107981616e14ea12c
This PS updates python modules and code to match Airflow 2.6.2:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was perfrmed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- postgresql image updated to 14.8
Change-Id: I65a1b86473ee3e988aae353b59fb5473d75851f9
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.
Also the latest osh-infra commit was used with kubernetes 1.27.1 for
integration tests.
Change-Id: Ia2c951f27b96774b553e7c0c7c6809172312f753
This PS delivers the following updates:
- fixed sample config and policy files generation in tox
- rolled back chart version incremention back to 0.2.0
Change-Id: I509030319a724b18bb21f45f7ede7c07ab18e894
- adjusted .gitignore to keep fresh egg-info and omit build artifacts
- fresh egg-info data is needed for promenade that depends on Deckhand
- restored deckhand-functional-uwsgi-py38 gate
- restored deckhand-integration-uwsgi-py38 gate
- made deckhand-airskiff-deployment gate voting ( treasuremap project
has been updated)
- removed bionic gates
- updated focal dockerfile
- added more binary deps into bindep.txt
- updated deckhand chart values to latest images - focal and wallaby
- fixed python code to compy with CVE's found by fresh version of bandit
- implemented pip freeze approach
- added tox -e freeze profile to manage it
- requirements-frozen.txt is now main file with requirements
- requirements-direct.txt is the file to control deps
- updated setup.cfg to adjust to newer version of setuptools
- fixed airskiff-deploy gate
- fixed docker-image-build playbook to restore Quay repo image publish
- updated other playbooks to include roles from zuul/base-jobs in order
to setup build hosts properly
- removed workaround with hardcoded dns resolver ip 10.96.0.10 as it
became obsolette due to recent fix in openstack-helm-infra
- adjusted tools/whitespace-linter.sh script
- tox.ini has been brought to compliance with tox4 requirements
- replaced str() calls with six.text_type() according to D325 Deckhand specific
commandment from Hacking.rst
- locked python-barbicanclient version with 5.2.0 because of breaking
changes in the upper versions
Change-Id: I1cd3c97e83569c4db7e958b3400bdd4b7ea5e668
update dockerfile for python deckhand install
add deckhand version to chart 1.0
add chart version 0.2.0
update all packages to latest in requirements.txt
update zuul jobs for focal and python 3.8
remove zuul job functional-uwsgi-py38 in favor of functional-docker-py38
update tox config
typecast to string in re.sub() function
add stestr to test-requirements.txt
add SQLAlchemy jsonpickle sphinx-rtd-theme stestr to requirements.txt
deprecated function: BarbicanException -> BarbicanClientException
fix mock import using unittest
fix import collections to collections.abc
fix for collections modules for older than python 3.10 versions.
deprecated function: json -> to_json
deprecated function: werkzeug.contrib.profiler ->
werkzeug.middleware.profiler
deprecated function: falcon.AIP -> falcon.App
deprecation warning: switch from resp.body to resp.text
rename fixtures to dh_fixtures because there is an imported module
fixtures
switch from stream.read to bounded_stream.read
deprecated function: falcon process_response needed additional parameter
deprecated function: falcon default_exception_handler changed parameter
order
move from MagicMock object to falcon test generated object to fix
incompatability with upgraded Falcon module.
Adjust gabbi tests to fix incompatability with upgraded DeepDiff module
update Makefile to execute ubuntu_focal
update HTK (helmtoolkit)
unpin barbican to pass integration tests
Use helm 3 in chart build.
`helm serve` is removed in helm 3 so this moves
to using local `file://` dependencies [0] instead.
Change-Id: I180416f480edea1b8968d80c993b3e1fcc95c08d
Update tox configuration and Zuul config to use Python 3.6, and make the
xenial and opensuse gates non-voting.
Change-Id: Iab98ad6cf269d57acf0598ae431b2f661673b234
This is a mechanically generated patch to ensure unit testing is in place
for all of the Tested Runtimes for Train.
See the Train python3-updates goal document for details:
https://governance.openstack.org/tc/goals/train/python3-updates.html
Change-Id: I13689a385bf237681bc96f3cc62fc17864f12659
This patch set uses concurrent.futures.ThreadPoolExecutor
[0] to retrieve multiple Barbican secrets concurrently.
This is because currently it is only possible to retrieve
1 secret payload from Barbican at a time -- for revisions
with several dozen secrets it is therefore too costly
to serially perform these API requests.
A new configuration option is added to the [barbican]
group called `max_workers` which specifies the number
of threads to use. The default value is 10. Note that:
"If max_workers is None or not given, it will default
to the number of processors on the machine, multiplied by 5"
[0] so the default is 10 for 2 * 5 which is overly
conservative if anything.
If any error occurs during any of the requests a 500
is raised with appropriate details.
[0] https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor
Change-Id: I76a5bb6c345054e160c14bdf9fb7087e3a746a5e
This patchset adds developer overview documentation for providing
a high-level introduction to Deckhand, including its architecture,
modules, test utilities, Helm utilities, and other errata. This
work is based off this Drydock patchset:
https://review.openstack.org/#/c/571298/
Change-Id: Ic3382d4e04edf02a65184651d272fe9cd1db56a4
Moving the whitespace linting job as part of PEP8 job to reduce the need to
create a VM just to find whitespaces in none *.py files. Python files'
whitespace violation is covered as part of PEP8 job.
Change-Id: I3bc6047c9ab65e8f55f91a949a6c05ef4e38ea16
Signed-off-by: Tin Lam <tin@irrational.io>
We want to default to running all tox environments under python 3, so
set the basepython value in each environment.
We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.
We do not want to set the override once in testenv, because that
breaks the more specific versions used in default environments like
py35 and py36.
Change-Id: I9a6e4950e2915fb551c35eba8c78a6b4bb4d932b
This patchset removes unncessary requirements in doc-specific targets in
the tox.ini file. It should only need a doc specific requirement file.
Change-Id: Ib61376b19fcd1c9d2af284e977d66e5abb6c31a0
Signed-off-by: Tin Lam <tin@irrational.io>
This patchset updates docs to doc to align with OpenStack
standard. Follow-up patchset will be needed to publish
documentation to OpenStack [0].
[0] https://docs.openstack.org/doc-contrib-guide/project-guides.html
Change-Id: Ia191ac1cc4536af1232aedd4bb491f3829651730
This patch set removes few pep8/flake8 ignored rules and implemented
the fix in the code to address those rules.
Change-Id: I2e613acd760818a6e18288d284f6224c38c4353a
Signed-off-by: Tin Lam <tin@irrational.io>
This adds a uwsgi functional test check to .zuul.yaml so that
deploying Deckhand via uwsgi (in a more standalone fashion,
sans containerization) works as intended.
Change-Id: I931ab4d11719daca7665d3a25b00e353c707237e
This patch set cleans up tox.ini and puts bandit as part of the pep8
target similar to keystone. This consolidates all the static code
analysis and linting to a single gate job or check.
Change-Id: I29b60fb9a2fbf14f321d9c2605216118f73f8bf7
Signed-off-by: Tin Lam <tin@irrational.io>
This patchset converts much of the previous logic in
functional-tests.sh into Ansible playbooks to be executed
by Zuul. This mainly includes all the Docker-related
deployment logic.
The functional-tests.sh script has been slimmed down to
just work with uwsgi so that a standalone functional
test deployment can be performed relatively easily,
mainly by developers.
Finally, py27 support for the gate has been dropped
as the Dockerfile in this project currently assumes
python3 for installing requirements and so forth,
leading to requirements issues blocking the gate.
Change-Id: I903a2845390061641d292fb0c016ba6a53723fc9
This patchset adds functional tests to .zuul.yaml. Additionally
it adds a functional-py35 job as well which will also be kicked
off via Zuul.
Change-Id: Ic2d1db4d3cd65c4d93c3a6f04e6efeeba9755f07
This PS adds noauth middleware to bypass keystone authentication
which will occur when Deckhand's server is executed in development
mode. Development mode is enabled by setting development_mode as True
in etc/deckhand/deckhand.conf.sample.
The logic is similar to Drydock's here: [0].
[0] 1c78477e95/drydock_provisioner/util.py (L43)
Co-Authored-By: Luna Das <luna.das@imaginea.com>
Co-Authored-By: Felipe Monteiro <felipe.monteiro@att.com>
Change-Id: I677d3d92768e0aa1a550772700403e0f028b0c59
This is to update releasenotes/docs tox jobs to remove need
to defined build_sphinx in setup.cfg and to ensure that they
both clean up prior to running via appropraite rm -rf commands
and to ensure all the requirements are being installed.
Change-Id: Iadd375dbb596151cb140fae03b82a728a64364a0
This patch set adds integration tests to Deckhand
where "integration" means the interaction between
Deckhand, Barbican and Keystone. OSH is used to
deploy Keystone and Barbican and Docker to deploy
PostgreSQL and Deckhand.
Unlike functional testing in Deckhand, all
integration tests use the default in-code policy
defaults and an admin token supplied by keystone
to validate authN and authZ.
The test scenarios consist of Deckhand secret
lifecycle management as well as document rendering
with secrets retrieved from Barbican.
Change-Id: Ib5ae1b345b2a4bd579671ec4ae9a232c2e3887dc
Updates Deckhand to use alembic to manage database upgrades.
Moves from creating tables at startup of Deckhand to the
db-sync job.
Change-Id: I6f4cb237fadc46fbee81d1c33096f48a720f589f
This is a trivial PS that fixes the tox -e cover job in
tox.ini which was recently broken with [0].
[0] https://review.gerrithub.io/#/c/405318/
Change-Id: Id50a6348e6f306c3d8d68fdd79eb331880e7498b
This PS fixes tox -v skipping over SQLite unit test jobs. tox -v
is used in CICD to run all jobs in tox envlist but currently
py{35,27}-{postgresql,} translates to:
py35-
py27-
py35-postgresql
py27-postgresql
Where the first two should instead be:
py35
py27
This PS also adds --regex flags to the unit test jobs so regular
expressions work with them.
Change-Id: Id468259a1b2e020494bdd58103d8750b4fac6000
Recently JSONB replaced a back-end agnostic data type
for the "data" column in the Document model. This
made it necessary to drop support for running Deckhand
unit tests with any other database store.
However, this arragenement is undesirable as a user
shouldn't need to have postgresql installed just to
kick off unit tests.
So, this PS re-adds support for running unit tests
via an in-memory sqlite database.
To run unit tests with sqlite:
tox -e py35
Unit tests still run against postgresql via:
tox -e py35-postgresql
Both jobs are executed in CICD already.
This PS also updates the remaining DB columns to use JSONB if
postgresql is enabled; else fallback columns are used for testing
with sqlite. This is a necessary change to make the column data
types consistent.
Change-Id: I951f2f04fd013d635bb7653a238ff1eb3725b5e1
For whatever reason, the following command:
pifpaf run postgresql -- <test command>
is not returning a non-zero error code on test failure.
(An example print out is included below.)
This PS updates pretty_tox.sh to forcibly raise a non-zero error
code in the event of test failure. It also renames the script
to run_pifpaf.sh to be more intuitive.
Example:
======
Totals
======
Ran: 7 tests in 5.7673 sec.
- Passed: 6
- Skipped: 0
- Expected Fail: 0
- Unexpected Success: 0
- Failed: 1
Sum of execute time for each test: 2.6962 sec.
==============
Worker Balance
==============
- Worker 0 (7 tests) => 0:00:02.698323
+ exit 0
py35: commands succeeded
congratulations :)
Change-Id: I7b1fa9d42295d06752997f251a0ec14082b44d03
This reverts https://review.gerrithub.io/#/c/393980/ which was
a temporary workaround to unblock the Deckhand gate. pifpaf should
be used to run unit tests as having to install Docker just to kick
off unit tests is excessive.
However, the unit-tests.sh script is maintained in tools/ directory
as a fallback.
Change-Id: I24a10d4b3ea00006004f27d0086719fb0bf86dd9
Unusual documents are documents with different data
types for the data field. The data types include:
object, array, string and integer.
This PS makes necessary ORM model and schema
changes needed to support the different data types.
The ORM data type for the data column has been changed
to JSONB for PostgreSQL. Thus, DH now only supports
PostgreSQL. As a result, the tox jobs have been updated
to only use postgre.
Change-Id: I53694d56bef71adacb5eb79162678be73acb4ad8
This PS unblocks the gate by replacing pifpaf to run postgresql
for unit tests with docker, as a workaround. This is because
"pifpaf run postgresql" is failing with pifpaf not being able
to find the command "pifpaf run". Steps to reproduce:
python3 -m virtualenv -p python3 /tmp/venv
source /tmp/venv/bin/activate
pip install -U pip wheel devpi-client setuptools
pip install pifpaf
$pifpaf run postgresql
>> pifpaf: 'run' is not a pifpaf command. See 'pifpaf --help'.
>> Did you mean one of these?
help
The unit test script for spinning up the docker postgresql container
and then running unit tests is very similar to the pre-existing
script for running functional tests located in tools/ directory.
Change-Id: Ib0f414ff58007037ac12161876dcd7a10e91f48c
This PS implements oslo.policy integration in Deckhand.
The policy.py file implements 2 types of functions for
performing policy enforcement in Deckhand: authorize,
which is a decorator that is used directly around
falcon on_HTTP_VERB methods that raises a 403 immediately
if policy enforcement fails; and conditional_authorize,
to be used inside controller code conditionally.
For example, since Deckhand has two types of documents
with respect to security -- encrypted and cleartext
documents -- policy enforcement is conditioned on the
type of the documents' metadata.storagePolicy.
Included in this PS:
- policy framework implementation
- policy in code and policy documentation for all
Deckhand policies
- modification of functional test script to override
default admin-only policies with custom policy file
dynamically created using lax permissions
- bug fix for filtering out deleted documents (and
its predecessors in previous revisions) for
PUT /revisions/{revision_id}/documents
- policy documentation
- basic unit tests for policy enforcement framework
- allow functional tests to be filtered via regex
Due to the size of this PS, functional tests related to
policy enforcement will be done in a follow up.
Change-Id: If418129f9b401091e098c0bd6c7336b8a5cd2359
Unskip some pep8 rules that aren't unreasonably annoying:
E121 - continuation line under-indented for hanging indent
E122 - continuation line missing indentation or outdented
E123 - closing bracket does not match indentation of opening bracket’s line
E124 - closing bracket does not match visual indentation
E125 - continuation line with same indent as next logical line
E126 - continuation line over-indented for hanging indent
E251 - unexpected spaces around keyword / parameter equals
Change-Id: Idf2640fc2d10715a687c46c3e853122ce38109ee
This PS revamps document hashing. Instead of relying on Python's
built-in hash function to hash the contents of a document (i.e.
metadata and data values), sha256 from hashlib is used instead,
mostly for security purposes.
Further, new parameters have been added to the document DB model:
data_hash and metadata_hash, and the old value hash has been
dropped. The data type for storing the hashes has been changed
to String from BigInt.
Finally, testing documentation was added.
Change-Id: I428ddcbce1007ea990ca0df1aa630072a050c722