This PS updates python modules and code to match Airflow 2.6.2:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was perfrmed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- postgresql image updated to 14.8
Change-Id: I65a1b86473ee3e988aae353b59fb5473d75851f9
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.
Also the latest osh-infra commit was used with kubernetes 1.27.1 for
integration tests.
Change-Id: Ia2c951f27b96774b553e7c0c7c6809172312f753
- adjusted .gitignore to keep fresh egg-info and omit build artifacts
- fresh egg-info data is needed for promenade that depends on Deckhand
- restored deckhand-functional-uwsgi-py38 gate
- restored deckhand-integration-uwsgi-py38 gate
- made deckhand-airskiff-deployment gate voting ( treasuremap project
has been updated)
- removed bionic gates
- updated focal dockerfile
- added more binary deps into bindep.txt
- updated deckhand chart values to latest images - focal and wallaby
- fixed python code to compy with CVE's found by fresh version of bandit
- implemented pip freeze approach
- added tox -e freeze profile to manage it
- requirements-frozen.txt is now main file with requirements
- requirements-direct.txt is the file to control deps
- updated setup.cfg to adjust to newer version of setuptools
- fixed airskiff-deploy gate
- fixed docker-image-build playbook to restore Quay repo image publish
- updated other playbooks to include roles from zuul/base-jobs in order
to setup build hosts properly
- removed workaround with hardcoded dns resolver ip 10.96.0.10 as it
became obsolette due to recent fix in openstack-helm-infra
- adjusted tools/whitespace-linter.sh script
- tox.ini has been brought to compliance with tox4 requirements
- replaced str() calls with six.text_type() according to D325 Deckhand specific
commandment from Hacking.rst
- locked python-barbicanclient version with 5.2.0 because of breaking
changes in the upper versions
Change-Id: I1cd3c97e83569c4db7e958b3400bdd4b7ea5e668
update dockerfile for python deckhand install
add deckhand version to chart 1.0
add chart version 0.2.0
update all packages to latest in requirements.txt
update zuul jobs for focal and python 3.8
remove zuul job functional-uwsgi-py38 in favor of functional-docker-py38
update tox config
typecast to string in re.sub() function
add stestr to test-requirements.txt
add SQLAlchemy jsonpickle sphinx-rtd-theme stestr to requirements.txt
deprecated function: BarbicanException -> BarbicanClientException
fix mock import using unittest
fix import collections to collections.abc
fix for collections modules for older than python 3.10 versions.
deprecated function: json -> to_json
deprecated function: werkzeug.contrib.profiler ->
werkzeug.middleware.profiler
deprecated function: falcon.AIP -> falcon.App
deprecation warning: switch from resp.body to resp.text
rename fixtures to dh_fixtures because there is an imported module
fixtures
switch from stream.read to bounded_stream.read
deprecated function: falcon process_response needed additional parameter
deprecated function: falcon default_exception_handler changed parameter
order
move from MagicMock object to falcon test generated object to fix
incompatability with upgraded Falcon module.
Adjust gabbi tests to fix incompatability with upgraded DeepDiff module
update Makefile to execute ubuntu_focal
update HTK (helmtoolkit)
unpin barbican to pass integration tests
Use helm 3 in chart build.
`helm serve` is removed in helm 3 so this moves
to using local `file://` dependencies [0] instead.
Change-Id: I180416f480edea1b8968d80c993b3e1fcc95c08d
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].
These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.
[0] https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020
Change-Id: Id75acea82ddf5d915a8b8805e076dac49cab800f
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.
https://pypi.org/project/pylibyaml/
Change-Id: Iebcc50b5db87518b3b7e0fac124c712afd06da2b
Use pip3 in event system has both pip2 and pip3 installed.
Use apt to install setuptools for Ansible's consumption.
Change-Id: I6929ecb0cce2ec8ac70e9261acb9f87dc7031153
Co-authored-by: Alexander Hughes <Alexander.Hughes@pm.me>
tox is failing because oslo.messaging 9.1.1 installs kombu 4.6.7, which
requires amqp >=2.5.2, but it is pinned to 2.5.1. Additionally, the
deckhand-integration-docker are failing because barbican is not being
deployed correctly.
This change updated the pinned version of amqp to 2.5.2, and uses the
barbican deployment script in openstack-helm.
Depends-On: I020632ad6a33d29a657164cf0e32baef6b670d5a
Change-Id: Idafe35d1496cd880e2d6a93c96afa724fd1c2e89
Fix problems with integration test zuul jobs which were made non-voting
by [0] and restore them to voting status. These jobs exercise some
secret-management functions that test integration between Deckhand and
Barbican that doesn't happen in the Airskiff jobs, so they still have
some test value.
- Some integration test scripts were pointing to old versions of
openstack-helm scripts that are no longer maintained. This PS
updates those scripts to point to current OSH scripts.
- Uplift openstack-helm-infra pin in helm_tk.sh to a recent
version
- Add build-images to uwsgi test jobs to ensure airship_deckhand_path
is set
[0]: https://review.opendev.org/660738
Change-Id: If83dead126307a98b44ad5d8f367a972c48a30d0
There are failures with the gates that need the
amqp version to be pinned back for now
reference b51d1d678e
Change-Id: Ifba9fb1f5a102d6d490a32be23ac23185089774d
This patch set pins down package requirements for the following
reasons:
* Inconsistencies between ranges and direct pins
* Deckhand isn't gated against requirements repo so changes to
some packages isn't actively tested
* Other Airship components use pins, so Deckhand might break them
when package requirements go out of sync between components
* Deckhand as of yet has no stable release so Airship relies on
master for stability so it is better to ensure stability over
anything else
Change-Id: I20ef4be3e01cd267771098d33447ccb61aee20b4
This simply updates Deckhand's package versions in
test-requirements.txt and requirements.txt using `pur -r` command
line tool.
Change-Id: If4bbd85190502c52a7f10adf6af750172b54c234
mox3 is not being used so remove it, apart of the community
goal for Rocky.
Change-Id: I4fd687391d83fd11665b287d1853a35fcdc1f9f9
Signed-off-by: Charles Short <zulcss@gmail.com>
This adds a uwsgi functional test check to .zuul.yaml so that
deploying Deckhand via uwsgi (in a more standalone fashion,
sans containerization) works as intended.
Change-Id: I931ab4d11719daca7665d3a25b00e353c707237e
This is to update releasenotes/docs tox jobs to remove need
to defined build_sphinx in setup.cfg and to ensure that they
both clean up prior to running via appropraite rm -rf commands
and to ensure all the requirements are being installed.
Change-Id: Iadd375dbb596151cb140fae03b82a728a64364a0
This reverts https://review.gerrithub.io/#/c/393980/ which was
a temporary workaround to unblock the Deckhand gate. pifpaf should
be used to run unit tests as having to install Docker just to kick
off unit tests is excessive.
However, the unit-tests.sh script is maintained in tools/ directory
as a fallback.
Change-Id: I24a10d4b3ea00006004f27d0086719fb0bf86dd9
This PS unblocks the gate by replacing pifpaf to run postgresql
for unit tests with docker, as a workaround. This is because
"pifpaf run postgresql" is failing with pifpaf not being able
to find the command "pifpaf run". Steps to reproduce:
python3 -m virtualenv -p python3 /tmp/venv
source /tmp/venv/bin/activate
pip install -U pip wheel devpi-client setuptools
pip install pifpaf
$pifpaf run postgresql
>> pifpaf: 'run' is not a pifpaf command. See 'pifpaf --help'.
>> Did you mean one of these?
help
The unit test script for spinning up the docker postgresql container
and then running unit tests is very similar to the pre-existing
script for running functional tests located in tools/ directory.
Change-Id: Ib0f414ff58007037ac12161876dcd7a10e91f48c
Currently readthedocs build is failing for DH because it runs
pip install -rtest-requirements.txt [0] when test-requirements.txt
doesn't include oslo.policy. And readthedocs can't run tox -e docs
which is what is intended to be run for documentation generation.
[0] https://readthedocs.org/projects/deckhand/builds/6351568/
Change-Id: I4ad93e41b0410c547c9fec7d6d38f3b7e7f6829f
This PS leverages pytest-html to generate an HTML report for
Deckhand functional tests. The test_gabbi.py file was updated to
use pytest as the test runner as it is compatible with pytest-html.
The report is saved in results/index.html. Test docs were updated.
Change-Id: I0b611bf124bf87d801ab93dd2a5d16f136e4801d
This PS revamps document hashing. Instead of relying on Python's
built-in hash function to hash the contents of a document (i.e.
metadata and data values), sha256 from hashlib is used instead,
mostly for security purposes.
Further, new parameters have been added to the document DB model:
data_hash and metadata_hash, and the old value hash has been
dropped. The data type for storing the hashes has been changed
to String from BigInt.
Finally, testing documentation was added.
Change-Id: I428ddcbce1007ea990ca0df1aa630072a050c722
Sphinx can be leveraged to auto-generate docs into feature-rich
HTML pages. Docstrings in modules and classes can be easily
auto-injected into documentation to produce high-level
documentation, yet with accompanying code blocks and
necessary docstrings.
This commit introduces the tox job for auto-generating docs,
as well as the foundational logic and documentation pages
(index, HACKING and glossary). While hacking rules are
introduced in HACKING.rst, they are not added in this
commit; that will be done in a follow-up patchset.
Additional documentation will also be included in a series
of future patchsets.
Change-Id: Iacd0e4542ebf481d66ab19dd43014b8d5bcc9e3f
Update test-requirements.txt to use latest version of:
* hacking
Enable the following off-by-default checks:
* [H203] Use assertIs(Not)None to check for None.
* [H204] Use assert(Not)Equal to check for equality.
* [H205] Use assert(Greater|Less)(Equal) for comparison.
* [H210] Require ‘autospec’, ‘spec’, or ‘spec_set’ in
mock.patch/mock.patch.object calls
* [H904] Delay string interpolations at logging calls.
Made minimal code changes to comply with changes.
Change-Id: I3559ead76b5476650d7193e7023d349175234922
This commit adds oslo.config options for keystone auth and
updates Deckhand's request context to use oslo_context for
facilitating integration with keystone auth options.
Change-Id: Ifd170e1a192402a970f8538f0c06bf017fe77f88
This commit fixes flake8 errors and fixes a minor bug related to
a schema version being v1 rather than v1.0.
OpenStack hacking rules are used to pin down flake8 to sane
standards using [0].
[0] 06e676c461/test-requirements.txt (L5)
Change-Id: Ib236df6f5ec9505c0e635f0faa9877d3397a2e55
Previously Deckhand was using manual validation of YAML files.
However, schema validation is much cleaner, robust and thorough.
So, this commit adds jsonschema as a dependence in Deckhand.
Tempest already uses it as a dependence as well -- so a precedence
already exists.
This commit also updates unit tests as needed. Documentation
changes will be made in a follow-up patch.
This commit adds the initial engine framework for Deckhand. Included
is the logic for parsing YAML files as well as validating them and
doing forward substitution as specified by the YAML file.
This commit also includes unit tests for the framework changes.
This commit implements the core Deckhand API framework.
It does not implement any real API routes. The core
framework is modeled after Drydock's [0].
This commit specifically:
- implements the core API framework which uses falcon
- implements errors.py for preliminary errors
- implements base resource class from which other API
resources will inherit to build out the API itself
- implements base API router
- implements entry-point for kicking off deckhand
- updates base README.rst with instructions on
- running and installing -- similar to Drydock's
- implements dummy API resource for secrets, to
be fleshed out further in a follow-up commit
[0] https://github.com/att-comdev/drydock
* DECKHAND-11: Add oslo.config integration to Deckhand
This commit adds oslo.config integration to Deckhand. It also
creates a lot of preliminary files/configuration settings
needed to run tox as well as lint and oslo-config-generator
jobs.
* Remove sample config file.
Sergiy Markin