This PS bumps up Airflow version to the latest
2.8.2 and also bumps up openstack dependences to
Antelope 2023.1
Change-Id: I1ab801527dacb7c667caf1d459d71f9df522cef6
This PS updates python modules and code to match Airflow 2.6.2
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
airflow-2.6.2 constraints
- a new pegleg validation gate was created based on airskiff deploy
pipeline from treasuremap v1.9
Change-Id: Icc96deb0b745eaab7a19d83b00aad909dcd28c25
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.
Change-Id: Id31d97ced8732d823937fb1f218e7ad8760d735c
This PS delivers focal version of Pegleg image and has the following updates:
- removed release-notes-jobs-python3 gate job because of incompartibility with Sphinx from current requirements
- added focal gate node and switched gates to use it
- added bindep.txt file into project root
- added bindep role into gate jobs
- added ubuntu_focal dockerfile for building focal pegleg image
- switched tox profiles to py38
- uplifted references to shipyard_client, promenade and deckhand projects
- resolved required dependencies conflicts by weakening constraints in Pipfile
- updated tox profile update-requirements for generate requirements.txt and test-requirements.txt
- generated new Pipfile.lock, requirements.txt and test-requirements.txt from Pipfile
- switched tox profiles to use requirements.txt and test-requirements.txt instead of pipenv because of upstream zuul nodes Pypi mirrorring issue
- updated reference to seaworthy site certificates in treasuremap repo
- fixed unit tests issues caused by pytest/mock updates and new openssl version
- fixed focal docker image publishing issue
- added multiprocessing into coverage tests running process
- made unit and coveraget tests more verbosive
Change-Id: I5c4c519dc725cfb8c7b4e14756347c9336028aff
This patch:
1. Updates the `tox -e safety` command to run the Safety tool [0]
against both the Pegleg requirements.txt and the
doc/requirements.txt file used only when `tox -e docs` is run.
2. Updates doc/requirements.txt with pinned dependencies so that
Safety can check those dependencies.
At the time of this patch it is expected for the non-voting Safety
gate to fail, because the Promenade version used by Pegleg requires
an insecure version of kubernetes, 3.0.0. This version of Kubernetes
is impacted by [1] due to the urllib version being used.
[0] https://pypi.org/project/safety/
[1] https://nvd.nist.gov/vuln/detail/CVE-2018-20060
Change-Id: I64fb4b1ae7c2814ad0ae11222bf8be531d4f40a5
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
Pipenv is a tool that brings better package dependency management to
python. It can automatically create and manage virtualenv as well as
managing package dependencies using Pipfile and Pipfile.lock. Adding
this dependency manager into Airship projects will decrease package
version conflicts between projects and help increase security through
hash validation of packages and vulnerability scans.
Changes:
- Imports requirements.txt type files into Pipfile
- Pipenv dependency management in tox
- Switches Safety package for "pipenv check", an implementation of
Safety
- Adds `-e` flag on VCS dependencies to resolve good versions on all
packages
- Unpins or loosens pins on "dev"-type packages
Pipenv Docs: https://docs.pipenv.org/en/latest/
Helpful Pipenv Guide: https://realpython.com/pipenv-guide/
Change-Id: I47e7e7b440d76103b4984499e6ffce4482a59353
Pytest includes a fixture that can be used to generate temporary
directories. Previously Pegleg had implemented a hombrewed version of a
temporary directory fixture. This change removes the homebrewed version
and replaces it with the tmpdir fixture.
Implement tmpdir fixture in tests
Upgrade all testing packages to use the latest features
Removes unused imports and organizes import lists
Removes mock package requirement and uses unittest.mock, included in
python >3.3
Implements a slightly cleaner method to get proxy info
Change-Id: If66e1cfba858d5fb8948529deb8fb2d32345f630
This patch addresses inconsistent code style and enforces it with a
gate for future submissions.
Separate work will be done in the future to address several of the
PEP8 ignores for docstrings, and attempt to bring the tests directory
to PEP8 compliance.
This patch:
1. Updates .style.yapf to set the knobs desired for YAPF.
2. Updates tox.ini to allow one of the knobs to work.
3. Removes unused code from several __init__.py files.
4. Updates the YAPF version in test-requirements.txt to latest (this
is needed for several knobs to work).
5. Stylistic changes to the python codebase in Pegleg.
6. Updates to tox.ini to run YAPF during PEP8 check.
Change-Id: Ieaa0fdef2b601d01c875d64b840986e54df73abf
Moves safety checks from pep8 linting to a new Zuul job. This allows
voting to be disabled on the job when the safety check database is down.
Changes basepython to python3 for safety job.
Change-Id: I29af8af244d45fbbe008c5efc341c1407c809539
Sphinx 2.1.0 has a bug [0] that causes whitespace to be excluded in
outputted docs. This change updates Sphinx to peg any version >2.1.0.
Safety dependency vulnerability checks now will also cover the doc
requirements.txt.
[0] https://github.com/sphinx-doc/sphinx/issues/6440
Change-Id: Ic4e69862781e35c8597c7eab67a270ba3da26a0d
This patchset adds support for globally encrypted secrets.
Documents with a "site" layer will be encrypted/decrypted with the
standard PEGLEG_PASSPHRASE and PEGLEG_SALT environment variables.
If any secrets exist for the site with a schema of "global_passphrase"
or "global_salt" their values will be captured and used to decrypt
any secrets that do not belong to "site" layer. If the global keys
do not exist, Pegleg will default to using site keys.
Expected usage:
1. Set site passphrase/salt environment variables
2. Select a global passphrase and salt
3. Use Pegleg's "wrap" command to wrap and encrypt the global keys
4. Encrypt or wrap documents with "global" layer
5. Provide Pegleg path to decrypt
In the case of (4) and (5) Pegleg will determine the correct keys
to use automatically
Change-Id: I5de6d63573619b346fe011628ae21e053e0711f6
This change adds dependency vulnerability checks into tox. These checks
are performed by the Safety package which checks requirements files
against a vulnerability database for any issues. If any vulnerabilities
are found, tox will fail its pep8 env.
Safety: https://pyup.io/safety/
Safety Repo: https://github.com/pyupio/safety
Safety Default Vulnerability DB: https://github.com/pyupio/safety-db
Change-Id: I70a10ecb01836373888bd87aa27953515accd4c4
This change depends on https://review.opendev.org/#/c/640645/
The changes in 640645 streamline the zuul jobs.
The changes in this patchset build off that, and adds py37 to tox
venv to ensure developers can perform the same tests that Zuul does
which includes both py36 and py37.
Change-Id: I07ad7524e007c02457bd738240e379b852179622
Currently tox coverage has a threshold of 85%, but unit tests are covering
88.44% of pegleg. Increase threshold from 85% to 87%.
Change-Id: Iee86b54763ea64bf0956a9928afbb35898460ac3
The openstack-python36-jobs template already exists; thus there
is no need for airship-pegleg-tox-py36. Since airship-pegleg-tox-py36
installs cfssl as a prerequisite for unit tests, move the install-cfssl.sh
command to tox.ini prior to unit test execution, allowing for the
airship-pegleg-tox-py36 Zuul job and its associated playbook to be
removed.
Change-Id: I66de957a1a57ef246476c1a81954cd0f822cb8be
This adds a custom yapf config file with
split_before_logical_operator knob set to false so that
pep8 doesn't complain when:
if (save_location and save_location != os.path.sep and
save_location.endswith(os.path.sep)):
is wrongly converted to:
if (save_location and save_location != os.path.sep
and save_location.endswith(os.path.sep)):
This keeps the and on the first line as in the first example.
Change-Id: Ibf2813c3c969fd5f1cfd14936f3fecc9100283f7
Previously tox coverage was set to 84 percent, it was updated to 86 percent
in https://review.openstack.org/#/c/634593/ this new threshold is too high
and causing failures with tox tests.
Change-Id: Ic741c55d2b5a555ec636320612a334672f1e0e7c
Per [0], fernet decrypt can never throw an InvalidSignature exception as that
is re-raised as InvalidToken. This patch set corrects the handling of the
exception, and added additional unit tests for coverage.
[0] https://cryptography.io/en/latest/fernet/#cryptography.fernet.Fernet.decrypt
Co-Authored-By: Drew Walters <drewwalters96@gmail.com>
Change-Id: Ic5ee7ef451a5657519c5397fc4b903b5adcb1d18
Signed-off-by: Tin Lam <tin@irrational.io>
This is patch set adds in a trivial fix for a mismatch between a comment
and the actual hacking rule that is being enabled.
Change-Id: I7b1b6146f0cc0222d050e9aabd42a965af63730f
Signed-off-by: Tin Lam <tin@irrational.io>
This patch set adds hacking rule to pegleg and fixes outstanding non-
docstring related violations.
Change-Id: I5bb5e78c211f24cf95669124bfcf9603bea8bf15
Signed-off-by: Tin Lam <tin@irrational.io>
This patch set adds releasenotes (reno) to the airship-pegleg repository.
Change-Id: I1ed98460f40eb851ca53f8a5b68ab17f8272a2e2
Signed-off-by: Tin Lam <tin@irrational.io>
This removes all PEP8 ignores and places in default settings for flake8.
Change-Id: I3c4df02dea959dfe58f44e7c0e0ac58078a81abc
Signed-off-by: Tin Lam <tin@irrational.io>
Consolidating all the linting to a single job rather than having a one-off
just to lint for trailing whitespaces. As most of this projects are python,
this should already be covered by PEP8, but will be scanning for yamls with
trailing whitespaces.
Change-Id: Iee33a69ff234d21c08217faa33a19e11dfef0ad9
Signed-off-by: Tin Lam <tin@irrational.io>
As the default Docker image support for Pegleg is using py36 as a base
image, this patch set simply removes the support for py35. This allows
us to use py36 features such as secrets and f-strings without needing to
cope with backward compatibility.
Change-Id: Ic156ca67d2a0fe6fdc74c2c9da253a1cb8a3f456
Signed-off-by: Tin Lam <tin@irrational.io>
This patch set adds in a python coverage job for pegleg using pytest.
Change-Id: Ice0332cc454f488e0372d5ab03e8b0a37db9efd1
Signed-off-by: Tin Lam <tin@irrational.io>
This patch set adds a shell wrapper script to allow
tox to be used to run a subset of unit tests via regexes:
To run all unit tests, execute::
$ tox -epy35
To run unit tests using a regex, execute::
$ tox -epy35 -- <regex>
Change-Id: I2ba1e18226d686cb549a075e020ba02e24204829
This patch set standardizes the Pegleg directory
structure because of the following reasons:
1) src/bin/pegleg is not necessary and only makes
building (e.g. documentation building) and running
of tox targets unnecessarily difficult.
2) src/bin/pegleg is a Java-like standard that
bears no relevance to Python.
Change-Id: I37d39d3d6186b92f8fbfe234221c9e44da48cf10
This patch set introduces a developer overview documentation
page to Pegleg to help developers onboard.
Change-Id: Ia453d76f024db39c6bdd97a44bfe1db1c25193f9
This patch set updates the various tox targets to use python 3 first
per Stein goal outlined in [0].
[0] https://governance.openstack.org/tc/goals/stein/python3-first.html
Change-Id: I7c6e2283ffaab74b72fe9aa3be2e040c3e338f22
Signed-off-by: Tin Lam <tin@irrational.io>
* Add support for URLs and directories including git clone support
* Add support for http://, https://, and ssh:// git cloning
* Add support for cloning behind proxy
* Add support for checking out references of cloned repos
* Add support for checking out references of local repos
* Add support for Pegleg Git exceptions
This patch set also adds support for including Pegleg
source code in documentation and adds exceptions
documentation.
Change-Id: I417a62c815f97a70f3abc432cc342707e8ce1f54
This patchset updates pegleg.engine.deckhand to also log the
errors associated with an exception object following a rendering
failure to aid with troubleshooting.
Change-Id: I9ca46e4b102bd39fb7c582bb1720702892df7f9e
This patchset updates docs to doc to align with OpenStack
standard. Follow-up patchset will be needed to publish
documentation to OpenStack [0].
[0] https://docs.openstack.org/doc-contrib-guide/project-guides.html
Change-Id: I90e5f9129207901402e26ed9488ec6e065568fe1
This patchset renames the tox.ini jobs:
* lint => pep8
to comply with OpenStack standards [0]. This is needed so that [1]
works seamlessly (it will enable py35 and cover jobs for Pegleg).
[0] e.g. 04469a5181/tox.ini (L119)
[1] I174d5df008f6e000da1a3878afe75919312ea7aa
Change-Id: Ic3a5f2391e129cae32f60c0fb17197169df1b4c6
This is to add a top-level lint/fmt job to Pegleg so that tox -e lint
is executed in CICD. lint is included under envlist to accomplish this.
fmt is excluded from envlist as it doesn't need to be executed in CICD
but is included nonetheless for convenience.
Also fixes some typos.
Change-Id: I76165704b32653c98e506ddde71c8240d5f28492
This is to add a tox target for running unit tests in
src/bin/pegleg and to add an alias to the Makefile in root
directory.
Change-Id: Ia138c57e8d732f6dbf19f00e452b0eac1a1aa9cb
Sergiy Markin