This PS updates python modules and code to match Airflow 2.6.2
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
airflow-2.6.2 constraints
- a new pegleg validation gate was created based on airskiff deploy
pipeline from treasuremap v1.9
Change-Id: Icc96deb0b745eaab7a19d83b00aad909dcd28c25
This PS delivers focal version of Pegleg image and has the following updates:
- removed release-notes-jobs-python3 gate job because of incompartibility with Sphinx from current requirements
- added focal gate node and switched gates to use it
- added bindep.txt file into project root
- added bindep role into gate jobs
- added ubuntu_focal dockerfile for building focal pegleg image
- switched tox profiles to py38
- uplifted references to shipyard_client, promenade and deckhand projects
- resolved required dependencies conflicts by weakening constraints in Pipfile
- updated tox profile update-requirements for generate requirements.txt and test-requirements.txt
- generated new Pipfile.lock, requirements.txt and test-requirements.txt from Pipfile
- switched tox profiles to use requirements.txt and test-requirements.txt instead of pipenv because of upstream zuul nodes Pypi mirrorring issue
- updated reference to seaworthy site certificates in treasuremap repo
- fixed unit tests issues caused by pytest/mock updates and new openssl version
- fixed focal docker image publishing issue
- added multiprocessing into coverage tests running process
- made unit and coveraget tests more verbosive
Change-Id: I5c4c519dc725cfb8c7b4e14756347c9336028aff
* disable python 35, leave only 36
* switch to bionic nodeset only
* update requirements.txt, Pipfile and lock
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: Ia3c4df9d1f39f0f2cebf8ba0d89aebc5eec6f674
Shpinx incorrectly claims it wants docutils >=0.18 but that is an error
and older versions fail with that, as mentioned here:
https://github.com/sphinx-doc/sphinx/issues/9841
Additionally, the repo URL for OpenSUSE 15.3 python has changed.
Change-Id: I9bee6cf3ad7aaba80a44f2bd2f917b16c776c0d7
Adding in note under environment variables that PROMENADE_ENCRYPTION_KEY environment variable is required
Change-Id: I52c24608b2ecf61cbeea693ccbae1d2dd081ac29
This change adds the site_type parameter in deployment_data
The value will be taken from respective site-definition.yaml
Change-Id: I8e65b39c73c94caf3ed4cc517520b9577160b20d
This patch:
1. Updates the `tox -e safety` command to run the Safety tool [0]
against both the Pegleg requirements.txt and the
doc/requirements.txt file used only when `tox -e docs` is run.
2. Updates doc/requirements.txt with pinned dependencies so that
Safety can check those dependencies.
At the time of this patch it is expected for the non-voting Safety
gate to fail, because the Promenade version used by Pegleg requires
an insecure version of kubernetes, 3.0.0. This version of Kubernetes
is impacted by [1] due to the urllib version being used.
[0] https://pypi.org/project/safety/
[1] https://nvd.nist.gov/vuln/detail/CVE-2018-20060
Change-Id: I64fb4b1ae7c2814ad0ae11222bf8be531d4f40a5
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
Deprecating cli options before removing them is a standard courtesy that
allows project consumers to handle such changes according to their
needs.
Option is removed here: https://review.opendev.org/#/c/701274
Change-Id: Iba4a4574426315a76294269d4287fd3ea18aa99e
Adds a path option to the encrypt command to specify what directory of
file to encrypt. If path is not specified, all applicable files in the
repositories will be encrypted (this is the current behavior).
Change-Id: Idd5e063a54cf157a8ec761de85cbd67edd05364c
'--' previously rendered as '-'
Fixes site secrets section of cli.rst being displayed as preformatted
text
Change-Id: Ibac3b3ec07f47cbb049140af6ab2d968e77b756d
Adds an option to specify a passphrase catalog to override catalogs
discovered in the site repository. This allows the generation of a
specified subset of passphrases instead of the entire site's catalog.
Change-Id: I797107234292eea8ca788b7a94ed5e2c90566bf5
During the recent Airship Working Committee meeting, the committee
addressed feedback from the Airship confirmation review [0]. One such
item was concerned with copyright footers mistakenly claiming rights to
all Airship documentation.
This change updates the footer to attribute documentation to the Pegleg
authors.
[0] https://etherpad.openstack.org/p/airship-wc-meeting-2019-12-09
Change-Id: I78fe864d832559afdbcd63dc255070f21a1608d1
Signed-off-by: Drew Walters <andrew.walters@att.com>
In [0] the secrets generate-pki command was moved to secrets
generate certificates. While release notes were added, this change
impacts automation set up for users of Pegleg. This change adds
back the generate-pki command but marks it as deprecated.
[0] https://review.opendev.org/#/c/694810/
Change-Id: I6a3841e5f5313511ec2afd8340bcae5857cd81fa
Updates the CLI documentation for addition of granular verbosity
option.
Related Change: https://review.opendev.org/#/c/684349/
Change-Id: I8ca1e6381e074d7821cd862222dfeac7537a3d1f
This change disables and skips input prompts for generate passphrases.
Using the -i option will now only enable prompts for passphrases that
are set to prompt=True.
Change-Id: Ia932305891259d9d1430e1d184dbf39892d4a5d3
This adds documentation for the PKI implementation
in Pegleg that was recently ported from Promenade.
Change-Id: I248ec62a0c3523fc5f3950a6a678cf6d998b093a
This patch adds functionality Pegleg currently lacks: the ability to
regenerate expired certificates.
This patch adds:
1. CLI toggle --regenerate-all to generate_pki. Default is False,
which means if no certificates are present, generate what is in
the pki catalogue. If new certs have been added to the catalogue
generate just those. If the --regenerate-all flag is True, then
Pegleg will ignore any existing certs and regenerate (or generate
for the first time) all certificates defined in the PKI catalogue.
2. Documentation updates for CLI change.
3. Updates to pki_utility to accomodate the new flag.
4. Updates pki_generator methods to use rendered documents to
accommodate documents that have to be layered.
5. Updates pki_generator unit tests to include a layering definition
which is now required to run the commands.
Change-Id: I2d8086770e9226e44598ef40eca790981279f626
Readthedocs failed to render Pegleg exceptions with error:
> WARNING: autodoc: failed to import exception ... from module
> 'pegleg'; the following exception was raised: No module named 'click'
Trying to add Pegleg requirements to the installed requirements list,
so that Readthedocs has all modules, including those needed for the
Pegleg itself.
Change-Id: I1780a9be3aeb0c3e2e775cbb1f713c2033f13e08
Sphinx 2.1.0 has a bug [0] that causes whitespace to be excluded in
outputted docs. This change updates Sphinx to peg any version >2.1.0.
Safety dependency vulnerability checks now will also cover the doc
requirements.txt.
[0] https://github.com/sphinx-doc/sphinx/issues/6440
Change-Id: Ic4e69862781e35c8597c7eab67a270ba3da26a0d
Added a force-cleartext option (false by default) which forces
passphrases to be generated in cleartext rather than encrypted.
Change-Id: I157a40103f67f85a24976b4f59aa46f2d4b92334
Multiple occurences of -p or -f in the same command is vague.
Removing duplicate shorthand flags in favor of the explicit long form
flags instead for several commands.
Change-Id: Ic26360e517ea8f7ad2e0e5354d34a61fd622e0f1
The Airship-In-A-Bottle repo was previously airship/in-a-bottle but has
been updated to airship/airship-in-a-bottle. Updating these references
to use the new name. This resolves both a consistency issue and failing
unit tests.
Change-Id: Ifa36e352a2718fb9998bf26f9f743ca2a7e8f8d4
Currently, using the upload command in Pegleg will upload all discovered
collections to Shipyard by repo. Uploading multiple of these repos can
result in 409 errors during uplift scenarios. This change compiles all
documents into a single collection document that can then be uploaded to
Shipyard.
Requires a collection name to be specified that will be used as the
'collection_id' for uploading to Shipyard.
Buffer mode is set by default to 'replace' instead of 'auto'.
Change-Id: I546b03fd82873296fff10aba355a50e4b11352d0
This change allows users to specify a directory or file to be decrypted.
Allows directory decryption.
Adds flag to overwrite encrypted file with decrypted data.
Intelligently recognizes paths vs files in CLI input and outputs data
accordingly.
Change-Id: I0d5e77f0eb1adb42165aa9b214aa90a0db0a3131
This patchset fixes:
1. -o and -f options are not available, removed both from documentation
2. Added documentation on the --author flag
3. Updated usage example for generate pki with above flag changes
Change-Id: I4ced0e0fb6bf4ea88d7dd4d3a02bb0825422566c
This patch:
1. Moves defaults for flags in line with the flag for improved
readability.
2. Removes an example that does not apply to that command.
Change-Id: I16575924c6c7cd6389a70cd687a21c2857889d03
This patch detects when a repository URL requires username substitution
and raises an exception when no username was specified.
Change-Id: Ia60982ecddd957cff8709118b3eb8a905258dd06
This change fixes incorrect references to option `--validators` and
changes them to the correct `--include-validators`.
Change-Id: I3cb8b86cc03d6e37c91e14d625fa20ad9488579b
Add DISTRO parameter to support multiple distros
Add Dockerfile for opensuse to build leap 15 image.
Change-Id: I7a529476937494e042a4801117489325aa6621c7
Decrypt command was previously requiring that specified files have
in their paths the site name. This isn't necessarily always the case
for example we can have global files that need to be decrypted and do
not contain the site name in the filepath, but the site name is
relevant in ensuring based on the site-definition.yaml file that
pegleg uses the correct revision of the global repository.
The end result should be that when decrypting a file, we specify the
site name, pegleg ensures we're on correct revisions of the repos
and if the file exists, decrypt and print to stdout
This patch addresses this by:
1. Updating pegleg.engine.secrets.decrypt to no longer require a
site name.
2. Updating pegleg.cli.decrypt to no longer pass a site name to
pegleg.engine.secrets.decrypt
3. Updating documentation for CLI.
4. Updating unit tests for CLI and secrets.
Change-Id: Ia97518b06a58b069a4d6c0b8d68a37f45e5d31bb
Add an option, -s, to write decrypted files to a file rather than
stdout. Decryptyed files have their mode set to 600. Also adds a few
improvements to files.write.
Change-Id: Ia1a6de78d401afbea6ee261652f4650071f54b60
Currently deckhand render validation is disabled by default with no
option to override that behavior from the command line. Resolve this
by:
1. Adding CLI render flag 'validate', default=True
2. Updating CLI documentation
3. Update pegleg.engine.site.render method to include configurable
validate flag
4. Update pegleg.engine.util.deckhand.deckhand_render method to
validate=True by default (previously False)
5. Update pegleg.engine.util.deckhand.deckhand_render method to
perform deckhand's validate all function on rendered documents
NOTE: Validation logic is handled in deckhand, see
https://opendev.org/airship/deckhand/src/branch/master/deckhand/engine/layering.pyhttps://opendev.org/airship/deckhand/src/branch/master/deckhand/engine/document_validation.py
Change-Id: I042fad4b2bf08c88e3a2eef6a54dede5d45c28f5
Shipyard helper's upload documents method supports two default modes
of buffer. None, and append. We want to allow the user to dictate
which mode is used to support the other methods Shipyard itself has.
To accomplish this we add a new command line argument, a new variable
and leave existing behavior intact via the new default 'auto' mode
Change-Id: I7a252efa7fe7a766152c42d9398c3290d7e52a13
The dockerfile and some unit tests were still pointing to review.openstack.org
update those references to review.opendev.org
Change-Id: I161158ac0d66533a1775957864d1bd69dfa9530b
https://review.openstack.org/#/c/639414/ was merged with outstanding
comments related to PKI cert expiration checks and PKI cert generation
This patch addresses those critiques to:
1. Make documentation clearer and more standard in format
2. Make code clearer by streamlining a validity check
Change-Id: If5352acd33cfd9e3e177aa11ff8f8ba74d5f55af
This patch:
1. Allows user to change valid duration of newly generated certs
default=1yr
2. Allows user to check certs that are expiring soon default=60d
Change-Id: Ia5c87a0c52b39b778f425599fa215fb67147c65b
Sergiy Markin