Debugging pegleg can currently be difficult and the Click CLI does not
easily allow debuggers like pdb or PyCharm to use breakpoints. By moving
all CLI command calls into singular functions, we can easily create an
"if __name__ == '__main__'" entry point to call these functions and
investigate any bugs that may arise.
We also gain the ability to reuse more portions of our code by
refactoring these methods.
Change-Id: Ia9739931273eb6458f82dbb7e702a505ae397ae3
Adds an option to specify a passphrase catalog to override catalogs
discovered in the site repository. This allows the generation of a
specified subset of passphrases instead of the entire site's catalog.
Change-Id: I797107234292eea8ca788b7a94ed5e2c90566bf5
Adds alternate message when no certificates are expiring
Exit code will now be 1 if there are certificates expiring and 0 if no
certificates are expiring
Change-Id: I94a7a5af0c5469b83001b5439f18691140de6245
In [0] the secrets generate-pki command was moved to secrets
generate certificates. While release notes were added, this change
impacts automation set up for users of Pegleg. This change adds
back the generate-pki command but marks it as deprecated.
[0] https://review.opendev.org/#/c/694810/
Change-Id: I6a3841e5f5313511ec2afd8340bcae5857cd81fa
Allows users to specify a save location for newly generated certificates
instead of always writing them to the site repository. The functionality
is identical to generate passphrase's save_location option.
Change-Id: I8625fba75160c441dbf3f345af99eb0733b2c37d
This patch updates layer for wrapped documents to preserve original
layer. Previously all encrypted documents had site layer.
Update encrypt/decrypt logic when determining global keys.
Update units tests.
Change-Id: I447aeaea08a4514655fcabfc7077b6d4b282e27f
Pegleg's current verbose option simply sets the logging level to either
DEBUG or ERROR. This change allows users to enter a specific logging
level anywhere between DEBUG and CRITICAL. The default logging level is
set to 40=ERROR.
The original verbose option will be kept in order to preserve backwards
compatibility on existing scripts.
Change-Id: I2cb81c55ab070380c4336ab8d75a9bf1c18b95fc
This change disables and skips input prompts for generate passphrases.
Using the -i option will now only enable prompts for passphrases that
are set to prompt=True.
Change-Id: Ia932305891259d9d1430e1d184dbf39892d4a5d3
This patch adds functionality Pegleg currently lacks: the ability to
regenerate expired certificates.
This patch adds:
1. CLI toggle --regenerate-all to generate_pki. Default is False,
which means if no certificates are present, generate what is in
the pki catalogue. If new certs have been added to the catalogue
generate just those. If the --regenerate-all flag is True, then
Pegleg will ignore any existing certs and regenerate (or generate
for the first time) all certificates defined in the PKI catalogue.
2. Documentation updates for CLI change.
3. Updates to pki_utility to accomodate the new flag.
4. Updates pki_generator methods to use rendered documents to
accommodate documents that have to be layered.
5. Updates pki_generator unit tests to include a layering definition
which is now required to run the commands.
Change-Id: I2d8086770e9226e44598ef40eca790981279f626
This patch addresses inconsistent code style and enforces it with a
gate for future submissions.
Separate work will be done in the future to address several of the
PEP8 ignores for docstrings, and attempt to bring the tests directory
to PEP8 compliance.
This patch:
1. Updates .style.yapf to set the knobs desired for YAPF.
2. Updates tox.ini to allow one of the knobs to work.
3. Removes unused code from several __init__.py files.
4. Updates the YAPF version in test-requirements.txt to latest (this
is needed for several knobs to work).
5. Stylistic changes to the python codebase in Pegleg.
6. Updates to tox.ini to run YAPF during PEP8 check.
Change-Id: Ieaa0fdef2b601d01c875d64b840986e54df73abf
files.write expects data, path
When using the -o option in decrypt, files.write was being called
with path, data leading to errors with filenames being too long
(because they were the contents of the file instead).
Change-Id: I2b00669227de94ebe6ced51d5ef25686de0c8a4b
Recently Shipyard was updated in [0] to allow a user to provide an
auth token manually, as an environment variable, instead of having
Shipyard obtain a token from Keystone using passed in auth variables
including a username and password.
This patch expands on that functionality to allow a user to either
set an OS_AUTH_TOKEN environment variable, or pass it during the
upload command with the --os-auth-token flag. If it is present, only
the token is used and the other auth variables are ignored.
[0] https://review.opendev.org/#/c/666402/
Change-Id: Iaf2a109022e8f5d496851ff43fcfa8198b5411c9
This patchset adds support for globally encrypted secrets.
Documents with a "site" layer will be encrypted/decrypted with the
standard PEGLEG_PASSPHRASE and PEGLEG_SALT environment variables.
If any secrets exist for the site with a schema of "global_passphrase"
or "global_salt" their values will be captured and used to decrypt
any secrets that do not belong to "site" layer. If the global keys
do not exist, Pegleg will default to using site keys.
Expected usage:
1. Set site passphrase/salt environment variables
2. Select a global passphrase and salt
3. Use Pegleg's "wrap" command to wrap and encrypt the global keys
4. Encrypt or wrap documents with "global" layer
5. Provide Pegleg path to decrypt
In the case of (4) and (5) Pegleg will determine the correct keys
to use automatically
Change-Id: I5de6d63573619b346fe011628ae21e053e0711f6
Some secrets are being created with undesirable permissions. Upon
inspection it was noticed that in general Pegleg is creating files,
then changing permissions after the fact. This leads to a small
window where the permissions on a file are overly permissive.
This patchset:
1. Sets default umask of 0o027 (640 permissions for files)
2. Explicitly adds the open flag ('r', 'w' etc.) to all open() calls.
3. Replaces sys.stdout.write calls with click.echo() calls to be more
in line with the rest of the project.
4. Re-orders methods that write so that data is always first, and the
path is always second.
5. Updates unit tests.
6. Adds unit tests for testing directory and file permissions.
7. Minor style changes.
Change-Id: I0c154aa311ea371940fd24b0aabf58fffaf1d231
Currently there isn't a uniform or easily expandable way to manage
how Pegleg gets credentials or enforces any complexity on them. This
patchset attempts to address this by:
1. Moving all logic for credentials into config.py
2. Using PeglegSecretManagement as the source of interfacing with
config.py as this code is the entry point for any encryption or
decryption work
3. Remove unnecessary code related to this change
4. Update unit tests
In future patchsets the goal is to use these changes to add in a global
passphrase and salt variable into config.py so that encrypt/decrypt type
commands can be executed one time against a site and intelligently
handle retrieval of global credentials for use with global secrets, site
credentials in the form of environment variables will remain used for
site secrets and will not be overridden by any global operations.
Change-Id: I0b6acd3ef5eab6b1f8931f46544bc53443f5c2c0
Added a force-cleartext option (false by default) which forces
passphrases to be generated in cleartext rather than encrypted.
Change-Id: I157a40103f67f85a24976b4f59aa46f2d4b92334
Previously the site PEGLEG_PASSPHRASE variable was used to encrypt the
genesis bundle. This is not always desired.
This patch:
1. Separates the Pegleg and Promenade encryption credentials
2. Simplifies the bundle code to avoid circular setting of environment
variables unnecessarily.
Change-Id: I2195cf8df81d3775402299d9a2b0aad4ba483b2c
Multiple occurences of -p or -f in the same command is vague.
Removing duplicate shorthand flags in favor of the explicit long form
flags instead for several commands.
Change-Id: Ic26360e517ea8f7ad2e0e5354d34a61fd622e0f1
Currently, using the upload command in Pegleg will upload all discovered
collections to Shipyard by repo. Uploading multiple of these repos can
result in 409 errors during uplift scenarios. This change compiles all
documents into a single collection document that can then be uploaded to
Shipyard.
Requires a collection name to be specified that will be used as the
'collection_id' for uploading to Shipyard.
Buffer mode is set by default to 'replace' instead of 'auto'.
Change-Id: I546b03fd82873296fff10aba355a50e4b11352d0
When the --save-location flag is used it attempts to keep the name
of the file being decrypted, and saving it to a desired location.
Previously this would cause
TypeError: join() argument must be str or bytes, not 'tuple'
This is resolved by grabbing the correct value from the os.path.split
command.
Change-Id: I9344bd07f5f8d03b50ac9fc004b79fefb024bbd6
This change allows users to specify a directory or file to be decrypted.
Allows directory decryption.
Adds flag to overwrite encrypted file with decrypted data.
Intelligently recognizes paths vs files in CLI input and outputs data
accordingly.
Change-Id: I0d5e77f0eb1adb42165aa9b214aa90a0db0a3131
This patch detects when a repository URL requires username substitution
and raises an exception when no username was specified.
Change-Id: Ia60982ecddd957cff8709118b3eb8a905258dd06
Decrypt command was previously requiring that specified files have
in their paths the site name. This isn't necessarily always the case
for example we can have global files that need to be decrypted and do
not contain the site name in the filepath, but the site name is
relevant in ensuring based on the site-definition.yaml file that
pegleg uses the correct revision of the global repository.
The end result should be that when decrypting a file, we specify the
site name, pegleg ensures we're on correct revisions of the repos
and if the file exists, decrypt and print to stdout
This patch addresses this by:
1. Updating pegleg.engine.secrets.decrypt to no longer require a
site name.
2. Updating pegleg.cli.decrypt to no longer pass a site name to
pegleg.engine.secrets.decrypt
3. Updating documentation for CLI.
4. Updating unit tests for CLI and secrets.
Change-Id: Ia97518b06a58b069a4d6c0b8d68a37f45e5d31bb
Add an option, -s, to write decrypted files to a file rather than
stdout. Decryptyed files have their mode set to 600. Also adds a few
improvements to files.write.
Change-Id: Ia1a6de78d401afbea6ee261652f4650071f54b60
Currently deckhand render validation is disabled by default with no
option to override that behavior from the command line. Resolve this
by:
1. Adding CLI render flag 'validate', default=True
2. Updating CLI documentation
3. Update pegleg.engine.site.render method to include configurable
validate flag
4. Update pegleg.engine.util.deckhand.deckhand_render method to
validate=True by default (previously False)
5. Update pegleg.engine.util.deckhand.deckhand_render method to
perform deckhand's validate all function on rendered documents
NOTE: Validation logic is handled in deckhand, see
https://opendev.org/airship/deckhand/src/branch/master/deckhand/engine/layering.pyhttps://opendev.org/airship/deckhand/src/branch/master/deckhand/engine/document_validation.py
Change-Id: I042fad4b2bf08c88e3a2eef6a54dede5d45c28f5
Shipyard helper's upload documents method supports two default modes
of buffer. None, and append. We want to allow the user to dictate
which mode is used to support the other methods Shipyard itself has.
To accomplish this we add a new command line argument, a new variable
and leave existing behavior intact via the new default 'auto' mode
Change-Id: I7a252efa7fe7a766152c42d9398c3290d7e52a13
This change changes all options that display a default in their help
messages to instead use click's `show_default` parameter. Using the
parameter instead of hard coding the help messages keeps styling uniform
throughout the CLI and adapts to future changes in default values.
Change-Id: Icedc20cca9605f4c7ae6a1b114a008f415a0c8c8
The dockerfile and some unit tests were still pointing to review.openstack.org
update those references to review.opendev.org
Change-Id: I161158ac0d66533a1775957864d1bd69dfa9530b
This patch:
1. Allows user to change valid duration of newly generated certs
default=1yr
2. Allows user to check certs that are expiring soon default=60d
Change-Id: Ia5c87a0c52b39b778f425599fa215fb67147c65b
Added a new command, site secrets wrap, to wrap bare files (e.g. pem or
crt) in a PeglegManagedDocument and optionally encrypt them.
Change-Id: I12689275c8e5a8854496fd6bbf69ce6e7cd9ad47
This patch:
1. Sets the salt in config when running genesis bundle
2. Updates the genesis bundle CLI method
3. Adds exception types for credentials
4. Updates unit tests to be compliant with new exceptions
Change-Id: I8869f897e2c25b98c30eaa6be52356aae4ac63b6
Added a pegleg cli command to build genesis.sh bundle for
a site deployment.
Pegleg imports promenade engine, and uses promenade to build
and encrypt the genesis.sh deployment bundle.
Change-Id: I1a489459b2c56b7b53018c32aab5e6550c69e1d2
This patch fixes a critical bug in decryption which prevents the
decrypted data from being output and adds a unit test to ensure the
output is being generated.
Change-Id: Ica791cd9d309dfff254fe7e35023d130b3d63153
Salts and Passphrases are both strings used in cryptography. This patch:
1. Adds CLI generation of salt
2. Adds unit test for CLI generation of salt
3. Updates passphrase.py code to be more generic as it is used to generate
both a passphrase and a salt
4. Update name of passphrase.py to be more generic
5. Update all references to, and tests of passphrase.py
6. Add documentation for CLI generation of salt
Co-Authored-By: chittibabu <cg329x@att.com>
Change-Id: I71858d63a2846290d22be96686ccfea3ba8aa6c0
1. Add support to pegleg to generate a passphrase from CLI
2. Update unit test to ensure encryption/decryption supports passphrase rotation
3. Update order of import statements to satisfy pep8
4. Add unit test for CLI passphrase generation
5. Resolve merge conflicts via rebase
Change-Id: I5cb9e41b2f0fac2451bd2b74f33c48cda417c22d
1. Adds the passphrases generation capability in Pegleg CLI,
so that pegleg can generation random passwords based on a
specification declared in pegleg/PassphrasesCatalog documents
2. Pegleg also wraps the generated passphrase documents in
pegleg managed documents, and encrypts the data.
3. Adds unit test cases for passphrase generation.
4. Updates pegleg CLI document.
Change-Id: I21d7668788cc24a8e0cc9cb0fb11df97600d0090
This patch set expands on the unit test coverage for lint checks
in test_selectable_linting which only covers a small subset of
the lint checks handled by Pegleg. This logic should be properly
tested as linting is fundamental to Pegleg functionality.
Change-Id: I6a59295982abd22bba8036827cefd4186b68e2fb
This PS enables Pegleg to upload documents directly to Shipyard
thus ensuring that unencrypted data never gets stored in disk.
The flow for this new CLI command is as follows:
- Collect documents as per the provided site repository
- Decrypt the collected documets(TODO)
- Upload document to Shipyard:
- one collection per repository will be uploaded to Shipyard
Eg-
pegleg site -r /opt/aic-clcp-site-manifests \
-e global=/opt/aic-clcp-manifests upload <site-name>
Two collections will be created in shipyard since there are two
repositories provided. The name of the collections will be the
name of repositories provided.
- Commit the documents in shipyard buffer.
Change-Id: I6275252b044ebb82d8bb2009c0bea6ebf7033bce
This removes all PEP8 ignores and places in default settings for flake8.
Change-Id: I3c4df02dea959dfe58f44e7c0e0ac58078a81abc
Signed-off-by: Tin Lam <tin@irrational.io>
Ian H. Pittwood