Debugging pegleg can currently be difficult and the Click CLI does not
easily allow debuggers like pdb or PyCharm to use breakpoints. By moving
all CLI command calls into singular functions, we can easily create an
"if __name__ == '__main__'" entry point to call these functions and
investigate any bugs that may arise.
We also gain the ability to reuse more portions of our code by
refactoring these methods.
Change-Id: Ia9739931273eb6458f82dbb7e702a505ae397ae3
Adds alternate message when no certificates are expiring
Exit code will now be 1 if there are certificates expiring and 0 if no
certificates are expiring
Change-Id: I94a7a5af0c5469b83001b5439f18691140de6245
This patch updates layer for wrapped documents to preserve original
layer. Previously all encrypted documents had site layer.
Update encrypt/decrypt logic when determining global keys.
Update units tests.
Change-Id: I447aeaea08a4514655fcabfc7077b6d4b282e27f
Pegleg has been using a year old revision of treasuremap for internal
testing using seaworthy. The age of this revision has become a problem
as the certificates in seaworthy expired on August 20th. This change
updates pegleg to the latest version of treasuremap and makes any
necessary path updates for the new version to work with Pegleg tests.
Change-Id: Icea70b8d9bf24c8f85360719945b5899cab6b396
Pytest includes a fixture that can be used to generate temporary
directories. Previously Pegleg had implemented a hombrewed version of a
temporary directory fixture. This change removes the homebrewed version
and replaces it with the tmpdir fixture.
Implement tmpdir fixture in tests
Upgrade all testing packages to use the latest features
Removes unused imports and organizes import lists
Removes mock package requirement and uses unittest.mock, included in
python >3.3
Implements a slightly cleaner method to get proxy info
Change-Id: If66e1cfba858d5fb8948529deb8fb2d32345f630
This patch addresses inconsistent code style and enforces it with a
gate for future submissions.
Separate work will be done in the future to address several of the
PEP8 ignores for docstrings, and attempt to bring the tests directory
to PEP8 compliance.
This patch:
1. Updates .style.yapf to set the knobs desired for YAPF.
2. Updates tox.ini to allow one of the knobs to work.
3. Removes unused code from several __init__.py files.
4. Updates the YAPF version in test-requirements.txt to latest (this
is needed for several knobs to work).
5. Stylistic changes to the python codebase in Pegleg.
6. Updates to tox.ini to run YAPF during PEP8 check.
Change-Id: Ieaa0fdef2b601d01c875d64b840986e54df73abf
This patchset adds support for globally encrypted secrets.
Documents with a "site" layer will be encrypted/decrypted with the
standard PEGLEG_PASSPHRASE and PEGLEG_SALT environment variables.
If any secrets exist for the site with a schema of "global_passphrase"
or "global_salt" their values will be captured and used to decrypt
any secrets that do not belong to "site" layer. If the global keys
do not exist, Pegleg will default to using site keys.
Expected usage:
1. Set site passphrase/salt environment variables
2. Select a global passphrase and salt
3. Use Pegleg's "wrap" command to wrap and encrypt the global keys
4. Encrypt or wrap documents with "global" layer
5. Provide Pegleg path to decrypt
In the case of (4) and (5) Pegleg will determine the correct keys
to use automatically
Change-Id: I5de6d63573619b346fe011628ae21e053e0711f6
Some secrets are being created with undesirable permissions. Upon
inspection it was noticed that in general Pegleg is creating files,
then changing permissions after the fact. This leads to a small
window where the permissions on a file are overly permissive.
This patchset:
1. Sets default umask of 0o027 (640 permissions for files)
2. Explicitly adds the open flag ('r', 'w' etc.) to all open() calls.
3. Replaces sys.stdout.write calls with click.echo() calls to be more
in line with the rest of the project.
4. Re-orders methods that write so that data is always first, and the
path is always second.
5. Updates unit tests.
6. Adds unit tests for testing directory and file permissions.
7. Minor style changes.
Change-Id: I0c154aa311ea371940fd24b0aabf58fffaf1d231
Multiple occurences of -p or -f in the same command is vague.
Removing duplicate shorthand flags in favor of the explicit long form
flags instead for several commands.
Change-Id: Ic26360e517ea8f7ad2e0e5354d34a61fd622e0f1
Unit tests are warning that yaml.load() without a loader are deprecated.
Switch these calls to yaml.safe_load() to resolve warnings.
Change-Id: Ia8e080fc5317eefe432eee984608df190546530c
The Airship-In-A-Bottle repo was previously airship/in-a-bottle but has
been updated to airship/airship-in-a-bottle. Updating these references
to use the new name. This resolves both a consistency issue and failing
unit tests.
Change-Id: Ifa36e352a2718fb9998bf26f9f743ca2a7e8f8d4
Currently, using the upload command in Pegleg will upload all discovered
collections to Shipyard by repo. Uploading multiple of these repos can
result in 409 errors during uplift scenarios. This change compiles all
documents into a single collection document that can then be uploaded to
Shipyard.
Requires a collection name to be specified that will be used as the
'collection_id' for uploading to Shipyard.
Buffer mode is set by default to 'replace' instead of 'auto'.
Change-Id: I546b03fd82873296fff10aba355a50e4b11352d0
This change allows users to specify a directory or file to be decrypted.
Allows directory decryption.
Adds flag to overwrite encrypted file with decrypted data.
Intelligently recognizes paths vs files in CLI input and outputs data
accordingly.
Change-Id: I0d5e77f0eb1adb42165aa9b214aa90a0db0a3131
The dockerfile and some unit tests were still pointing to review.openstack.org
update those references to review.opendev.org
Change-Id: I161158ac0d66533a1775957864d1bd69dfa9530b
This patch:
1. Allows user to change valid duration of newly generated certs
default=1yr
2. Allows user to check certs that are expiring soon default=60d
Change-Id: Ia5c87a0c52b39b778f425599fa215fb67147c65b
Added a new command, site secrets wrap, to wrap bare files (e.g. pem or
crt) in a PeglegManagedDocument and optionally encrypt them.
Change-Id: I12689275c8e5a8854496fd6bbf69ce6e7cd9ad47
This patch:
1. Sets the salt in config when running genesis bundle
2. Updates the genesis bundle CLI method
3. Adds exception types for credentials
4. Updates unit tests to be compliant with new exceptions
Change-Id: I8869f897e2c25b98c30eaa6be52356aae4ac63b6
This patch fixes a critical bug in decryption which prevents the
decrypted data from being output and adds a unit test to ensure the
output is being generated.
Change-Id: Ica791cd9d309dfff254fe7e35023d130b3d63153
Salts and Passphrases are both strings used in cryptography. This patch:
1. Adds CLI generation of salt
2. Adds unit test for CLI generation of salt
3. Updates passphrase.py code to be more generic as it is used to generate
both a passphrase and a salt
4. Update name of passphrase.py to be more generic
5. Update all references to, and tests of passphrase.py
6. Add documentation for CLI generation of salt
Co-Authored-By: chittibabu <cg329x@att.com>
Change-Id: I71858d63a2846290d22be96686ccfea3ba8aa6c0
1. Add support to pegleg to generate a passphrase from CLI
2. Update unit test to ensure encryption/decryption supports passphrase rotation
3. Update order of import statements to satisfy pep8
4. Add unit test for CLI passphrase generation
5. Resolve merge conflicts via rebase
Change-Id: I5cb9e41b2f0fac2451bd2b74f33c48cda417c22d
1. Adds the passphrases generation capability in Pegleg CLI,
so that pegleg can generation random passwords based on a
specification declared in pegleg/PassphrasesCatalog documents
2. Pegleg also wraps the generated passphrase documents in
pegleg managed documents, and encrypts the data.
3. Adds unit test cases for passphrase generation.
4. Updates pegleg CLI document.
Change-Id: I21d7668788cc24a8e0cc9cb0fb11df97600d0090
This PS enables Pegleg to upload documents directly to Shipyard
thus ensuring that unencrypted data never gets stored in disk.
The flow for this new CLI command is as follows:
- Collect documents as per the provided site repository
- Decrypt the collected documets(TODO)
- Upload document to Shipyard:
- one collection per repository will be uploaded to Shipyard
Eg-
pegleg site -r /opt/aic-clcp-site-manifests \
-e global=/opt/aic-clcp-manifests upload <site-name>
Two collections will be created in shipyard since there are two
repositories provided. The name of the collections will be the
name of repositories provided.
- Commit the documents in shipyard buffer.
Change-Id: I6275252b044ebb82d8bb2009c0bea6ebf7033bce
This patch set refactors uses of tempfile.mkdtemp() used
throughout many tests in Pegleg which leaves lingering temporary
directories around. Recently a fixture was introduced in [0] which
automatically cleans up after itself. This patch set applies the
fixture everywhere possible to minimize the testing footprint.
[0] https://review.openstack.org/#/c/609818/20/tests/unit/fixtures.py
Change-Id: Id4c1195c4f248b974a5396a429d651943a84ee83
This patch set adds additional logic to properly handle parsing
out the revision from an SSH repo url. The issue was being
masked by unit tests whose automated logic for calculating
the expected revision mirrored the actual implementation.
Thus, the unit tests have also been refactored to take in
hardcoded expected values to ensure that the assertions are
foolproof around validating expected revisions.
Change-Id: I7aacb4792f6b2dfc08d3a7bb4c3f18bbcfc95b8a
This patch set fixes a recent issue that was introducing
following a refactor Pegleg table output. The related
CLI unit tests were changed to stop checking for an exit_code
of 0 due to improper mocking causing an exit_code of -1
to be returned instead.
This patch set corrects the issue by refactoring the code to
allow for exit_code 0 to always be checked.
Further refactoring was done in places to reduce code
duplication via helpers.
Change-Id: Ib9cf7c71b02d74de152b7e34911c867d2251a18a
As it currently stands, Pegleg clones site repositories into the /tmp
directory. Even if the site repository already exists in the /tmp
directory it is still cloned there which results in wasted disk space.
This commit allows users to pass in a `clone_path` (-p) option to Pegleg
CLI commands that specify where to clone a site repository. If the clone
path matches the path of an existing repository, then a error message is
logged stating so. If the repository already exists in the clone path, the
user can either specify to use that local repo by passing it as the site
repository or they proceed by passing in a different clone path.
This commit also updates the logic that deletes the copy of the repo that
is created in the temporary folder to also delete the parent folder that
contains the copied repo. This scenario happens when using a local
repository as the site repository.
Addionally, this commit adds a cleanup fixture that removes files and
directories created in the temporary folder by the unit tests.
Change-Id: I1b2943493b8f201f337ea60006c009973dd941b3
This patch set enables Pegleg to support repos like Airship in a
Bottle -- those that have site/ type/ global/ folders nested
under deployment_files/. Very particular logic is needed in order
to handle that. CLI unit tests included for validation/regression.
Change-Id: I9f13f59738599f07329ad3e3274eb4590e8638f9
This commit leverages python prettytable to create tables and output
CLI information for the following:
- site:
- lint
- list
- show
- type:
- list
- repo:
- lint
Addtionally, this commit changes the verbosity settings for pegleg CLI.
When verbosity is not set, only error logs will be shown as this would
be useful to users when errors do occur. Otherwise, no logs should be
shown in the CLI output unless the user passes the `verbose` flag.
Change-Id: Ic7782e9e383a1d6a7e31ff7cce025beb53c7db01
Currently Pegleg really needs CLI tests that test the CLI workflow
end to end. This continues off the initial legwork for it here:
https://review.openstack.org/#/c/605289/
This adds positive tests for all CLI site actions. Negative tests
will be added in the future.
Change-Id: Icf6737c6204edf518f2b384dd8369de793cb303a
With the implementation of revisioned repository to the CLI in
https://review.openstack.org/#/c/577886 there was a change to
the lint command [0], which changed it from being a global lint
to a site-level (targetted lint)... kind of: Only the CLI logic
was modified to support targetted single-site linting. Thus,
the first issue this patch set addresses is implementing the
back-end logic to realize targetted, single-site linting.
The second issue this patch set addresses is re-supporting global
linting (linting all sites within a repository) which means that
this partially reverts [0] which had (kind of) replaced global
linting with per-site linting.
So, this patch set:
1) Implements targetted, single-site linting back-end logic
2) Re-implements global linting for all sites in a repo
3) Adds unit tests for both
4) Adds some helper functions to util.engine.definition to
help with 1) and 2)
[0] https://review.openstack.org/#/c/577886/4/src/bin/pegleg/pegleg/cli.py@191
Change-Id: I5147282556763d93dfaf06912d2c4c876e1bd69f
This patch set adds support for passing in URLs ending in .git
via the --site-repository (-r) flag, e.g.:
pegleg site -r https://github.com/openstack/project.git@foo [...]
Unit test added to validate behavior.
Change-Id: If822ed195f07dc25e0590e42d046730632fdaafd
This patch set tidies up collect output file names for cloned
repos. As an example, previously the output file could be named
something like tmp5ak7409aic-clcp-security-manifests.yaml -- after
this change it will be aic-clcp-security-manifests.yaml instead
which is much cleaner.
Pegleg "collect" looks at the last piece in a file path to
determine the filename to write to: [0]
This means that changing the clone path here from something like:
/tmp/tmp54d13yairship-pegleg (which will result in collection
output to tmp54d13yairship-pegleg.yaml)
To:
/tmp/tmp54d13y/airship-pegleg (which will result in collection
output to airship-pegleg.yaml)
[0] 2ea1a55a2d/pegleg/engine/site.py (L71)
Change-Id: I2198e299d392e24d376ccfa53bef57fcabf0d41b
This patch set fixes an issue where currently -r flag won't
work with remote repository URLs. site_repository.process_repositories
will blow up with an error (for example):
Error: https://github.com/openstack/airship-treasuremap/site/
airship-seaworthy/site-definition.yaml not found. Pegleg must be
run from the root of a configuration repository.
It is apparent that the URL should not be treated as a local path
which is what is happening behind the scenes with this bug.
Also, CLI unit tests are added to validate the intended behavior.
They will be expanded on in future patch sets.
Change-Id: I618465841f1e455c8f00f046b3c5d22348b99396
Ian H. Pittwood