This patch updates layer for wrapped documents to preserve original
layer. Previously all encrypted documents had site layer.
Update encrypt/decrypt logic when determining global keys.
Update units tests.
Change-Id: I447aeaea08a4514655fcabfc7077b6d4b282e27f
One of the well-known issues of Python is that dictionaries do not
maintain order in their keys once created. This causes YAML data dumps
to output in a seemingly random order or alphabetically. As these output
files are often kept in their own repositories, they must go through
review or comparison in VCS. If the order of keys is switching for these
files every time Pegleg is ran, it makes it difficult for a user to
compare newly generated files with the old.
To fix this issue, we can change all dictionaries used to template
YAML files into OrderedDict objects. The OrderedDict objects will
maintain order through their dumping to YAML.
Change-Id: I0c1ee3f3f37ed8598d2ba81528d5c61447cbd0d0
Pytest includes a fixture that can be used to generate temporary
directories. Previously Pegleg had implemented a hombrewed version of a
temporary directory fixture. This change removes the homebrewed version
and replaces it with the tmpdir fixture.
Implement tmpdir fixture in tests
Upgrade all testing packages to use the latest features
Removes unused imports and organizes import lists
Removes mock package requirement and uses unittest.mock, included in
python >3.3
Implements a slightly cleaner method to get proxy info
Change-Id: If66e1cfba858d5fb8948529deb8fb2d32345f630
This patch addresses inconsistent code style and enforces it with a
gate for future submissions.
Separate work will be done in the future to address several of the
PEP8 ignores for docstrings, and attempt to bring the tests directory
to PEP8 compliance.
This patch:
1. Updates .style.yapf to set the knobs desired for YAPF.
2. Updates tox.ini to allow one of the knobs to work.
3. Removes unused code from several __init__.py files.
4. Updates the YAPF version in test-requirements.txt to latest (this
is needed for several knobs to work).
5. Stylistic changes to the python codebase in Pegleg.
6. Updates to tox.ini to run YAPF during PEP8 check.
Change-Id: Ieaa0fdef2b601d01c875d64b840986e54df73abf
This patchset adds support for globally encrypted secrets.
Documents with a "site" layer will be encrypted/decrypted with the
standard PEGLEG_PASSPHRASE and PEGLEG_SALT environment variables.
If any secrets exist for the site with a schema of "global_passphrase"
or "global_salt" their values will be captured and used to decrypt
any secrets that do not belong to "site" layer. If the global keys
do not exist, Pegleg will default to using site keys.
Expected usage:
1. Set site passphrase/salt environment variables
2. Select a global passphrase and salt
3. Use Pegleg's "wrap" command to wrap and encrypt the global keys
4. Encrypt or wrap documents with "global" layer
5. Provide Pegleg path to decrypt
In the case of (4) and (5) Pegleg will determine the correct keys
to use automatically
Change-Id: I5de6d63573619b346fe011628ae21e053e0711f6
Some secrets are being created with undesirable permissions. Upon
inspection it was noticed that in general Pegleg is creating files,
then changing permissions after the fact. This leads to a small
window where the permissions on a file are overly permissive.
This patchset:
1. Sets default umask of 0o027 (640 permissions for files)
2. Explicitly adds the open flag ('r', 'w' etc.) to all open() calls.
3. Replaces sys.stdout.write calls with click.echo() calls to be more
in line with the rest of the project.
4. Re-orders methods that write so that data is always first, and the
path is always second.
5. Updates unit tests.
6. Adds unit tests for testing directory and file permissions.
7. Minor style changes.
Change-Id: I0c154aa311ea371940fd24b0aabf58fffaf1d231
Currently there isn't a uniform or easily expandable way to manage
how Pegleg gets credentials or enforces any complexity on them. This
patchset attempts to address this by:
1. Moving all logic for credentials into config.py
2. Using PeglegSecretManagement as the source of interfacing with
config.py as this code is the entry point for any encryption or
decryption work
3. Remove unnecessary code related to this change
4. Update unit tests
In future patchsets the goal is to use these changes to add in a global
passphrase and salt variable into config.py so that encrypt/decrypt type
commands can be executed one time against a site and intelligently
handle retrieval of global credentials for use with global secrets, site
credentials in the form of environment variables will remain used for
site secrets and will not be overridden by any global operations.
Change-Id: I0b6acd3ef5eab6b1f8931f46544bc53443f5c2c0
This patch handles the case where CA certs or authorities are loaded as
byte strings. It also disables parsing YAML documents with python/object
types directly into (non-dict) Python objects (which is PyYaml's
default behavior), as it creates issues with the PeglegManagedDocument
module.
The patch also fixes a bug where attempting to re-encrypt an already
encrypted file would result in a serialized python object being written
rather than the expected output YAML.
Change-Id: I4b84ee8f9922ae042411e70242ffda4622647e86
Some applications, such as k8s, require a base64 encoded string.
This patch updates the passphrase catalog such that the user can
specify to use base64 encoding on one or more passphrases found in
the passphrase catalog.
We add that support by:
1. Updating pegleg.engine.catalogs.passphrase_catalog to include a
method which determines what encoding type to use, if any.
2. Updating pegleg.engine.generators.passphrase_generator.generate
to encode the passphrase in base64 if detected. This change is
designed to easily add other supported encoding methods in the
future if desired.
3. Updating tests.unit.engine.test_generate_passphrases to
demonstrate that the encoding field in passphrase catalog is
being used, and that the resultant passphrase is in fact base64
encoded. Also show that when encoding type is not specified, or
is set to 'none' that base64 encoding does not take place.
4. Updating tests.unit.engine.test_generate_passphrases to
demonstrate that the encoding field in passphrase catalog is
being used, and that the resultant passphrase is in fact base64
encoded. We also demonstrate the flow from original passphrase
to bytes, to base64 encoded, to encrypted, and back again yields
the expected values at each step of encoding/decoding/encryption
and decryption.
Change-Id: I47c740ca13be57ed74b6780f80c90b39e935708b
The three lines of code in pegleg.engine.errorcodes, and
pegleg.engine.util.pegleg_secret_management are giving false positive
bandit errors. This patchset address these by adding # nosec label
to each line, instructing Bandit to ignore that line of code.
The three errors detected are all B105, details below from Bandit:
>> Issue: [B105:hardcoded_password_string] Possible hardcoded password:
'P009'
Severity: Low Confidence: Medium
Location: pegleg/engine/errorcodes.py:22
20 FILE_CONTAINS_INVALID_YAML = 'P007'
21 DOCUMENT_LAYER_MISMATCH = 'P008'
22 SECRET_NOT_ENCRYPTED_POLICY = 'P009'
23
24 ALL_CODES = (
25 SCHEMA_STORAGE_POLICY_MISMATCH_FLAG,
# nosec reasoning: The variable 'SECRET_NOT_ENCRYPTED_POLICY' does not
map to a hardcoded password.
--------------------------------------------------
>> Issue: [B105:hardcoded_password_string] Possible hardcoded password:
'^.{24,}$'
Severity: Low Confidence: Medium
Location: pegleg/engine/util/pegleg_secret_management.py:30
28
29 LOG = logging.getLogger(__name__)
30 PASSPHRASE_PATTERN = '^.{24,}$'
31 ENV_PASSPHRASE = 'PEGLEG_PASSPHRASE'
32 ENV_SALT = 'PEGLEG_SALT'
# nosec reasoning: The variable 'PASSPHRASE_PATTERN' does not map to a
hardcoded password
--------------------------------------------------
>> Issue: [B105:hardcoded_password_string] Possible hardcoded password:
'PEGLEG_PASSPHRASE'
Severity: Low Confidence: Medium
Location: pegleg/engine/util/pegleg_secret_management.py:31
29 LOG = logging.getLogger(__name__)
30 PASSPHRASE_PATTERN = '^.{24,}$'
31 ENV_PASSPHRASE = 'PEGLEG_PASSPHRASE'
32 ENV_SALT = 'PEGLEG_SALT'
33
# nosec reasoning: The variable 'ENV_PASSPHRASE' does not map to a
hardcoded password. This is setting the environment variable name that
passwords are stored in as 'PEGLEG_PASSPHRASE'. The passphrases are not
hardcoded on disk, but retrieved from environment variables later via
os.environ.get(ENV_PASSPHRASE)
Change-Id: I4508b30b763f25e4466c2e2159fbaf3c7df68b5b
Added a pegleg cli command to build genesis.sh bundle for
a site deployment.
Pegleg imports promenade engine, and uses promenade to build
and encrypt the genesis.sh deployment bundle.
Change-Id: I1a489459b2c56b7b53018c32aab5e6550c69e1d2
1. Adds the passphrases generation capability in Pegleg CLI,
so that pegleg can generation random passwords based on a
specification declared in pegleg/PassphrasesCatalog documents
2. Pegleg also wraps the generated passphrase documents in
pegleg managed documents, and encrypts the data.
3. Adds unit test cases for passphrase generation.
4. Updates pegleg CLI document.
Change-Id: I21d7668788cc24a8e0cc9cb0fb11df97600d0090
This patch set adds hacking rule to pegleg and fixes outstanding non-
docstring related violations.
Change-Id: I5bb5e78c211f24cf95669124bfcf9603bea8bf15
Signed-off-by: Tin Lam <tin@irrational.io>
This removes all PEP8 ignores and places in default settings for flake8.
Change-Id: I3c4df02dea959dfe58f44e7c0e0ac58078a81abc
Signed-off-by: Tin Lam <tin@irrational.io>
This patch set changes Pegleg in two similar ways:
1) Ignore certain types of files altogether:
- those located in hidden folders
- those prefixed with "." (files like .zuul.yaml)
2) Only read Deckhand-formatted documents for lint/collect/etc.
commands as Pegleg need not consider other types of documents
(it separately reads the site-definition.yaml for internal
processing still).
The tools/ subfolder is also ignored as it can contain
.yaml files which are not Deckhand-formatted documents,
so need not be processed by pegleg.engine.
Change-Id: I8996b5d430cf893122af648ef8e5805b36c1bfd9
1. Added the method to decrypt a secret file and return its contents
as a list of documents (instead of printing out the file content).
2. Added clarifications for a encrypt and decrypt commands.
Change-Id: I77bce21be214c880c8413f5e6a2d0c2d1993fc8e
Alexander Hughes