One of the well-known issues of Python is that dictionaries do not
maintain order in their keys once created. This causes YAML data dumps
to output in a seemingly random order or alphabetically. As these output
files are often kept in their own repositories, they must go through
review or comparison in VCS. If the order of keys is switching for these
files every time Pegleg is ran, it makes it difficult for a user to
compare newly generated files with the old.
To fix this issue, we can change all dictionaries used to template
YAML files into OrderedDict objects. The OrderedDict objects will
maintain order through their dumping to YAML.
Change-Id: I0c1ee3f3f37ed8598d2ba81528d5c61447cbd0d0
This patch addresses inconsistent code style and enforces it with a
gate for future submissions.
Separate work will be done in the future to address several of the
PEP8 ignores for docstrings, and attempt to bring the tests directory
to PEP8 compliance.
This patch:
1. Updates .style.yapf to set the knobs desired for YAPF.
2. Updates tox.ini to allow one of the knobs to work.
3. Removes unused code from several __init__.py files.
4. Updates the YAPF version in test-requirements.txt to latest (this
is needed for several knobs to work).
5. Stylistic changes to the python codebase in Pegleg.
6. Updates to tox.ini to run YAPF during PEP8 check.
Change-Id: Ieaa0fdef2b601d01c875d64b840986e54df73abf
Pegleg currently raises an exception when a filepath exists before
writing to it in the case of yaml.dump actions. This is not always
desired behavior, and with the addition of the flags passed into the
files methods for write/append is no longer necessary.
Removing these so that overwrite options can take place for example
when rotating passphrases or secrets.
Change-Id: I2200cc681a797d91a90f6237662fd1ee77954906
Previously Pegleg would attempt to create directories using the path
specified directly. This path wasn't always an absolute path,
resulting in errors such as:
File "/opt/pegleg/pegleg/engine/util/files.py", line 265, in dump_all
os.makedirs(os.path.dirname(path), exist_ok=True)
File "/usr/local/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: ''
This bugfix determines an absolute path before attempting to create
any directories.
Change-Id: I84a0e7bc63d6f56a56b9c5c41de1ede99dfbacc7
Some secrets are being created with undesirable permissions. Upon
inspection it was noticed that in general Pegleg is creating files,
then changing permissions after the fact. This leads to a small
window where the permissions on a file are overly permissive.
This patchset:
1. Sets default umask of 0o027 (640 permissions for files)
2. Explicitly adds the open flag ('r', 'w' etc.) to all open() calls.
3. Replaces sys.stdout.write calls with click.echo() calls to be more
in line with the rest of the project.
4. Re-orders methods that write so that data is always first, and the
path is always second.
5. Updates unit tests.
6. Adds unit tests for testing directory and file permissions.
7. Minor style changes.
Change-Id: I0c154aa311ea371940fd24b0aabf58fffaf1d231
Added a force-cleartext option (false by default) which forces
passphrases to be generated in cleartext rather than encrypted.
Change-Id: I157a40103f67f85a24976b4f59aa46f2d4b92334
This patch handles the case where CA certs or authorities are loaded as
byte strings. It also disables parsing YAML documents with python/object
types directly into (non-dict) Python objects (which is PyYaml's
default behavior), as it creates issues with the PeglegManagedDocument
module.
The patch also fixes a bug where attempting to re-encrypt an already
encrypted file would result in a serialized python object being written
rather than the expected output YAML.
Change-Id: I4b84ee8f9922ae042411e70242ffda4622647e86
Add an option, -s, to write decrypted files to a file rather than
stdout. Decryptyed files have their mode set to 600. Also adds a few
improvements to files.write.
Change-Id: Ia1a6de78d401afbea6ee261652f4650071f54b60
This PS adds a DeploymentData document to sites collected by Pegleg.
This document describes the repos Pegleg collected, including their
commit SHA, tag, and whether the repo was dirty.
If the source directory is not a git repo, these values will be
None.
Change-Id: I7919b02d70c9797f689cdad85066d3953b978901
Fix an error where secrets in global directories are erroneously
flagged for being outside a secrets directory. Now, any file
that is a child of a directory called secrets should be handled
correctly.
Change-Id: I827aa75110d761601dc65df64e1accf1b1a54544
This patch set adds hacking rule to pegleg and fixes outstanding non-
docstring related violations.
Change-Id: I5bb5e78c211f24cf95669124bfcf9603bea8bf15
Signed-off-by: Tin Lam <tin@irrational.io>
This PS enables Pegleg to upload documents directly to Shipyard
thus ensuring that unencrypted data never gets stored in disk.
The flow for this new CLI command is as follows:
- Collect documents as per the provided site repository
- Decrypt the collected documets(TODO)
- Upload document to Shipyard:
- one collection per repository will be uploaded to Shipyard
Eg-
pegleg site -r /opt/aic-clcp-site-manifests \
-e global=/opt/aic-clcp-manifests upload <site-name>
Two collections will be created in shipyard since there are two
repositories provided. The name of the collections will be the
name of repositories provided.
- Commit the documents in shipyard buffer.
Change-Id: I6275252b044ebb82d8bb2009c0bea6ebf7033bce
This patch set changes Pegleg in two similar ways:
1) Ignore certain types of files altogether:
- those located in hidden folders
- those prefixed with "." (files like .zuul.yaml)
2) Only read Deckhand-formatted documents for lint/collect/etc.
commands as Pegleg need not consider other types of documents
(it separately reads the site-definition.yaml for internal
processing still).
The tools/ subfolder is also ignored as it can contain
.yaml files which are not Deckhand-formatted documents,
so need not be processed by pegleg.engine.
Change-Id: I8996b5d430cf893122af648ef8e5805b36c1bfd9
This patch set removes Pegleg stub logic from the CLI and engine
because:
1) It's currently broken following revisioned repository support
with https://review.openstack.org/#/c/577886/
2) The functionality isn't used anywhere in Airship, either upstream
or downstream
3) The functionality is, in any case, obsolete following the
revision repository change
As such, it is better if we simply remove the stub logic out for
now and revisit the implementation later, once the requirements
for it are fully fleshed out to align with modern Pegleg architecture.
Change-Id: If82b91bcdd39555596f2a4003bfda0167142fddd
This patch set standardizes the Pegleg directory
structure because of the following reasons:
1) src/bin/pegleg is not necessary and only makes
building (e.g. documentation building) and running
of tox targets unnecessarily difficult.
2) src/bin/pegleg is a Java-like standard that
bears no relevance to Python.
Change-Id: I37d39d3d6186b92f8fbfe234221c9e44da48cf10
Ian H Pittwood