Updated basedeployment_data to have version from site-definition.yaml
and implemented exception if no global revision existed.
Change-Id: I14caf8244a7d36201e9cc4ebc90e310d594743ec
This change adds the site_type parameter in deployment_data
The value will be taken from respective site-definition.yaml
Change-Id: I8e65b39c73c94caf3ed4cc517520b9577160b20d
This patchset aims to address least privileged concerns, namely that
Pegleg's current behavior is to require decryption of all site
documents prior to rendering. Failure to do so leads to a duplicate
document error.
Operators of Pegleg may not have a valid reason to access secrets
that are not being modified during their current workflow, their
work may be limited to non-secrets but need to test their changes by
rendering the site manifests.
To enable this, the get_rendered_documents function has been updated
such that if a document is encrypted, the secret value will be
converted to a string to pass schema validation, and then used for
rendering. This will allow operators of Pegleg to render documents
without decrypting secrets. Instead the encrypted string value
of the secret will be used.
Change-Id: I8656b5496e2225e6eb59727c4f79326a1406147c
A bug was found where the deployment_data document was not being
included in the upload to shipyard. Upon investigation it was also
noticed that deployment_data wasn't being rendered.
This patch updates the render and upload commands to include the
deployment data document.
Change-Id: I916132e80ac13546468f171a75517a9159e48ea6
It is possible for Pegleg to generate a commit on top of a repo if
the repo is dirty (aka, has uncommited/untracked files). This
effectively makes the repo appear "clean", and also changes the head
of the repo. This can potentially interfere with the deployment_data
generation that analyzes the cleanliness of the repo as well as the
commit at the head of the repo.
This patch set updates the deployment_data generation logic, to be
able to detect Pegleg-generated commits at the head of a repo, and
instead go off of the Pegleg-generated commit's parent commit when
generating the data. It also ensures the repo in the data is always
marked dirty if a Pegleg-generated commit is seen, because the
Pegleg-generated commit would not exist unless the repo was dirty.
Change-Id: I863b3f2f661f11c36ba939ee3023f78733021b96
This patch adds functionality Pegleg currently lacks: the ability to
regenerate expired certificates.
This patch adds:
1. CLI toggle --regenerate-all to generate_pki. Default is False,
which means if no certificates are present, generate what is in
the pki catalogue. If new certs have been added to the catalogue
generate just those. If the --regenerate-all flag is True, then
Pegleg will ignore any existing certs and regenerate (or generate
for the first time) all certificates defined in the PKI catalogue.
2. Documentation updates for CLI change.
3. Updates to pki_utility to accomodate the new flag.
4. Updates pki_generator methods to use rendered documents to
accommodate documents that have to be layered.
5. Updates pki_generator unit tests to include a layering definition
which is now required to run the commands.
Change-Id: I2d8086770e9226e44598ef40eca790981279f626
One of the well-known issues of Python is that dictionaries do not
maintain order in their keys once created. This causes YAML data dumps
to output in a seemingly random order or alphabetically. As these output
files are often kept in their own repositories, they must go through
review or comparison in VCS. If the order of keys is switching for these
files every time Pegleg is ran, it makes it difficult for a user to
compare newly generated files with the old.
To fix this issue, we can change all dictionaries used to template
YAML files into OrderedDict objects. The OrderedDict objects will
maintain order through their dumping to YAML.
Change-Id: I0c1ee3f3f37ed8598d2ba81528d5c61447cbd0d0
This patch addresses inconsistent code style and enforces it with a
gate for future submissions.
Separate work will be done in the future to address several of the
PEP8 ignores for docstrings, and attempt to bring the tests directory
to PEP8 compliance.
This patch:
1. Updates .style.yapf to set the knobs desired for YAPF.
2. Updates tox.ini to allow one of the knobs to work.
3. Removes unused code from several __init__.py files.
4. Updates the YAPF version in test-requirements.txt to latest (this
is needed for several knobs to work).
5. Stylistic changes to the python codebase in Pegleg.
6. Updates to tox.ini to run YAPF during PEP8 check.
Change-Id: Ieaa0fdef2b601d01c875d64b840986e54df73abf
Render when no save location is saved is attempting to do a
yaml.dump_all to None. Changing to a click.echo()
Change-Id: I3e1bd9e1e9bfd09ab234e38f95cd4561bc2b41c9
Some secrets are being created with undesirable permissions. Upon
inspection it was noticed that in general Pegleg is creating files,
then changing permissions after the fact. This leads to a small
window where the permissions on a file are overly permissive.
This patchset:
1. Sets default umask of 0o027 (640 permissions for files)
2. Explicitly adds the open flag ('r', 'w' etc.) to all open() calls.
3. Replaces sys.stdout.write calls with click.echo() calls to be more
in line with the rest of the project.
4. Re-orders methods that write so that data is always first, and the
path is always second.
5. Updates unit tests.
6. Adds unit tests for testing directory and file permissions.
7. Minor style changes.
Change-Id: I0c154aa311ea371940fd24b0aabf58fffaf1d231
This patch handles the case where CA certs or authorities are loaded as
byte strings. It also disables parsing YAML documents with python/object
types directly into (non-dict) Python objects (which is PyYaml's
default behavior), as it creates issues with the PeglegManagedDocument
module.
The patch also fixes a bug where attempting to re-encrypt an already
encrypted file would result in a serialized python object being written
rather than the expected output YAML.
Change-Id: I4b84ee8f9922ae042411e70242ffda4622647e86
Reorganized DeploymentData to match spec, added Deckhand validation to
unit tests for Pegleg collect.
Change-Id: I9d8d5819bfd55b960ae7dd3c93f5a03b4ea364f1
Currently deckhand render validation is disabled by default with no
option to override that behavior from the command line. Resolve this
by:
1. Adding CLI render flag 'validate', default=True
2. Updating CLI documentation
3. Update pegleg.engine.site.render method to include configurable
validate flag
4. Update pegleg.engine.util.deckhand.deckhand_render method to
validate=True by default (previously False)
5. Update pegleg.engine.util.deckhand.deckhand_render method to
perform deckhand's validate all function on rendered documents
NOTE: Validation logic is handled in deckhand, see
https://opendev.org/airship/deckhand/src/branch/master/deckhand/engine/layering.pyhttps://opendev.org/airship/deckhand/src/branch/master/deckhand/engine/document_validation.py
Change-Id: I042fad4b2bf08c88e3a2eef6a54dede5d45c28f5
This PS adds a DeploymentData document to sites collected by Pegleg.
This document describes the repos Pegleg collected, including their
commit SHA, tag, and whether the repo was dirty.
If the source directory is not a git repo, these values will be
None.
Change-Id: I7919b02d70c9797f689cdad85066d3953b978901
This patch set adds hacking rule to pegleg and fixes outstanding non-
docstring related violations.
Change-Id: I5bb5e78c211f24cf95669124bfcf9603bea8bf15
Signed-off-by: Tin Lam <tin@irrational.io>
This patch set simplifies some code related to site_by_params
by allowing it to take an iterable argument called *fields
which specifies exactly which parameters to include from
the site-definition.yaml. This means that no hard-coding
is required to manually filter the params into the exact
parameters required by follow-up function calls. This is
done for better code maintenance.
Change-Id: Ief6483dfbf3759204106330284e8e9b824b5567e
This patch set enables Pegleg to support repos like Airship in a
Bottle -- those that have site/ type/ global/ folders nested
under deployment_files/. Very particular logic is needed in order
to handle that. CLI unit tests included for validation/regression.
Change-Id: I9f13f59738599f07329ad3e3274eb4590e8638f9
This commit leverages python prettytable to create tables and output
CLI information for the following:
- site:
- lint
- list
- show
- type:
- list
- repo:
- lint
Addtionally, this commit changes the verbosity settings for pegleg CLI.
When verbosity is not set, only error logs will be shown as this would
be useful to users when errors do occur. Otherwise, no logs should be
shown in the CLI output unless the user passes the `verbose` flag.
Change-Id: Ic7782e9e383a1d6a7e31ff7cce025beb53c7db01
When performing pegleg <options> site <site_name> render -o <output>
the resulting output doesn't include a "start_end" or "start_end"
(meaning '---' at the beginning and '...' at the end). This corrects
that issue.
Change-Id: Ife8c3a74413957b9ac4be187047ce5543a399b8e
This patch set standardizes the Pegleg directory
structure because of the following reasons:
1) src/bin/pegleg is not necessary and only makes
building (e.g. documentation building) and running
of tox targets unnecessarily difficult.
2) src/bin/pegleg is a Java-like standard that
bears no relevance to Python.
Change-Id: I37d39d3d6186b92f8fbfe234221c9e44da48cf10
francisy