diff options
authorMatt McEuen <>2018-09-06 19:10:16 -0500
committerMatt McEuen <>2018-09-12 18:19:50 -0500
commitd1fca1269328a1a30d0b65feec64a4cf9fa04f7f (patch)
parent145307c84b5cdd10b3e63cf126cc675e6986d1ab (diff)
Spec: Pegleg secret generation and encryption
This patchset adds a spec outlining the work effort and high-level implementation considerations for adding the ability to generate, encrypt, and decrypt secrets from Pegleg. Change-Id: Iacc99008be96a4b0bac145d5fa84143cc64e9b67 Story: 2003708 Task: 26363
Notes (review): Code-Review+2: Scott Hussey <> Code-Review+2: Mark Burnett <> Workflow+1: Mark Burnett <> Verified+2: Zuul Submitted-by: Zuul Submitted-at: Fri, 14 Sep 2018 18:05:37 +0000 Reviewed-on: Project: openstack/airship-specs Branch: refs/heads/master
2 files changed, 372 insertions, 8 deletions
diff --git a/specs/approved/_placeholder.rst b/specs/approved/_placeholder.rst
deleted file mode 100644
index 0cd761b..0000000
--- a/specs/approved/_placeholder.rst
+++ /dev/null
@@ -1,8 +0,0 @@
1.. placeholder:
7This file is a placeholder and should be deleted when the first spec is moved
8to this directory.
diff --git a/specs/approved/pegleg-secrets.rst b/specs/approved/pegleg-secrets.rst
new file mode 100644
index 0000000..ba40145
--- /dev/null
+++ b/specs/approved/pegleg-secrets.rst
@@ -0,0 +1,372 @@
2 This work is licensed under a Creative Commons Attribution 3.0 Unported
3 License.
7.. index::
8 single: template
9 single: creating specs
12Pegleg Secret Generation and Encryption
15Pegleg is responsible for shepherding deployment manifest documents from their
16resting places in Git repositories to a consumable format that is ready
17for ingestion into Airship. This spec expands its responsibility to
18account for secure generation and encryption of secrets that are
19required within an Airship-based deployment.
24The work to author and implement this spec will be tracked under this
25`Storyboard Story`_.
27Problem description
30Airship supports the ability to identify secret information
31required for functioning deployments, such as passwords and keys; to
32ingest it into the site in a least-privilege-oriented fashion; and
33to encrypt it at rest within Deckhand. However, lifecycle management of
34the secrets outside the site should be made automatable and
35repeatable, to facilitate operational needs such as periodic password
36rotation, and to ensure that unencrypted secrets are only accessible by
37authorized individuals.
39Impacted components
42The following Airship components will be impacted by this solution:
44#. Pegleg: enhanced to generate, rotate, encrypt, and decrypt secrets.
45#. Promenade: PKICatalog will move to Pegleg.
46#. Treasuremap: site manifests augmented to support the updated Secrets schema.
47#. Airship-in-a-Bottle: site manifests augmented to support the updated
48 Secrets schema.
50Proposed change
56With this spec, the role of Pegleg grows from being a custodian of deployment
57manifests to additionally being the author of certain manifests. A new YAML
58schema will be created to describe these documents:
60Documents of this type will have one or both of the following data elements,
61although more may be added in the future: ``generated``, ``encrypted``.
62PeglegManagedDocuments serve as wrappers around other documents, and the
63wrapping serves to capture additional metadata that is necessary, but
64separate from the managed document proper.
65The managed document data will live in the ``data.managedDocument`` portion
66of a PeglegManagedDocument.
68If a PeglegManagedDocument is ``generated``, then its contents have been
69created by Pegleg, and it must include provenance information per this
72 schema: pegleg/PeglegManagedDocument/v1
73 metadata:
74 name: matches-document-name
75 schema: deckhand/Document/v1
76 labels:
77 matching: wrapped-doc
78 layeringDefinition:
79 abstract: false
80 # Pegleg will initially support generation at site level only
81 layer: site
82 storagePolicy: encrypted
83 data:
84 generated:
85 at: <timestamp>
86 by: <author>
87 specifiedBy:
88 repo: <...>
89 reference: <git ref-head or similar>
90 path: <PKICatalog/PassphraseCatalog details>
91 managedDocument:
92 metadata:
93 storagePolicy: encrypted
94 schema: <as appropriate for wrapped document>
95 <metadata from parent PeglegManagedDocument>
96 <any other metadata as appropriate>
97 data: <generated data>
99If a PeglegManagedDocument is ``encrypted``, then its contents have been
100encrypted by Pegleg, and it must include provenance information per this
103 schema: pegleg/PeglegManagedDocument/v1
104 metadata:
105 name: matches-document-name
106 schema: deckhand/Document/v1
107 labels:
108 matching: wrapped-doc
109 layeringDefinition:
110 abstract: false
111 layer: matching-wrapped-doc
112 storagePolicy: encrypted
113 data:
114 encrypted:
115 at: <timestamp>
116 by: <author>
117 managedDocument:
118 metadata:
119 storagePolicy: encrypted
120 schema: <as appropriate for wrapped document>
121 <metadata from parent PeglegManagedDocument>
122 <any other metadata as appropriate>
123 data: <encrypted string blob>
125A PeglegManagedDocument that is both generated via a Catalog, and encrypted
126(as specified by the catalog) will contain both ``generated`` and
127``encrypted`` stanzas.
129Note that this ``encrypted`` has a different purpose than the Deckhand
130``storagePolicy: encrypted`` metadata, which indicates an *intent* for Deckhand
131to store a document encrypted at rest in the cluster. The two can be used
132together to ensure security, however: if a document is marked as
133``storagePolicy: encrypted``, then automation may validate that it is only
134persisted (e.g. to a Git repository) if it is in fact encrypted within
135a PeglegManagedDocument.
137Document Generation
140Document generation will follow the pattern established by Promenade's
141PKICatalog pattern. In fact, PKICatalog management responsibility will move
142to Pegleg as part of this effort. The types of documents that are expected
143to be generated are certificates and keys, which are defined via PKICatalog
144documents now, and passphrases, which will be defined via a new
145``pegleg/PassphraseCatalog/v1`` document. Longer-term, these specifications
146may be combined, or split further (into a CertificateCatalog and
147KeypairCatalog), but this is not needed in the initial implementation in
148Pegleg. A collection of manifests
149may define more than one of each of these secret catalog documents if desired.
151The documents generated via PKICatalog and PassphraseCatalog will follow the
152PeglegManagedDocument schema above; note that this is a change to existing
153PKICatalog behavior. The PKICatalog schema and associated code should be
154copied to Pegleg (and renamed to ``pegleg/PKICatalog/v1``), and during a
155transition period the old and new PKICatalog implementations will exist
156side-by-side with slightly different semantics. Promenade's PKICatalog can
157be removed once all deployment manifests have been updated to use the new one.
159Pegleg will place generated document files in ``<site>/secrets/passphrases/``,
160``<site>/secrets/certificates``, or ``<site>/secrets/keypairs`` as appropriate:
162* The generated filenames for passphrases will follow the pattern
163 ``<passphrase-doc-name>.yaml``.
164* The generated filenames for certificate authorities will follow the pattern
165 ``<ca-name>_ca.yaml``.
166* The generated filenames for certificates will follow the pattern
167 ``<ca-name>_<certificate-doc-name>_certificate.yaml``.
168* The generated filenames for certificate keys will follow the pattern
169 ``<ca-name>_<certificate-doc-name>_key.yaml``.
170* The generated filenames for keypairs will follow the pattern
171 ``<keypair-doc-name>.yaml``.
172* Dashes in the document names will be converted to underscores for consistency.
174A PassphraseCatalog will capture the following example structure::
176 schema: pegleg/PassphraseCatalog/v1
177 metadata:
178 schema: metadata/Document/v1
179 name: cluster-passphrases
180 layeringDefinition:
181 abstract: false
182 layer: site
183 storagePolicy: cleartext
184 data:
185 passphrases:
186 - document_name: osh-nova-password
187 description: Service password for Nova
188 encrypted: true
189 - document_name: osh-nova-oslo-db-password
190 description: Database password for Nova
191 encrypted: true
192 length: 12
194The nonobvious bits of the document described above are:
196* ``encrypted`` is optional, and denotes whether the generated
197 PeglegManagedDocument will be ``encrypted``, as well as whether the wrapped
198 document will have ``storagePolicy: encrypted`` or
199 ``storagePolicy: cleartext`` metadata.
200 If absent, ``encrypted`` defaults to ``true``.
201* ``document_name`` is required, and is used to create the filename of the
202 generated PeglegManagedDocument manifest, and the ```` of
203 the wrapped ``deckhand/Passphrase/v1`` document. In both cases, Pegleg will
204 replace dashes in the ``document_name`` with underscores.
205* ``length`` is optional, and denotes the length in characters of the
206 generated cleartext passphrase data. If absent, ``length`` defaults
207 to ``24``.
208* ``description`` is optional.
210The ``encrypted`` key will be added to the PKICatalog schema, and adds the same
211semantics to PKICatalog-based generation as are described above for
214Pegleg CLI Changes
217The Pegleg CLI interface will be extended as follows. These
218commands will create PeglegManagedDocument manifests in the local repository.
219Committing and pushing the changes will be left to the
220operator or to script-based automation.
222For the CLI commands below which encrypt or decrypt secrets, an environment
223variable (e.g. ``$PEGLEG_KEY`` will be use to capture the key/passphrase to use.
224``pegleg site secrets rotate`` will use a second variable
225(e.g. ``$PEGLEG_PREVIOUS_KEY``) to hold the key/passphrase being rotated
228``pegleg site secrets generate passphrases``: Generate passphrases according to
229all PassphraseCatalog documents in the site.
230Note that regenerating passphrases can be accomplished
231simply by re-running ``pegleg site secrets generate passphrases``.
233``pegleg site secrets generate pki``: Generate certificates and keys according
234to all PKICatalog documents in the site.
235Note that regenerating certificates can be accomplished
236simply by re-running ``pegleg site secrets generate pki``.
238``pegleg site secrets generate``: Combines the two commands above.
239May be expanded in the future to include other manifest generation activities.
241``pegleg site bootstrap``: For now, a synonym for
242``pegleg site secrets generate``,
243and may be expanded in the future to include other bootstrapping activities.
245``pegleg site secrets encrypt``: Encrypt all site documents which have
246``metadata.storagePolicy: encrypted``, and which are not already encrypted
247within a wrapping PeglegManagedDocument. Note that the
248``pegleg site secrets generate`` commands encrypt generated secrets as
249specified, so ``pegleg site secrets encrypt`` is intended mainly for
250external-facing secrets which a deployment engineer brings to the site
252The output PeglegManagedDocument will be written back to the filename that
253served as its source.
255``pegleg site secrets decrypt <document YAML file>``: Decrypt a specific
256PeglegManagedDocument manifest, unwrapping it and outputting the cleartext
257original document YAML to standard output. This is intended to be used when
258an authorized deployment engineer needs to determine a particular cleartext
259secret for a specific operational purpose.
261``pegleg site secrets rotate``: This action re-encrypts encrypted secrets
262with a new key/passphrase, and it takes the previously-used key and a new
263key as input. It accomplishes its task via two activities:
265* For encrypted secrets that were imported from outside of Pegleg
266 (i.e. PeglegManagedDocuments which lack the ``generated`` stanza),
267 decrypt them with the old key (in-memory), re-encrypt them with
268 the new key, and output the results.
269* Perform a fresh ``pegleg site secrets generate`` process using the new key.
270 This will replace all ``generated`` secrets with new secret values
271 for added security. There is an assumption here that the only actors
272 that need to know generated secrets are the services within the
273 Airship-managed cluster, not external services or deployment engineers,
274 except perhaps for point-in-time troubleshooting or operational
275 exercises.
277Driving deployment of a site directly via Pegleg is follow-on functionality
278which will
279collect site documents, use them to create the ```` script, and then
280interact directly with Shipyard to drive deployments. Its details are beyond
281the scope of this spec, but when implemented, it should decrypt documents
282wrapped by applicable PeglegManagedDocuments at the lst responsible moment,
283and take care not to write, log, or stdout them to disk as cleartext.
285Note that existing ``pegleg collect`` functionality should **not** be changed
286to decrypt encrypted secrets; this is because it writes its output to disk.
287If ``pegleg collect`` is called, at this point in time, the
288PeglegManagedDocuments will be written (encrypted) to disk.
289To enable special case full site secret decryption, a ``--force-decrypt`` flag
290will be added to ``pegleg collect`` to do this under controlled circumstances,
291and to help bridge the gap with existing CICD pipelines until Pegleg-driven
292site deployment is in place. It will leverage the ``$PEGLEG_KEY``
293variable described above.
295Secret Generation
298The ``rstr`` library should be invoked to generate secrets of the
299appropriate length and character set.
300This library uses the ``os.urandom()`` function,
301which in turn leverages ``/dev/urandom`` on Linux,
302and it is suitable for cryptographic purposes.
304Characters in generated secrets will be evenly distributed across lower-
305and upper-case letters, digits, and punctuation in
306!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~. Note this is equivalent to the union of
307Python string.ascii_letters, string.digits, and string.punctuation.
309Secret Encryption
312Details around encryption will be defined in a follow-on patch set to this spec.
314Security impact
317These changes will result in a system that handles site secrets in a highly
318secure manner, in the face of multiple roles and day 2 operational needs.
320Performance impact
323Performance impact to existing flows will be minimal. Pegleg will need to
324additionally decrypt secrets as part of site deployment, but this will be
325an efficient operation performed once per deployment.
330The Python ``secrets`` library presents a convenient interface for generating
331random strings. However, it was introduced in Python 3.6, and it would be
332limiting to introduce this constraint on Airship CICD pipelines.
334The ``strgen`` library presents an even more convenient interface for
335generating pseudo-random strings; however, it leverages the Python ``random``
336library, which is unsuitably random for cryptographic purposes.
338Deckhand already supports a ``storagePolicy`` element which indicates whether
339whether Deckhand will persist document data in an encrypted state, and this
340flag could have been re-used by Pegleg to indicate whether a secret is
341(or should be) encrypted. However, "should this data be encrypted" is a
342fundamentally different question than "is this data encrypted now", and
343additional metadata-esque parameters (``generated``, ``generatedLength``)
344were desired as well, so this proposal adds ``data.encrypted`` to indicate
345the point-in-time encryption status. ``storagePolicy`` is still valuable
346in this context to make sure everything that *should* be encrypted *is*,
347prior to performing actions with it (e.g. Git commits).
349This proposed implementation writes the output of generation/encryption events
350back to the same source files from which the original data came. This is a
351destructive operation; however, it wasn't evident that it is problematic in
352any anticipated workflow. In addition, it sidesteps challenges around
353naming of generated files, and cleanup of original files.
358Please refer to the `Storyboard Story`_ for implementation planning information.
363This work should be based on the patchset to add `Git branch and revision
364support`_ to Pegleg, if it is not merged by the time implementation begins.
365This patchset alters the CLI interface and Git repository management code,
366and basing on it will avoid future refactoring.
371.. _Storyboard Story:!/story/2003708
372.. _Git branch and revision support: