summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorFelipe Monteiro <felipe.monteiro@att.com>2018-09-28 17:24:30 +0100
committerLev Morgan <morgan.lev@gmail.com>2019-01-15 13:29:21 -0600
commit2a8d2638b3fb760ed50eb0504b39d24e8e39b499 (patch)
tree36a2c37e6554c73248be196581906d7b04d532b7
parent40da373023c508f216a4246ee04efe5ba4bd05de (diff)
pki: Port Promenade's PKI catalog into Pegleg
This patch set implements the PKICatalog [0] requirements as well as PeglegManagedDocument [1] generation requirements outlined in the spec [2]. Included in this patch set: * New CLI entry point called "pegleg site secrets generate-pki" * PeglegManagedDocument generation logic in engine.cache.managed_document * Refactored PKICatalog logic in engine.cache.pki_catalog derived from the Promenade PKI implementation [3], responsible for generating certificates, CAs, and keypairs * Refactored PKIGenerator logic in engine.cache.pki_generator derived from Promenade Generator implementation [4], responsible for reading in pegleg/PKICatalog/v1 documents (as well as promenade/PKICatalog/v1 documents for backwards compatibility) and generating required secrets and storing them into the paths specified under [0] * Unit tests for all of the above [5] * Example pki-catalog.yaml document under pegleg/site_yamls * Validation schema for pki-catalog.yaml (TODO: implement validation logic here: [6]) * Updates to CLI documentation and inclusion of PKICatalog and PeglegManagedDocument documentation * Documentation updates with PKI information [7] TODO (in follow-up patch sets): * Expand on overview documentation to include new Pegleg responsibilities * Allow the original repository (not the copied one) to be the destination where the secrets are written to * Finish up cert expiry/revocation logic [0] https://airship-specs.readthedocs.io/en/latest/specs/approved/pegleg-secrets.html#document-generation [1] https://airship-specs.readthedocs.io/en/latest/specs/approved/pegleg-secrets.html#peglegmanageddocument [2] https://airship-specs.readthedocs.io/en/latest/specs/approved/pegleg-secrets.html [3] https://github.com/openstack/airship-promenade/blob/master/promenade/pki.py [4] https://github.com/openstack/airship-promenade/blob/master/promenade/generator.py [5] https://review.openstack.org/#/c/611739/ [6] https://review.openstack.org/#/c/608159/ [7] https://review.openstack.org/#/c/611738/ Change-Id: I3010d04cac6d22c656d144f0dafeaa5e19a13068
Notes
Notes (review): Code-Review+1: Michael Beaver <michaelbeaver64@gmail.com> Code-Review+1: Alexander Hughes <Alexander.Hughes@pm.me> Code-Review+1: chittibabu <chittibabu1299@gmail.com> Code-Review+1: Nishant Kumar <nishant.e.kumar@ericsson.com> Code-Review+2: Bryan Strassner <strassner.bryan@gmail.com> Code-Review-1: Tin Lam <tin@irrational.io> Code-Review+2: Matt McEuen <matt.mceuen@att.com> Workflow+1: Matt McEuen <matt.mceuen@att.com> Verified+2: Zuul Submitted-by: Zuul Submitted-at: Sat, 26 Jan 2019 00:29:39 +0000 Reviewed-on: https://review.openstack.org/606131 Project: openstack/airship-pegleg Branch: refs/heads/master
-rw-r--r--.zuul.yaml16
-rw-r--r--doc/source/cli/cli.rst65
-rw-r--r--doc/source/developer-overview.rst8
-rw-r--r--doc/source/exceptions.rst8
-rw-r--r--doc/source/getting_started.rst7
-rw-r--r--doc/source/images/architecture-pegleg.pngbin37604 -> 37604 bytes
-rw-r--r--images/pegleg/Dockerfile4
-rw-r--r--pegleg/cli.py53
-rw-r--r--pegleg/config.py13
-rw-r--r--pegleg/engine/catalog/__init__.py17
-rw-r--r--pegleg/engine/catalog/pki_generator.py307
-rw-r--r--pegleg/engine/catalog/pki_utility.py330
-rw-r--r--pegleg/engine/common/__init__.py0
-rw-r--r--pegleg/engine/common/managed_document.py115
-rw-r--r--pegleg/engine/exceptions.py10
-rw-r--r--pegleg/engine/repository.py22
-rw-r--r--pegleg/engine/secrets.py5
-rw-r--r--pegleg/engine/util/__init__.py9
-rw-r--r--pegleg/engine/util/catalog.py52
-rw-r--r--pegleg/engine/util/deckhand.py4
-rw-r--r--pegleg/engine/util/definition.py1
-rw-r--r--pegleg/engine/util/git.py47
-rw-r--r--pegleg/engine/util/pegleg_secret_management.py8
-rw-r--r--pegleg/schemas/PKICatalog.yaml44
-rw-r--r--requirements.txt3
-rw-r--r--site_yamls/site/pki-catalog.yaml23
-rw-r--r--site_yamls/site/site-definition.yaml1
-rw-r--r--tests/unit/engine/test_secrets.py (renamed from tests/unit/engine/test_encryption.py)107
-rw-r--r--tests/unit/fixtures.py6
-rw-r--r--tests/unit/test_cli.py111
-rw-r--r--tools/gate/playbooks/install-cfssl.yaml23
-rwxr-xr-xtools/gate/whitespace-linter.sh1
-rwxr-xr-xtools/install-cfssl.sh22
-rw-r--r--tox.ini7
34 files changed, 1374 insertions, 75 deletions
diff --git a/.zuul.yaml b/.zuul.yaml
index 9356aca..5b3c8c5 100644
--- a/.zuul.yaml
+++ b/.zuul.yaml
@@ -18,11 +18,13 @@
18 check: 18 check:
19 jobs: 19 jobs:
20 - openstack-tox-pep8 20 - openstack-tox-pep8
21 - airship-pegleg-tox-py36
21 - airship-pegleg-doc-build 22 - airship-pegleg-doc-build
22 - airship-pegleg-docker-build-gate 23 - airship-pegleg-docker-build-gate
23 gate: 24 gate:
24 jobs: 25 jobs:
25 - openstack-tox-pep8 26 - openstack-tox-pep8
27 - airship-pegleg-tox-py36
26 - airship-pegleg-doc-build 28 - airship-pegleg-doc-build
27 - airship-pegleg-docker-build-gate 29 - airship-pegleg-docker-build-gate
28 post: 30 post:
@@ -36,6 +38,20 @@
36 label: ubuntu-xenial 38 label: ubuntu-xenial
37 39
38- job: 40- job:
41 name: airship-pegleg-tox-py36
42 description: |
43 Executes unit tests under Python 3.6
44 parent: openstack-tox-py36
45 pre-run:
46 - tools/gate/playbooks/install-cfssl.yaml
47 irrelevant-files:
48 - ^.*\.rst$
49 - ^doc/.*$
50 - ^etc/.*$
51 - ^releasenotes/.*$
52 - ^setup.cfg$
53
54- job:
39 name: airship-pegleg-doc-build 55 name: airship-pegleg-doc-build
40 description: | 56 description: |
41 Locally build the documentation to check for errors 57 Locally build the documentation to check for errors
diff --git a/doc/source/cli/cli.rst b/doc/source/cli/cli.rst
index 2e73125..02dcfda 100644
--- a/doc/source/cli/cli.rst
+++ b/doc/source/cli/cli.rst
@@ -81,10 +81,10 @@ CLI Options
81 81
82Enable debug logging. 82Enable debug logging.
83 83
84.. _site: 84.. _repo-group:
85 85
86Repo 86Repo Group
87==== 87==========
88 88
89Allows you to perform repository-level operations. 89Allows you to perform repository-level operations.
90 90
@@ -127,8 +127,10 @@ a specific site, see :ref:`site-level linting <cli-site-lint>`.
127 127
128See :ref:`linting` for more information. 128See :ref:`linting` for more information.
129 129
130Site 130.. _site-group:
131==== 131
132Site Group
133==========
132 134
133Allows you to perform site-level operations. 135Allows you to perform site-level operations.
134 136
@@ -303,7 +305,7 @@ Show details for one site.
303 305
304Name of site. 306Name of site.
305 307
306**-o /--output** (Optional). 308**-o/--output** (Optional).
307 309
308Where to output. 310Where to output.
309 311
@@ -331,7 +333,7 @@ Render documents via `Deckhand`_ for one site.
331 333
332Name of site. 334Name of site.
333 335
334**-o /--output** (Optional). 336**-o/--output** (Optional).
335 337
336Where to output. 338Where to output.
337 339
@@ -418,6 +420,39 @@ Usage:
418 420
419 ./pegleg.sh site <options> upload <site_name> --context-marker=<uuid> 421 ./pegleg.sh site <options> upload <site_name> --context-marker=<uuid>
420 422
423Site Secrets Group
424==================
425
426Subgroup of :ref:`site-group`.
427
428Generate PKI
429------------
430
431Generate certificates and keys according to all PKICatalog documents in the
432site using the PKI module. Regenerating certificates can be
433accomplished by re-running this command.
434
435Pegleg places generated document files in ``<site>/secrets/passphrases``,
436``<site>/secrets/certificates``, or ``<site>/secrets/keypairs`` as
437appropriate:
438
439* The generated filenames for passphrases will follow the pattern
440 :file:`<passphrase-doc-name>.yaml`.
441* The generated filenames for certificate authorities will follow the pattern
442 :file:`<ca-name>_ca.yaml`.
443* The generated filenames for certificates will follow the pattern
444 :file:`<ca-name>_<certificate-doc-name>_certificate.yaml`.
445* The generated filenames for certificate keys will follow the pattern
446 :file:`<ca-name>_<certificate-doc-name>_key.yaml`.
447* The generated filenames for keypairs will follow the pattern
448 :file:`<keypair-doc-name>.yaml`.
449
450Dashes in the document names will be converted to underscores for consistency.
451
452**site_name** (Required).
453
454Name of site.
455
421Examples 456Examples
422^^^^^^^^ 457^^^^^^^^
423 458
@@ -427,6 +462,14 @@ Examples
427 upload <site_name> <options> 462 upload <site_name> <options>
428 463
429 464
465::
466
467 ./pegleg.sh site -r <site_repo> -e <extra_repo> \
468 secrets generate-pki \
469 <site_name> \
470 -o <output> \
471 -f <filename>
472
430.. _command-line-repository-overrides: 473.. _command-line-repository-overrides:
431 474
432Secrets 475Secrets
@@ -571,13 +614,13 @@ Example:
571 614
572 615
573CLI Repository Overrides 616CLI Repository Overrides
574------------------------ 617========================
575 618
576Repository overrides should only be used for entries included underneath 619Repository overrides should only be used for entries included underneath
577the ``repositories`` field for a given :file:`site-definition.yaml`. 620the ``repositories`` field for a given :file:`site-definition.yaml`.
578 621
579Overrides are specified via the ``-e`` flag for all :ref:`site` commands. They 622Overrides are specified via the ``-e`` flag for all :ref:`site-group` commands.
580have the following format: 623They have the following format:
581 624
582:: 625::
583 626
@@ -611,7 +654,7 @@ Where:
611.. _self-contained-repo: 654.. _self-contained-repo:
612 655
613Self-Contained Repository 656Self-Contained Repository
614^^^^^^^^^^^^^^^^^^^^^^^^^ 657-------------------------
615 658
616For self-contained repositories, specification of extra repositories is 659For self-contained repositories, specification of extra repositories is
617unnecessary. The following command can be used to deploy the manifests in 660unnecessary. The following command can be used to deploy the manifests in
diff --git a/doc/source/developer-overview.rst b/doc/source/developer-overview.rst
index a1e5d08..e5735ec 100644
--- a/doc/source/developer-overview.rst
+++ b/doc/source/developer-overview.rst
@@ -100,8 +100,8 @@ directory):
100 100
101.. code-block:: console 101.. code-block:: console
102 102
103 # Quick way of building a venv and installing all required dependencies into 103 # Quick way of building a virtualenv and installing all required
104 # it. 104 # dependencies into it.
105 tox -e py36 --notest 105 tox -e py36 --notest
106 source .tox/py36/bin/activate 106 source .tox/py36/bin/activate
107 pip install -e . 107 pip install -e .
@@ -128,11 +128,11 @@ Unit Tests
128 128
129To run all unit tests, execute:: 129To run all unit tests, execute::
130 130
131 $ tox -epy36 131 $ tox -e py36
132 132
133To run unit tests using a regex, execute:: 133To run unit tests using a regex, execute::
134 134
135 $ tox -epy36 -- <regex> 135 $ tox -e py36 -- <regex>
136 136
137.. _Airship: https://airshipit.readthedocs.io 137.. _Airship: https://airshipit.readthedocs.io
138.. _Deckhand: https://airship-deckhand.readthedocs.io/ 138.. _Deckhand: https://airship-deckhand.readthedocs.io/
diff --git a/doc/source/exceptions.rst b/doc/source/exceptions.rst
index 8fb8577..ef2f1ad 100644
--- a/doc/source/exceptions.rst
+++ b/doc/source/exceptions.rst
@@ -63,3 +63,11 @@ Authentication Exceptions
63.. autoexception:: pegleg.engine.util.shipyard_helper.AuthValuesError 63.. autoexception:: pegleg.engine.util.shipyard_helper.AuthValuesError
64 :members: 64 :members:
65 :undoc-members: 65 :undoc-members:
66
67PKI Exceptions
68--------------
69
70.. autoexception:: pegleg.engine.exceptions.IncompletePKIPairError
71 :members:
72 :show-inheritance:
73 :undoc-members:
diff --git a/doc/source/getting_started.rst b/doc/source/getting_started.rst
index bd54cbb..39c07d7 100644
--- a/doc/source/getting_started.rst
+++ b/doc/source/getting_started.rst
@@ -21,13 +21,14 @@ Getting Started
21What is Pegleg? 21What is Pegleg?
22--------------- 22---------------
23 23
24Pegleg is a document aggregator that will aggregate all the documents in a 24Pegleg is a document aggregator that aggregates all the documents in a
25repository and pack them into a single YAML file. This allows for operators to 25repository and packs them into a single YAML file. This allows for operators to
26structure their site definitions in a maintainable directory layout, while 26structure their site definitions in a maintainable directory layout, while
27providing them with the automation and tooling needed to aggregate, lint, and 27providing them with the automation and tooling needed to aggregate, lint, and
28render those documents for deployment. 28render those documents for deployment.
29 29
30For more information on the documents that Pegleg works on see `Document Fundamentals`_. 30For more information on the documents that Pegleg works on see
31`Document Fundamentals`_.
31 32
32Basic Usage 33Basic Usage
33----------- 34-----------
diff --git a/doc/source/images/architecture-pegleg.png b/doc/source/images/architecture-pegleg.png
index 8b526cc..c872f55 100644
--- a/doc/source/images/architecture-pegleg.png
+++ b/doc/source/images/architecture-pegleg.png
Binary files differ
diff --git a/images/pegleg/Dockerfile b/images/pegleg/Dockerfile
index 62c4dc0..a7ede75 100644
--- a/images/pegleg/Dockerfile
+++ b/images/pegleg/Dockerfile
@@ -1,5 +1,6 @@
1ARG FROM=python:3.6 1ARG FROM=python:3.6
2FROM ${FROM} 2FROM ${FROM}
3ARG CFSSLURL=https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
3 4
4LABEL org.opencontainers.image.authors='airship-discuss@lists.airshipit.org, irc://#airshipit@freenode' 5LABEL org.opencontainers.image.authors='airship-discuss@lists.airshipit.org, irc://#airshipit@freenode'
5LABEL org.opencontainers.image.url='https://airshipit.org' 6LABEL org.opencontainers.image.url='https://airshipit.org'
@@ -14,5 +15,8 @@ WORKDIR /var/pegleg
14COPY requirements.txt /opt/pegleg/requirements.txt 15COPY requirements.txt /opt/pegleg/requirements.txt
15RUN pip3 install --no-cache-dir -r /opt/pegleg/requirements.txt 16RUN pip3 install --no-cache-dir -r /opt/pegleg/requirements.txt
16 17
18COPY tools/install-cfssl.sh /opt/pegleg/tools/install-cfssl.sh
19RUN /opt/pegleg/tools/install-cfssl.sh ${CFSSLURL}
20
17COPY . /opt/pegleg 21COPY . /opt/pegleg
18RUN pip3 install -e /opt/pegleg 22RUN pip3 install -e /opt/pegleg
diff --git a/pegleg/cli.py b/pegleg/cli.py
index 6a03a70..7897802 100644
--- a/pegleg/cli.py
+++ b/pegleg/cli.py
@@ -20,6 +20,7 @@ import click
20 20
21from pegleg import config 21from pegleg import config
22from pegleg import engine 22from pegleg import engine
23from pegleg.engine import catalog
23from pegleg.engine.util.shipyard_helper import ShipyardHelper 24from pegleg.engine.util.shipyard_helper import ShipyardHelper
24 25
25LOG = logging.getLogger(__name__) 26LOG = logging.getLogger(__name__)
@@ -130,7 +131,6 @@ def main(*, verbose):
130 131
131 * site: site-level actions 132 * site: site-level actions
132 * repo: repository-level actions 133 * repo: repository-level actions
133 * stub (DEPRECATED)
134 134
135 """ 135 """
136 136
@@ -208,7 +208,7 @@ def site(*, site_repository, clone_path, extra_repositories, repo_key,
208 * list: list available sites in a manifests repo 208 * list: list available sites in a manifests repo
209 * lint: lint a site along with all its dependencies 209 * lint: lint a site along with all its dependencies
210 * render: render a site using Deckhand 210 * render: render a site using Deckhand
211 * show: show a sites' files 211 * show: show a site's files
212 212
213 """ 213 """
214 214
@@ -375,6 +375,39 @@ def upload(ctx, *, os_project_domain_name,
375 click.echo(ShipyardHelper(ctx).upload_documents()) 375 click.echo(ShipyardHelper(ctx).upload_documents())
376 376
377 377
378@site.group(name='secrets', help='Commands to manage site secrets documents')
379def secrets():
380 pass
381
382
383@secrets.command(
384 'generate-pki',
385 help="""
386Generate certificates and keys according to all PKICatalog documents in the
387site. Regenerating certificates can be accomplished by re-running this command.
388""")
389@click.option(
390 '-a',
391 '--author',
392 'author',
393 help="""Identifying name of the author generating new certificates. Used
394for tracking provenance information in the PeglegManagedDocuments. An attempt
395is made to automatically determine this value, but should be provided.""")
396@click.argument('site_name')
397def generate_pki(site_name, author):
398 """Generate certificates, certificate authorities and keypairs for a given
399 site.
400
401 """
402
403 engine.repository.process_repositories(site_name,
404 overwrite_existing=True)
405 pkigenerator = catalog.pki_generator.PKIGenerator(site_name, author=author)
406 output_paths = pkigenerator.generate()
407
408 click.echo("Generated PKI files written to:\n%s" % '\n'.join(output_paths))
409
410
378@main.group(help='Commands related to types') 411@main.group(help='Commands related to types')
379@MAIN_REPOSITORY_OPTION 412@MAIN_REPOSITORY_OPTION
380@REPOSITORY_CLONE_PATH_OPTION 413@REPOSITORY_CLONE_PATH_OPTION
@@ -409,11 +442,6 @@ def list_types(*, output_stream):
409 engine.type.list_types(output_stream) 442 engine.type.list_types(output_stream)
410 443
411 444
412@site.group(name='secrets', help='Commands to manage site secrets documents')
413def secrets():
414 pass
415
416
417@secrets.command( 445@secrets.command(
418 'encrypt', 446 'encrypt',
419 help='Command to encrypt and wrap site secrets ' 447 help='Command to encrypt and wrap site secrets '
@@ -437,7 +465,9 @@ def secrets():
437 'documents') 465 'documents')
438@click.argument('site_name') 466@click.argument('site_name')
439def encrypt(*, save_location, author, site_name): 467def encrypt(*, save_location, author, site_name):
440 engine.repository.process_repositories(site_name) 468 engine.repository.process_repositories(site_name, overwrite_existing=True)
469 if save_location is None:
470 save_location = config.get_site_repo()
441 engine.secrets.encrypt(save_location, author, site_name) 471 engine.secrets.encrypt(save_location, author, site_name)
442 472
443 473
@@ -453,4 +483,9 @@ def encrypt(*, save_location, author, site_name):
453@click.argument('site_name') 483@click.argument('site_name')
454def decrypt(*, file_name, site_name): 484def decrypt(*, file_name, site_name):
455 engine.repository.process_repositories(site_name) 485 engine.repository.process_repositories(site_name)
456 engine.secrets.decrypt(file_name, site_name) 486 try:
487 click.echo(engine.secrets.decrypt(file_name, site_name))
488 except FileNotFoundError:
489 raise click.exceptions.FileError("Couldn't find file %s, "
490 "check your arguments and try "
491 "again." % file_name)
diff --git a/pegleg/config.py b/pegleg/config.py
index 8cf0a61..560f437 100644
--- a/pegleg/config.py
+++ b/pegleg/config.py
@@ -25,7 +25,8 @@ except NameError:
25 'extra_repos': [], 25 'extra_repos': [],
26 'clone_path': None, 26 'clone_path': None,
27 'site_path': 'site', 27 'site_path': 'site',
28 'type_path': 'type' 28 'site_rev': None,
29 'type_path': 'type',
29 } 30 }
30 31
31 32
@@ -49,6 +50,16 @@ def set_clone_path(p):
49 GLOBAL_CONTEXT['clone_path'] = p 50 GLOBAL_CONTEXT['clone_path'] = p
50 51
51 52
53def get_site_rev():
54 """Get site revision derived from the site repo URL/path, if provided."""
55 return GLOBAL_CONTEXT['site_rev']
56
57
58def set_site_rev(r):
59 """Set site revision derived from the site repo URL/path."""
60 GLOBAL_CONTEXT['site_rev'] = r
61
62
52def get_extra_repo_overrides(): 63def get_extra_repo_overrides():
53 """Get extra repository overrides specified via ``-e`` CLI flag.""" 64 """Get extra repository overrides specified via ``-e`` CLI flag."""
54 return GLOBAL_CONTEXT.get('extra_repo_overrides', []) 65 return GLOBAL_CONTEXT.get('extra_repo_overrides', [])
diff --git a/pegleg/engine/catalog/__init__.py b/pegleg/engine/catalog/__init__.py
new file mode 100644
index 0000000..37cc7e8
--- /dev/null
+++ b/pegleg/engine/catalog/__init__.py
@@ -0,0 +1,17 @@
1# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15# flake8: noqa
16from pegleg.engine.catalog import pki_utility
17from pegleg.engine.catalog import pki_generator
diff --git a/pegleg/engine/catalog/pki_generator.py b/pegleg/engine/catalog/pki_generator.py
new file mode 100644
index 0000000..7ef26e6
--- /dev/null
+++ b/pegleg/engine/catalog/pki_generator.py
@@ -0,0 +1,307 @@
1# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15import collections
16import itertools
17import logging
18import os
19
20import yaml
21
22from pegleg import config
23from pegleg.engine.catalog import pki_utility
24from pegleg.engine.common import managed_document as md
25from pegleg.engine import exceptions
26from pegleg.engine import util
27from pegleg.engine.util.pegleg_managed_document import \
28 PeglegManagedSecretsDocument
29
30__all__ = ['PKIGenerator']
31
32LOG = logging.getLogger(__name__)
33
34
35class PKIGenerator(object):
36 """Generates certificates, certificate authorities and keypairs using
37 the ``PKIUtility`` class.
38
39 Pegleg searches through a given "site" to derive all the documents
40 of kind ``PKICatalog``, which are in turn parsed for information related
41 to the above secret types and passed to ``PKIUtility`` for generation.
42
43 These secrets are output to various subdirectories underneath
44 ``<site>/secrets/<subpath>``.
45
46 """
47
48 def __init__(self, sitename, block_strings=True, author=None):
49 """Constructor for ``PKIGenerator``.
50
51 :param str sitename: Site name for which to retrieve documents used for
52 certificate and keypair generation.
53 :param bool block_strings: Whether to dump out certificate data as
54 block-style YAML string. Defaults to true.
55 :param str author: Identifying name of the author generating new
56 certificates.
57
58 """
59
60 self._sitename = sitename
61 self._documents = util.definition.documents_for_site(sitename)
62 self._author = author
63
64 self.keys = pki_utility.PKIUtility(block_strings=block_strings)
65 self.outputs = collections.defaultdict(dict)
66
67 # Maps certificates to CAs in order to derive certificate paths.
68 self._cert_to_ca_map = {}
69
70 def generate(self):
71 for catalog in util.catalog.iterate(
72 documents=self._documents, kind='PKICatalog'):
73 for ca_name, ca_def in catalog['data'].get(
74 'certificate_authorities', {}).items():
75 ca_cert, ca_key = self.get_or_gen_ca(ca_name)
76
77 for cert_def in ca_def.get('certificates', []):
78 document_name = cert_def['document_name']
79 self._cert_to_ca_map.setdefault(document_name, ca_name)
80 cert, key = self.get_or_gen_cert(
81 document_name,
82 ca_cert=ca_cert,
83 ca_key=ca_key,
84 cn=cert_def['common_name'],
85 hosts=_extract_hosts(cert_def),
86 groups=cert_def.get('groups', []))
87
88 for keypair_def in catalog['data'].get('keypairs', []):
89 document_name = keypair_def['name']
90 self.get_or_gen_keypair(document_name)
91
92 return self._write(config.get_site_repo())
93
94 def get_or_gen_ca(self, document_name):
95 kinds = [
96 'CertificateAuthority',
97 'CertificateAuthorityKey',
98 ]
99 return self._get_or_gen(self.gen_ca, kinds, document_name)
100
101 def get_or_gen_cert(self, document_name, **kwargs):
102 kinds = [
103 'Certificate',
104 'CertificateKey',
105 ]
106 return self._get_or_gen(self.gen_cert, kinds, document_name, **kwargs)
107
108 def get_or_gen_keypair(self, document_name):
109 kinds = [
110 'PublicKey',
111 'PrivateKey',
112 ]
113 return self._get_or_gen(self.gen_keypair, kinds, document_name)
114
115 def gen_ca(self, document_name, **kwargs):
116 return self.keys.generate_ca(document_name, **kwargs)
117
118 def gen_cert(self, document_name, *, ca_cert, ca_key, **kwargs):
119 ca_cert_data = ca_cert['data']['managedDocument']['data']
120 ca_key_data = ca_key['data']['managedDocument']['data']
121 return self.keys.generate_certificate(
122 document_name, ca_cert=ca_cert_data, ca_key=ca_key_data, **kwargs)
123
124 def gen_keypair(self, document_name):
125 return self.keys.generate_keypair(document_name)
126
127 def _get_or_gen(self, generator, kinds, document_name, *args, **kwargs):
128 docs = self._find_docs(kinds, document_name)
129 if not docs:
130 docs = generator(document_name, *args, **kwargs)
131 else:
132 docs = [PeglegManagedSecretsDocument(doc).pegleg_document
133 for doc in docs]
134
135 # Adding these to output should be idempotent, so we use a dict.
136
137 for wrapper_doc in docs:
138 wrapped_doc = wrapper_doc['data']['managedDocument']
139 schema = wrapped_doc['schema']
140 name = wrapped_doc['metadata']['name']
141 self.outputs[schema][name] = wrapper_doc
142
143 return docs
144
145 def _find_docs(self, kinds, document_name):
146 schemas = ['deckhand/%s/v1' % k for k in kinds]
147 docs = self._find_among_collected(schemas, document_name)
148 if docs:
149 if len(docs) == len(kinds):
150 LOG.debug('Found docs in input config named %s, kinds: %s',
151 document_name, kinds)
152 return docs
153 else:
154 raise exceptions.IncompletePKIPairError(
155 kinds=kinds, name=document_name)
156
157 else:
158 docs = self._find_among_outputs(schemas, document_name)
159 if docs:
160 LOG.debug('Found docs in current outputs named %s, kinds: %s',
161 document_name, kinds)
162 return docs
163 # TODO(felipemonteiro): Should this be a critical error?
164 LOG.debug('No docs existing docs named %s, kinds: %s', document_name,
165 kinds)
166 return []
167
168 def _find_among_collected(self, schemas, document_name):
169 result = []
170 for schema in schemas:
171 doc = _find_document_by(
172 self._documents, schema=schema, name=document_name)
173 # If the document wasn't found, then means it needs to be
174 # generated.
175 if doc:
176 result.append(doc)
177 return result
178
179 def _find_among_outputs(self, schemas, document_name):
180 result = []
181 for schema in schemas:
182 if document_name in self.outputs.get(schema, {}):
183 result.append(self.outputs[schema][document_name])
184 return result
185
186 def _write(self, output_dir):
187 documents = self.get_documents()
188 output_paths = set()
189
190 # First, delete each of the output paths below because we do an append
191 # action in the `open` call below. This means that for regeneration
192 # of certs, the original paths must be deleted.
193 for document in documents:
194 output_file_path = md.get_document_path(
195 sitename=self._sitename,
196 wrapper_document=document,
197 cert_to_ca_map=self._cert_to_ca_map)
198 output_path = os.path.join(output_dir, 'site', output_file_path)
199 # NOTE(felipemonteiro): This is currently an entirely safe
200 # operation as these files are being removed in the temporarily
201 # replicated versions of the local repositories.
202 if os.path.exists(output_path):
203 os.remove(output_path)
204
205 # Next, generate (or regenerate) the certificates.
206 for document in documents:
207 output_file_path = md.get_document_path(
208 sitename=self._sitename,
209 wrapper_document=document,
210 cert_to_ca_map=self._cert_to_ca_map)
211 output_path = os.path.join(output_dir, 'site', output_file_path)
212 dir_name = os.path.dirname(output_path)
213
214 if not os.path.exists(dir_name):
215 LOG.debug('Creating secrets path: %s', dir_name)
216 os.makedirs(dir_name)
217
218 with open(output_path, 'a') as f:
219 # Don't use safe_dump so we can block format certificate
220 # data.
221 yaml.dump(
222 document,
223 stream=f,
224 default_flow_style=False,
225 explicit_start=True,
226 indent=2)
227
228 output_paths.add(output_path)
229 return output_paths
230
231 def get_documents(self):
232 return list(
233 itertools.chain.from_iterable(
234 v.values() for v in self.outputs.values()))
235
236
237def get_host_list(service_names):
238 service_list = []
239 for service in service_names:
240 parts = service.split('.')
241 for i in range(len(parts)):
242 service_list.append('.'.join(parts[:i + 1]))
243 return service_list
244
245
246def _extract_hosts(cert_def):
247 hosts = cert_def.get('hosts', [])
248 hosts.extend(get_host_list(cert_def.get('kubernetes_service_names', [])))
249 return hosts
250
251
252def _find_document_by(documents, **kwargs):
253 try:
254 return next(_iterate(documents, **kwargs))
255 except StopIteration:
256 return None
257
258
259def _iterate(documents, *, kind=None, schema=None, labels=None, name=None):
260 if kind is not None:
261 if schema is not None:
262 raise AssertionError('Logic error: specified both kind and schema')
263 schema = 'promenade/%s/v1' % kind
264
265 for document in documents:
266 if _matches_filter(document, schema=schema, labels=labels, name=name):
267 yield document
268
269
270def _matches_filter(document, *, schema, labels, name):
271 matches = True
272
273 if md.is_managed_document(document):
274 document = document['data']['managedDocument']
275 else:
276 document_schema = document['schema']
277 if document_schema in md.SUPPORTED_SCHEMAS:
278 # Can't use the filter value as they might not be an exact match.
279 document_metadata = document['metadata']
280 document_labels = document_metadata.get('labels', {})
281 document_name = document_metadata['name']
282 LOG.warning('Detected deprecated unmanaged document during PKI '
283 'generation. Details: schema=%s, name=%s, labels=%s.',
284 document_schema, document_labels, document_name)
285
286 if schema is not None and not document.get('schema',
287 '').startswith(schema):
288 matches = False
289
290 if labels is not None:
291 document_labels = _mg(document, 'labels', [])
292 for key, value in labels.items():
293 if key not in document_labels:
294 matches = False
295 else:
296 if document_labels[key] != value:
297 matches = False
298
299 if name is not None:
300 if _mg(document, 'name') != name:
301 matches = False
302
303 return matches
304
305
306def _mg(document, field, default=None):
307 return document.get('metadata', {}).get(field, default)
diff --git a/pegleg/engine/catalog/pki_utility.py b/pegleg/engine/catalog/pki_utility.py
new file mode 100644
index 0000000..780370f
--- /dev/null
+++ b/pegleg/engine/catalog/pki_utility.py
@@ -0,0 +1,330 @@
1# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15from datetime import datetime
16import json
17import logging
18import os
19# Ignore bandit false positive: B404:blacklist
20# The purpose of this module is to safely encapsulate calls via fork.
21import subprocess # nosec
22import tempfile
23
24from dateutil import parser
25import pytz
26import yaml
27
28from pegleg.engine.util.pegleg_managed_document import \
29 PeglegManagedSecretsDocument
30
31LOG = logging.getLogger(__name__)
32_ONE_YEAR_IN_HOURS = '8760h' # 365 * 24
33
34__all__ = ['PKIUtility']
35
36
37# TODO(felipemonteiro): Create an abstract base class for other future Catalog
38# classes.
39
40
41class PKIUtility(object):
42 """Public Key Infrastructure utility class.
43
44 Responsible for generating certificate and CA documents using ``cfssl`` and
45 keypairs using ``openssl``. These secrets are all wrapped in instances
46 of ``pegleg/PeglegManagedDocument/v1``.
47
48 """
49
50 @staticmethod
51 def cfssl_exists():
52 """Checks whether cfssl command exists. Useful for testing."""
53 try:
54 subprocess.check_output( # nosec
55 ['which', 'cfssl'], stderr=subprocess.STDOUT)
56 return True
57 except subprocess.CalledProcessError:
58 return False
59
60 def __init__(self, *, block_strings=True):
61 self.block_strings = block_strings
62 self._ca_config_string = None
63
64 @property
65 def ca_config(self):
66 if not self._ca_config_string:
67 self._ca_config_string = json.dumps({
68 'signing': {
69 'default': {
70 # TODO(felipemonteiro): Make this configurable.
71 'expiry':
72 _ONE_YEAR_IN_HOURS,
73 'usages': [
74 'signing', 'key encipherment', 'server auth',
75 'client auth'
76 ],
77 },
78 },
79 })
80 return self._ca_config_string
81
82 def generate_ca(self, ca_name):
83 """Generate CA cert and associated key.
84
85 :param str ca_name: Name of Certificate Authority in wrapped document.
86 :returns: Tuple of (wrapped CA cert, wrapped CA key)
87 :rtype: tuple[dict, dict]
88
89 """
90
91 result = self._cfssl(
92 ['gencert', '-initca', 'csr.json'],
93 files={
94 'csr.json': self.csr(name=ca_name),
95 })
96
97 return (self._wrap_ca(ca_name, result['cert']),
98 self._wrap_ca_key(ca_name, result['key']))
99
100 def generate_keypair(self, name):
101 """Generate keypair.
102
103 :param str name: Name of keypair in wrapped document.
104 :returns: Tuple of (wrapped public key, wrapped private key)
105 :rtype: tuple[dict, dict]
106
107 """
108
109 priv_result = self._openssl(['genrsa', '-out', 'priv.pem'])
110 pub_result = self._openssl(
111 ['rsa', '-in', 'priv.pem', '-pubout', '-out', 'pub.pem'],
112 files={
113 'priv.pem': priv_result['priv.pem'],
114 })
115
116 return (self._wrap_pub_key(name, pub_result['pub.pem']),
117 self._wrap_priv_key(name, priv_result['priv.pem']))
118
119 def generate_certificate(self,
120 name,
121 *,
122 ca_cert,
123 ca_key,
124 cn,
125 groups=None,
126 hosts=None):
127 """Generate certificate and associated key given CA cert and key.
128
129 :param str name: Name of certificate in wrapped document.
130 :param str ca_cert: CA certificate.
131 :param str ca_key: CA certificate key.
132 :param str cn: Common name associated with certificate.
133 :param list groups: List of groups associated with certificate.
134 :param list hosts: List of hosts associated with certificate.
135 :returns: Tuple of (wrapped certificate, wrapped certificate key)
136 :rtype: tuple[dict, dict]
137
138 """
139
140 if groups is None:
141 groups = []
142 if hosts is None:
143 hosts = []
144
145 result = self._cfssl(
146 [
147 'gencert', '-ca', 'ca.pem', '-ca-key', 'ca-key.pem', '-config',
148 'ca-config.json', 'csr.json'
149 ],
150 files={
151 'ca-config.json': self.ca_config,
152 'ca.pem': ca_cert,
153 'ca-key.pem': ca_key,
154 'csr.json': self.csr(name=cn, groups=groups, hosts=hosts),
155 })
156
157 return (self._wrap_cert(name, result['cert']),
158 self._wrap_cert_key(name, result['key']))
159
160 def csr(self,
161 *,
162 name,
163 groups=None,
164 hosts=None,
165 key={
166 'algo': 'rsa',
167 'size': 2048
168 }):
169 if groups is None:
170 groups = []
171 if hosts is None:
172 hosts = []
173
174 return json.dumps({
175 'CN': name,
176 'key': key,
177 'hosts': hosts,
178 'names': [{
179 'O': g
180 } for g in groups],
181 })
182
183 def cert_info(self, cert):
184 """Retrieve certificate info via ``cfssl``.
185
186 :param str cert: Client certificate that contains the public key.
187 :returns: Information related to certificate.
188 :rtype: dict
189
190 """
191
192 return self._cfssl(
193 ['certinfo', '-cert', 'cert.pem'], files={
194 'cert.pem': cert,
195 })
196
197 def check_expiry(self, cert):
198 """Chek whether a given certificate is expired.
199
200 :param str cert: Client certificate that contains the public key.
201 :returns: True if certificate is expired, else False.
202 :rtype: bool
203
204 """
205
206 info = self.cert_info(cert)
207 expiry_str = info['not_after']
208 expiry = parser.parse(expiry_str)
209 # expiry is timezone-aware; do the same for `now`.
210 now = pytz.utc.localize(datetime.utcnow())
211 return now > expiry
212
213 def _cfssl(self, command, *, files=None):
214 """Executes ``cfssl`` command via ``subprocess`` call."""
215 if not files:
216 files = {}
217 with tempfile.TemporaryDirectory() as tmp:
218 for filename, data in files.items():
219 with open(os.path.join(tmp, filename), 'w') as f:
220 f.write(data)
221
222 # Ignore bandit false positive:
223 # B603:subprocess_without_shell_equals_true
224 # This method wraps cfssl calls originating from this module.
225 result = subprocess.check_output( # nosec
226 ['cfssl'] + command, cwd=tmp, stderr=subprocess.PIPE)
227 if not isinstance(result, str):
228 result = result.decode('utf-8')
229 return json.loads(result)
230
231 def _openssl(self, command, *, files=None):
232 """Executes ``openssl`` command via ``subprocess`` call."""
233 if not files:
234 files = {}
235
236 with tempfile.TemporaryDirectory() as tmp:
237 for filename, data in files.items():
238 with open(os.path.join(tmp, filename), 'w') as f:
239 f.write(data)
240
241 # Ignore bandit false positive:
242 # B603:subprocess_without_shell_equals_true
243 # This method wraps openssl calls originating from this module.
244 subprocess.check_call( # nosec
245 ['openssl'] + command,
246 cwd=tmp,
247 stderr=subprocess.PIPE)
248
249 result = {}
250 for filename in os.listdir(tmp):
251 if filename not in files:
252 with open(os.path.join(tmp, filename)) as f:
253 result[filename] = f.read()
254
255 return result
256
257 def _wrap_ca(self, name, data):
258 return self.wrap_document(kind='CertificateAuthority', name=name,
259 data=data, block_strings=self.block_strings)
260
261 def _wrap_ca_key(self, name, data):
262 return self.wrap_document(kind='CertificateAuthorityKey', name=name,
263 data=data, block_strings=self.block_strings)
264
265 def _wrap_cert(self, name, data):
266 return self.wrap_document(kind='Certificate', name=name, data=data,
267 block_strings=self.block_strings)
268
269 def _wrap_cert_key(self, name, data):
270 return self.wrap_document(kind='CertificateKey', name=name, data=data,
271 block_strings=self.block_strings)
272
273 def _wrap_priv_key(self, name, data):
274 return self.wrap_document(kind='PrivateKey', name=name, data=data,
275 block_strings=self.block_strings)
276
277 def _wrap_pub_key(self, name, data):
278 return self.wrap_document(kind='PublicKey', name=name, data=data,
279 block_strings=self.block_strings)
280
281 @staticmethod
282 def wrap_document(kind, name, data, block_strings=True):
283 """Wrap document ``data`` with PeglegManagedDocument pattern.
284
285 :param str kind: The kind of document (found in ``schema``).
286 :param str name: Name of the document.
287 :param dict data: Document data.
288 :param bool block_strings: Whether to dump out certificate data as
289 block-style YAML string. Defaults to true.
290 :return: the wrapped document
291 :rtype: dict
292 """
293
294 wrapped_schema = 'deckhand/%s/v1' % kind
295 wrapped_metadata = {
296 'schema': 'metadata/Document/v1',
297 'name': name,
298 'layeringDefinition': {
299 'abstract': False,
300 'layer': 'site',
301 }
302 }
303 wrapped_data = PKIUtility._block_literal(
304 data, block_strings=block_strings)
305
306 document = {
307 "schema": wrapped_schema,
308 "metadata": wrapped_metadata,
309 "data": wrapped_data
310 }
311
312 return PeglegManagedSecretsDocument(document).pegleg_document
313
314 @staticmethod
315 def _block_literal(data, block_strings=True):
316 if block_strings:
317 return block_literal(data)
318 else:
319 return data
320
321
322class block_literal(str):
323 pass
324
325
326def block_literal_representer(dumper, data):
327 return dumper.represent_scalar('tag:yaml.org,2002:str', data, style='|')
328
329
330yaml.add_representer(block_literal, block_literal_representer)
diff --git a/pegleg/engine/common/__init__.py b/pegleg/engine/common/__init__.py
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/pegleg/engine/common/__init__.py
diff --git a/pegleg/engine/common/managed_document.py b/pegleg/engine/common/managed_document.py
new file mode 100644
index 0000000..76b3946
--- /dev/null
+++ b/pegleg/engine/common/managed_document.py
@@ -0,0 +1,115 @@
1# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15import os
16
17from pegleg import config
18from pegleg.engine.util import git
19
20MANAGED_DOCUMENT_SCHEMA = 'pegleg/PeglegManagedDocument/v1'
21SUPPORTED_SCHEMAS = (
22 'deckhand/CertificateAuthority/v1',
23 'deckhand/CertificateAuthorityKey/v1',
24 'deckhand/Certificate/v1',
25 'deckhand/CertificateKey/v1',
26 'deckhand/PublicKey/v1',
27 'deckhand/PrivateKey/v1',
28)
29
30_KIND_TO_PATH = {
31 'CertificateAuthority': 'certificates',
32 'CertificateAuthorityKey': 'certificates',
33 'Certificate': 'certificates',
34 'CertificateKey': 'certificates',
35 'PublicKey': 'keypairs',
36 'PrivateKey': 'keypairs'
37}
38
39
40def is_managed_document(document):
41 """Utility for determining whether a document is wrapped by
42 ``pegleg/PeglegManagedDocument/v1`` pattern.
43
44 :param dict document: Document to check.
45 :returns: True if document is managed, else False.
46 :rtype: bool
47
48 """
49
50 return document.get('schema') == "pegleg/PeglegManagedDocument/v1"
51
52
53def get_document_path(sitename, wrapper_document, cert_to_ca_map=None):
54 """Get path for outputting generated certificates or keys to.
55
56 Also updates the provenance path (``data.generated.specifiedBy.path``)
57 for ``wrapper_document``.
58
59 * Certificates ar written to: ``<site>/secrets/certificates``
60 * Keypairs are written to: ``<site>/secrets/keypairs``
61 * Passphrases are written to: ``<site>/secrets/passphrases``
62
63 * The generated filenames for passphrases will follow the pattern
64 ``<passphrase-doc-name>.yaml``.
65 * The generated filenames for certificate authorities will follow the
66 pattern ``<ca-name>_ca.yaml``.
67 * The generated filenames for certificates will follow the pattern
68 ``<ca-name>_<certificate-doc-name>_certificate.yaml``.
69 * The generated filenames for certificate keys will follow the pattern
70 ``<ca-name>_<certificate-doc-name>_key.yaml``.
71 * The generated filenames for keypairs will follow the pattern
72 ``<keypair-doc-name>.yaml``.
73
74 :param str sitename: Name of site.
75 :param dict wrapper_document: Generated ``PeglegManagedDocument``.
76 :param dict cert_to_ca_map: Dict that maps certificate names to
77 their respective CA name.
78 :returns: Path to write document out to.
79 :rtype: str
80
81 """
82
83 cert_to_ca_map = cert_to_ca_map or {}
84
85 managed_document = wrapper_document['data']['managedDocument']
86 kind = managed_document['schema'].split("/")[1]
87 name = managed_document['metadata']['name']
88
89 path = "%s/secrets/%s" % (sitename, _KIND_TO_PATH[kind])
90
91 if 'authority' in kind.lower():
92 filename_structure = '%s_ca.yaml'
93 elif 'certificate' in kind.lower():
94 ca_name = cert_to_ca_map[name]
95 filename_structure = ca_name + '_%s_certificate.yaml'
96 elif 'public' in kind.lower() or 'private' in kind.lower():
97 filename_structure = '%s.yaml'
98
99 # Dashes in the document names are converted to underscores for
100 # consistency.
101 filename = (filename_structure % name).replace('-', '_')
102 fullpath = os.path.join(path, filename)
103
104 # Not all managed documents are generated. Only update path provenance
105 # information for those that are.
106 if wrapper_document['data'].get('generated'):
107 wrapper_document['data']['generated']['specifiedBy']['path'] = fullpath
108 return fullpath
109
110
111def _get_repo_url_and_rev():
112 repo_path_or_url = config.get_site_repo()
113 repo_url = git.repo_url(repo_path_or_url)
114 repo_rev = config.get_site_rev()
115 return repo_url, repo_rev
diff --git a/pegleg/engine/exceptions.py b/pegleg/engine/exceptions.py
index 6ad7892..2539101 100644
--- a/pegleg/engine/exceptions.py
+++ b/pegleg/engine/exceptions.py
@@ -65,3 +65,13 @@ class GitConfigException(PeglegBaseException):
65class GitInvalidRepoException(PeglegBaseException): 65class GitInvalidRepoException(PeglegBaseException):
66 """Exception raised when an invalid repository is detected.""" 66 """Exception raised when an invalid repository is detected."""
67 message = 'The repository path or URL is invalid: %(repo_path)s' 67 message = 'The repository path or URL is invalid: %(repo_path)s'
68
69
70#
71# PKI EXCEPTIONS
72#
73
74
75class IncompletePKIPairError(PeglegBaseException):
76 """Exception for incomplete private/public keypair."""
77 message = ("Incomplete keypair set %(kinds)s for name: %(name)s")
diff --git a/pegleg/engine/repository.py b/pegleg/engine/repository.py
index 7c8bdfd..b7ea063 100644
--- a/pegleg/engine/repository.py
+++ b/pegleg/engine/repository.py
@@ -42,18 +42,19 @@ def _clean_temp_folders():
42 shutil.rmtree(r, ignore_errors=True) 42 shutil.rmtree(r, ignore_errors=True)
43 43
44 44
45def process_repositories(site_name): 45def process_repositories(site_name, overwrite_existing=False):
46 """Process and setup all repositories including ensuring we are at the 46 """Process and setup all repositories including ensuring we are at the
47 right revision based on the site's own site-definition.yaml file. 47 right revision based on the site's own site-definition.yaml file.
48 48
49 :param site_name: Site name for which to clone relevant repos. 49 :param site_name: Site name for which to clone relevant repos.
50 :param overwrite_existing: Whether to overwrite an existing directory
50 51
51 """ 52 """
52 53
53 # Only tracks extra repositories - not the site (primary) repository. 54 # Only tracks extra repositories - not the site (primary) repository.
54 extra_repos = [] 55 extra_repos = []
55 56
56 site_repo = process_site_repository() 57 site_repo = process_site_repository(overwrite_existing=overwrite_existing)
57 58
58 # Retrieve extra repo data from site-definition.yaml files. 59 # Retrieve extra repo data from site-definition.yaml files.
59 site_data = util.definition.load_as_params( 60 site_data = util.definition.load_as_params(
@@ -94,7 +95,9 @@ def process_repositories(site_name):
94 "repo_username=%s, revision=%s", repo_alias, repo_url_or_path, 95 "repo_username=%s, revision=%s", repo_alias, repo_url_or_path,
95 repo_key, repo_user, repo_revision) 96 repo_key, repo_user, repo_revision)
96 97
97 temp_extra_repo = _process_repository(repo_url_or_path, repo_revision) 98 temp_extra_repo = _process_repository(
99 repo_url_or_path, repo_revision,
100 overwrite_existing=overwrite_existing)
98 extra_repos.append(temp_extra_repo) 101 extra_repos.append(temp_extra_repo)
99 102
100 # Overwrite the site repo and extra repos in the config because further 103 # Overwrite the site repo and extra repos in the config because further
@@ -105,12 +108,13 @@ def process_repositories(site_name):
105 config.set_extra_repo_list(extra_repos) 108 config.set_extra_repo_list(extra_repos)
106 109
107 110
108def process_site_repository(update_config=False): 111def process_site_repository(update_config=False, overwrite_existing=False):
109 """Process and setup site repository including ensuring we are at the right 112 """Process and setup site repository including ensuring we are at the right
110 revision based on the site's own site-definition.yaml file. 113 revision based on the site's own site-definition.yaml file.
111 114
112 :param bool update_config: Whether to update Pegleg config with computed 115 :param bool update_config: Whether to update Pegleg config with computed
113 site repo path. 116 site repo path.
117 :param overwrite_existing: Whether to overwrite an existing directory
114 118
115 """ 119 """
116 120
@@ -122,8 +126,10 @@ def process_site_repository(update_config=False):
122 126
123 repo_url_or_path, repo_revision = _extract_repo_url_and_revision( 127 repo_url_or_path, repo_revision = _extract_repo_url_and_revision(
124 site_repo_or_path) 128 site_repo_or_path)
129 config.set_site_rev(repo_revision)
125 repo_url_or_path = _format_url_with_repo_username(repo_url_or_path) 130 repo_url_or_path = _format_url_with_repo_username(repo_url_or_path)
126 new_repo_path = _process_repository(repo_url_or_path, repo_revision) 131 new_repo_path = _process_repository(repo_url_or_path, repo_revision,
132 overwrite_existing=overwrite_existing)
127 133
128 if update_config: 134 if update_config:
129 # Overwrite the site repo in the config because further processing will 135 # Overwrite the site repo in the config because further processing will
@@ -134,17 +140,19 @@ def process_site_repository(update_config=False):
134 return new_repo_path 140 return new_repo_path
135 141
136 142
137def _process_repository(repo_url_or_path, repo_revision): 143def _process_repository(repo_url_or_path, repo_revision,
144 overwrite_existing=False):
138 """Process a repository located at ``repo_url_or_path``. 145 """Process a repository located at ``repo_url_or_path``.
139 146
140 :param str repo_url_or_path: Path to local repo or URL of remote URL. 147 :param str repo_url_or_path: Path to local repo or URL of remote URL.
141 :param str repo_revision: branch, commit or ref in the repo to checkout. 148 :param str repo_revision: branch, commit or ref in the repo to checkout.
149 :param overwrite_existing: Whether to overwrite an existing directory
142 150
143 """ 151 """
144 152
145 global __REPO_FOLDERS 153 global __REPO_FOLDERS
146 154
147 if os.path.exists(repo_url_or_path): 155 if os.path.exists(repo_url_or_path) and not overwrite_existing:
148 repo_name = util.git.repo_name(repo_url_or_path) 156 repo_name = util.git.repo_name(repo_url_or_path)
149 parent_temp_path = tempfile.mkdtemp() 157 parent_temp_path = tempfile.mkdtemp()
150 __REPO_FOLDERS.setdefault(repo_name, parent_temp_path) 158 __REPO_FOLDERS.setdefault(repo_name, parent_temp_path)
diff --git a/pegleg/engine/secrets.py b/pegleg/engine/secrets.py
index 19afb59..6d89974 100644
--- a/pegleg/engine/secrets.py
+++ b/pegleg/engine/secrets.py
@@ -75,12 +75,13 @@ def decrypt(file_path, site_name):
75 :type file_path: string 75 :type file_path: string
76 :param site_name: The name of the site to search for the file. 76 :param site_name: The name of the site to search for the file.
77 :type site_name: string 77 :type site_name: string
78 :return: The decrypted secrets
79 :rtype: list
78 """ 80 """
79
80 LOG.info('Started decrypting...') 81 LOG.info('Started decrypting...')
81 if (os.path.isfile(file_path) and 82 if (os.path.isfile(file_path) and
82 [s for s in file_path.split(os.path.sep) if s == site_name]): 83 [s for s in file_path.split(os.path.sep) if s == site_name]):
83 PeglegSecretManagement(file_path).decrypt_secrets() 84 return PeglegSecretManagement(file_path).decrypt_secrets()
84 else: 85 else:
85 LOG.info('File: {} was not found. Check your file path and name, ' 86 LOG.info('File: {} was not found. Check your file path and name, '
86 'and try again.'.format(file_path)) 87 'and try again.'.format(file_path))
diff --git a/pegleg/engine/util/__init__.py b/pegleg/engine/util/__init__.py
index 7168f04..eee108d 100644
--- a/pegleg/engine/util/__init__.py
+++ b/pegleg/engine/util/__init__.py
@@ -13,7 +13,8 @@
13# limitations under the License. 13# limitations under the License.
14 14
15# flake8: noqa 15# flake8: noqa
16from . import definition 16from pegleg.engine.util import catalog
17from . import files 17from pegleg.engine.util import definition
18from . import deckhand 18from pegleg.engine.util import deckhand
19from . import git \ No newline at end of file 19from pegleg.engine.util import files
20from pegleg.engine.util import git
diff --git a/pegleg/engine/util/catalog.py b/pegleg/engine/util/catalog.py
new file mode 100644
index 0000000..245d033
--- /dev/null
+++ b/pegleg/engine/util/catalog.py
@@ -0,0 +1,52 @@
1# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14"""Utility functions for catalog files such as pki-catalog.yaml."""
15
16import logging
17
18from pegleg.engine.util import definition
19
20LOG = logging.getLogger(__name__)
21
22__all__ = ('iterate', )
23
24
25def iterate(kind, sitename=None, documents=None):
26 """Retrieve the list of catalog documents by catalog schema ``kind``.
27
28 :param str kind: The schema kind of the catalog. For example, for schema
29 ``pegleg/PKICatalog/v1`` kind should be "PKICatalog".
30 :param str sitename: (optional) Site name for retrieving documents.
31 Multually exclusive with ``documents``.
32 :param str documents: (optional) Documents to search through. Mutually
33 exclusive with ``sitename``.
34 :return: All catalog documents for ``kind``.
35 :rtype: generator[dict]
36
37 """
38
39 if not any([sitename, documents]):
40 raise ValueError('Either `sitename` or `documents` must be specified')
41
42 documents = documents or definition.documents_for_site(sitename)
43 for document in documents:
44 schema = document.get('schema')
45 # TODO(felipemonteiro): Remove 'promenade/%s/v1' once site manifest
46 # documents switch to new 'pegleg' namespace.
47 if schema == 'pegleg/%s/v1' % kind:
48 yield document
49 elif schema == 'promenade/%s/v1' % kind:
50 LOG.warning('The schema promenade/%s/v1 is deprecated. Use '
51 'pegleg/%s/v1 instead.', kind, kind)
52 yield document
diff --git a/pegleg/engine/util/deckhand.py b/pegleg/engine/util/deckhand.py
index c1d42fd..8fb244f 100644
--- a/pegleg/engine/util/deckhand.py
+++ b/pegleg/engine/util/deckhand.py
@@ -41,10 +41,10 @@ def load_schemas_from_docs(documents):
41 return schema_set, errors 41 return schema_set, errors
42 42
43 43
44def deckhand_render(documents=[], 44def deckhand_render(documents=None,
45 fail_on_missing_sub_src=False, 45 fail_on_missing_sub_src=False,
46 validate=False): 46 validate=False):
47 47 documents = documents or []
48 errors = [] 48 errors = []
49 rendered_documents = [] 49 rendered_documents = []
50 50
diff --git a/pegleg/engine/util/definition.py b/pegleg/engine/util/definition.py
index 8ef67f3..07e25d2 100644
--- a/pegleg/engine/util/definition.py
+++ b/pegleg/engine/util/definition.py
@@ -11,6 +11,7 @@
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and 12# See the License for the specific language governing permissions and
13# limitations under the License. 13# limitations under the License.
14"""Utility functions for site-definition.yaml files."""
14 15
15import os 16import os
16 17
diff --git a/pegleg/engine/util/git.py b/pegleg/engine/util/git.py
index 616e0a6..217ee8f 100644
--- a/pegleg/engine/util/git.py
+++ b/pegleg/engine/util/git.py
@@ -26,7 +26,8 @@ from pegleg.engine import exceptions
26 26
27LOG = logging.getLogger(__name__) 27LOG = logging.getLogger(__name__)
28 28
29__all__ = ('git_handler', ) 29__all__ = ('git_handler', 'is_repository', 'is_equal', 'repo_url', 'repo_name',
30 'normalize_repo_path')
30 31
31 32
32def git_handler(repo_url, 33def git_handler(repo_url,
@@ -377,21 +378,26 @@ def is_equal(first_repo, other_repo):
377 return False 378 return False
378 379
379 380
380def repo_name(repo_path): 381def repo_url(repo_url_or_path):
381 """Get the repository name for local repo at ``repo_path``. 382 """Get the repository URL for the local or remote repo at
383 ``repo_url_or_path``.
382 384
383 :param repo_path: Path to local Git repo. 385 :param repo_url_or_path: URL of remote Git repo or path to local Git repo.
384 :returns: Corresponding repo name. 386 :returns: Corresponding repo name.
385 :rtype: str 387 :rtype: str
386 :raises GitConfigException: If the path is not a valid Git repo. 388 :raises GitConfigException: If the path is not a valid Git repo.
387 389
388 """ 390 """
389 391
390 if not is_repository(normalize_repo_path(repo_path)[0]): 392 # If ``repo_url_or_path`` is already a URL, no point in checking.
391 raise exceptions.GitConfigException(repo_path=repo_path) 393 if not os.path.exists(repo_url_or_path):
394 return repo_url_or_path
395
396 if not is_repository(normalize_repo_path(repo_url_or_path)[0]):
397 raise exceptions.GitConfigException(repo_url=repo_url_or_path)
392 398
393 # TODO(felipemonteiro): Support this for remote URLs too? 399 # TODO(felipemonteiro): Support this for remote URLs too?
394 repo = Repo(repo_path, search_parent_directories=True) 400 repo = Repo(repo_url_or_path, search_parent_directories=True)
395 config_reader = repo.config_reader() 401 config_reader = repo.config_reader()
396 section = 'remote "origin"' 402 section = 'remote "origin"'
397 option = 'url' 403 option = 'url'
@@ -408,9 +414,24 @@ def repo_name(repo_path):
408 else: 414 else:
409 return repo_url.split('/')[-1] 415 return repo_url.split('/')[-1]
410 except Exception: 416 except Exception:
411 raise exceptions.GitConfigException(repo_path=repo_path) 417 raise exceptions.GitConfigException(repo_url=repo_url_or_path)
418
419 raise exceptions.GitConfigException(repo_url=repo_url_or_path)
420
421
422def repo_name(repo_url_or_path):
423 """Get the repository name for the local or remote repo at
424 ``repo_url_or_path``.
425
426 :param repo_url_or_path: URL of remote Git repo or path to local Git repo.
427 :returns: Corresponding repo name.
428 :rtype: str
429 :raises GitConfigException: If the path is not a valid Git repo.
430
431 """
412 432
413 raise exceptions.GitConfigException(repo_path=repo_path) 433 _repo_url = repo_url(repo_url_or_path)
434 return _repo_url.split('/')[-1].split('.git')[0]
414 435
415 436
416def normalize_repo_path(repo_url_or_path): 437def normalize_repo_path(repo_url_or_path):
@@ -435,7 +456,7 @@ def normalize_repo_path(repo_url_or_path):
435 """ 456 """
436 457
437 repo_url_or_path = repo_url_or_path.rstrip('/') 458 repo_url_or_path = repo_url_or_path.rstrip('/')
438 orig_repo_path = repo_url_or_path 459 orig_repo_url_or_path = repo_url_or_path
439 sub_path = "" 460 sub_path = ""
440 is_local_repo = os.path.exists(repo_url_or_path) 461 is_local_repo = os.path.exists(repo_url_or_path)
441 462
@@ -455,8 +476,10 @@ def normalize_repo_path(repo_url_or_path):
455 repo_url_or_path = os.path.abspath(repo_url_or_path) 476 repo_url_or_path = os.path.abspath(repo_url_or_path)
456 477
457 if not repo_url_or_path or not is_repository(repo_url_or_path): 478 if not repo_url_or_path or not is_repository(repo_url_or_path):
458 msg = "The repo_path=%s is not a valid Git repo" % (orig_repo_path) 479 msg = "The repo_path=%s is not a valid Git repo" % (
480 orig_repo_url_or_path)
459 LOG.error(msg) 481 LOG.error(msg)
460 raise exceptions.GitInvalidRepoException(repo_path=repo_url_or_path) 482 raise exceptions.GitInvalidRepoException(
483 repo_path=orig_repo_url_or_path)
461 484
462 return repo_url_or_path, sub_path 485 return repo_url_or_path, sub_path
diff --git a/pegleg/engine/util/pegleg_secret_management.py b/pegleg/engine/util/pegleg_secret_management.py
index 364fab4..870ccf4 100644
--- a/pegleg/engine/util/pegleg_secret_management.py
+++ b/pegleg/engine/util/pegleg_secret_management.py
@@ -15,7 +15,6 @@
15import logging 15import logging
16import os 16import os
17import re 17import re
18import sys
19 18
20import click 19import click
21import yaml 20import yaml
@@ -130,9 +129,10 @@ class PeglegSecretManagement(object):
130 included in a site secrets file, and print the result to the standard 129 included in a site secrets file, and print the result to the standard
131 out.""" 130 out."""
132 131
133 yaml.safe_dump_all( 132 secrets = self.get_decrypted_secrets()
134 self.get_decrypted_secrets(), 133
135 sys.stdout, 134 return yaml.safe_dump_all(
135 secrets,
136 explicit_start=True, 136 explicit_start=True,
137 explicit_end=True, 137 explicit_end=True,
138 default_flow_style=False) 138 default_flow_style=False)
diff --git a/pegleg/schemas/PKICatalog.yaml b/pegleg/schemas/PKICatalog.yaml
new file mode 100644
index 0000000..2662ad2
--- /dev/null
+++ b/pegleg/schemas/PKICatalog.yaml
@@ -0,0 +1,44 @@
1# TODO(felipemonteiro): Implement validation and use this.
2---
3schema: deckhand/DataSchema/v1
4metadata:
5 schema: metadata/Control/v1
6 name: pegleg/PKICatalog/v1
7 labels:
8 application: pegleg
9data:
10 $schema: http://json-schema.org/schema#
11 certificate_authorities:
12 type: array
13 items:
14 type: object
15 properties:
16 description:
17 type: string
18 certificates:
19 type: array
20 items:
21 type: object
22 properties:
23 document_name:
24 type: string
25 description:
26 type: string
27 common_name:
28 type: string
29 hosts:
30 type: array
31 items: string
32 groups:
33 type: array
34 items: string
35 keypairs:
36 type: array
37 items:
38 type: object
39 properties:
40 name:
41 type: string
42 description:
43 type: string
44...
diff --git a/requirements.txt b/requirements.txt
index b019100..00af399 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -3,5 +3,8 @@ click==6.7
3jsonschema==2.6.0 3jsonschema==2.6.0
4pyyaml==3.12 4pyyaml==3.12
5cryptography==2.3.1 5cryptography==2.3.1
6python-dateutil==2.7.3
7
8# External dependencies
6git+https://github.com/openstack/airship-deckhand.git@7d697012fcbd868b14670aa9cf895acfad5a7f8d 9git+https://github.com/openstack/airship-deckhand.git@7d697012fcbd868b14670aa9cf895acfad5a7f8d
7git+https://github.com/openstack/airship-shipyard.git@44f7022df6438de541501c2fdd5c46df198b82bf#egg=shipyard_client&subdirectory=src/bin/shipyard_client 10git+https://github.com/openstack/airship-shipyard.git@44f7022df6438de541501c2fdd5c46df198b82bf#egg=shipyard_client&subdirectory=src/bin/shipyard_client
diff --git a/site_yamls/site/pki-catalog.yaml b/site_yamls/site/pki-catalog.yaml
new file mode 100644
index 0000000..9395858
--- /dev/null
+++ b/site_yamls/site/pki-catalog.yaml
@@ -0,0 +1,23 @@
1# Basic example of pki-catalog.yaml for k8s.
2---
3schema: promenade/PKICatalog/v1
4metadata:
5 schema: metadata/Document/v1
6 name: cluster-certificates-addition
7 layeringDefinition:
8 abstract: false
9 layer: site
10 storagePolicy: cleartext
11data:
12 certificate_authorities:
13 kubernetes:
14 description: CA for Kubernetes components
15 certificates:
16 - document_name: kubelet-n3
17 common_name: system:node:n3
18 hosts:
19 - n3
20 - 192.168.77.13
21 groups:
22 - system:nodes
23...
diff --git a/site_yamls/site/site-definition.yaml b/site_yamls/site/site-definition.yaml
index 3005e26..94d3fb9 100644
--- a/site_yamls/site/site-definition.yaml
+++ b/site_yamls/site/site-definition.yaml
@@ -1,3 +1,4 @@
1# TODO(felipemonteiro): Update `data` section below with new values.
1--- 2---
2data: 3data:
3 revision: v1.0 4 revision: v1.0
diff --git a/tests/unit/engine/test_encryption.py b/tests/unit/engine/test_secrets.py
index 33e00f8..4c8dbd0 100644
--- a/tests/unit/engine/test_encryption.py
+++ b/tests/unit/engine/test_secrets.py
@@ -12,23 +12,29 @@
12# See the License for the specific language governing permissions and 12# See the License for the specific language governing permissions and
13# limitations under the License. 13# limitations under the License.
14 14
15import click
16import os 15import os
17import tempfile 16from os import listdir
18 17
18import click
19import mock 19import mock
20import pytest 20import pytest
21import yaml 21import yaml
22import tempfile
22 23
23from pegleg.engine.util import encryption as crypt 24from pegleg import config
24from tests.unit import test_utils 25from pegleg.engine import secrets
26from pegleg.engine.catalog import pki_utility
27from pegleg.engine.catalog.pki_generator import PKIGenerator
28from pegleg.engine.util import encryption as crypt, catalog, git
29from pegleg.engine.util import files
25from pegleg.engine.util.pegleg_managed_document import \ 30from pegleg.engine.util.pegleg_managed_document import \
26 PeglegManagedSecretsDocument 31 PeglegManagedSecretsDocument
27from pegleg.engine.util.pegleg_secret_management import PeglegSecretManagement
28from pegleg.engine.util.pegleg_secret_management import ENV_PASSPHRASE 32from pegleg.engine.util.pegleg_secret_management import ENV_PASSPHRASE
29from pegleg.engine.util.pegleg_secret_management import ENV_SALT 33from pegleg.engine.util.pegleg_secret_management import ENV_SALT
30from tests.unit.fixtures import temp_path 34from pegleg.engine.util.pegleg_secret_management import PeglegSecretManagement
31from pegleg.engine.util import files 35from tests.unit import test_utils
36from tests.unit.fixtures import temp_path, create_tmp_deployment_files, _gen_document
37from tests.unit.test_cli import TestSiteSecretsActions, BaseCLIActionTest, TEST_PARAMS
32 38
33TEST_DATA = """ 39TEST_DATA = """
34--- 40---
@@ -69,6 +75,44 @@ def test_short_passphrase():
69 PeglegSecretManagement('file_path') 75 PeglegSecretManagement('file_path')
70 76
71 77
78@mock.patch.dict(os.environ, {
79 ENV_PASSPHRASE: 'ytrr89erARAiPE34692iwUMvWqqBvC',
80 ENV_SALT: 'MySecretSalt'})
81def test_secret_encrypt_and_decrypt(create_tmp_deployment_files, tmpdir):
82 site_dir = tmpdir.join("deployment_files", "site", "cicd")
83 passphrase_doc = """---
84schema: deckhand/Passphrase/v1
85metadata:
86 schema: metadata/Document/v1
87 name: {0}
88 storagePolicy: {1}
89 layeringDefinition:
90 abstract: False
91 layer: {2}
92data: {0}-password
93...
94""".format("cicd-passphrase-encrypted", "encrypted",
95 "site")
96 with open(os.path.join(str(site_dir), 'secrets',
97 'passphrases',
98 'cicd-passphrase-encrypted.yaml'), "w") \
99 as outfile:
100 outfile.write(passphrase_doc)
101
102 save_location = tmpdir.mkdir("encrypted_files")
103 save_location_str = str(save_location)
104
105 secrets.encrypt(save_location_str, "pytest", "cicd")
106 encrypted_files = listdir(save_location_str)
107 assert len(encrypted_files) > 0
108
109 # for _file in encrypted_files:
110 decrypted = secrets.decrypt(str(save_location.join(
111 "site/cicd/secrets/passphrases/"
112 "cicd-passphrase-encrypted.yaml")), "cicd")
113 assert yaml.load(decrypted) == yaml.load(passphrase_doc)
114
115
72def test_pegleg_secret_management_constructor(): 116def test_pegleg_secret_management_constructor():
73 test_data = yaml.load(TEST_DATA) 117 test_data = yaml.load(TEST_DATA)
74 doc = PeglegManagedSecretsDocument(test_data) 118 doc = PeglegManagedSecretsDocument(test_data)
@@ -141,3 +185,52 @@ def test_encrypt_decrypt_using_docs(temp_path):
141 'name'] 185 'name']
142 assert test_data[0]['metadata']['storagePolicy'] == decrypted_data[0][ 186 assert test_data[0]['metadata']['storagePolicy'] == decrypted_data[0][
143 'metadata']['storagePolicy'] 187 'metadata']['storagePolicy']
188
189
190@pytest.mark.skipif(
191 not pki_utility.PKIUtility.cfssl_exists(),
192 reason='cfssl must be installed to execute these tests')
193def test_generate_pki_using_local_repo_path(create_tmp_deployment_files):
194 """Validates ``generate-pki`` action using local repo path."""
195 # Scenario:
196 #
197 # 1) Generate PKI using local repo path
198
199 repo_path = str(git.git_handler(TEST_PARAMS["repo_url"],
200 ref=TEST_PARAMS["repo_rev"]))
201 with mock.patch.dict(config.GLOBAL_CONTEXT, {"site_repo": repo_path}):
202 pki_generator = PKIGenerator(sitename=TEST_PARAMS["site_name"])
203 generated_files = pki_generator.generate()
204
205 assert len(generated_files), 'No secrets were generated'
206 for generated_file in generated_files:
207 with open(generated_file, 'r') as f:
208 result = yaml.safe_load_all(f) # Validate valid YAML.
209 assert list(result), "%s file is empty" % generated_file.name
210
211
212@pytest.mark.skipif(
213 not pki_utility.PKIUtility.cfssl_exists(),
214 reason='cfssl must be installed to execute these tests')
215def test_check_expiry(create_tmp_deployment_files):
216 """ Validates check_expiry """
217 repo_path = str(git.git_handler(TEST_PARAMS["repo_url"],
218 ref=TEST_PARAMS["repo_rev"]))
219 with mock.patch.dict(config.GLOBAL_CONTEXT, {"site_repo": repo_path}):
220 pki_generator = PKIGenerator(sitename=TEST_PARAMS["site_name"])
221 generated_files = pki_generator.generate()
222
223 pki_util = pki_utility.PKIUtility()
224
225 assert len(generated_files), 'No secrets were generated'
226 for generated_file in generated_files:
227 if "certificate" not in generated_file:
228 continue
229 with open(generated_file, 'r') as f:
230 results = yaml.safe_load_all(f) # Validate valid YAML.
231 for result in results:
232 if result['data']['managedDocument']['schema'] == \
233 "deckhand/Certificate/v1":
234 cert = result['data']['managedDocument']['data']
235 assert not pki_util.check_expiry(cert), \
236 "%s is expired!" % generated_file.name
diff --git a/tests/unit/fixtures.py b/tests/unit/fixtures.py
index b47c4e0..2ca81db 100644
--- a/tests/unit/fixtures.py
+++ b/tests/unit/fixtures.py
@@ -30,7 +30,7 @@ schema: deckhand/Passphrase/v1
30metadata: 30metadata:
31 schema: metadata/Document/v1 31 schema: metadata/Document/v1
32 name: %(name)s 32 name: %(name)s
33 storagePolicy: cleartext 33 storagePolicy: %(storagePolicy)s
34 layeringDefinition: 34 layeringDefinition:
35 abstract: False 35 abstract: False
36 layer: %(layer)s 36 layer: %(layer)s
@@ -40,6 +40,8 @@ data: %(name)s-password
40 40
41 41
42def _gen_document(**kwargs): 42def _gen_document(**kwargs):
43 if "storagePolicy" not in kwargs:
44 kwargs["storagePolicy"] = "cleartext"
43 test_document = TEST_DOCUMENT % kwargs 45 test_document = TEST_DOCUMENT % kwargs
44 return yaml.load(test_document) 46 return yaml.load(test_document)
45 47
@@ -154,7 +156,7 @@ schema: pegleg/SiteDefinition/v1
154 cicd_path = os.path.join(str(p), files._site_path(site)) 156 cicd_path = os.path.join(str(p), files._site_path(site))
155 files._create_tree(cicd_path, tree=test_structure) 157 files._create_tree(cicd_path, tree=test_structure)
156 158
157 yield 159 yield tmpdir
158 160
159 161
160@pytest.fixture() 162@pytest.fixture()
diff --git a/tests/unit/test_cli.py b/tests/unit/test_cli.py
index 59323c4..7d09d1d 100644
--- a/tests/unit/test_cli.py
+++ b/tests/unit/test_cli.py
@@ -19,14 +19,25 @@ from click.testing import CliRunner
19from mock import ANY 19from mock import ANY
20import mock 20import mock
21import pytest 21import pytest
22import yaml
22 23
23from pegleg import cli 24from pegleg import cli
25from pegleg.engine.catalog import pki_utility
24from pegleg.engine import errorcodes 26from pegleg.engine import errorcodes
25from pegleg.engine.util import git 27from pegleg.engine.util import git
26from tests.unit import test_utils 28from tests.unit import test_utils
27from tests.unit.fixtures import temp_path 29from tests.unit.fixtures import temp_path
28 30
29 31
32TEST_PARAMS = {
33 "site_name": "airship-seaworthy",
34 "site_type": "foundry",
35 "repo_rev": '6b183e148b9bb7ba6f75c98dd13451088255c60b',
36 "repo_name": "airship-treasuremap",
37 "repo_url": "https://github.com/openstack/airship-treasuremap.git",
38}
39
40
30@pytest.mark.skipif( 41@pytest.mark.skipif(
31 not test_utils.is_connected(), 42 not test_utils.is_connected(),
32 reason='git clone requires network connectivity.') 43 reason='git clone requires network connectivity.')
@@ -50,13 +61,13 @@ class BaseCLIActionTest(object):
50 cls.runner = CliRunner() 61 cls.runner = CliRunner()
51 62
52 # Pin so we know that airship-seaworthy is a valid site. 63 # Pin so we know that airship-seaworthy is a valid site.
53 cls.site_name = "airship-seaworthy" 64 cls.site_name = TEST_PARAMS["site_name"]
54 cls.site_type = "foundry" 65 cls.site_type = TEST_PARAMS["site_type"]
55 66
56 cls.repo_rev = '6b183e148b9bb7ba6f75c98dd13451088255c60b' 67 cls.repo_rev = TEST_PARAMS["repo_rev"]
57 cls.repo_name = "airship-treasuremap" 68 cls.repo_name = TEST_PARAMS["repo_name"]
58 repo_url = "https://github.com/openstack/%s.git" % cls.repo_name 69 cls.treasuremap_path = git.git_handler(TEST_PARAMS["repo_url"],
59 cls.treasuremap_path = git.git_handler(repo_url, ref=cls.repo_rev) 70 ref=TEST_PARAMS["repo_rev"])
60 71
61 72
62class TestSiteCLIOptions(BaseCLIActionTest): 73class TestSiteCLIOptions(BaseCLIActionTest):
@@ -428,6 +439,94 @@ class TestRepoCliActions(BaseCLIActionTest):
428 assert not result.output 439 assert not result.output
429 440
430 441
442class TestSiteSecretsActions(BaseCLIActionTest):
443 """Tests site secrets-related CLI actions."""
444
445 def _validate_generate_pki_action(self, result):
446 assert result.exit_code == 0
447
448 generated_files = []
449 output_lines = result.output.split("\n")
450 for line in output_lines:
451 if self.repo_name in line:
452 generated_files.append(line)
453
454 assert len(generated_files), 'No secrets were generated'
455 for generated_file in generated_files:
456 with open(generated_file, 'r') as f:
457 result = yaml.safe_load_all(f) # Validate valid YAML.
458 assert list(result), "%s file is empty" % filename
459
460 @pytest.mark.skipif(
461 not pki_utility.PKIUtility.cfssl_exists(),
462 reason='cfssl must be installed to execute these tests')
463 def test_site_secrets_generate_pki_using_remote_repo_url(self):
464 """Validates ``generate-pki`` action using remote repo URL."""
465 # Scenario:
466 #
467 # 1) Generate PKI using remote repo URL
468
469 repo_url = 'https://github.com/openstack/%s@%s' % (self.repo_name,
470 self.repo_rev)
471
472 secrets_opts = ['secrets', 'generate-pki', self.site_name]
473
474 result = self.runner.invoke(cli.site, ['-r', repo_url] + secrets_opts)
475 self._validate_generate_pki_action(result)
476
477 @pytest.mark.skipif(
478 not pki_utility.PKIUtility.cfssl_exists(),
479 reason='cfssl must be installed to execute these tests')
480 def test_site_secrets_generate_pki_using_local_repo_path(self):
481 """Validates ``generate-pki`` action using local repo path."""
482 # Scenario:
483 #
484 # 1) Generate PKI using local repo path
485
486 repo_path = self.treasuremap_path
487 secrets_opts = ['secrets', 'generate-pki', self.site_name]
488
489 result = self.runner.invoke(cli.site, ['-r', repo_path] + secrets_opts)
490 self._validate_generate_pki_action(result)
491
492 @pytest.mark.skipif(
493 not pki_utility.PKIUtility.cfssl_exists(),
494 reason='cfssl must be installed to execute these tests')
495 @mock.patch.dict(os.environ, {
496 "PEGLEG_PASSPHRASE": "123456789012345678901234567890",
497 "PEGLEG_SALT": "123456"
498 })
499 def test_site_secrets_encrypt_local_repo_path(self):
500 """Validates ``generate-pki`` action using local repo path."""
501 # Scenario:
502 #
503 # 1) Encrypt a file in a local repo
504
505 repo_path = self.treasuremap_path
506 with open(os.path.join(repo_path, "site", "airship-seaworthy",
507 "secrets", "passphrases", "ceph_fsid.yaml"), "r") \
508 as ceph_fsid_fi:
509 ceph_fsid = yaml.load(ceph_fsid_fi)
510 ceph_fsid["metadata"]["storagePolicy"] = "encrypted"
511
512 with open(os.path.join(repo_path, "site", "airship-seaworthy",
513 "secrets", "passphrases", "ceph_fsid.yaml"), "w") \
514 as ceph_fsid_fi:
515 yaml.dump(ceph_fsid, ceph_fsid_fi)
516
517 secrets_opts = ['secrets', 'encrypt', '-a', 'test', self.site_name]
518 result = self.runner.invoke(cli.site, ['-r', repo_path] + secrets_opts)
519
520 assert result.exit_code == 0
521
522 with open(os.path.join(repo_path, "site", "airship-seaworthy",
523 "secrets", "passphrases", "ceph_fsid.yaml"), "r") \
524 as ceph_fsid_fi:
525 ceph_fsid = yaml.load(ceph_fsid_fi)
526 assert "encrypted" in ceph_fsid["data"]
527 assert "managedDocument" in ceph_fsid["data"]
528
529
431class TestTypeCliActions(BaseCLIActionTest): 530class TestTypeCliActions(BaseCLIActionTest):
432 """Tests type-level CLI actions.""" 531 """Tests type-level CLI actions."""
433 532
diff --git a/tools/gate/playbooks/install-cfssl.yaml b/tools/gate/playbooks/install-cfssl.yaml
new file mode 100644
index 0000000..360888f
--- /dev/null
+++ b/tools/gate/playbooks/install-cfssl.yaml
@@ -0,0 +1,23 @@
1# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15- hosts: all
16 gather_facts: False
17 tasks:
18 - name: Install cfssl for Ubuntu
19 shell: |-
20 ./tools/install-cfssl.sh
21 become: yes
22 args:
23 chdir: "{{ zuul.project.src_dir }}"
diff --git a/tools/gate/whitespace-linter.sh b/tools/gate/whitespace-linter.sh
index 031a6b4..e96708c 100755
--- a/tools/gate/whitespace-linter.sh
+++ b/tools/gate/whitespace-linter.sh
@@ -8,6 +8,7 @@ RES=$(find . \
8 -not -path "*/htmlcov/*" \ 8 -not -path "*/htmlcov/*" \
9 -not -name "*.tgz" \ 9 -not -name "*.tgz" \
10 -not -name "*.pyc" \ 10 -not -name "*.pyc" \
11 -not -name "*.html" \
11 -type f -exec egrep -l " +$" {} \;) 12 -type f -exec egrep -l " +$" {} \;)
12 13
13if [[ -n $RES ]]; then 14if [[ -n $RES ]]; then
diff --git a/tools/install-cfssl.sh b/tools/install-cfssl.sh
new file mode 100755
index 0000000..e1994ee
--- /dev/null
+++ b/tools/install-cfssl.sh
@@ -0,0 +1,22 @@
1#!/usr/bin/env bash
2
3set -ex
4
5if [ $# -eq 1 ]; then
6 CFSSLURL=$1
7else
8 CFSSLURL=${CFSSLURL:="http://pkg.cfssl.org/R1.2/cfssl_linux-amd64"}
9fi
10
11if [ -z $(which cfssl) ]; then
12 if [ $(whoami) == "root" ]; then
13 curl -Lo /usr/local/bin/cfssl ${CFSSLURL}
14 chmod 555 /usr/local/bin/cfssl
15 else
16 if [ ! -d ~/.local/bin ]; then
17 mkdir -p ~/.local/bin
18 fi
19 curl -Lo ~/.local/bin/cfssl ${CFSSLURL}
20 chmod 555 ~/.local/bin/cfssl
21 fi
22fi
diff --git a/tox.ini b/tox.ini
index 75efc85..73fe93b 100644
--- a/tox.ini
+++ b/tox.ini
@@ -57,7 +57,12 @@ deps =
57 -r{toxinidir}/requirements.txt 57 -r{toxinidir}/requirements.txt
58 -r{toxinidir}/test-requirements.txt 58 -r{toxinidir}/test-requirements.txt
59commands = 59commands =
60 pytest --cov=pegleg --cov-report html:cover --cov-report xml:cover/coverage.xml --cov-report term --cov-fail-under 84 tests/ 60 {toxinidir}/tools/install-cfssl.sh
61 bash -c 'PATH=$PATH:~/.local/bin; pytest --cov=pegleg --cov-report \
62 html:cover --cov-report xml:cover/coverage.xml --cov-report term \
63 --cov-fail-under 84 tests/'
64whitelist_externals =
65 bash
61 66
62[testenv:releasenotes] 67[testenv:releasenotes]
63basepython = python3 68basepython = python3