docs: Elaborate on document layering in documentation

This patchset elaborates on document layering in the documentation
to provide much greater clarity into what layering is and its
associated concepts, including: layer, layer order, layering policy,
layering definition, document abstraction, parent selection,
layering actions, etc.

Change-Id: I584e67b7984fa4035cef481a116ae3b8a3eb2906
This commit is contained in:
Felipe Monteiro 2018-06-27 16:37:42 -04:00
parent f9e4b5993f
commit 1ac9abb555
6 changed files with 373 additions and 23 deletions

Binary file not shown.

Before

Width:  |  Height:  |  Size: 37 KiB

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 23 KiB

After

Width:  |  Height:  |  Size: 23 KiB

View File

@ -111,9 +111,14 @@ correct schemas.
.. _JSON schema: http://json-schema.org .. _JSON schema: http://json-schema.org
.. _layering-policy:
LayeringPolicy LayeringPolicy
^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^
This document defines the strict order in which documents are layered together
from their component parts.
Only one ``LayeringPolicy`` document can exist within the system at any time. Only one ``LayeringPolicy`` document can exist within the system at any time.
It is an error to attempt to insert a new ``LayeringPolicy`` document if it has It is an error to attempt to insert a new ``LayeringPolicy`` document if it has
a different ``metadata.name`` than the existing document. If the names match, a different ``metadata.name`` than the existing document. If the names match,
@ -127,8 +132,8 @@ it is treated as an update to the existing document.
the new ``LayeringPolicy``. the new ``LayeringPolicy``.
This document defines the strict order in which documents are merged together This document defines the strict order in which documents are merged together
from their component parts. It should result in a validation error if a from their component parts. An error is raised if a document refers to a layer
document refers to a layer not specified in the ``LayeringPolicy``. not specified in the ``LayeringPolicy``.
Below is an example of a ``LayeringPolicy`` document: Below is an example of a ``LayeringPolicy`` document:

View File

@ -34,6 +34,8 @@ Detailed documentation for :ref:`layering`, :ref:`substitution`,
:ref:`revision-history` and :ref:`validation` should be reviewed for a more :ref:`revision-history` and :ref:`validation` should be reviewed for a more
thorough understanding of each concept. thorough understanding of each concept.
.. _document-format:
Document Format Document Format
--------------- ---------------

View File

@ -23,38 +23,376 @@ Introduction
------------ ------------
Layering provides a restricted data inheritance model intended to help reduce Layering provides a restricted data inheritance model intended to help reduce
duplication in configuration. Documents with different ``schema``'s are never duplication in configuration. With layering, child documents can inherit
layered together (see the :ref:`substitution` section if you need to combine data data from parent documents. Through :ref:`layering-actions`, child documents
from multiple types of documents). can control exactly what they inherit from their parent. Document layering,
conceptually speaking, works much like class inheritance: A child class
inherits all variables and methods from its parent, but can elect to override
its parent's functionality.
Layering is controlled in two places: Goals behind layering include:
1. The ``LayeringPolicy`` control document (described below), which defines the * model site deployment data hierarchically
valid layers and their order of precedence. * lessen data duplication across site layers (as well as other conceptual
2. In the ``metadata.layeringDefinition`` section of normal layers)
(``metadata.schema=metadata/Document/v1``) documents.
When rendering a particular document, you resolve the chain of parents upward Document Abstraction
through the layers, and begin working back down each layer rendering at each ^^^^^^^^^^^^^^^^^^^^
document in the chain.
When rendering each layer, the parent document is used as the starting point, Layering works with :ref:`document-abstraction`: child documents can inherit
so the entire contents of the parent are brought forward. Then the list of from abstract as well as concrete parent documents.
`actions` will be applied in order. Supported actions are:
* ``merge`` - "deep" merge child data at the specified path into the existing Pre-Conditions
data ^^^^^^^^^^^^^^
* ``replace`` - overwrite existing data with child data at the specified path
* ``delete`` - remove the existing data at the specified path A document only has one parent, but its parent is computed dynamically using
the :ref:`parent-selection` algorithm. That is, the notion of
"multiple inheritance" **does not** apply to document layering.
Documents with different ``schema`` values are never layered together (see the
:ref:`substitution` section if you need to combine data from multiple types of
documents).
Document layering requires a :ref:`layering-policy` to exist in the revision
whose documents will be layered together (rendered). An error will be issued
otherwise.
Terminology
-----------
.. note::
Whether a layer is "lower" or "higher" has entirely to do with its order of
initialization in a ``layerOrder`` and, by extension, its precedence in the
:ref:`parent-selection` algorithm described below.
* Layer - A position in a hierarchy used to control :ref:`parent-selection` by
the :ref:`layering-algorithm`. It can be likened to a position in an
inheritance hierarchy, where ``object`` in Python can be likened to the
highest layer in a ``layerOrder`` in Deckhand and a leaf class can be likened
to the lowest layer in a ``layerOrder``.
* Child - Meaningful only in a parent-child document relationship. A document
with a lower layer (but higher priority) than its parent, determined using
using :ref:`parent-selection`.
* Parent - Meaningful only in a parent-child document relationship. A document
with a higher layer (but lower priority) than its child.
* Layering Policy - A :ref:`control document <control-documents>` that defines
the strict ``layerOrder`` in which documents are layered together. See
:ref:`layering-policy` documentation for more information.
* Layer Order (``layerOrder``) - Corresponds to the ``data.layerOrder`` of the
:ref:`layering-policy` document. Establishes the layering hierarchy for a
set of layers in the system.
* Layering Definition (``layeringDefinition``) - Metadata in each document for
controlling the following:
* ``layer``: the document layer itself
* ``parentSelector``: :ref:`parent-selection`
* ``abstract``: :ref:`document-abstraction`
* ``actions``: :ref:`layering-actions`
* Parent Selector (``parentSelector``) - Key-value pairs or labels for
identifying the document's parent. Note that these key-value pairs are not
unique and that multiple documents can use them. All the key-value pairs
in the ``parentSelector`` must be found among the target parent's
``metadata.labels``: this means that the ``parentSelector`` key-value pairs
must be a subset of the target parent's ``metadata.labels`` key-value
pairs. See :ref:`parent-selection` for further details.
* Layering Actions (``actions``) - A list of actions that control what data
are inherited from the parent by the child. See :ref:`layering-actions`
for further details.
.. _layering-algorithm:
Algorithm
---------
Layering is applied at the bottommost layer of the ``layerOrder`` first and
at the topmost layer of the ``layerOrder`` last, such that the "base" layers
are processed first and the "leaf" layers are processed last. For each
layer in the ``layerOrder``, the documents that correspond to that layer
are retrieved. For each document retrieved, the ``layerOrder`` hierarchy
is resolved using :ref:`parent-selection` to identify the parent document.
Finally, the current document is layered with its parent using
:ref:`layering-actions`.
After layering is complete, the :ref:`substitution` algorithm is applied to the
*current* document, if applicable.
.. _layering-configuration:
Layering Configuration
----------------------
Layering is configured in 2 places:
#. The ``LayeringPolicy`` control document (described in
:ref:`layering-policy`), which defines the valid layers and their order of
precedence.
#. In the ``metadata.layeringDefinition`` section of normal
(``metadata.schema=metadata/Document/v1``) documents. For more information
about document structure, reference :ref:`document-format`.
An example ``layeringDefinition`` may look like::
layeringDefinition:
# Controls whether the document is abstract or concrete.
abstract: true
# A layer in the ``layerOrder``. Must be valid or else an error is raised.
layer: region
# Key-value pairs or labels for identifying the document's parent.
parentSelector:
required_key_a: required_label_a
required_key_b: required_label_b
# Actions which specify which data to add to the child document.
actions:
- method: merge
path: .path.to.merge.into.parent
- method: delete
path: .path.to.delete
.. _layering-actions:
Layering Actions
----------------
Introduction
^^^^^^^^^^^^
Layering actions allow child documents to modify data that is inherited from
the parent. What if the child document should only inherit some of the parent
data? No problem. A merge action can be performed, followed by ``delete``
and ``replace`` actions to trim down on what should be inherited.
Each layer action consists of an ``action`` and a ``path``. Whenever *any*
action is specified, *all* the parent data is automatically inherited by the
child document. The ``path`` specifies which data from the *child* document to
**prioritize over** that of the parent document. Stated differently, all data
from the parent is considered while *only* the *child* data at ``path`` is
considered during an action. However, whenever a conflict occurs during an
action, the *child* data takes priority over that of the parent.
Layering actions are queued -- meaning that if a ``merge`` is
specified before a ``replace`` then the ``merge`` will *necessarily* be
applied before the ``replace``. For example, a ``merge`` followed by a
``replace`` **is not necessarily** the same as a ``replace`` followed by a
``merge``.
Layering actions can be applied to primitives, lists and dictionaries alike.
Action Types
^^^^^^^^^^^^
Supported actions are:
* ``merge`` - "deep" merge child data and parent data into the child ``data``,
at the specified `JSONPath`_
.. note::
For conflicts between the child and parent data, the child document's
data is **always** prioritized. No other conflict resolution strategy for
this action currently exists.
``merge`` behavior depends upon the data types getting merged. For objects
and lists, Deckhand uses `JSONPath`_ resolution to retrieve data from those
entities, after which Deckhand applies merge strategies (see below) to
combine merge child and parent data into the child document's ``data``
section.
**Merge Strategies**
Deckhand applies the following merge strategies for each data type:
* object: "Deep-merge" child and parent data together; conflicts are resolved
by prioritizing child data over parent data. "Deep-merge" means
recursively combining data for each key-value pair in both objects.
* array: The merge strategy involves:
* When using an index in the action ``path`` (e.g. ``a[0]``):
#. Copying the parent array into the child's ``data`` section at the
specified JSONPath.
#. Appending each child entry in the original child array into the parent
array. This behavior is synonymous with the ``extend`` list function
in Python.
* When not using an index in the action ``path`` (e.g. ``a``):
#. The child's array replaces the parent's array.
* primitives: Includes all other data types, except for ``null``. In this
case JSONPath resolution is impossible, so child data is prioritized over
that of the parent.
**Examples**
Given::
Child Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
Parent Data: ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
* When::
Merge Path: ``.``
Then::
Rendered Data: ``{'a': {'x': 7, 'y': 2, 'z': 3}, 'b': 4, 'c': 9}``
All data from parent is automatically considered, all data from child
is considered due to ``.`` (selects everything), then both merged.
* When::
Merge Path: ``.a``
Then::
Rendered Data: ``{'a': {'x': 7, 'y': 2, 'z': 3}, 'c': 9}``
All data from parent is automatically considered, all data from child
at ``.a`` is considered, then both merged.
* When::
Merge Path: ``.b``
Then::
Rendered Data: ``{'a': {'x': 1, 'y': 2}, 'b': 4, 'c': 9}``
All data from parent is automatically considered, all data from child
at ``.b`` is considered, then both merged.
* When::
Merge Path: ``.c``
Then::
Error raised (``.c`` missing in child).
* ``replace`` - overwrite existing data with child data at the specified
JSONPath.
**Examples**
Given::
Child Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
Parent Data: ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
* When::
Replace Path: ``.``
Then::
Rendered Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
All data from parent is automatically considered, but is replaced by all
data from child at ``.`` (selects everything), so replaces everything
in parent.
* When::
Replace Path: ``.a``
Then::
Rendered Data: ``{'a': {'x': 7, 'z': 3}, 'c': 9}``
All data from parent is automatically considered, but is replaced by all
data from child at ``.a``, so replaces all parent data at ``.a``.
* When::
Replace Path: ``.b``
Then::
Rendered Data: ``{'a': {'x': 1, 'y': 2}, 'b': 4, 'c': 9}``
All data from parent is automatically considered, but is replaced by all
data from child at ``.b``, so replaces all parent data at ``.b``.
While ``.b`` isn't in the parent, it only needs to exist in the child.
In this case, something (from the child) replaces nothing (from the
parent).
* When::
Replace Path: ``.c``
Then::
Error raised (``.c`` missing in child).
* ``delete`` - remove the existing data at the specified JSONPath.
**Examples**
Given::
Child Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
Parent Data: ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
* When::
Delete Path: ``.``
Then::
Rendered Data: ``{}``
Note that deletion of everything results in an empty dictionary by
default.
* When::
Delete Path: ``.a``
Then::
Rendered Data: ``{'c': 9}``
All data from Parent Data at ``.a`` was deleted, rest copied over.
* When::
Delete Path: ``.c``
Then::
Rendered Data: ``{'a': {'x': 1, 'y': 2}}``
All data from Parent Data at ``.c`` was deleted, rest copied over.
* When::
Replace Path: ``.b``
Then::
Error raised (``.b`` missing in child).
After actions are applied for a given layer, substitutions are applied (see After actions are applied for a given layer, substitutions are applied (see
the Substitution section for details). the :ref:`substitution` section for details).
.. _JSONPath: http://goessner.net/articles/JsonPath/
.. _parent-selection: .. _parent-selection:
Parent Selection Parent Selection
---------------- ----------------
Parent selection is performed dynamically. Unlike :ref:`substitution`,
parent selection does not target a specific document using ``schema`` and
``name`` identifiers. Rather, parent selection respects the ``layerOrder``,
selecting the highest precedence parent in accordance with the algorithm that
follows. This allows flexibility in parent selection: if a document's immediate
parent is removed in a revision, then, if applicable, the grandparent (in the
previous revision) can become the document's parent (in the latest revision).
Selection of document parents is controlled by the ``parentSelector`` field and Selection of document parents is controlled by the ``parentSelector`` field and
works as follows: works as follows:

View File

@ -4,8 +4,13 @@
# files. Must be run from root project directory. # files. Must be run from root project directory.
set -ex set -ex
# Generate architectural diagrams.
mkdir -p doc/source/images
python -m plantuml doc/source/diagrams/*.uml
mv doc/source/diagrams/*.png doc/source/images
# Generate documentation.
rm -rf doc/build rm -rf doc/build
rm -rf releasenotes/build rm -rf releasenotes/build
sphinx-build -W -b html doc/source doc/build/html sphinx-build -W -b html doc/source doc/build/html
python -m plantuml doc/source/diagrams/*.uml
mv doc/source/diagrams/*.png doc/source/images