From 1ac9abb5551f9bec311e55f9846b6d428a1311ec Mon Sep 17 00:00:00 2001 From: Felipe Monteiro Date: Wed, 27 Jun 2018 16:37:42 -0400 Subject: [PATCH] docs: Elaborate on document layering in documentation This patchset elaborates on document layering in the documentation to provide much greater clarity into what layering is and its associated concepts, including: layer, layer order, layering policy, layering definition, document abstraction, parent selection, layering actions, etc. Change-Id: I584e67b7984fa4035cef481a116ae3b8a3eb2906 --- doc/source/images/architecture-pegleg.png | Bin 37598 -> 37600 bytes doc/source/images/architecture.png | Bin 23227 -> 23230 bytes doc/source/users/document-types.rst | 9 +- doc/source/users/documents.rst | 2 + doc/source/users/layering.rst | 376 ++++++++++++++++++++-- tools/build-docs.sh | 9 +- 6 files changed, 373 insertions(+), 23 deletions(-) diff --git a/doc/source/images/architecture-pegleg.png b/doc/source/images/architecture-pegleg.png index f35812da5a7c19578fb37279499b61a643b9812c..ed0a368d7dc2d6c20368e0d49c1c24f9d6e05e70 100644 GIT binary patch delta 531 zcmV+u0_^?Xr2^ok0+3mM_py0%5 z#KDS;-Fy_QZgJd$Wkm~2$SSaFEsI;L>VNO70mqRdq}9&Hd;8`wx3%HgG*#&^`|->u z4eo62It--|<^b;fYPCEIlOOR}^eef0a4qsh^9l)x+1o3ZxeI@P9R{$8lr&nETk!n} zZ052M61NWfcLrHvQDV^p+6bcfp~G~EPodv;C=jBdG}@6{R=R-)o4lLyIZu0#b;*_*>j)*g z3wYk-{RLQvJTLKoH`n&DPt!+$g*wchma_Md5S6)|w63k6e4Nd} zn|$!wjzM?w1+*w$jU9M44`GJ8QXMKXx1&OB7a(tY9d=9Saf8^h^VqTBMC3f+2h4dk#Mv zTDJ;Au+d<(7(z1dlcZ@cw31dIdt5KtCb(B7*(q~uzP|FN!alD_7pxa8>AKYzlE^Rq z#q((FrNNldN@;F{EFhtugH;Hzkj)=9!x!F^)`3jrR1Qh~SMP?!Tow&4NE=EPDbl15 VR7==@{xSpw;4-KbWwTfUhL9By1L*(& delta 529 zcmV+s0`C3br2^ii0+3mM_IgxUbZ~58Zgh2RYybdwoNZEFYuhjseHZvYT>O+g*wkg& zN*+SV$NDjcWpk1}givJb))ld(Aj@gP*ni(uY^Ny{fzbUpN9P{twzAyVx-2~A7SF6U z;7@11$52^mkKo@gmy3%q`4wM8f0DZg-yxq3FOiU#?OwPNKK%867{F$tl{LE9LKu!< zGgG-#xb@h-3&;ZV0`m^gL@eX5skS?WyV!YlVE1h zLPGN)j0qB;65CVf4?I>@zmp4YdCDs%O;=>!36L_6h^NIQ^o$(3s72qoJC z1i|S22AE4B3Vh{%#$EPl`UG%LkGb7Zb{-O?@4O_h(yQqDJtqF@OClRN=!vj{&TvT3Mrit2Kn-C9GzulnU1# z`}jtA+@RuCT{7DNN!Y@YTZ}_?jH~Ym&Xo}%A%<7O&Hhgzi>xzRFAp8 zlRPYf82c^ss$|B0VPAXdz^}1Ac106*Ct>qSDEpKqutn2Itsn|VC(%V1MFTEqDY${r z2%_`&>@*&q!qZ|3E|SgI{BG%%HOAng!6F?3pB+G$ zsjW0x)pQhgzfAigxYd?#wC#$|&w`0`?G5>YW!j4_dyg?hnfwu+Jdegf7L6I5+6XIE z37nRObqtAA?K`{17tvOhy5+i{bWHAl1{W+5s%%9`-cYfrCQoKZInF)-9z@X35lY_` IvljxSSP~=Pwg3PC delta 463 zcmV;=0Wki)wE?@e0gzdLwR%)obZ~58Zgh2RYybdwoNZA}Z`wc*y|3i|F!Biww&JL? z5r-&((4;A-v<7-WC|cVSy274C>oF;+`rkWiL&%}BWbJ&sH*enD)vO&iRT(J%A+g~j zgpZ5GJiLVI_vyRnXZr9Z^xZE`R|w%f=DS~-iV*$;3Sd39#Lkv~8;IjuSWjqc2seTH z`wsGLkCd{`cQCfq-jYz)XZT3F%;cI*_!M)83@KZOFcL_FRBYPLuXwE4UdRj0I@7iA z&uz_HhhBHbDldp&U7Vyo7ugx)SM{aFuO-RT1ET2mG680$C`x?Q&JX)p#0xc zo{bR1$%T$p#Q5BQn@?@>ZS6qa$%gGo)V$Qh%h?nT=xSpLlKA>6xrvh`L)OWlx1=4} zSAtw@G$jI?3j3<}FrPwg8UgrY(PY|q diff --git a/doc/source/users/document-types.rst b/doc/source/users/document-types.rst index a84e7433..2a702b35 100644 --- a/doc/source/users/document-types.rst +++ b/doc/source/users/document-types.rst @@ -111,9 +111,14 @@ correct schemas. .. _JSON schema: http://json-schema.org +.. _layering-policy: + LayeringPolicy ^^^^^^^^^^^^^^ +This document defines the strict order in which documents are layered together +from their component parts. + Only one ``LayeringPolicy`` document can exist within the system at any time. It is an error to attempt to insert a new ``LayeringPolicy`` document if it has a different ``metadata.name`` than the existing document. If the names match, @@ -127,8 +132,8 @@ it is treated as an update to the existing document. the new ``LayeringPolicy``. This document defines the strict order in which documents are merged together -from their component parts. It should result in a validation error if a -document refers to a layer not specified in the ``LayeringPolicy``. +from their component parts. An error is raised if a document refers to a layer +not specified in the ``LayeringPolicy``. Below is an example of a ``LayeringPolicy`` document: diff --git a/doc/source/users/documents.rst b/doc/source/users/documents.rst index e20e4a8c..cedc2f85 100644 --- a/doc/source/users/documents.rst +++ b/doc/source/users/documents.rst @@ -34,6 +34,8 @@ Detailed documentation for :ref:`layering`, :ref:`substitution`, :ref:`revision-history` and :ref:`validation` should be reviewed for a more thorough understanding of each concept. +.. _document-format: + Document Format --------------- diff --git a/doc/source/users/layering.rst b/doc/source/users/layering.rst index 178cfaef..bc7d905b 100644 --- a/doc/source/users/layering.rst +++ b/doc/source/users/layering.rst @@ -23,38 +23,376 @@ Introduction ------------ Layering provides a restricted data inheritance model intended to help reduce -duplication in configuration. Documents with different ``schema``'s are never -layered together (see the :ref:`substitution` section if you need to combine data -from multiple types of documents). +duplication in configuration. With layering, child documents can inherit +data from parent documents. Through :ref:`layering-actions`, child documents +can control exactly what they inherit from their parent. Document layering, +conceptually speaking, works much like class inheritance: A child class +inherits all variables and methods from its parent, but can elect to override +its parent's functionality. -Layering is controlled in two places: +Goals behind layering include: -1. The ``LayeringPolicy`` control document (described below), which defines the - valid layers and their order of precedence. -2. In the ``metadata.layeringDefinition`` section of normal - (``metadata.schema=metadata/Document/v1``) documents. +* model site deployment data hierarchically +* lessen data duplication across site layers (as well as other conceptual + layers) -When rendering a particular document, you resolve the chain of parents upward -through the layers, and begin working back down each layer rendering at each -document in the chain. +Document Abstraction +^^^^^^^^^^^^^^^^^^^^ -When rendering each layer, the parent document is used as the starting point, -so the entire contents of the parent are brought forward. Then the list of -`actions` will be applied in order. Supported actions are: +Layering works with :ref:`document-abstraction`: child documents can inherit +from abstract as well as concrete parent documents. -* ``merge`` - "deep" merge child data at the specified path into the existing - data -* ``replace`` - overwrite existing data with child data at the specified path -* ``delete`` - remove the existing data at the specified path +Pre-Conditions +^^^^^^^^^^^^^^ + +A document only has one parent, but its parent is computed dynamically using +the :ref:`parent-selection` algorithm. That is, the notion of +"multiple inheritance" **does not** apply to document layering. + +Documents with different ``schema`` values are never layered together (see the +:ref:`substitution` section if you need to combine data from multiple types of +documents). + +Document layering requires a :ref:`layering-policy` to exist in the revision +whose documents will be layered together (rendered). An error will be issued +otherwise. + +Terminology +----------- + +.. note:: + + Whether a layer is "lower" or "higher" has entirely to do with its order of + initialization in a ``layerOrder`` and, by extension, its precedence in the + :ref:`parent-selection` algorithm described below. + +* Layer - A position in a hierarchy used to control :ref:`parent-selection` by + the :ref:`layering-algorithm`. It can be likened to a position in an + inheritance hierarchy, where ``object`` in Python can be likened to the + highest layer in a ``layerOrder`` in Deckhand and a leaf class can be likened + to the lowest layer in a ``layerOrder``. +* Child - Meaningful only in a parent-child document relationship. A document + with a lower layer (but higher priority) than its parent, determined using + using :ref:`parent-selection`. +* Parent - Meaningful only in a parent-child document relationship. A document + with a higher layer (but lower priority) than its child. +* Layering Policy - A :ref:`control document ` that defines + the strict ``layerOrder`` in which documents are layered together. See + :ref:`layering-policy` documentation for more information. +* Layer Order (``layerOrder``) - Corresponds to the ``data.layerOrder`` of the + :ref:`layering-policy` document. Establishes the layering hierarchy for a + set of layers in the system. +* Layering Definition (``layeringDefinition``) - Metadata in each document for + controlling the following: + + * ``layer``: the document layer itself + * ``parentSelector``: :ref:`parent-selection` + * ``abstract``: :ref:`document-abstraction` + * ``actions``: :ref:`layering-actions` + +* Parent Selector (``parentSelector``) - Key-value pairs or labels for + identifying the document's parent. Note that these key-value pairs are not + unique and that multiple documents can use them. All the key-value pairs + in the ``parentSelector`` must be found among the target parent's + ``metadata.labels``: this means that the ``parentSelector`` key-value pairs + must be a subset of the target parent's ``metadata.labels`` key-value + pairs. See :ref:`parent-selection` for further details. +* Layering Actions (``actions``) - A list of actions that control what data + are inherited from the parent by the child. See :ref:`layering-actions` + for further details. + +.. _layering-algorithm: + +Algorithm +--------- + +Layering is applied at the bottommost layer of the ``layerOrder`` first and +at the topmost layer of the ``layerOrder`` last, such that the "base" layers +are processed first and the "leaf" layers are processed last. For each +layer in the ``layerOrder``, the documents that correspond to that layer +are retrieved. For each document retrieved, the ``layerOrder`` hierarchy +is resolved using :ref:`parent-selection` to identify the parent document. +Finally, the current document is layered with its parent using +:ref:`layering-actions`. + +After layering is complete, the :ref:`substitution` algorithm is applied to the +*current* document, if applicable. + +.. _layering-configuration: + +Layering Configuration +---------------------- + +Layering is configured in 2 places: + +#. The ``LayeringPolicy`` control document (described in + :ref:`layering-policy`), which defines the valid layers and their order of + precedence. +#. In the ``metadata.layeringDefinition`` section of normal + (``metadata.schema=metadata/Document/v1``) documents. For more information + about document structure, reference :ref:`document-format`. + +An example ``layeringDefinition`` may look like:: + + layeringDefinition: + # Controls whether the document is abstract or concrete. + abstract: true + # A layer in the ``layerOrder``. Must be valid or else an error is raised. + layer: region + # Key-value pairs or labels for identifying the document's parent. + parentSelector: + required_key_a: required_label_a + required_key_b: required_label_b + # Actions which specify which data to add to the child document. + actions: + - method: merge + path: .path.to.merge.into.parent + - method: delete + path: .path.to.delete + +.. _layering-actions: + +Layering Actions +---------------- + +Introduction +^^^^^^^^^^^^ + +Layering actions allow child documents to modify data that is inherited from +the parent. What if the child document should only inherit some of the parent +data? No problem. A merge action can be performed, followed by ``delete`` +and ``replace`` actions to trim down on what should be inherited. + +Each layer action consists of an ``action`` and a ``path``. Whenever *any* +action is specified, *all* the parent data is automatically inherited by the +child document. The ``path`` specifies which data from the *child* document to +**prioritize over** that of the parent document. Stated differently, all data +from the parent is considered while *only* the *child* data at ``path`` is +considered during an action. However, whenever a conflict occurs during an +action, the *child* data takes priority over that of the parent. + +Layering actions are queued -- meaning that if a ``merge`` is +specified before a ``replace`` then the ``merge`` will *necessarily* be +applied before the ``replace``. For example, a ``merge`` followed by a +``replace`` **is not necessarily** the same as a ``replace`` followed by a +``merge``. + +Layering actions can be applied to primitives, lists and dictionaries alike. + +Action Types +^^^^^^^^^^^^ + +Supported actions are: + +* ``merge`` - "deep" merge child data and parent data into the child ``data``, + at the specified `JSONPath`_ + + .. note:: + + For conflicts between the child and parent data, the child document's + data is **always** prioritized. No other conflict resolution strategy for + this action currently exists. + + ``merge`` behavior depends upon the data types getting merged. For objects + and lists, Deckhand uses `JSONPath`_ resolution to retrieve data from those + entities, after which Deckhand applies merge strategies (see below) to + combine merge child and parent data into the child document's ``data`` + section. + + **Merge Strategies** + + Deckhand applies the following merge strategies for each data type: + + * object: "Deep-merge" child and parent data together; conflicts are resolved + by prioritizing child data over parent data. "Deep-merge" means + recursively combining data for each key-value pair in both objects. + * array: The merge strategy involves: + + * When using an index in the action ``path`` (e.g. ``a[0]``): + + #. Copying the parent array into the child's ``data`` section at the + specified JSONPath. + #. Appending each child entry in the original child array into the parent + array. This behavior is synonymous with the ``extend`` list function + in Python. + + * When not using an index in the action ``path`` (e.g. ``a``): + + #. The child's array replaces the parent's array. + * primitives: Includes all other data types, except for ``null``. In this + case JSONPath resolution is impossible, so child data is prioritized over + that of the parent. + + **Examples** + + Given:: + + Child Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}`` + Parent Data: ``{'a': {'x': 1, 'y': 2}, 'c': 9}`` + + * When:: + + Merge Path: ``.`` + + Then:: + + Rendered Data: ``{'a': {'x': 7, 'y': 2, 'z': 3}, 'b': 4, 'c': 9}`` + + All data from parent is automatically considered, all data from child + is considered due to ``.`` (selects everything), then both merged. + + * When:: + + Merge Path: ``.a`` + + Then:: + + Rendered Data: ``{'a': {'x': 7, 'y': 2, 'z': 3}, 'c': 9}`` + + All data from parent is automatically considered, all data from child + at ``.a`` is considered, then both merged. + + * When:: + + Merge Path: ``.b`` + + Then:: + + Rendered Data: ``{'a': {'x': 1, 'y': 2}, 'b': 4, 'c': 9}`` + + All data from parent is automatically considered, all data from child + at ``.b`` is considered, then both merged. + + * When:: + + Merge Path: ``.c`` + + Then:: + + Error raised (``.c`` missing in child). + +* ``replace`` - overwrite existing data with child data at the specified + JSONPath. + + **Examples** + + Given:: + + Child Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}`` + Parent Data: ``{'a': {'x': 1, 'y': 2}, 'c': 9}`` + + * When:: + + Replace Path: ``.`` + + Then:: + + Rendered Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}`` + + All data from parent is automatically considered, but is replaced by all + data from child at ``.`` (selects everything), so replaces everything + in parent. + + * When:: + + Replace Path: ``.a`` + + Then:: + + Rendered Data: ``{'a': {'x': 7, 'z': 3}, 'c': 9}`` + + All data from parent is automatically considered, but is replaced by all + data from child at ``.a``, so replaces all parent data at ``.a``. + + * When:: + + Replace Path: ``.b`` + + Then:: + + Rendered Data: ``{'a': {'x': 1, 'y': 2}, 'b': 4, 'c': 9}`` + + All data from parent is automatically considered, but is replaced by all + data from child at ``.b``, so replaces all parent data at ``.b``. + + While ``.b`` isn't in the parent, it only needs to exist in the child. + In this case, something (from the child) replaces nothing (from the + parent). + + * When:: + + Replace Path: ``.c`` + + Then:: + + Error raised (``.c`` missing in child). + +* ``delete`` - remove the existing data at the specified JSONPath. + + **Examples** + + Given:: + + Child Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}`` + Parent Data: ``{'a': {'x': 1, 'y': 2}, 'c': 9}`` + + * When:: + + Delete Path: ``.`` + + Then:: + + Rendered Data: ``{}`` + + Note that deletion of everything results in an empty dictionary by + default. + + * When:: + + Delete Path: ``.a`` + + Then:: + + Rendered Data: ``{'c': 9}`` + + All data from Parent Data at ``.a`` was deleted, rest copied over. + + * When:: + + Delete Path: ``.c`` + + Then:: + + Rendered Data: ``{'a': {'x': 1, 'y': 2}}`` + + All data from Parent Data at ``.c`` was deleted, rest copied over. + + * When:: + + Replace Path: ``.b`` + + Then:: + + Error raised (``.b`` missing in child). After actions are applied for a given layer, substitutions are applied (see -the Substitution section for details). +the :ref:`substitution` section for details). + +.. _JSONPath: http://goessner.net/articles/JsonPath/ .. _parent-selection: Parent Selection ---------------- +Parent selection is performed dynamically. Unlike :ref:`substitution`, +parent selection does not target a specific document using ``schema`` and +``name`` identifiers. Rather, parent selection respects the ``layerOrder``, +selecting the highest precedence parent in accordance with the algorithm that +follows. This allows flexibility in parent selection: if a document's immediate +parent is removed in a revision, then, if applicable, the grandparent (in the +previous revision) can become the document's parent (in the latest revision). + Selection of document parents is controlled by the ``parentSelector`` field and works as follows: diff --git a/tools/build-docs.sh b/tools/build-docs.sh index 8cd81e74..35d42447 100755 --- a/tools/build-docs.sh +++ b/tools/build-docs.sh @@ -4,8 +4,13 @@ # files. Must be run from root project directory. set -ex + +# Generate architectural diagrams. +mkdir -p doc/source/images +python -m plantuml doc/source/diagrams/*.uml +mv doc/source/diagrams/*.png doc/source/images + +# Generate documentation. rm -rf doc/build rm -rf releasenotes/build sphinx-build -W -b html doc/source doc/build/html -python -m plantuml doc/source/diagrams/*.uml -mv doc/source/diagrams/*.png doc/source/images