summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorZuul <zuul@review.openstack.org>2018-10-24 14:03:35 +0000
committerGerrit Code Review <review@openstack.org>2018-10-24 14:03:35 +0000
commit1d4cc81dfaa9a5280625074e569fa4ab9dc7fd09 (patch)
treeb3f29aa59a9080cf87d91ab495edfc534439171d
parentb4cd48cde0225417da8ab149539ee87b1571ee1c (diff)
parent1ac9abb5551f9bec311e55f9846b6d428a1311ec (diff)
Merge "docs: Elaborate on document layering in documentation"
-rw-r--r--doc/source/users/document-types.rst9
-rw-r--r--doc/source/users/documents.rst2
-rw-r--r--doc/source/users/layering.rst376
-rwxr-xr-xtools/build-docs.sh9
4 files changed, 373 insertions, 23 deletions
diff --git a/doc/source/users/document-types.rst b/doc/source/users/document-types.rst
index a84e743..2a702b3 100644
--- a/doc/source/users/document-types.rst
+++ b/doc/source/users/document-types.rst
@@ -111,9 +111,14 @@ correct schemas.
111 111
112.. _JSON schema: http://json-schema.org 112.. _JSON schema: http://json-schema.org
113 113
114.. _layering-policy:
115
114LayeringPolicy 116LayeringPolicy
115^^^^^^^^^^^^^^ 117^^^^^^^^^^^^^^
116 118
119This document defines the strict order in which documents are layered together
120from their component parts.
121
117Only one ``LayeringPolicy`` document can exist within the system at any time. 122Only one ``LayeringPolicy`` document can exist within the system at any time.
118It is an error to attempt to insert a new ``LayeringPolicy`` document if it has 123It is an error to attempt to insert a new ``LayeringPolicy`` document if it has
119a different ``metadata.name`` than the existing document. If the names match, 124a different ``metadata.name`` than the existing document. If the names match,
@@ -127,8 +132,8 @@ it is treated as an update to the existing document.
127 the new ``LayeringPolicy``. 132 the new ``LayeringPolicy``.
128 133
129This document defines the strict order in which documents are merged together 134This document defines the strict order in which documents are merged together
130from their component parts. It should result in a validation error if a 135from their component parts. An error is raised if a document refers to a layer
131document refers to a layer not specified in the ``LayeringPolicy``. 136not specified in the ``LayeringPolicy``.
132 137
133Below is an example of a ``LayeringPolicy`` document: 138Below is an example of a ``LayeringPolicy`` document:
134 139
diff --git a/doc/source/users/documents.rst b/doc/source/users/documents.rst
index e20e4a8..cedc2f8 100644
--- a/doc/source/users/documents.rst
+++ b/doc/source/users/documents.rst
@@ -34,6 +34,8 @@ Detailed documentation for :ref:`layering`, :ref:`substitution`,
34:ref:`revision-history` and :ref:`validation` should be reviewed for a more 34:ref:`revision-history` and :ref:`validation` should be reviewed for a more
35thorough understanding of each concept. 35thorough understanding of each concept.
36 36
37.. _document-format:
38
37Document Format 39Document Format
38--------------- 40---------------
39 41
diff --git a/doc/source/users/layering.rst b/doc/source/users/layering.rst
index 178cfae..bc7d905 100644
--- a/doc/source/users/layering.rst
+++ b/doc/source/users/layering.rst
@@ -23,38 +23,376 @@ Introduction
23------------ 23------------
24 24
25Layering provides a restricted data inheritance model intended to help reduce 25Layering provides a restricted data inheritance model intended to help reduce
26duplication in configuration. Documents with different ``schema``'s are never 26duplication in configuration. With layering, child documents can inherit
27layered together (see the :ref:`substitution` section if you need to combine data 27data from parent documents. Through :ref:`layering-actions`, child documents
28from multiple types of documents). 28can control exactly what they inherit from their parent. Document layering,
29conceptually speaking, works much like class inheritance: A child class
30inherits all variables and methods from its parent, but can elect to override
31its parent's functionality.
32
33Goals behind layering include:
34
35* model site deployment data hierarchically
36* lessen data duplication across site layers (as well as other conceptual
37 layers)
38
39Document Abstraction
40^^^^^^^^^^^^^^^^^^^^
41
42Layering works with :ref:`document-abstraction`: child documents can inherit
43from abstract as well as concrete parent documents.
44
45Pre-Conditions
46^^^^^^^^^^^^^^
47
48A document only has one parent, but its parent is computed dynamically using
49the :ref:`parent-selection` algorithm. That is, the notion of
50"multiple inheritance" **does not** apply to document layering.
51
52Documents with different ``schema`` values are never layered together (see the
53:ref:`substitution` section if you need to combine data from multiple types of
54documents).
55
56Document layering requires a :ref:`layering-policy` to exist in the revision
57whose documents will be layered together (rendered). An error will be issued
58otherwise.
59
60Terminology
61-----------
62
63.. note::
64
65 Whether a layer is "lower" or "higher" has entirely to do with its order of
66 initialization in a ``layerOrder`` and, by extension, its precedence in the
67 :ref:`parent-selection` algorithm described below.
68
69* Layer - A position in a hierarchy used to control :ref:`parent-selection` by
70 the :ref:`layering-algorithm`. It can be likened to a position in an
71 inheritance hierarchy, where ``object`` in Python can be likened to the
72 highest layer in a ``layerOrder`` in Deckhand and a leaf class can be likened
73 to the lowest layer in a ``layerOrder``.
74* Child - Meaningful only in a parent-child document relationship. A document
75 with a lower layer (but higher priority) than its parent, determined using
76 using :ref:`parent-selection`.
77* Parent - Meaningful only in a parent-child document relationship. A document
78 with a higher layer (but lower priority) than its child.
79* Layering Policy - A :ref:`control document <control-documents>` that defines
80 the strict ``layerOrder`` in which documents are layered together. See
81 :ref:`layering-policy` documentation for more information.
82* Layer Order (``layerOrder``) - Corresponds to the ``data.layerOrder`` of the
83 :ref:`layering-policy` document. Establishes the layering hierarchy for a
84 set of layers in the system.
85* Layering Definition (``layeringDefinition``) - Metadata in each document for
86 controlling the following:
87
88 * ``layer``: the document layer itself
89 * ``parentSelector``: :ref:`parent-selection`
90 * ``abstract``: :ref:`document-abstraction`
91 * ``actions``: :ref:`layering-actions`
92
93* Parent Selector (``parentSelector``) - Key-value pairs or labels for
94 identifying the document's parent. Note that these key-value pairs are not
95 unique and that multiple documents can use them. All the key-value pairs
96 in the ``parentSelector`` must be found among the target parent's
97 ``metadata.labels``: this means that the ``parentSelector`` key-value pairs
98 must be a subset of the target parent's ``metadata.labels`` key-value
99 pairs. See :ref:`parent-selection` for further details.
100* Layering Actions (``actions``) - A list of actions that control what data
101 are inherited from the parent by the child. See :ref:`layering-actions`
102 for further details.
103
104.. _layering-algorithm:
105
106Algorithm
107---------
108
109Layering is applied at the bottommost layer of the ``layerOrder`` first and
110at the topmost layer of the ``layerOrder`` last, such that the "base" layers
111are processed first and the "leaf" layers are processed last. For each
112layer in the ``layerOrder``, the documents that correspond to that layer
113are retrieved. For each document retrieved, the ``layerOrder`` hierarchy
114is resolved using :ref:`parent-selection` to identify the parent document.
115Finally, the current document is layered with its parent using
116:ref:`layering-actions`.
117
118After layering is complete, the :ref:`substitution` algorithm is applied to the
119*current* document, if applicable.
120
121.. _layering-configuration:
122
123Layering Configuration
124----------------------
125
126Layering is configured in 2 places:
127
128#. The ``LayeringPolicy`` control document (described in
129 :ref:`layering-policy`), which defines the valid layers and their order of
130 precedence.
131#. In the ``metadata.layeringDefinition`` section of normal
132 (``metadata.schema=metadata/Document/v1``) documents. For more information
133 about document structure, reference :ref:`document-format`.
134
135An example ``layeringDefinition`` may look like::
136
137 layeringDefinition:
138 # Controls whether the document is abstract or concrete.
139 abstract: true
140 # A layer in the ``layerOrder``. Must be valid or else an error is raised.
141 layer: region
142 # Key-value pairs or labels for identifying the document's parent.
143 parentSelector:
144 required_key_a: required_label_a
145 required_key_b: required_label_b
146 # Actions which specify which data to add to the child document.
147 actions:
148 - method: merge
149 path: .path.to.merge.into.parent
150 - method: delete
151 path: .path.to.delete
152
153.. _layering-actions:
154
155Layering Actions
156----------------
157
158Introduction
159^^^^^^^^^^^^
160
161Layering actions allow child documents to modify data that is inherited from
162the parent. What if the child document should only inherit some of the parent
163data? No problem. A merge action can be performed, followed by ``delete``
164and ``replace`` actions to trim down on what should be inherited.
165
166Each layer action consists of an ``action`` and a ``path``. Whenever *any*
167action is specified, *all* the parent data is automatically inherited by the
168child document. The ``path`` specifies which data from the *child* document to
169**prioritize over** that of the parent document. Stated differently, all data
170from the parent is considered while *only* the *child* data at ``path`` is
171considered during an action. However, whenever a conflict occurs during an
172action, the *child* data takes priority over that of the parent.
173
174Layering actions are queued -- meaning that if a ``merge`` is
175specified before a ``replace`` then the ``merge`` will *necessarily* be
176applied before the ``replace``. For example, a ``merge`` followed by a
177``replace`` **is not necessarily** the same as a ``replace`` followed by a
178``merge``.
179
180Layering actions can be applied to primitives, lists and dictionaries alike.
181
182Action Types
183^^^^^^^^^^^^
184
185Supported actions are:
186
187* ``merge`` - "deep" merge child data and parent data into the child ``data``,
188 at the specified `JSONPath`_
189
190 .. note::
191
192 For conflicts between the child and parent data, the child document's
193 data is **always** prioritized. No other conflict resolution strategy for
194 this action currently exists.
195
196 ``merge`` behavior depends upon the data types getting merged. For objects
197 and lists, Deckhand uses `JSONPath`_ resolution to retrieve data from those
198 entities, after which Deckhand applies merge strategies (see below) to
199 combine merge child and parent data into the child document's ``data``
200 section.
201
202 **Merge Strategies**
203
204 Deckhand applies the following merge strategies for each data type:
205
206 * object: "Deep-merge" child and parent data together; conflicts are resolved
207 by prioritizing child data over parent data. "Deep-merge" means
208 recursively combining data for each key-value pair in both objects.
209 * array: The merge strategy involves:
210
211 * When using an index in the action ``path`` (e.g. ``a[0]``):
212
213 #. Copying the parent array into the child's ``data`` section at the
214 specified JSONPath.
215 #. Appending each child entry in the original child array into the parent
216 array. This behavior is synonymous with the ``extend`` list function
217 in Python.
218
219 * When not using an index in the action ``path`` (e.g. ``a``):
220
221 #. The child's array replaces the parent's array.
222 * primitives: Includes all other data types, except for ``null``. In this
223 case JSONPath resolution is impossible, so child data is prioritized over
224 that of the parent.
225
226 **Examples**
227
228 Given::
229
230 Child Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
231 Parent Data: ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
232
233 * When::
234
235 Merge Path: ``.``
236
237 Then::
238
239 Rendered Data: ``{'a': {'x': 7, 'y': 2, 'z': 3}, 'b': 4, 'c': 9}``
240
241 All data from parent is automatically considered, all data from child
242 is considered due to ``.`` (selects everything), then both merged.
243
244 * When::
245
246 Merge Path: ``.a``
247
248 Then::
249
250 Rendered Data: ``{'a': {'x': 7, 'y': 2, 'z': 3}, 'c': 9}``
251
252 All data from parent is automatically considered, all data from child
253 at ``.a`` is considered, then both merged.
254
255 * When::
256
257 Merge Path: ``.b``
258
259 Then::
260
261 Rendered Data: ``{'a': {'x': 1, 'y': 2}, 'b': 4, 'c': 9}``
262
263 All data from parent is automatically considered, all data from child
264 at ``.b`` is considered, then both merged.
265
266 * When::
267
268 Merge Path: ``.c``
269
270 Then::
271
272 Error raised (``.c`` missing in child).
273
274* ``replace`` - overwrite existing data with child data at the specified
275 JSONPath.
29 276
30Layering is controlled in two places: 277 **Examples**
31 278
321. The ``LayeringPolicy`` control document (described below), which defines the 279 Given::
33 valid layers and their order of precedence.
342. In the ``metadata.layeringDefinition`` section of normal
35 (``metadata.schema=metadata/Document/v1``) documents.
36 280
37When rendering a particular document, you resolve the chain of parents upward 281 Child Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
38through the layers, and begin working back down each layer rendering at each 282 Parent Data: ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
39document in the chain.
40 283
41When rendering each layer, the parent document is used as the starting point, 284 * When::
42so the entire contents of the parent are brought forward. Then the list of
43`actions` will be applied in order. Supported actions are:
44 285
45* ``merge`` - "deep" merge child data at the specified path into the existing 286 Replace Path: ``.``
46 data 287
47* ``replace`` - overwrite existing data with child data at the specified path 288 Then::
48* ``delete`` - remove the existing data at the specified path 289
290 Rendered Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
291
292 All data from parent is automatically considered, but is replaced by all
293 data from child at ``.`` (selects everything), so replaces everything
294 in parent.
295
296 * When::
297
298 Replace Path: ``.a``
299
300 Then::
301
302 Rendered Data: ``{'a': {'x': 7, 'z': 3}, 'c': 9}``
303
304 All data from parent is automatically considered, but is replaced by all
305 data from child at ``.a``, so replaces all parent data at ``.a``.
306
307 * When::
308
309 Replace Path: ``.b``
310
311 Then::
312
313 Rendered Data: ``{'a': {'x': 1, 'y': 2}, 'b': 4, 'c': 9}``
314
315 All data from parent is automatically considered, but is replaced by all
316 data from child at ``.b``, so replaces all parent data at ``.b``.
317
318 While ``.b`` isn't in the parent, it only needs to exist in the child.
319 In this case, something (from the child) replaces nothing (from the
320 parent).
321
322 * When::
323
324 Replace Path: ``.c``
325
326 Then::
327
328 Error raised (``.c`` missing in child).
329
330* ``delete`` - remove the existing data at the specified JSONPath.
331
332 **Examples**
333
334 Given::
335
336 Child Data: ``{'a': {'x': 7, 'z': 3}, 'b': 4}``
337 Parent Data: ``{'a': {'x': 1, 'y': 2}, 'c': 9}``
338
339 * When::
340
341 Delete Path: ``.``
342
343 Then::
344
345 Rendered Data: ``{}``
346
347 Note that deletion of everything results in an empty dictionary by
348 default.
349
350 * When::
351
352 Delete Path: ``.a``
353
354 Then::
355
356 Rendered Data: ``{'c': 9}``
357
358 All data from Parent Data at ``.a`` was deleted, rest copied over.
359
360 * When::
361
362 Delete Path: ``.c``
363
364 Then::
365
366 Rendered Data: ``{'a': {'x': 1, 'y': 2}}``
367
368 All data from Parent Data at ``.c`` was deleted, rest copied over.
369
370 * When::
371
372 Replace Path: ``.b``
373
374 Then::
375
376 Error raised (``.b`` missing in child).
49 377
50After actions are applied for a given layer, substitutions are applied (see 378After actions are applied for a given layer, substitutions are applied (see
51the Substitution section for details). 379the :ref:`substitution` section for details).
380
381.. _JSONPath: http://goessner.net/articles/JsonPath/
52 382
53.. _parent-selection: 383.. _parent-selection:
54 384
55Parent Selection 385Parent Selection
56---------------- 386----------------
57 387
388Parent selection is performed dynamically. Unlike :ref:`substitution`,
389parent selection does not target a specific document using ``schema`` and
390``name`` identifiers. Rather, parent selection respects the ``layerOrder``,
391selecting the highest precedence parent in accordance with the algorithm that
392follows. This allows flexibility in parent selection: if a document's immediate
393parent is removed in a revision, then, if applicable, the grandparent (in the
394previous revision) can become the document's parent (in the latest revision).
395
58Selection of document parents is controlled by the ``parentSelector`` field and 396Selection of document parents is controlled by the ``parentSelector`` field and
59works as follows: 397works as follows:
60 398
diff --git a/tools/build-docs.sh b/tools/build-docs.sh
index 8cd81e7..35d4244 100755
--- a/tools/build-docs.sh
+++ b/tools/build-docs.sh
@@ -4,8 +4,13 @@
4# files. Must be run from root project directory. 4# files. Must be run from root project directory.
5 5
6set -ex 6set -ex
7
8# Generate architectural diagrams.
9mkdir -p doc/source/images
10python -m plantuml doc/source/diagrams/*.uml
11mv doc/source/diagrams/*.png doc/source/images
12
13# Generate documentation.
7rm -rf doc/build 14rm -rf doc/build
8rm -rf releasenotes/build 15rm -rf releasenotes/build
9sphinx-build -W -b html doc/source doc/build/html 16sphinx-build -W -b html doc/source doc/build/html
10python -m plantuml doc/source/diagrams/*.uml
11mv doc/source/diagrams/*.png doc/source/images