Add concept of buckets

This enables more declarative delivery of documents from multiple
owners.

- Updates design document:
  - Removes Tombstone concept from the API.
  - Adds and descritbes Bucket concept.
  - Removes planned DELETE endpoints.
  - Adds "purge" endpoint to simplify functional testing.
- Adds functional tests for document crud using buckets.

Change-Id: I7987deb417d34fbd80b888e41e52fabaf330d544
This commit is contained in:
Mark Burnett 2017-08-18 15:58:40 -05:00
parent ee3a96d518
commit bfe930bb8c
12 changed files with 629 additions and 66 deletions

View File

@ -0,0 +1,33 @@
# Tests error paths for document crud.
#
# 1. Purges existing data to ensure test isolation
# 2. Creates a single initial document in one bucket
# 3. Attempts to create the same document in a separate bucket.
# - Verifies 409 response code
defaults:
request_headers:
content-type: application/x-yaml
response_headers:
content-type: application/x-yaml
tests:
- name: purge
desc: Begin testing from known state.
DELETE: /api/v1.0/revisions
status: 204
skip: Not implemented.
- name: create
desc: Create initial documents
PUT: /api/v1.0/bucket/a/documents
status: 201
data: <@resources/sample-document.yaml
skip: Not implemented.
- name: error
desc: Trigger error case
PUT: /api/v1.0/bucket/b/documents
status: 409
data: <@resources/sample-document.yaml
skip: Not implemented.

View File

@ -0,0 +1,101 @@
# Test success paths for document create, read, update and delete using
# multiple buckets.
#
# 1. Purges existing data to ensure test isolation
# 2. Adds documents to bucket a (partial layering sample from the design doc)
# 3. Verifies:
# - The documents have the expected revisions
# - The documents have the specified bucket
# 4. Adds documents to bucket b (remaining layering sample from design doc)
# 5. Verifies:
# - The documents have the expected revisions
# - The documents have the expected buckets
# 6. Verifies the previous revision is unchanged.
defaults:
request_headers:
content-type: application/x-yaml
response_headers:
content-type: application/x-yaml
tests:
- name: purge
desc: Begin testing from known state.
DELETE: /api/v1.0/revisions
status: 204
skip: Not implemented.
- name: create_a
desc: Create documents in one bucket (a)
PUT: /api/v1.0/bucket/a/documents
status: 201
data: <@resources/layering-needs-substitution-source.yaml
skip: Not implemented.
- name: verify_first_bucket
desc: Verify document count and revisions
GET: /api/v1.0/revisions/$RESPONSE['$.documents[0].revision']/documents
status: 200
response_multidoc_jsonpaths:
$.documents[*].metadata.name:
- layering-policy
- abstract-1234
- concrete-1234
$.documents[*].status.revision:
- "$RESPONSE['$.documents[0].revision']"
- "$RESPONSE['$.documents[0].revision']"
- "$RESPONSE['$.documents[0].revision']"
$.documents[*].status.bucket:
- a
- a
- a
skip: Not implemented.
- name: create_b
desc: Create documents in a second bucket (b)
PUT: /api/v1.0/bucket/b/documents
status: 201
data: <@resources/passphrase.yaml
skip: Not implemented.
- name: verify_second_bucket
desc: Verify document count and revisions
GET: /api/v1.0/revisions/$RESPONSE['$.documents[0].revision']/documents
status: 200
response_multidoc_jsonpaths:
$.documents[*].metadata.name:
- layering-policy
- abstract-1234
- concrete-1234
- my-passphrase
$.documents[*].status.revision:
- "$HISTORY['create_a'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['create_a'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['create_a'].$RESPONSE['$.documents[0].revision']"
- "$RESPONSE['$.documents[0].revision']"
$.documents[*].status.bucket:
- a
- a
- a
- b
skip: Not implemented.
- name: verify_first_revision_unchanged
desc: Verify initial revision is unchanged
GET: /api/v1.0/revisions/$HISTORY['create_a'].$RESPONSE['$.documents[0].revision']/documents
status: 200
response_multidoc_jsonpaths:
$.documents[*].metadata.name:
- layering-policy
- abstract-1234
- concrete-1234
$.documents[*].status.revision:
- "$HISTORY['create_a'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['create_a'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['create_a'].$RESPONSE['$.documents[0].revision']"
$.documents[*].status.bucket:
- a
- a
- a
skip: Not implemented.

View File

@ -0,0 +1,230 @@
# Test success paths for document create, read, update and delete using a
# single bucket.
#
# 1. Purges existing data to ensure test isolation
# 2. Creates documents specified in the layering sample of the design document
# 3. Verifies the documents:
# - Have the revision in the response for (2)
# - Have the bucket specified in the request for (2)
# 4. Attempts to PUT the layering sample again (unchanged)
# 5. Verifies that no changes took place.
# 6. Updates a single document in the layering sample.
# 7. Verifies:
# - The documents have the expected revisions
# - The documents have the specified bucket
# - The updated document has the correct content
# 8. Verifies that the initial revision is unmodified:
# - The documents have the expected revisions
# - The documents have the specified bucket
# - The updated document has its original content
# 9. Deletes a single document from the layering sample
# 10. Verifies the state of documents after the delete
# - The correct number of documents is returned
# - The remaining documents have the expected revisions
# - The remaining documents have the specified bucket
# - The updated document has the correct content
# 11. Verifies that the initial revision is unmodified after delete
# - The documents have the expected revisions
# - The documents have the specified bucket
# - The updated document has its original content
# 12. Verifies the "updated" revision is unmodified after delete
# - The documents have the expected revisions
# - The documents have the specified bucket
# - The updated document has the correct content
defaults:
request_headers:
content-type: application/x-yaml
response_headers:
content-type: application/x-yaml
tests:
- name: purge
desc: Begin testing from known state.
DELETE: /api/v1.0/revisions
status: 204
skip: Not implemented.
- name: initialize
desc: Create initial documents
PUT: /api/v1.0/bucket/mop/documents
status: 201
data: <@resources/design-doc-layering-sample.yaml
skip: Not implemented.
- name: verify_initial
desc: Verify initial document count and revisions
GET: /api/v1.0/revisions/$RESPONSE['$.documents[0].revision']/documents
status: 200
response_multidoc_jsonpaths:
$.documents[*].metadata.name:
- layering-policy
- global-1234
- region-1234
- site-1234
$.documents[*].status.revision:
- "$RESPONSE['$.documents[0].revision']"
- "$RESPONSE['$.documents[0].revision']"
- "$RESPONSE['$.documents[0].revision']"
- "$RESPONSE['$.documents[0].revision']"
$.documents[*].status.bucket:
- mop
- mop
- mop
- mop
skip: Not implemented.
- name: ignore_duplicate
desc: Push a duplicate bucket of documents
PUT: /api/v1.0/bucket/mop/documents
status: 200
data: <@resources/design-doc-layering-sample.yaml
skip: Not implemented.
- name: verify_ignore
desc: Verify duplicate documents were ignored
GET: /api/v1.0/revisions/$RESPONSE['$.documents[0].revision']/documents
status: 200
response_multidoc_jsonpaths:
$.documents[*].metadata.name:
- layering-policy
- global-1234
- region-1234
- site-1234
$.documents[*].status.revision:
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
$.documents[*].status.bucket:
- mop
- mop
- mop
- mop
skip: Not implemented.
- name: update_single_document
desc: Update a single document, ignore other documents in the bucket
PUT: /api/v1.0/bucket/mop/documents
status: 201
data: <@resources/design-doc-layering-sample-with-update.yaml
skip: Not implemented.
- name: verify_update
desc: Verify updated document count and revisions
GET: /api/v1.0/revisions/$RESPONSE['$.documents[0].revision']/documents
status: 200
response_multidoc_jsonpaths:
$.documents[*].metadata.name:
- layering-policy
- global-1234
- region-1234
- site-1234
$.documents[*].status.revision:
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$RESPONSE['$.documents[0].revision']"
$.documents[*].status.bucket:
- mop
- mop
- mop
- mop
$.documents[3].data.b: 5
skip: Not implemented.
- name: verify_initial_documents_preserved_after_update
desc: Verify initial documents count and revisions preserved after update
GET: /api/v1.0/revisions/$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']/documents
status: 200
response_multidoc_jsonpaths:
$.documents[*].metadata.name:
- layering-policy
- global-1234
- region-1234
- site-1234
$.documents[*].status.revision:
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
$.documents[*].status.bucket:
- mop
- mop
- mop
- mop
$.documents[3].data.b: 4
skip: Not implemented.
- name: delete_document
desc: Delete a single document
PUT: /api/v1.0/bucket/mop/documents
status: 201
data: <@resources/design-doc-layering-sample-with-delete.yaml
skip: Not implemented.
- name: verify_delete
desc: Verify document deletion
GET: /api/v1.0/revisions/$RESPONSE['$.documents[0].revision']/documents
status: 200
response_multidoc_jsonpaths:
$.documents[*].status.revision:
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['update_with_ignore'].$RESPONSE['$.documents[0].revision']"
$.documents[*].metadata.name:
- layering-policy
- global-1234
- site-1234
$.documents[*].status.bucket:
- mop
- mop
- mop
$.documents[2].data.b: 5
skip: Not implemented.
- name: verify_initial_documents_preserved_after_delete
desc: Verify initial documents count and revisions
GET: /api/v1.0/revisions/$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']/documents
status: 200
response_multidoc_jsonpaths:
$.documents[*].metadata.name:
- layering-policy
- global-1234
- region-1234
- site-1234
$.documents[*].status.revision:
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
$.documents[*].status.bucket:
- mop
- mop
- mop
- mop
$.documents[3].data.b: 4
skip: Not implemented.
- name: verify_updated_documents_preserved_after_delete
desc: Verify updated documents count and revisions preserved after delete
GET: /api/v1.0/revisions/$HISTORY['update_with_ignore'].$RESPONSE['$.documents[0].revision']/documents
status: 200
response_multidoc_jsonpaths:
$.documents[*].metadata.name:
- layering-policy
- global-1234
- region-1234
- site-1234
$.documents[*].status.revision:
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['initialize'].$RESPONSE['$.documents[0].revision']"
- "$HISTORY['update_with_ignore'].$RESPONSE['$.documents[0].revision']"
$.documents[*].status.bucket:
- mop
- mop
- mop
- mop
$.documents[3].data.b: 5
skip: Not implemented.

View File

@ -1,30 +0,0 @@
# Test creation of a single document and verify duplciates will be ignored
defaults:
request_headers:
content-type: application/x-yaml
tests:
- name: create_single_document
desc: Create a sample document
POST: /api/v1.0/documents
status: 201
data: <@resources/sample-doc.yaml
response_headers:
content-type: application/x-yaml
- name: verify_single_document
desc: Check that a single document was created above
GET: /api/v1.0/revisions/$RESPONSE['$.documents[0].revision_id']/documents
status: 200
response_headers:
content-type: application/x-yaml
response_multidoc_jsonpaths:
$.documents[*].metadata.name: a-unique-config-name-12345
- name: create_duplicate_document
desc: Attempt to duplicate sample document
POST: /api/v1.0/documents
status: 204
data: <@resources/sample-doc.yaml

View File

@ -0,0 +1,13 @@
defaults:
request_headers:
content-type: application/x-yaml
response_headers:
content-type: application/x-yaml
tests:
- name: placeholder
desc: |
There must be at least one passing test to make the test harness happy.
This should be removed as soon as there is a passing functional test.
GET: /api/v1.0/revisions
status: 200

View File

@ -0,0 +1,39 @@
---
schema: deckhand/LayeringPolicy/v1
metadata:
schema: metadata/Control/v1
name: layering-policy
data:
layerOrder:
- global
- region
- site
---
schema: example/Kind/v1
metadata:
schema: metadata/Document/v1
name: global-1234
labels:
key1: value1
layeringDefinition:
abstract: true
layer: global
data:
a:
x: 1
y: 2
---
schema: example/Kind/v1
metadata:
schema: metadata/Document/v1
name: site-1234
layeringDefinition:
layer: site
parentSelector:
key1: value1
actions:
- method: merge
path: .
data:
b: 5
...

View File

@ -0,0 +1,57 @@
---
schema: deckhand/LayeringPolicy/v1
metadata:
schema: metadata/Control/v1
name: layering-policy
data:
layerOrder:
- global
- region
- site
---
schema: example/Kind/v1
metadata:
schema: metadata/Document/v1
name: global-1234
labels:
key1: value1
layeringDefinition:
abstract: true
layer: global
data:
a:
x: 1
y: 2
---
schema: example/Kind/v1
metadata:
schema: metadata/Document/v1
name: region-1234
labels:
key1: value1
layeringDefinition:
abstract: true
layer: region
parentSelector:
key1: value1
actions:
- method: replace
path: .a
data:
a:
z: 3
---
schema: example/Kind/v1
metadata:
schema: metadata/Document/v1
name: site-1234
layeringDefinition:
layer: site
parentSelector:
key1: value1
actions:
- method: merge
path: .
data:
b: 5 # This value is updated
...

View File

@ -0,0 +1,57 @@
---
schema: deckhand/LayeringPolicy/v1
metadata:
schema: metadata/Control/v1
name: layering-policy
data:
layerOrder:
- global
- region
- site
---
schema: example/Kind/v1
metadata:
schema: metadata/Document/v1
name: global-1234
labels:
key1: value1
layeringDefinition:
abstract: true
layer: global
data:
a:
x: 1
y: 2
---
schema: example/Kind/v1
metadata:
schema: metadata/Document/v1
name: region-1234
labels:
key1: value1
layeringDefinition:
abstract: true
layer: region
parentSelector:
key1: value1
actions:
- method: replace
path: .a
data:
a:
z: 3
---
schema: example/Kind/v1
metadata:
schema: metadata/Document/v1
name: site-1234
layeringDefinition:
layer: site
parentSelector:
key1: value1
actions:
- method: merge
path: .
data:
b: 4
...

View File

@ -0,0 +1,46 @@
---
schema: deckhand/LayeringPolicy/v1
metadata:
schema: metadata/Control/v1
name: layering-policy
data:
layerOrder:
- global
- region
- site
---
schema: example/Kind/v1
metadata:
schema: metadata/Document/v1
name: abstract-1234
labels:
key1: value1
layeringDefinition:
abstract: true
layer: global
data:
a:
x: 1
y: 2
---
schema: example/Kind/v1
metadata:
schema: metadata/Document/v1
name: concrete-1234
layeringDefinition:
layer: site
parentSelector:
key1: value1
actions:
- method: merge
path: .
substitutions:
- dest:
path: .sub
src:
schema: deckhand/Passphrase/v1
name: my-passphrase
path: .
data:
b: 4
...

View File

@ -0,0 +1,7 @@
---
schema: deckhand/Passphrase/v1
metadata:
schema: metadata/Document/v1
name: my-passphrase
data: not-a-real-password
...

View File

@ -1,7 +1,7 @@
---
schema: promenade/ResourceType/v1.0
schema: promenade/ResourceType/v1
metadata:
schema: metadata/Document/v1.0
schema: metadata/Document/v1
name: a-unique-config-name-12345
labels:
component: apiserver
@ -21,20 +21,20 @@ metadata:
- dest:
path: .chart.values.tls.certificate
src:
schema: deckhand/Certificate/v1.0
schema: deckhand/Certificate/v1
name: example-cert
path: .
- dest:
path: .chart.values.tls.key
src:
schema: deckhand/CertificateKey/v1.0
schema: deckhand/CertificateKey/v1
name: example-key
path: .
- dest:
path: .chart.values.some_url
pattern: INSERT_[A-Z]+_HERE
src:
schema: deckhand/Passphrase/v1.0
schema: deckhand/Passphrase/v1
name: example-password
path: .
data:
@ -43,3 +43,4 @@ data:
data: here
values:
some_url: http://admin:INSERT_PASSWORD_HERE@service-name:8080/v1
...

View File

@ -46,11 +46,10 @@ level of each document is a dictionary with 3 keys: `schema`, `metadata`, and
#### Document Metadata
There are 3 supported kinds of document metadata. Documents with `Document`
There are 2 supported kinds of document metadata. Documents with `Document`
metadata are the most common, and are used for normal configuration data.
Documents with `Control` metadata are used to customize the behavior of
Deckhand. Documents with `Tombstone` metadata are used to delete pre-existing
documents with either `Document` or `Control` metadata.
Deckhand.
##### schema: metadata/Document/v1
@ -142,11 +141,6 @@ actions are not supported on `Control` documents.
The complete list of valid `Control` document kinds is specified below along
with descriptions of each document kind.
##### schema: metadata/Tombstone/v1
The only valid key in a `Tombstone` metadata section is `name`. Additionally,
the top-level `data` section should be omitted.
### Layering
Layering provides a restricted data inheritance model intended to help reduce
@ -554,6 +548,20 @@ data: some-password
...
```
## Buckets
Collections of documents, called buckets, are managed together. All documents
belong to a bucket and all documents that are part of a bucket must be fully
specified together.
To create or update a new document in, e.g. bucket `mop`, one must PUT the
entire set of documents already in `mop` along with the new or modified
document. Any documents not included in that PUT will be automatically
deleted in the created revision.
This feature allows the separation of concerns when delivering different
categories of documents, while making the delivered payload more declarative.
## Revision History
Documents will be ingested in batches which will be given a revision index.
@ -597,6 +605,7 @@ Here is a list of internal validations:
Deckhand will use standard OpenStack Role Based Access Control using the
following actions:
- `purge_database` - Remove all documents and revisions from the database.
- `read_cleartext_document` - Read unencrypted documents.
- `read_encrypted_document` - Read (including substitution and layering)
encrypted documents.
@ -616,21 +625,20 @@ does not provide an official media type for YAML, this API will use
This is a description of the `v1.0` API. Documented paths are considered
relative to `/api/v1.0`.
### POST `/documents`
### PUT `/bucket/{bucket_name}/documents`
Accepts a multi-document YAML body and creates a new revision which adds
those documents. Updates are detected based on exact match to an existing
document of `schema` + `metadata.name`. Documents are "deleted" by including
documents with the tombstone metadata schema, such as:
Accepts a multi-document YAML body and creates a new revision that updates the
contents of the `bucket_name` bucket. Documents from the specified bucket that
exist in previous revisions, but are absent from the request are removed from
that revision (though still accessible via older revisions).
```yaml
---
schema: any-namespace/AnyKind/v1
metadata:
schema: metadata/Tombstone/v1
name: name-to-delete
...
```
Documents in other buckets are not changed and will be included in queries for
documents of the newly created revision.
Updates are detected based on exact match to an existing document of `schema` +
`metadata.name`. It is an error that responds with `409 Conflict` to attempt
to PUT a document with the same `schema` + `metadata.name` as an existing
document from a different bucket in the most-recent revision.
This endpoint is the only way to add, update, and delete documents. This
triggers Deckhand's internal schema validations for all documents.
@ -642,19 +650,11 @@ unnecessary revisions.
This endpoint uses the `write_cleartext_document` and
`write_encrypted_document` actions.
### DELETE `/documents/{{schema}}/{{name}}`
Delete the specified document. This is equivalent to posting a tombstone for
the document.
This endpoint uses the `write_cleartext_document` and
`write_encrypted_document` actions.
### GET `/revisions/{revision_id}/documents`
Returns a multi-document YAML response containing all the documents matching
the filters specified via query string parameters. Returned documents will be
as originally posted with no substitutions or layering applied.
as originally added with no substitutions or layering applied.
Supported query string parameters:
@ -673,6 +673,9 @@ Supported query string parameters:
* `metadata.label` - string, optional, repeatable - Uses the format
`metadata.label=key=value`. Repeating this parameter indicates all
requested labels must apply (AND not OR).
* `status.bucket` - string, optional, repeatable - Used to select documents
only from a particular bucket. Repeating this parameter indicates documents
from any of the specified buckets should be returned.
This endpoint uses the `read_cleartext_document` and
`read_encrypted_document` actions.
@ -721,6 +724,13 @@ results:
This endpoint uses the `read_revision` action.
### DELETE `/revisions`
Permanently delete all documents. This removes all revisions and resets the
data store.
This endpoint uses the `purge_database` action.
### GET `/revisions/{{revision_id}}`
Get a detailed description of a particular revision. The status of each
@ -980,7 +990,6 @@ metadata:
This endpoint uses the `read_tag` action.
### DELETE `/revisions/{{revision_id}}/tags/{{tag}}`
Delete tag associated with a revision.