diff --git a/.gitignore b/.gitignore
index c487c53e..7a17c73d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -107,4 +107,5 @@ AUTHORS
# build/lint artifacts
/charts/shipyard/charts
-/charts/shipyard/requirements.lock
\ No newline at end of file
+/charts/shipyard/requirements.lock
+.DS_Store
\ No newline at end of file
diff --git a/docs/API.md b/docs/API.md
deleted file mode 100644
index 353a9f42..00000000
--- a/docs/API.md
+++ /dev/null
@@ -1,755 +0,0 @@
-# Shipyard API
-
-Logically, the API has three parts to handle the three areas of functionality
-in Shipyard.
-
-1. Document Staging
-2. Action Handling
-3. Airflow Monitoring
-
-
-## Standards used by the API
-See [UCP API conventions](https://github.com/att-comdev/ucp-integration/blob/master/docs/api-conventions.md)
-
-
-## Notes on examples
-Examples assume the following environment variables are set before issuing
-the curl commands shown:
-```
-$TOKEN={a valid keystone token}
-$URL={the url and port of the shipyard api}
-```
-* Examples will use json formatted by the jq command for sake of presentation.
-* Actual responses will not formatted.
-* The use of ellipsis indicate a repeated structure in the case of lists, or
-prior/subsequent structure unimportant to the example (or considered
-understood).
-* The content-length response headers have been removed so as to not cause
-confusion with the listed output.
-
-### Example response for an invalid token:
-
-```
-HTTP/1.1 401 Unauthorized
-content-type: application/json
-x-shipyard-req: a8194b97-8973-4b04-a3b3-2bd319024c5d
-WWW-Authenticate: Keystone uri='http://keystone-api.ucp.svc.cluster.local:80/v3'
-
-{
- "apiVersion": "v1.0",
- "status": "Failure",
- "metadata": {},
- "message": "Unauthenticated",
- "code": "401 Unauthorized",
- "details": {
- "errorList": [
- {
- "message": "Credentials are not established"
- }
- ],
- "errorCount": 1,
- "errorType": "ApiError"
- },
- "kind": "status",
- "reason": "Credentials are not established"
-}
-```
-
-
----
-## Document Staging API
->Shipyard will serve as the entrypoint for documents (designs, secrets,
-configurations, etc...) into a site. Documents are posted to Shipyard in
-collections, rather than individually. At any point in time, there will be two
-represented versions of documents in a site that are accessible via this API:
->
->* The "Committed Documents" version, which represents the last version of
-documents that were successfully commited with a commit_configdocs action.
->* The "Shipyard Buffer" version, which represents the collection of documents
-that have been ingested by this API since the last commited version. Note that
-only one set of documents maybe posted to the buffer at a time by default.
-(This behavior can be overridden by query parameters issued by the user of
-Shipyard)
->
-> All versions of documents rely upon Deckhand for storage. Shipyard uses the
-tagging features of Deckhand of to find the appropriate Committed Documents
-and Shipyard Buffer version.
-
----
-### /v1.0/configdocs/{collection_id}
-Represents the site configuration documents
-
-#### Entity Structure
-The documents as noted above (commonly yaml), in a format understood by
-Deckhand
-
-
-#### POST /v1.0/configdocs/{collection_id}
-Ingests a collection of documents. Synchronous. POSTing an empty body
-indicates that the specified collection should be deleted when the Shipyard
-Buffer is committed. If a POST to the commitconfigdocs is in progress, this
-POST should be rejected with a 409 error.
-
-Important:
-The expected input type for this request is 'Content-Type: application/x-yaml'
-
-##### Query Parameters
-* bufferMode=append|replace|**rejectOnContents**
-Indicates how the existing Shipyard Buffer should be handled. By default,
-Shipyard will reject the POST if contents already exist in the Shipyard
-Buffer.
- * append: Add the collection to the Shipyard Buffer, only if that
- collection doesn't already exist in the Shipyard Buffer. If the collection
- is already present, the request will be rejected and a 409 Conflict will be
- returned.
- * replace: Clear the Shipyard Buffer before adding the specified collection.
-##### Responses
-* 201 Created
-If the documents are successfully ingested, even with validation failures.
-Response message includes:
- * a list of validation results
- * The response headers will include a Location indicating the GET endpoint
- to retrieve the configDocs
-
-* 409 Conflict
- * If a commitconfigdocs POST is in progress.
- * If any collections exist in the Shipyard Buffer unless bufferMode=replace
- or bufferMode=append.
- * If bufferMode=append, and the collection being posted is already in the
- Shipyard Buffer
-
-#### GET /v1.0/configdocs/{collection_id}
-Returns the source documents for a collection of documents
-##### Query Parameters
-* version=committed|**buffer**
-Return the documents for the version specified - buffer by default.
-
-Important:
-The output type for this request is 'Content-Type: application/x-yaml'
-
-##### Responses
-* 200 OK
-If documents can be retrieved.
- * If the response is 200 with an empty response body, this indicates that
-the buffer version is attempting to 'delete' the collection when it is
-committed. An empty response body will only be possible for version=buffer.
-* 404 Not Found
-If the collection is not represented
- * When version=buffer, this indicates that no representations of this
-collection have been POSTed since the last committed version.
- * When version=committed, this indicates that either the collection has
-never existed or has been deleted by a prior commit
-
----
-### /v1.0/renderedconfigdocs
-Represents the site configuration documents, as a whole set - does not
-consider collections in any way.
-
-#### GET /v1.0/renderedconfigdocs
-Returns the full set of configdocs in their rendered form.
-
-Important:
-The output type for this request is 'Content-Type: application/x-yaml'
-
-##### Query Parameters
-* version=committed|**buffer**
-Return the documents for the version specified - buffer by default.
-##### Responses
-* 200 OK
-If documents can be retrieved.
-
----
-### /v1.0/commitconfigdocs
-A RPC style command to trigger a commit of the configuration documents from the
-Shipyard Buffer to the Committed Documents. This resource will support POST
-only.
-#### Entity Structure
-The response will be the list of validations from all downstream systems that
-perform validation during the commit process. The structure will match the
-error response object described in the [UCP API conventions](https://github.com/att-comdev/ucp-integration/blob/master/docs/api-conventions.md)
-and will be an aggregation of each UCP component's responses.
-
-#### POST /v1.0/commitconfigdocs
-Performs the commit of the Shipyard Buffer to the Committed Documents.
-Synchronous. This invokes each of the UCP components to examine the Shipyard
-Buffer version of the configuration documents and aggregate the responses.
-While performing this commit, further POSTing of configdocs, or other commits
-may not be invoked (Shipyard will block those requests with a 409 response).
-If there are any failures to validate, the Shipyard Buffer and Commited
-Documents will remain unchanged. If successful, the Shipyard Buffer will be
-cleared, and the Committed documents will be updated.
-
-**Note**: if there are unhandled runtime errors during the commitconfigdocs POST,
-a deadlock situation may be possible. Future enhancements may improve this
-handling.
-
-##### Query Parameters
-* force=true|**false**
-By default, false, if there are validation failures the POST will fail with
-a 400 response. With force=true, allows for the commit to succeed (with a 200
-response) even if there are validation failures from downstream components. The
-aggregate response of validation failures will be returned in this case, but
-the invalid documents will still be moved from the Shipyard Buffer to the
-Committed Documents.
-
-##### Responses
-* 200 OK
-If the validations are successful. Returns an "empty" structure as as response
-indicating no errors. A 200 may also be returned if there are validation
-failures, but the force=true query parameter was specified. In this case, the
-response will contain the list of validations.
-* 400 Bad Request
-If the validations fail. Returns a populated response structure containing
-the aggregation of the failed validations.
-* 409 Conflict
-If the there is a POST to commitconfigdocs in progress.
----
-## Action API
->The Shipyard Action API is a resource that allows for creation, control and
-investigation of triggered workflows. These actions encapsulate a command
-interface for the Undercloud Platform.
-See [Action Commands](API_action_commands.md) for supported actions
-
----
-### /v1.0/actions
-
-#### Entity Structure
-A list of actions that have been executed through shipyard's action API.
-```
-[
- { Action objects summarized, See below},
- ...
-]
-```
-
-#### GET /v1.0/actions
-Returns the list of actions in the system that have been posted, and are
-accessible to the current user.
-
-##### Responses
-* 200 OK
-If the actions can be retrieved.
-
-##### Example
-```
-$ curl -X GET $URL/api/v1.0/actions -H "X-Auth-Token:$TOKEN"
-
-HTTP/1.1 200 OK
-x-shipyard-req: 0804d13e-08fc-4e60-a819-3b7532cac4ec
-content-type: application/json; charset=UTF-8
-
-[
- {
- "dag_status": "failed",
- "parameters": {},
- "steps": [
- {
- "id": "action_xcom",
- "url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/action_xcom",
- "index": 1,
- "state": "success"
- },
- {
- "id": "dag_concurrency_check",
- "url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/dag_concurrency_check",
- "index": 2,
- "state": "success"
- },
- {
- "id": "preflight",
- "url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/preflight",
- "index": 3,
- "state": "failed"
- },
- ...
- ],
- "action_lifecycle": "Failed",
- "dag_execution_date": "2017-09-23T02:42:12",
- "id": "01BTP9T2WCE1PAJR2DWYXG805V",
- "dag_id": "deploy_site",
- "datetime": "2017-09-23 02:42:06.860597+00:00",
- "user": "shipyard",
- "context_marker": "416dec4b-82f9-4339-8886-3a0c4982aec3",
- "name": "deploy_site"
- },
- ...
-]
-```
-
-#### POST /v1.0/actions
-Creates an action in the system. This will cause some action to start.
-The input body to this post will represent an action object that has at least
-these fields:
-* name
-The name of the action to invoke, as noted in
-[Action Commands](API_action_commands.md)
-* parameters
-A dictionary of parameters to use for the trigger invocation. The supported
-parameters will vary for the action invoked.
-```
-{
- "name" : "action name",
- "parameters" : { varies by action }
-}
-```
-
-The POST will synchronously create the action (a shell object that represents
-a DAG invocation), perform any checks to validate the preconditions to run the
-DAG, and trigger the invocation of the DAG. The DAG will run asynchronously in
-airflow.
-##### Responses
-* 201 Created
-If the action is created successfully, and all preconditions to run the DAG
-are successful. The response body is the action entity created.
-* 400 Bad Request
-If the action name doesn't exist, or the input entity is otherwise malformed.
-* 409 Conflict
-For any failed pre-run validations. The response body is the action entity
-created, with the failed validations. The DAG will not begin execution in this
-case.
-
-##### Example
-```
-$ curl -D - -d '{"name":"deploy_site"}' -X POST $URL/api/v1.0/actions \
- -H "X-Auth-Token:$TOKEN" -H "content-type:application/json"
-
-HTTP/1.1 201 Created
-location: {$URL}/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN
-x-shipyard-req: 629f2ea2-c59d-46b9-8641-7367a91a7016
-content-type: application/json; charset=UTF-8
-
-{
- "dag_status": "SCHEDULED",
- "parameters": {},
- "dag_execution_date": "2017-09-24T19:05:49",
- "id": "01BTTMFVDKZFRJM80FGD7J1AKN",
- "dag_id": "deploy_site",
- "name": "deploy_site",
- "user": "shipyard",
- "context_marker": "629f2ea2-c59d-46b9-8641-7367a91a7016",
- "timestamp": "2017-09-24 19:05:43.603591"
-}
-```
----
-### /v1.0/actions/{action_id}
-Each action will be assigned an unique id that can be used to get details for
-the action, including the execution status.
-
-#### Entity Structure
-All actions will include fields that indicate the following data:
-
- - action_lifecycle
- - A summarized value indicating the status or lifecycle phase of the
- action.
-
- - Pending - The action is scheduled or preparing for execution.
- - Processing - The action is underway.
- - Complete - The action has completed successfully.
- - Failed - The action has encountered an error, and has failed.
- - Paused - The action has been paused by a user.
-
-
- - command audit
- - A list of commands that have been issued against the action. Initially,
- the action listed will be "invoke", but may include "pause", "unpause",
- or "stop" if those commands are issued.
- - context_marker
- - The user supplied or system assigned context marker associated with the
- action
- - dag_execution_date
- - The execution date assigned by the workflow system during action
- creation.
- - dag_status
- - Represents the status that airflow provides for an executing DAG.
- - datetime
- - The time at which the action was invoked.
- - id
- - The identifier for the action, a 26 character ULID assigned during the
-creation of the action.
- - name
- - The name of the action, e.g.: deploy_site.
- - parameters
- - The parameters configuring the action that were supplied by the
-user during action creation.
- - steps
- - The list of steps for the action, including the status for that
- step.
- - user
- - The user who has invoked this action, as acquired from the authorization
- token.
- - validations
- - A list of validations that have been done, including any status
- information for those validations. During the lifecycle of the action,
- this list of validations may continue to grow.
-
-
-#### GET /v1.0/actions/{action_id}
-Returns the action entity for the specified id.
-##### Responses
-* 200 OK
-
-##### Example
-```
-$ curl -D - -X GET $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN \
- -H "X-Auth-Token:$TOKEN"
-
-HTTP/1.1 200 OK
-x-shipyard-req: eb3eacb3-4206-40df-bd91-2a3a6d81cd02
-content-type: application/json; charset=UTF-8
-
-{
- "name": "deploy_site",
- "dag_execution_date": "2017-09-24T19:05:49",
- "validations": [],
- "id": "01BTTMFVDKZFRJM80FGD7J1AKN",
- "dag_id": "deploy_site",
- "command_audit": [
- {
- "id": "01BTTMG16R9H3Z4JVQNBMRV1MZ",
- "action_id": "01BTTMFVDKZFRJM80FGD7J1AKN",
- "datetime": "2017-09-24 19:05:49.530223+00:00",
- "user": "shipyard",
- "command": "invoke"
- }
- ],
- "user": "shipyard",
- "context_marker": "629f2ea2-c59d-46b9-8641-7367a91a7016",
- "datetime": "2017-09-24 19:05:43.603591+00:00",
- "dag_status": "failed",
- "parameters": {},
- "steps": [
- {
- "id": "action_xcom",
- "url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/action_xcom",
- "index": 1,
- "state": "success"
- },
- {
- "id": "dag_concurrency_check",
- "url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/dag_concurrency_check",
- "index": 2,
- "state": "success"
- },
- {
- "id": "preflight",
- "url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/preflight",
- "index": 3,
- "state": "failed"
- },
- {
- "id": "deckhand_get_design_version",
- "url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/deckhand_get_design_version",
- "index": 4,
- "state": null
- },
- ...
- ],
- "action_lifecycle": "Failed"
-}
-
-```
-
----
-### /v1.0/actions/{action_id}/validationdetails/{validation_id}
-Allows for drilldown to validation detailed info.
-
-#### Entity Structure
-The detailed information for a validation
-```
-{ TBD }
-```
-
-#### GET /v1.0/actions/{action_id}/validationdetails/{validation_id}
-Returns the validation detail by Id for the supplied action Id.
-##### Responses
-* 200 OK
-
----
-### /v1.0/actions/{action_id}/steps/{step_id}
-Allow for drilldown to step information. The step information includes details
-of the steps excution, successful or not, and enough to facilitate
-troubleshooting in as easy a fashion as possible.
-
-#### Entity Structure
-A step entity represents detailed information representing a single step of
-execution as part of an action. Not all fields are necessarily represented in
-every returned entity.
-
-
- - dag_id
- - The name/id of the workflow DAG that contains this step.
- - duration
- - The duration (seconds) for the step.
- - end_date
- - The timestamp of the completion of the step.
- - execution_date
- - The execution date of the workflow that contains this step.
- - index
- - The ordinal value representing the position of this step in the sequence
- of steps associated with this step.
- - operator
- - The name of the processing facility used by the workflow system.
- - queued_dttm
- - The timestamp when the step was enqueued by the workflow system.
- - start_date
- - The timestamp for the beginning of execution for this step.
- - state
- - The execution state of the step.
- - task_id
- - The name of the task used by the workflow system (and also representing
- this step name queried in the reqeust.
- - try_number
- - A number of retries taken in the case of failure. Some workflow steps may
- be configured to retry before considering the step truly failed.
-
-
-#### GET /v1.0/actions/{action_id}/steps/{step_id}
-Returns the details for a step by id for the given action by Id.
-##### Responses
-* 200 OK
-
-##### Example
-```
-$ curl -D - \
- -X GET $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/action_xcom \
- -H "X-Auth-Token:$TOKEN"
-
-HTTP/1.1 200 OK
-x-shipyard-req: 72daca4d-1f79-4e08-825f-2ad181912a47
-content-type: application/json; charset=UTF-8
-
-{
- "end_date": "2017-09-24 19:05:59.446213",
- "duration": 0.165181,
- "queued_dttm": "2017-09-24 19:05:52.993983",
- "operator": "PythonOperator",
- "try_number": 1,
- "task_id": "action_xcom",
- "state": "success",
- "execution_date": "2017-09-24 19:05:49",
- "dag_id": "deploy_site",
- "index": 1,
- "start_date": "2017-09-24 19:05:59.281032"
-}
-
-```
----
-### /v1.0/actions/{action_id}/control/{control_verb}
-Allows for issuing DAG controls against an action.
-
-#### Entity Structure
-None, there is no associated response entity for this resource
-
-#### POST /v1.0/actions/{action_id}/{control_verb}
-Trigger a control action against an activity.- this includes: pause, unpause
-##### Responses
-* 202 Accepted
-##### Example
-Failure case - command is invalid for the execution state of the action.
-```
-$ curl -D - \
- -X POST $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/control/pause \
- -H "X-Auth-Token:$TOKEN"
-
-HTTP/1.1 409 Conflict
-content-type: application/json
-x-shipyard-req: 9c9551e0-335c-4297-af93-8440cc6b324f
-
-{
- "apiVersion": "v1.0",
- "status": "Failure",
- "metadata": {},
- "message": "Unable to pause action",
- "code": "409 Conflict",
- "details": {
- "errorList": [
- {
- "message": "dag_run state must be running, but is failed"
- }
- ],
- "errorCount": 1,
- "errorType": "ApiError"
- },
- "kind": "status",
- "reason": "dag_run state must be running, but is failed"
-}
-```
-
-Success case
-```
-$ curl -D - \
- -X POST $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/control/pause \
- -H "X-Auth-Token:$TOKEN"
-
-HTTP/1.1 202 Accepted
-content-length: 0
-x-shipyard-req: 019fae1c-03b0-4af1-b57d-451ae6ddac77
-content-type: application/json; charset=UTF-8
-```
-
-
-
----
-## Airflow Monitoring API
->Airflow has a primary function of scheduling DAGs, as opposed to Shipyard's
-primary case of triggering DAGs.
-Shipyard provides functionality to allow for an operator to monitor and review
-these scheduled workflows (DAGs) in addition to the ones triggered by Shipyard.
-This API will allow for accessing Airflow DAGs of any type -- providing a peek
-into the totality of what is happening in Airflow.
-
----
-### /v1.0/workflows
-The resource that represents DAGs (workflows) in airflow
-
-#### Entity Structure
-A list of objects representing the DAGs that have run in airflow.
-
-#### GET /v1.0/workflows
-Queries airflow for DAGs that are running or have run (successfully or
-unsuccessfully) and provides a summary of those things.
-##### Query parameters
-* since={iso8601 date (past) or duration}
-optional, a boundary in the past within which to retrieve results. Default is
-30 days in the past.
-##### Responses
-* 200 OK
-
-##### Example
-
-Note the workflow_id values, these can be used for drilldown.
-
-```
-curl -D - -X GET $URL/api/v1.0/workflows -H "X-Auth-Token:$TOKEN"
-
-HTTP/1.1 200 OK
-content-type: application/json; charset=UTF-8
-x-shipyard-req: 3ab4ccc6-b956-4c7a-9ae6-183c562d8297
-
-[
- {
- "execution_date": "2017-10-09 21:18:56",
- "end_date": null,
- "workflow_id": "deploy_site__2017-10-09T21:18:56.000000",
- "start_date": "2017-10-09 21:18:56.685999",
- "external_trigger": true,
- "dag_id": "deploy_site",
- "state": "failed",
- "run_id": "manual__2017-10-09T21:18:56"
- },
- {
- "execution_date": "2017-10-09 21:19:03",
- "end_date": null,
- "workflow_id": "deploy_site__2017-10-09T21:19:03.000000",
- "start_date": "2017-10-09 21:19:03.361522",
- "external_trigger": true,
- "dag_id": "deploy_site",
- "state": "failed",
- "run_id": "manual__2017-10-09T21:19:03"
- }
- ...
-]
-```
-
----
-### /v1.0/workflows/{workflow_id}
-
-#### Entity Structure
-An object representing the information available from airflow regarding a DAG's
-execution
-
-#### GET /v1.0/workflows/{id}
-Further details of a particular workflow's steps. All steps of all sub-dags
-will be included in the list of steps, as well as section indicating the
-sub-dags for this parent workflow.
-##### Responses
-* 200 OK
-##### Example
-
-Note that sub_dags can be queried to restrict to only that sub-dag's steps.
-e.g. using this as {workflow_id}:
-deploy_site.preflight.armada_preflight_check__2017-10-09T21:19:03.000000
-
-
-```
-curl -D - \
- -X GET $URL/api/v1.0/workflows/deploy_site__2017-10-09T21:19:03.000000 \
- -H "X-Auth-Token:$TOKEN"
-
-HTTP/1.1 200 OK
-content-type: application/json; charset=UTF-8
-x-shipyard-req: 98d71530-816a-4692-9df2-68f22c057467
-
-{
- "execution_date": "2017-10-09 21:19:03",
- "end_date": null,
- "workflow_id": "deploy_site__2017-10-09T21:19:03.000000",
- "start_date": "2017-10-09 21:19:03.361522",
- "external_trigger": true,
- "steps": [
- {
- "end_date": "2017-10-09 21:19:14.916220",
- "task_id": "action_xcom",
- "start_date": "2017-10-09 21:19:14.798053",
- "duration": 0.118167,
- "queued_dttm": "2017-10-09 21:19:08.432582",
- "try_number": 1,
- "state": "success",
- "operator": "PythonOperator",
- "dag_id": "deploy_site",
- "execution_date": "2017-10-09 21:19:03"
- },
- {
- "end_date": "2017-10-09 21:19:25.283785",
- "task_id": "dag_concurrency_check",
- "start_date": "2017-10-09 21:19:25.181492",
- "duration": 0.102293,
- "queued_dttm": "2017-10-09 21:19:19.283132",
- "try_number": 1,
- "state": "success",
- "operator": "ConcurrencyCheckOperator",
- "dag_id": "deploy_site",
- "execution_date": "2017-10-09 21:19:03"
- },
- {
- "end_date": "2017-10-09 21:20:05.394677",
- "task_id": "preflight",
- "start_date": "2017-10-09 21:19:34.994775",
- "duration": 30.399902,
- "queued_dttm": "2017-10-09 21:19:28.449848",
- "try_number": 1,
- "state": "failed",
- "operator": "SubDagOperator",
- "dag_id": "deploy_site",
- "execution_date": "2017-10-09 21:19:03"
- },
- ...
- ],
- "dag_id": "deploy_site",
- "state": "failed",
- "run_id": "manual__2017-10-09T21:19:03",
- "sub_dags": [
- {
- "execution_date": "2017-10-09 21:19:03",
- "end_date": null,
- "workflow_id": "deploy_site.preflight__2017-10-09T21:19:03.000000",
- "start_date": "2017-10-09 21:19:35.082479",
- "external_trigger": false,
- "dag_id": "deploy_site.preflight",
- "state": "failed",
- "run_id": "backfill_2017-10-09T21:19:03"
- },
- ...,
- {
- "execution_date": "2017-10-09 21:19:03",
- "end_date": null,
- "workflow_id": "deploy_site.preflight.armada_preflight_check__2017-10-09T21:19:03.000000",
- "start_date": "2017-10-09 21:19:48.265023",
- "external_trigger": false,
- "dag_id": "deploy_site.preflight.armada_preflight_check",
- "state": "failed",
- "run_id": "backfill_2017-10-09T21:19:03"
- }
- ]
-}
-```
diff --git a/docs/API_action_commands.md b/docs/API_action_commands.md
deleted file mode 100644
index 64a2410e..00000000
--- a/docs/API_action_commands.md
+++ /dev/null
@@ -1,49 +0,0 @@
-# Action API supported commands
-
-## Supported actions
-These actions are currently supported using the [Action API](API.md#ActionAPI)
-
-None at this time.
-
-----
-## Actions under development
-These actions are under active development
-
-* deploy_site
-Triggers the initial deployment of a site, using the latest committed
-configuration documents. Steps:
- 1) Concurrency check - to prevent concurrent site modifications by
- conflicting actions/workflows.
- 2) Preflight checks - ensures all UCP components are in a responsive state.
- 3) Get design version - uses Deckhand to discover the latest committed
- version of design for the site.
- 4) Validate design - asks each involved UCP component to validate the design.
- This ensures that the design, which was previously committed is valid at the
- present time.
- 5) Drydock build - orchestrates the Drydock component to configure hardware
- and the Kubernetes environment (Drydock -> Promenade)
- 6) Check deployed node status - checks that the deployment of nodes is
- successful.
- 7) Armada build - orchestrates Armada to configure software on the nodes as
- designed.
-
-* update_site
-Triggers the initial deployment of a site, using the latest committed
-configuration documents. Steps: (same as deploy_site)
-
-* redeploy_server
-Using parameters to indicate which server(s), triggers a redeployment of server
-to the last known good design and secrets
-
----
-## Future actions
-These actions are anticipated for development
-* test region
-Invoke site validation testing - perhaps baseline is a invocation of all
-components regular "component" tests. This test would be used as a
-preflight-style test to ensure all components are in a working state.
-
-* test component
-Invoke a particular platform component to test it. This test would be used to
-interrogate a particular platform component to ensure it is in a working state,
-and that its own downstream dependencies are also operational
diff --git a/docs/CLI.md b/docs/CLI.md
deleted file mode 100644
index f008107b..00000000
--- a/docs/CLI.md
+++ /dev/null
@@ -1,644 +0,0 @@
-# Supported Environment Variables
-
-All commands will utilize the following environment variables to determine
-necessary information for execution, unless otherwise noted.
-
-
- - Openstack Keystone Authorization environment variables
- -
- The Shipyard CLI/API Client will check for the presence of appropriate
- environment setup to do authentication on behalf of the user.
- The openrc variables that will be used are as follows:
- OS_PROJECT_DOMAIN_NAME ("default" if not specified)
- OS_USER_DOMAIN_NAME ("default" if not specified)
- OS_PROJECT_NAME
- OS_USERNAME
- OS_PASSWORD
- OS_AUTH_URL - The fully qualified identity endpoint.
- E.g. http://keystone.ucp.fully.qualified.name:80/v3
-
- - Openstack Keystone Authorization environment variables not used
- -
- OS_IDENTITY_API_VERSION -- this value will be ignored as Shipyard only
- supports version 3 at this time
-
-
-
-# Shipyard command options
-The base shipyard command supports options that determine cross-CLI behaviors.
-These options are positionally immediately following the shipyard command as
-shown here:
-```
-shipyard <--these options> subcommands...
-
-shipyard
- [--context-marker=]
- [--debug/--no-debug]
- [--os_{various}=]
- [--output-format=[format | raw | cli]] (default = cli)
-
-```
-
- - --context-marker=<uuid>
- -
- Specifies a UUID (8-4-4-4-12 format) that will be used to correlate logs,
- transactions, etc... in downstream activities triggered by this
- interaction. If not specified, Shipyard will supply a new UUID to serve
- as this marker. (optional)
-
- - --debug | --no-debug (default)
- -
- Enable/disable debugging of this CLI and API client.
-
- - --os_{various}=<value>
- -
- See supported Openstack Keystone Authorization Environment variables above
- for the list of supported names, converting to a downcase version of the
- environment variable.
- E.g.: --os_auth_url=http://keystone.ucp:80/v3
- If not specified, the environment variables matching these actions will be
- used instead. The Keystone os_auth_url should reference the exposed
- keystone:port for the target Shipyard environment, as this Keystone will
- be used to discover the instance of Shipyard. For most invocations other
- than help, a valid combination of values must be resolved to authenticate
- and authorize the user's invocation.
-
- - --output-format=[format | raw | cli] (default = cli)
- -
- Specifies the desired output formating such that:
- format: Display the raw ouptut from the invoked Shipyard API in a
- column restricted mode.
- raw: Display the result from the invoked Shipyard API as-is, without
- modification.
- cli: (default) Display results in a plain text interpretation of the
- response from the invoked Shipyard API.
-
-
-
-# Commit Commands
-## commit configdocs
-Attempts to commit the Shipyard Buffer documents, first invoking validation
-by downstream components.
-```
-shipyard commit configdocs
- [--force]
-
-Example:
- shipyard commit configdocs
-```
-
- - --force
- -
- Force the commit to occur, even if validations fail. Note that while this
- allows for bypassing validations, it does not bypass a failure to
- communicate with the systems that provide the validations.
-
-
-
-### Sample
-```
-TBD
-```
-
----
-
-# Control commands
-## pause, unpause, stop
-Three separate commands with a common format that allow for controlling
-the processing of actions created in Shipyard.
-
- - pause
- - pause something in progress e.g. an executing action
- - unpause
- - unpause something paused e.g. a paused action
- - stop
- - stops an executing or paused item e.g. an action
-
-
-```
-shipyard pause
-
-
-
-shipyard unpause
-
-
-
-shipyard stop
-
-
-
-shipyard
- pause|unpause|stop
-
-
-Example:
-
- shipyard pause action 01BTG32JW87G0YKA1K29TKNAFX
-
- shipyard unpause action 01BTG32JW87G0YKA1K29TKNAFX
-
- shipyard stop action 01BTG32JW87G0YKA1K29TKNAFX
-
- shipyard pause action/01BTG32JW87G0YKA1K29TKNAFX
-```
-
- - <type>
- -
- The type of entity to take action upon. Currently supports: action
-
- - <id>
- -
- The id of the entity to take action upon.
-
- - <qualified name>
- -
- The qualified name of the item to take the specified action upon
-
-
-
-### Sample
-```
-$ shipyard pause action/01BZZMEXAVYGG7BT0BMA3RHYY7
-pause successfully submitted for action 01BZZMEXAVYGG7BT0BMA3RHYY7
-```
-
-A failed command:
-```
-$ shipyard pause action/01BZZK07NF04XPC5F4SCTHNPKN
-Error: Unable to pause action
-Reason: dag_run state must be running, but is failed
-- Error: dag_run state must be running, but is failed
-```
-
----
-
-# Create Commands
-## create action
-Invokes the specified workflow through Shipyard.
-See [Action Commands](API_action_commands.md) for supported actions
-Returns the id of the action invoked so that it can be queried subsequently.
-```
-shipyard create action
-
- --param= (repeatable)
-
-Example:
- shipyard create action redeploy_server --param="server-name=mcp"
-```
-
- - <action command>
- -
- The action to invoke.
-
- - --param=<parameter>
- -
- A parameter to be provided to the action being invoked. (repeatable)
-
-
-
-### Sample
-```
-$ shipyard create action deploy_site
-Name Action Lifecycle
-deploy_site action/01BZZK07NF04XPC5F4SCTHNPKN None
-```
-
----
-
-## create configdocs
-Load documents into the Shipyard Buffer. The use of one or more filename or
-a single directory option must be specified.
-
-```
-shipyard create configdocs
-
- [--append | --replace]
- --filename= (repeatable)
- |
- --directory=
-
-Example:
- shipyard create configdocs design --append --filename=site_design.yaml
-```
-Note: If neither append or replace are specified, the Shipyard API default
-value of rejectoncontents will be used.
-
-Note: Either --filename or --directory must be specified, but both may not be
-specified for the same invocation of shipyard.
-
- - <collection>
- -
- The collection to load.
-
- - --append
- -
- Add the collection to the Shipyard Buffer. This will fail if the
- collection already exists.
-
- - --replace
- -
- Clear the shipyard buffer and replace it with the specified contents.
-
- - --filename=<filename>
- -
- The file name to use as the contents of the collection. (repeatable)
- If any documents specified fail basic validation, all of the documents
- will be rejected. Use of filename parameters may not be used in
- conjunction with the directory parameter.
-
- - --directory=<directory>
- -
- A directory containing documents that will be joined and loaded as a
- collection. Any documents that fail basic validation will reject the
- whole set. Use of the directory parameter may not be used with the filename
- parameter.
-
-
-
-### Sample
-```
-$ shipyard create configdocs coll1 --filename=/home/ubuntu/yaml/coll1.yaml
-Configuration documents added.
-Status: Validations succeeded
-Reason: Validation
-```
-Attempting to load the same collection into the uncommitted buffer.
-```
-$ shipyard create configdocs coll1 --filename=/home/ubuntu/yaml/coll1.yaml
-Error: Invalid collection specified for buffer
-Reason: Buffermode : rejectoncontents
-- Error: Buffer is either not empty or the collection already exists in buffer. Setting a different buffermode may provide the desired functionality
-```
-Replace the buffer with --replace
-```
-$ shipyard create configdocs coll1 --replace --filename=/home/ubuntu/yaml/coll1.yaml
-Configuration documents added.
-Status: Validations succeeded
-Reason: Validation
-```
-
----
-
-# Describe Commands
-## describe
-Retrieves the detailed information about the supplied namespaced item
-```
-shipyard describe
-
-
-Example:
- shipyard describe action/01BTG32JW87G0YKA1K29TKNAFX
- Equivalent to:
- shipyard describe action 01BTG32JW87G0YKA1K29TKNAFX
-
- shipyard describe step/01BTG32JW87G0YKA1K29TKNAFX/preflight
- Equivalent to:
- shipyard describe step preflight --action=01BTG32JW87G0YKA1K29TKNAFX
-
- shipyard describe validation/01BTG32JW87G0YKA1K29TKNAFX/01BTG3PKBS15KCKFZ56XXXBGF2
- Equivalent to:
- shipyard describe validation 01BTG3PKBS15KCKFZ56XXXBGF2 \
- --action=01BTG32JW87G0YKA1K29TKNAFX
-
- shipyard describe workflow/deploy_site__2017-01-01T12:34:56.123456
- Equivalent to:
- shipyard describe workflow deploy_site__2017-01-01T12:34:56.123456
-```
-
----
-
-## describe action
-Retrieves the detailed information about the supplied action id.
-```
-shipyard describe action
-
-
-Example:
- shipyard describe action 01BTG32JW87G0YKA1K29TKNAFX
-```
-
-### Sample
-```
-$ shipyard describe action/01BZZK07NF04XPC5F4SCTHNPKN
-Name: deploy_site
-Action: action/01BZZK07NF04XPC5F4SCTHNPKN
-Lifecycle: Failed
-Parameters: {}
-Datetime: 2017-11-27 20:34:24.610604+00:00
-Dag Status: failed
-Context Marker: 71d4112e-8b6d-44e8-9617-d9587231ffba
-User: shipyard
-
-Steps Index State
-step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom 1 success
-step/01BZZK07NF04XPC5F4SCTHNPKN/dag_concurrency_check 2 success
-step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 3 failed
-step/01BZZK07NF04XPC5F4SCTHNPKN/validate_site_design 4 None
-step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 5 failed
-step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 6 failed
-step/01BZZK07NF04XPC5F4SCTHNPKN/drydock_build 7 None
-
-Commands User Datetime
-invoke shipyard 2017-11-27 20:34:34.443053+00:00
-
-Validations: None
-```
-
----
-
-## describe step
-Retrieves the step details associated with an action and step.
-```
-shipyard describe step
-
- --action=
-
-Example:
- shipyard describe step preflight --action=01BTG32JW87G0YKA1K29TKNAFX
-```
-
- - <step id>
- -
- The id of the step found in the describe action response.
-
- - --action=<action id>
- -
- The action id that provides the context for this step.
-
-
-
-### Sample
-```
-$ shipyard describe step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom
-Name: action_xcom
-Task ID: step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom
-Index: 1
-State: success
-Start Date: 2017-11-27 20:34:45.604109
-End Date: 2017-11-27 20:34:45.818946
-Duration: 0.214837
-Try Number: 1
-Operator: PythonOperator
-```
-
----
-
-## describe validation
-Retrieves the validation details associated with an action and validation id
-```
-shipyard describe validation
-
- --action=
-
-Example:
- shipyard describe validation 01BTG3PKBS15KCKFZ56XXXBGF2 \
- --action=01BTG32JW87G0YKA1K29TKNAFX
-```
-
- - <validation id>
- -
- The id of the validation found in the describe action response.
-
- - --action=<action id>
- -
- The action id that provides the context for this validation.
-
-
-
-### Sample
-```
-TBD
-```
-
----
-
-## describe workflow
-Retrieves the details for a workflow that is running or has run in the workflow
-engine.
-```
-shipyard describe workflow
-
-
-Example:
- shipyard describe workflow deploy_site__2017-01-01T12:34:56.123456
-```
-
- - <workflow id>
- -
- The id of the workflow found in the get workflows response.
-
-
-
-### Sample
-```
-$ shipyard describe workflow deploy_site__2017-11-27T20:34:33.000000
-Workflow: deploy_site__2017-11-27T20:34:33.000000
-State: failed
-Dag ID: deploy_site
-Execution Date: 2017-11-27 20:34:33
-Start Date: 2017-11-27 20:34:33.979594
-End Date: None
-External Trigger: True
-
-Steps State
-action_xcom success
-dag_concurrency_check success
-deckhand_get_design_version failed
-validate_site_design None
-deckhand_get_design_version failed
-deckhand_get_design_version failed
-drydock_build None
-
-Subworkflows:
-Workflow: deploy_site.deckhand_get_design_version__2017-11-27T20:34:33.000000
-State: failed
-Dag ID: deploy_site.deckhand_get_design_version
-Execution Date: 2017-11-27 20:34:33
-Start Date: 2017-11-27 20:35:06.281825
-End Date: None
-External Trigger: False
-
-Workflow: deploy_site.deckhand_get_design_version.deckhand_get_design_version__2017-11-27T20:34:33.000000
-State: failed
-Dag ID: deploy_site.deckhand_get_design_version.deckhand_get_design_version
-Execution Date: 2017-11-27 20:34:33
-Start Date: 2017-11-27 20:35:20.725506
-End Date: None
-External Trigger: False
-```
-
----
-
-# Get Commands
-## get actions
-Lists the actions that have been invoked.
-```
-shipyard get actions
-```
-
-### Sample
-```
-$ shipyard get actions
-Name Action Lifecycle
-deploy_site action/01BZZK07NF04XPC5F4SCTHNPKN Failed
-update_site action/01BZZKMW60DV2CJZ858QZ93HRS Processing
-```
-
----
-
-## get configdocs
-Retrieve documents loaded into Shipyard, either committed or from the
-Shipyard Buffer.
-```
-shipyard get configdocs
-
- [--committed | --buffer]
-
-Example:
- shipyard get configdocs design
-```
-
- - <collection>
- -
- The collection to retrieve for viewing.
-
- - --committed
- -
- Retrieve the documents that have last been committed for this
- collection
-
- - --buffer
- -
- Retrive the documents that have been loaded into Shipyard
- since the prior commit. If no documents have been loaded
- into the buffer for this collection, this will return an empty
- response (default)
-
-
-
-### Sample
-```
-$ shipyard get configdocs coll1
-data:
- chart_groups: [kubernetes-proxy, container-networking, dns, kubernetes, kubernetes-rbac]
- release_prefix: ucp
-id: 1
-metadata:
- layeringDefinition: {abstract: false, layer: site}
- name: cluster-bootstrap-1
- schema: metadata/Document/v1.0
- storagePolicy: cleartext
-schema: armada/Manifest/v1.0
-status: {bucket: coll1, revision: 1}
-```
-
----
-
-## get renderedconfigdocs
-Retrieve the rendered version of documents loaded into Shipyard. Rendered
-documents are the "final" version of the documents after applying Deckhand
-layering and substitution.
-```
-shipyard get renderedconfigdocs
- [--committed | --buffer]
-
-Example:
- shipyard get renderedconfigdocs
-```
-
- - --committed
- -
- Retrieve the documents that have last been committed.
-
- - --buffer
- -
- Retrieve the documents that have been loaded into Shipyard
- since the prior commit. (default)
-
-
-
-### Sample
-```
-$ shipyard get renderedconfigdocs
-data:
- chart_groups: [kubernetes-proxy, container-networking, dns, kubernetes, kubernetes-rbac]
- release_prefix: ucp
-id: 1
-metadata:
- layeringDefinition: {abstract: false, layer: site}
- name: cluster-bootstrap-1
- schema: metadata/Document/v1.0
- storagePolicy: cleartext
-schema: armada/Manifest/v1.0
-status: {bucket: coll1, revision: 1}
-```
-
----
-
-## get workflows
-Retrieve workflows that are running or have run in the workflow engine. This
-includes processses that may not have been started as an action
-(e.g. scheduled tasks).
-```
-shipyard get workflows
- [--since=]
-
-Example:
- shipyard get workflows
-
- shipyard get workflows --since=2017-01-01T12:34:56.123456
-```
-
- - --since=<date>
- -
- The historical cutoff date to limit the results of of this response.
-
-
-
-### Sample
-```
-$ shipyard get workflows
-Workflows State
-deploy_site__2017-11-27T20:34:33.000000 failed
-update_site__2017-11-27T20:45:47.000000 running
-```
-
----
-
-# help commands
-Provides topical help for shipyard. Note that --help will provide more
-specific command help.
-```
-shipyard help
- []
-
-Example:
- shipyard help configdocs
-```
-
- - <topic>
- -
- The topic of the help to be displayed. If this parameter is not specified
- the list of avaialable topics will be displayed.
-
-
-
-### Sample
-```
-$ shipyard help
-THE SHIPYARD COMMAND
-The base shipyard command supports options that determine cross-CLI behaviors.
-
-FORMAT
-shipyard [--context-marker=] [--os_{various}=]
- [--debug/--no-debug] [--output-format]
-
-Please Note: --os_auth_url is required for every command except shipyard help
- .
-
-TOPICS
-For information of the following topics, run shipyard help
- actions
- configdocs
-```
\ No newline at end of file
diff --git a/docs/README.md b/docs/README.md
index 811779a5..59b9a3e2 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -1,49 +1,55 @@
-## Shipyard ##
+## Shipyard
-Shipyard adopts the Falcon web framework and uses Apache Airflow as the backend engine to programmatically
-author, schedule and monitor workflows.
+Shipyard adopts the Falcon web framework and uses Apache Airflow as the backend
+engine to programmatically author, schedule and monitor workflows.
The current workflow is as follows:
-1. Inital region/site data will be passed to Shipyard from either a human operator or Jenkins
-2. The data (in YAML format) will be sent to [DeckHand](https://github.com/att-comdev/deckhand) for processing
-3. Shipyard will make use of the post-processed data from DeckHand to interact with [DryDock](https://github.com/att-comdev/drydock)
-4. DryDock will interact with [Promenade](https://github.com/att-comdev/promenade) to provision and deploy
- bare metal nodes using Ubuntu MAAS and a resilient Kubernetes cluster will be created at the end of the
- process
-5. Once the Kubernetes clusters are up and validated to be working properly, Shipyard will interact with
- [Armada](https://github.com/att-comdev/armada) to deploy OpenStack using [OpenStack Helm](https://github.com/openstack/openstack-helm)
-6. Once the OpenStack cluster is deployed, Shipyard will trigger a workflow to perform basic sanity health
- checks on the cluster
+1. Inital region/site data will be passed to Shipyard from either a human
+operator or Jenkins
+2. The data (in YAML format) will be sent to
+ [DeckHand](https://github.com/att-comdev/deckhand) for validation and
+ storage
+3. Shipyard will make use of the post-processed data from DeckHand to interact
+ with [DryDock](https://github.com/att-comdev/drydock)
+4. DryDock will interact with
+ [Promenade](https://github.com/att-comdev/promenade) to provision and deploy
+ bare metal nodes using Ubuntu MAAS and a resilient Kubernetes cluster will
+ be created at the end of the process
+5. Once the Kubernetes clusters are up and validated to be working properly,
+ Shipyard will interact with [Armada](https://github.com/att-comdev/armada)
+ to deploy OpenStack using
+ [OpenStack Helm](https://github.com/openstack/openstack-helm)
+6. Once the OpenStack cluster is deployed, Shipyard will trigger a workflow to
+ perform basic sanity health checks on the cluster
+
+Note: This project, along with the tools used within are community-based and
+open sourced.
+
+### Mission
+
+The goal for Shipyard is to provide a customizable *framework* for operators
+and developers alike. This framework will enable end-users to orchestrate and
+deploy a fully functional container-based Cloud.
-Note: This project, along with the tools used within are community-based and open sourced.
+### Roadmap
+The detailed Roadmap can be viewed on the
+[LCOO JIRA](https://openstack-lcoo.atlassian.net/projects/SHIPYARD/issues/)
-### Mission ###
-
-The goal for Shipyard is to provide a customizable *framework* for operators and developers alike. This
-framework will enable end-users to orchestrate and deploy a fully functional container-based Cloud
-
-
-### Roadmap ###
-
-The detailed Roadmap can be viewed on the [LCOO JIRA](https://openstack-lcoo.atlassian.net/projects/SHIPYARD/issues/)
-
-- Create Helm Charts for Airflow
-- Create Helm Charts for Shipyard
-- Create Falcon API Framework
- Integrate with DeckHand, DryDock/Promenade, Armada
+- Expand funcitionality to provide visibility into and options for operation of
+ a UCP installation
+
+### Getting Started
+
+This project is under development at the moment. We encourage anyone who is
+interested in Shipyard to review our
+[documentation](http://shipyard.readthedocs.io/en/latest/)
-### Getting Started ###
-
-This project is under development at the moment. We encourage anyone who is interested in Shipyard to review
-our [Installation](https://github.com/att-comdev/shipyard/blob/master/docs/deployment_guide.md) documentation
-
-
-### Bugs ###
-
-Bugs are traced in [LCOO JIRA](https://openstack-lcoo.atlassian.net/projects/SHIPYARD/issues/). If you find
-a bug, please feel free to create a [GitHub issue](https://github.com/att-comdev/shipyard_issues) and it will be synced to JIRA.
+### Bugs
+If you find a bug, please feel free to create a
+[GitHub issue](https://github.com/att-comdev/shipyard/issues)
diff --git a/docs/source/API.rst b/docs/source/API.rst
new file mode 100644
index 00000000..d75286c4
--- /dev/null
+++ b/docs/source/API.rst
@@ -0,0 +1,894 @@
+.. _shipyard_api:
+
+Shipyard API
+============
+Logically, the API has three parts to handle the three areas of
+functionality in Shipyard.
+
+1. Document Staging
+2. Action Handling
+3. Airflow Monitoring
+
+Standards used by the API
+-------------------------
+See `UCP API
+conventions `__
+
+Notes on examples
+-----------------
+Examples assume the following environment variables are set before
+issuing the curl commands shown:
+
+::
+
+ $TOKEN={a valid keystone token}
+ $URL={the url and port of the shipyard api}
+
+- Examples will use json formatted by the jq command for sake of
+ presentation.
+- Actual responses will not formatted.
+- The use of ellipsis indicate a repeated structure in the case of
+ lists, or prior/subsequent structure unimportant to the example (or
+ considered understood).
+- The content-length response headers have been removed so as to not
+ cause confusion with the listed output.
+
+Example response for an invalid token:
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ HTTP/1.1 401 Unauthorized
+ content-type: application/json
+ x-shipyard-req: a8194b97-8973-4b04-a3b3-2bd319024c5d
+ WWW-Authenticate: Keystone uri='http://keystone-api.ucp.svc.cluster.local:80/v3'
+
+ {
+ "apiVersion": "v1.0",
+ "status": "Failure",
+ "metadata": {},
+ "message": "Unauthenticated",
+ "code": "401 Unauthorized",
+ "details": {
+ "errorList": [
+ {
+ "message": "Credentials are not established"
+ }
+ ],
+ "errorCount": 1,
+ "errorType": "ApiError"
+ },
+ "kind": "status",
+ "reason": "Credentials are not established"
+ }
+
+Document Staging API
+--------------------
+Shipyard will serve as the entrypoint for documents (designs, secrets,
+configurations, etc...) into a site. Documents are posted to Shipyard in
+collections, rather than individually. At any point in time, there will be two
+represented versions of documents in a site that are accessible via this API:
+
+- The "Committed Documents" version, which represents the last version of
+ documents that were successfully commited with a commit_configdocs action.
+- The "Shipyard Buffer" version, which represents the collection of documents
+ that have been ingested by this API since the last commited version. Note
+ that only one set of documents maybe posted to the buffer at a time by
+ default. (This behavior can be overridden by query parameters issued by the
+ user of Shipyard)
+
+All versions of documents rely upon Deckhand for storage. Shipyard uses the
+tagging features of Deckhand of to find the appropriate Committed Documents
+and Shipyard Buffer version.
+
+/v1.0/configdocs/{collection_id}
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Represents the site configuration documents
+
+Entity Structure
+^^^^^^^^^^^^^^^^
+The documents as noted above (commonly yaml), in a format understood by
+Deckhand
+
+POST /v1.0/configdocs/{collection_id}
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Ingests a collection of documents. Synchronous. POSTing an empty body
+indicates that the specified collection should be deleted when the
+Shipyard Buffer is committed. If a POST to the commitconfigdocs is in
+progress, this POST should be rejected with a 409 error.
+
+.. note::
+
+ The expected input type for this request is ‘Content-Type: application/x-yaml’
+
+
+Query Parameters
+''''''''''''''''
+
+- bufferMode=append|replace\|\ **rejectOnContents**
+ Indicates how the existing Shipyard Buffer should be handled. By default,
+ Shipyard will reject the POST if contents already exist in the Shipyard
+ Buffer.
+
+ - append: Add the collection to the Shipyard Buffer, only if that
+ collection doesn’t already exist in the Shipyard Buffer. If the
+ collection is already present, the request will be rejected and a 409
+ Conflict will be returned.
+ - replace: Clear the Shipyard Buffer before adding the specified
+ collection.
+
+Responses
+'''''''''
+201 Created
+ If the documents are successfully ingested, even with validation failures.
+ Response message includes:
+
+ - a list of validation results
+ - The response headers will include a Location indicating the GET
+ endpoint to retrieve the configDocs
+
+409 Conflict
+ A condition in the system is blocking this document ingestion
+
+ - If a commitconfigdocs POST is in progress.
+ - If any collections exist in the Shipyard Buffer unless bufferMode=replace
+ or bufferMode=append.
+ - If bufferMode=append, and the collection being posted is already in the
+ Shipyard Buffer
+
+GET /v1.0/configdocs/{collection_id}
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Returns the source documents for a collection of documents
+
+.. note::
+
+ The output type for this request is ‘Content-Type: application/x-yaml’
+
+Query Parameters
+''''''''''''''''
+version=committed | **buffer**
+ Return the documents for the version specified - buffer by default.
+
+Responses
+'''''''''
+200 OK
+ If documents can be retrieved.
+
+ - If the response is 200 with an empty response body, this indicates
+ that the buffer version is attempting to ‘delete’ the collection
+ when it is committed. An empty response body will only be possible
+ for version=buffer.
+
+404 Not Found
+ If the collection is not represented
+
+ - When version=buffer, this indicates that no representations of this
+ collection have been POSTed since the last committed version.
+ - When version=committed, this indicates that either the collection has
+ never existed or has been deleted by a prior commit.
+
+/v1.0/renderedconfigdocs
+~~~~~~~~~~~~~~~~~~~~~~~~
+Represents the site configuration documents, as a whole set - does not
+consider collections in any way.
+
+GET /v1.0/renderedconfigdocs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Returns the full set of configdocs in their rendered form.
+
+.. note::
+
+ The output type for this request is 'Content-Type: application/x-yaml'
+
+Query Parameters
+''''''''''''''''
+version=committed|**buffer**
+ Return the documents for the version specified - buffer by default.
+
+Responses
+'''''''''
+200 OK
+ If documents can be retrieved.
+
+
+/v1.0/commitconfigdocs
+~~~~~~~~~~~~~~~~~~~~~~
+An RPC style command to trigger a commit of the configuration documents from
+the Shipyard Buffer to the Committed Documents. This resource will support POST
+only.
+
+Entity Structure
+^^^^^^^^^^^^^^^^
+The response will be the list of validations from all downstream systems that
+perform validation during the commit process. The structure will match the
+error response object described in the `UCP API
+conventions `__
+and will be an aggregation of each UCP component’s responses.
+
+POST /v1.0/commitconfigdocs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Synchronous. Performs the commit of the Shipyard Buffer to the Committed
+Documents. This invokes each of the UCP components to examine the Shipyard
+Buffer version of the configuration documents and aggregate the responses.
+While performing this commit, further POSTing of configdocs, or other commits
+may not be invoked (Shipyard will block those requests with a 409 response). If
+there are any failures to validate, the Shipyard Buffer and Commited Documents
+will remain unchanged. If successful, the Shipyard Buffer will be cleared, and
+the Committed documents will be updated.
+
+.. note::
+
+ If there are unhandled runtime errors during the commitconfigdocs POST, a
+ deadlock situation may be possible. Future enhancements may improve this
+ handling.
+
+Query Parameters
+''''''''''''''''
+force=true | **false**
+ By default, false, if there are validation failures the POST will
+ fail with a 400 response. With force=true, allows for the commit to
+ succeed (with a 200 response) even if there are validation failures
+ from downstream components. The aggregate response of validation
+ failures will be returned in this case, but the invalid documents
+ will still be moved from the Shipyard Buffer to the Committed
+ Documents.
+
+Responses
+'''''''''
+200 OK
+ If the validations are successful. Returns an “empty” structure as as
+ response indicating no errors. A 200 may also be returned if there
+ are validation failures, but the force=true query parameter was
+ specified. In this case, the response will contain the list of
+ validations.
+400 Bad Request
+ If the validations fail. Returns a populated response structure
+ containing the aggregation of the failed validations.
+409 Conflict
+ If the there is a POST to commitconfigdocs in progress.
+
+Example
+'''''''
+
+::
+
+ {
+ "apiVersion": "v1",
+ "code": "400 Bad Request",
+ "details": {
+ "errorCount": 2,
+ "messageList": [
+ {
+ "error": true,
+ "message": "Error loading effective site: 'NoneType' object is not iterable",
+ "name": "Drydock"
+ },
+ {
+ "error": true,
+ "message": "Armada unable to validate configdocs",
+ "name": "Armada"
+ }
+ ]
+ },
+ "kind": "Status",
+ "message": "Validations failed",
+ "metadata": {},
+ "reason": "Validation",
+ "status": "Invalid"
+ }
+
+Action API
+----------
+The Shipyard Action API is a resource that allows for creation, control and
+investigation of triggered workflows. These actions encapsulate a command
+interface for the Undercloud Platform. See :ref:`shipyard_action_commands` for
+supported actions
+
+/v1.0/actions
+~~~~~~~~~~~~~
+
+Entity Structure
+^^^^^^^^^^^^^^^^
+A list of actions that have been executed through shipyard's action API.
+
+::
+
+ [
+ { Action objects summarized, See below},
+ ...
+ ]
+
+
+GET /v1.0/actions
+^^^^^^^^^^^^^^^^^
+Returns the list of actions in the system that have been posted, and are
+accessible to the current user.
+
+Responses
+'''''''''
+200 OK
+ If the actions can be retrieved.
+
+Example
+'''''''
+
+::
+
+ $ curl -X GET $URL/api/v1.0/actions -H "X-Auth-Token:$TOKEN"
+
+ HTTP/1.1 200 OK
+ x-shipyard-req: 0804d13e-08fc-4e60-a819-3b7532cac4ec
+ content-type: application/json; charset=UTF-8
+
+ [
+ {
+ "dag_status": "failed",
+ "parameters": {},
+ "steps": [
+ {
+ "id": "action_xcom",
+ "url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/action_xcom",
+ "index": 1,
+ "state": "success"
+ },
+ {
+ "id": "dag_concurrency_check",
+ "url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/dag_concurrency_check",
+ "index": 2,
+ "state": "success"
+ },
+ {
+ "id": "preflight",
+ "url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/preflight",
+ "index": 3,
+ "state": "failed"
+ },
+ ...
+ ],
+ "action_lifecycle": "Failed",
+ "dag_execution_date": "2017-09-23T02:42:12",
+ "id": "01BTP9T2WCE1PAJR2DWYXG805V",
+ "dag_id": "deploy_site",
+ "datetime": "2017-09-23 02:42:06.860597+00:00",
+ "user": "shipyard",
+ "context_marker": "416dec4b-82f9-4339-8886-3a0c4982aec3",
+ "name": "deploy_site"
+ },
+ ...
+ ]
+
+POST /v1.0/actions
+^^^^^^^^^^^^^^^^^^
+Creates an action in the system. This will cause some action to start. The
+input body to this post will represent an action object that has at least these
+fields:
+
+name
+ The name of the action to invoke, as noted in :ref:`shipyard_action_commands`
+
+parameters
+ A dictionary of parameters to use for the trigger invocation. The supported
+ parameters will vary for the action invoked.
+
+ ::
+
+ {
+ "name" : "action name",
+ "parameters" : { varies by action }
+ }
+
+The POST will synchronously create the action (a shell object that represents
+a DAG invocation), perform any checks to validate the preconditions to run the
+DAG, and trigger the invocation of the DAG. The DAG will run asynchronously in
+airflow.
+
+Responses
+'''''''''
+201 Created
+ If the action is created successfully, and all preconditions to run the DAG
+ are successful. The response body is the action entity created.
+400 Bad Request
+ If the action name doesn't exist, or the input entity is otherwise malformed.
+409 Conflict
+ For any failed pre-run validations. The response body is the action entity
+ created, with the failed validations. The DAG will not begin execution in
+ this case.
+
+Example
+'''''''
+
+::
+
+ $ curl -D - -d '{"name":"deploy_site"}' -X POST $URL/api/v1.0/actions \
+ -H "X-Auth-Token:$TOKEN" -H "content-type:application/json"
+
+ HTTP/1.1 201 Created
+ location: {$URL}/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN
+ x-shipyard-req: 629f2ea2-c59d-46b9-8641-7367a91a7016
+ content-type: application/json; charset=UTF-8
+
+ {
+ "dag_status": "SCHEDULED",
+ "parameters": {},
+ "dag_execution_date": "2017-09-24T19:05:49",
+ "id": "01BTTMFVDKZFRJM80FGD7J1AKN",
+ "dag_id": "deploy_site",
+ "name": "deploy_site",
+ "user": "shipyard",
+ "context_marker": "629f2ea2-c59d-46b9-8641-7367a91a7016",
+ "timestamp": "2017-09-24 19:05:43.603591"
+ }
+
+/v1.0/actions/{action_id}
+~~~~~~~~~~~~~~~~~~~~~~~~~
+Each action will be assigned an unique id that can be used to get
+details for the action, including the execution status.
+
+Entity Structure
+^^^^^^^^^^^^^^^^
+All actions will include fields that indicate the following data:
+
+action_lifecycle
+ A summarized value indicating the status or lifecycle phase of the action.
+
+ - Pending - The action is scheduled or preparing for execution.
+ - Processing - The action is underway.
+ - Complete - The action has completed successfully.
+ - Failed - The action has encountered an error, and has failed.
+ - Paused - The action has been paused by a user.
+
+command audit
+ A list of commands that have been issued against the action. Initially,
+ the action listed will be “invoke”, but may include “pause”, “unpause”,
+ or “stop” if those commands are issued.
+
+context_marker
+ The user supplied or system assigned context marker associated with the
+ action
+
+dag_execution_date
+ The execution date assigned by the workflow system during action
+ creation.
+
+dag_status
+ Represents the status that airflow provides for an executing DAG.
+
+datetime
+ The time at which the action was invoked.
+
+id
+ The identifier for the action, a 26 character ULID assigned during the
+ creation of the action.
+
+name
+ The name of the action, e.g.: deploy_site.
+
+parameters
+ The parameters configuring the action that were supplied by the user
+ during action creation.
+
+steps
+ The list of steps for the action, including the status for that step.
+
+user
+ The user who has invoked this action, as acquired from the authorization
+ token.
+
+validations
+ A list of validations that have been done, including any status
+ information for those validations. During the lifecycle of the action,
+ this list of validations may continue to grow.
+
+GET /v1.0/actions/{action_id}
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Returns the action entity for the specified id.
+
+Responses
+'''''''''
+200 OK
+
+Example
+'''''''
+
+::
+
+ $ curl -D - -X GET $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN \
+ -H "X-Auth-Token:$TOKEN"
+
+ HTTP/1.1 200 OK
+ x-shipyard-req: eb3eacb3-4206-40df-bd91-2a3a6d81cd02
+ content-type: application/json; charset=UTF-8
+
+ {
+ "name": "deploy_site",
+ "dag_execution_date": "2017-09-24T19:05:49",
+ "validations": [],
+ "id": "01BTTMFVDKZFRJM80FGD7J1AKN",
+ "dag_id": "deploy_site",
+ "command_audit": [
+ {
+ "id": "01BTTMG16R9H3Z4JVQNBMRV1MZ",
+ "action_id": "01BTTMFVDKZFRJM80FGD7J1AKN",
+ "datetime": "2017-09-24 19:05:49.530223+00:00",
+ "user": "shipyard",
+ "command": "invoke"
+ }
+ ],
+ "user": "shipyard",
+ "context_marker": "629f2ea2-c59d-46b9-8641-7367a91a7016",
+ "datetime": "2017-09-24 19:05:43.603591+00:00",
+ "dag_status": "failed",
+ "parameters": {},
+ "steps": [
+ {
+ "id": "action_xcom",
+ "url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/action_xcom",
+ "index": 1,
+ "state": "success"
+ },
+ {
+ "id": "dag_concurrency_check",
+ "url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/dag_concurrency_check",
+ "index": 2,
+ "state": "success"
+ },
+ {
+ "id": "preflight",
+ "url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/preflight",
+ "index": 3,
+ "state": "failed"
+ },
+ {
+ "id": "deckhand_get_design_version",
+ "url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/deckhand_get_design_version",
+ "index": 4,
+ "state": null
+ },
+ ...
+ ],
+ "action_lifecycle": "Failed"
+ }
+
+/v1.0/actions/{action_id}/validationdetails/{validation_id}
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Allows for drilldown to validation detailed info.
+
+Entity Structure
+^^^^^^^^^^^^^^^^
+The detailed information for a validation
+
+::
+
+ { TBD }
+
+GET /v1.0/actions/{action_id}/validationdetails/{validation_id}
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Returns the validation detail by Id for the supplied action Id.
+
+Responses
+'''''''''
+200 OK
+
+/v1.0/actions/{action_id}/steps/{step_id}
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Allow for drilldown to step information. The step information includes
+details of the steps excution, successful or not, and enough to
+facilitate troubleshooting in as easy a fashion as possible.
+
+Entity Structure
+^^^^^^^^^^^^^^^^
+A step entity represents detailed information representing a single step
+of execution as part of an action. Not all fields are necessarily
+represented in every returned entity.
+
+dag_id
+ The name/id of the workflow DAG that contains this step.
+
+duration
+ The duration (seconds) for the step.
+
+end_date
+ The timestamp of the completion of the step.
+
+execution_date
+ The execution date of the workflow that contains this step.
+
+index
+ The numeric value representing the position of this step in the sequence
+ of steps associated with this step.
+
+operator
+ The name of the processing facility used by the workflow system.
+
+queued_dttm
+ The timestamp when the step was enqueued by the workflow system.
+
+start_date
+ The timestamp for the beginning of execution for this step.
+
+state
+ The execution state of the step.
+
+task_id
+ The name of the task used by the workflow system (and also representing
+ this step name queried in the reqeust.
+
+try_number
+ A number of retries taken in the case of failure. Some workflow steps
+ may be configured to retry before considering the step truly failed.
+
+
+GET /v1.0/actions/{action_id}/steps/{step_id}
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Returns the details for a step by id for the given action by Id. #####
+
+Responses
+'''''''''
+200 OK
+
+Example
+'''''''
+
+::
+
+ $ curl -D - \
+ -X GET $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/action_xcom \
+ -H "X-Auth-Token:$TOKEN"
+
+ HTTP/1.1 200 OK
+ x-shipyard-req: 72daca4d-1f79-4e08-825f-2ad181912a47
+ content-type: application/json; charset=UTF-8
+
+ {
+ "end_date": "2017-09-24 19:05:59.446213",
+ "duration": 0.165181,
+ "queued_dttm": "2017-09-24 19:05:52.993983",
+ "operator": "PythonOperator",
+ "try_number": 1,
+ "task_id": "action_xcom",
+ "state": "success",
+ "execution_date": "2017-09-24 19:05:49",
+ "dag_id": "deploy_site",
+ "index": 1,
+ "start_date": "2017-09-24 19:05:59.281032"
+ }
+
+/v1.0/actions/{action_id}/control/{control_verb}
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Allows for issuing DAG controls against an action.
+
+Entity Structure
+^^^^^^^^^^^^^^^^
+None, there is no associated response entity for this resource
+
+POST /v1.0/actions/{action_id}/{control_verb}
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Trigger a control action against an activity.- this includes: pause, unpause
+
+Responses
+'''''''''
+202 Accepted
+
+Example
+'''''''
+Failure case - command is invalid for the execution state of the action.
+
+::
+
+ $ curl -D - \
+ -X POST $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/control/pause \
+ -H "X-Auth-Token:$TOKEN"
+
+ HTTP/1.1 409 Conflict
+ content-type: application/json
+ x-shipyard-req: 9c9551e0-335c-4297-af93-8440cc6b324f
+
+ {
+ "apiVersion": "v1.0",
+ "status": "Failure",
+ "metadata": {},
+ "message": "Unable to pause action",
+ "code": "409 Conflict",
+ "details": {
+ "errorList": [
+ {
+ "message": "dag_run state must be running, but is failed"
+ }
+ ],
+ "errorCount": 1,
+ "errorType": "ApiError"
+ },
+ "kind": "status",
+ "reason": "dag_run state must be running, but is failed"
+ }
+
+Success case
+
+::
+
+ $ curl -D - \
+ -X POST $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/control/pause \
+ -H "X-Auth-Token:$TOKEN"
+
+ HTTP/1.1 202 Accepted
+ content-length: 0
+ x-shipyard-req: 019fae1c-03b0-4af1-b57d-451ae6ddac77
+ content-type: application/json; charset=UTF-8
+
+
+Airflow Monitoring API
+----------------------
+Airflow has a primary function of scheduling DAGs, as opposed to Shipyard’s
+primary case of triggering DAGs. Shipyard provides functionality to allow for
+an operator to monitor and review these scheduled workflows (DAGs) in addition
+to the ones triggered by Shipyard. This API will allow for accessing Airflow
+DAGs of any type – providing a peek into the totality of what is happening in
+Airflow.
+
+/v1.0/workflows
+~~~~~~~~~~~~~~~
+The resource that represents DAGs (workflows) in airflow
+
+Entity Structure
+^^^^^^^^^^^^^^^^
+A list of objects representing the DAGs that have run in airflow.
+
+GET /v1.0/workflows
+^^^^^^^^^^^^^^^^^^^
+Queries airflow for DAGs that are running or have run (successfully or
+unsuccessfully) and provides a summary of those things.
+
+Query parameters
+''''''''''''''''
+since={iso8601 date (past) or duration}
+ optional, a boundary in the past within which to retrieve results. Default is
+ 30 days in the past.
+
+Responses
+'''''''''
+200 OK
+
+Example
+'''''''
+Notice the workflow_id values, these can be used for drilldown.
+
+::
+
+ curl -D - -X GET $URL/api/v1.0/workflows -H "X-Auth-Token:$TOKEN"
+
+ HTTP/1.1 200 OK
+ content-type: application/json; charset=UTF-8
+ x-shipyard-req: 3ab4ccc6-b956-4c7a-9ae6-183c562d8297
+
+ [
+ {
+ "execution_date": "2017-10-09 21:18:56",
+ "end_date": null,
+ "workflow_id": "deploy_site__2017-10-09T21:18:56.000000",
+ "start_date": "2017-10-09 21:18:56.685999",
+ "external_trigger": true,
+ "dag_id": "deploy_site",
+ "state": "failed",
+ "run_id": "manual__2017-10-09T21:18:56"
+ },
+ {
+ "execution_date": "2017-10-09 21:19:03",
+ "end_date": null,
+ "workflow_id": "deploy_site__2017-10-09T21:19:03.000000",
+ "start_date": "2017-10-09 21:19:03.361522",
+ "external_trigger": true,
+ "dag_id": "deploy_site",
+ "state": "failed",
+ "run_id": "manual__2017-10-09T21:19:03"
+ }
+ ...
+ ]
+
+/v1.0/workflows/{workflow_id}
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Entity Structure
+^^^^^^^^^^^^^^^^
+An object representing the information available from airflow regarding
+a DAG’s execution
+
+GET /v1.0/workflows/{id}
+^^^^^^^^^^^^^^^^^^^^^^^^
+Further details of a particular workflow’s steps. All steps of all
+sub-dags will be included in the list of steps, as well as section
+indicating the sub-dags for this parent workflow.
+
+Responses
+'''''''''
+200 OK
+
+Example
+'''''''
+.. note::
+
+ Sub_dags can be queried to restrict to only that sub-dag’s steps. e.g. using
+ this as {workflow_id}:
+ deploy_site.preflight.armada_preflight_check__2017-10-09T21:19:03.000000
+
+::
+
+ curl -D - \
+ -X GET $URL/api/v1.0/workflows/deploy_site__2017-10-09T21:19:03.000000 \
+ -H "X-Auth-Token:$TOKEN"
+
+ HTTP/1.1 200 OK
+ content-type: application/json; charset=UTF-8
+ x-shipyard-req: 98d71530-816a-4692-9df2-68f22c057467
+
+ {
+ "execution_date": "2017-10-09 21:19:03",
+ "end_date": null,
+ "workflow_id": "deploy_site__2017-10-09T21:19:03.000000",
+ "start_date": "2017-10-09 21:19:03.361522",
+ "external_trigger": true,
+ "steps": [
+ {
+ "end_date": "2017-10-09 21:19:14.916220",
+ "task_id": "action_xcom",
+ "start_date": "2017-10-09 21:19:14.798053",
+ "duration": 0.118167,
+ "queued_dttm": "2017-10-09 21:19:08.432582",
+ "try_number": 1,
+ "state": "success",
+ "operator": "PythonOperator",
+ "dag_id": "deploy_site",
+ "execution_date": "2017-10-09 21:19:03"
+ },
+ {
+ "end_date": "2017-10-09 21:19:25.283785",
+ "task_id": "dag_concurrency_check",
+ "start_date": "2017-10-09 21:19:25.181492",
+ "duration": 0.102293,
+ "queued_dttm": "2017-10-09 21:19:19.283132",
+ "try_number": 1,
+ "state": "success",
+ "operator": "ConcurrencyCheckOperator",
+ "dag_id": "deploy_site",
+ "execution_date": "2017-10-09 21:19:03"
+ },
+ {
+ "end_date": "2017-10-09 21:20:05.394677",
+ "task_id": "preflight",
+ "start_date": "2017-10-09 21:19:34.994775",
+ "duration": 30.399902,
+ "queued_dttm": "2017-10-09 21:19:28.449848",
+ "try_number": 1,
+ "state": "failed",
+ "operator": "SubDagOperator",
+ "dag_id": "deploy_site",
+ "execution_date": "2017-10-09 21:19:03"
+ },
+ ...
+ ],
+ "dag_id": "deploy_site",
+ "state": "failed",
+ "run_id": "manual__2017-10-09T21:19:03",
+ "sub_dags": [
+ {
+ "execution_date": "2017-10-09 21:19:03",
+ "end_date": null,
+ "workflow_id": "deploy_site.preflight__2017-10-09T21:19:03.000000",
+ "start_date": "2017-10-09 21:19:35.082479",
+ "external_trigger": false,
+ "dag_id": "deploy_site.preflight",
+ "state": "failed",
+ "run_id": "backfill_2017-10-09T21:19:03"
+ },
+ ...,
+ {
+ "execution_date": "2017-10-09 21:19:03",
+ "end_date": null,
+ "workflow_id": "deploy_site.preflight.armada_preflight_check__2017-10-09T21:19:03.000000",
+ "start_date": "2017-10-09 21:19:48.265023",
+ "external_trigger": false,
+ "dag_id": "deploy_site.preflight.armada_preflight_check",
+ "state": "failed",
+ "run_id": "backfill_2017-10-09T21:19:03"
+ }
+ ]
+ }
diff --git a/docs/source/API_action_commands.rst b/docs/source/API_action_commands.rst
new file mode 100644
index 00000000..1faaaffc
--- /dev/null
+++ b/docs/source/API_action_commands.rst
@@ -0,0 +1,68 @@
+.. _shipyard_action_commands:
+
+Action Commands
+===============
+
+Supported actions
+-----------------
+
+These actions are currently supported using the Action API
+
+deploy_site
+~~~~~~~~~~~
+
+Triggers the initial deployment of a site, using the latest committed
+configuration documents. Steps:
+
+#. Concurrency check
+ Prevents concurrent site modifications by conflicting
+ actions/workflows.
+#. Preflight checks
+ Ensures all UCP components are in a responsive state.
+#. Get design version
+ Uses Deckhand to discover the latest committed version of design for
+ the site.
+#. Validate design
+ Asks each involved UCP component to validate the design. This ensures
+ that the previously committed design is valid at the present time.
+#. Drydock build
+ Orchestrates the Drydock component to configure hardware and the
+ Kubernetes environment (Drydock -> Promenade)
+#. Check deployed node status
+ Checks that the deployment of nodes is successful.
+#. Armada build
+ Orchestrates Armada to configure software on the nodes as designed.
+
+
+Actions under development
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+These actions are under active development
+
+- update_site
+
+ Triggers the initial deployment of a site, using the latest committed
+ configuration documents. Steps: (a superset of deploy_site)
+
+- redeploy_server
+
+ Using parameters to indicate which server(s), triggers a redeployment of
+ server to the last known good design and secrets
+
+Future actions
+~~~~~~~~~~~~~~
+
+These actions are anticipated for development
+
+- test region
+
+ Invoke site validation testing - perhaps baseline is a invocation of all
+ components regular “component” tests. This test would be used as a
+ preflight-style test to ensure all components are in a working state.
+
+- test component
+
+ Invoke a particular platform component to test it. This test would be
+ used to interrogate a particular platform component to ensure it is in a
+ working state, and that its own downstream dependencies are also
+ operational
diff --git a/docs/source/CLI.rst b/docs/source/CLI.rst
new file mode 100644
index 00000000..c76291fe
--- /dev/null
+++ b/docs/source/CLI.rst
@@ -0,0 +1,666 @@
+.. _shipyard_cli:
+
+Shipyard CLI
+============
+
+Environment Variables
+---------------------
+All commands will utilize the following environment variables to
+determine necessary information for execution, unless otherwise noted.
+
+Openstack Keystone Authorization environment variables
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The Shipyard CLI/API Client will check for the presence of appropriate
+environment setup to do authentication on behalf of the user. The openrc
+variables that will be used are as follows:
+
+- OS_PROJECT_DOMAIN_NAME ("default" if not specified)
+- OS_USER_DOMAIN_NAME ("default" if not specified)
+- OS_PROJECT_NAME
+- OS_USERNAME
+- OS_PASSWORD
+- OS_AUTH_URL The fully qualified identity endpoint. E.g. http://keystone.ucp.fully.qualified.name:80/v3
+
+Openstack Keystone Authorization environment variables *not* used
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+These openstack identity variables are not supported by shipyard.
+
+- OS_IDENTITY_API_VERSION
+ This value will be ignored as Shipyard only supports version 3 at this time
+
+Shipyard command options
+------------------------
+The base shipyard command supports options that determine cross-CLI behaviors.
+These options are positionally immediately following the shipyard command as
+shown here:
+
+::
+
+ shipyard <--these options> subcommands...
+
+ shipyard
+ [--context-marker=]
+ [--debug/--no-debug]
+ [--os-{various}=]
+ [--output-format=[format | raw | cli]] (default = cli)
+
+
+
+\--context-marker=
+ Specifies a UUID (8-4-4-4-12 format) that will be used to correlate logs,
+ transactions, etc... in downstream activities triggered by this interaction.
+ If not specified, Shipyard will supply a new UUID to serve as this marker.
+ (optional)
+
+\--debug | --no-debug
+ Enable/disable debugging of this CLI and API client. Defaults to no debug
+
+\--os-=
+ See supported Openstack Keystone Authorization Environment variables above
+ for the list of supported names, converting to a downcase version of the
+ environment variable. E.g.: --os-auth-url=http://keystone.ucp:80/v3
+ If not specified, the environment variables matching these options will be
+ used instead. The Keystone os-auth-url should reference the exposed
+ keystone:port for the target Shipyard environment, as this Keystone will be
+ used to discover the instance of Shipyard. For most invocations other than
+ help, a valid combination of values must be resolved to authenticate and
+ authorize the user's invocation.
+
+\--output-format=
+ Specifies the desired output formating such that:
+
+ - format
+ Display the raw ouptut from the invoked Shipyard API in a column
+ restricted mode.
+ - raw
+ Display the result from the invoked Shipyard API as-is, without
+ modification.
+ - cli (default)
+ Display results in a plain text interpretation of the response from the
+ invoked Shipyard API.
+
+Commit Commands
+---------------
+
+commit configdocs
+~~~~~~~~~~~~~~~~~
+Attempts to commit the Shipyard Buffer documents, first invoking validation by
+downstream components.
+
+::
+
+ shipyard commit configdocs
+ [--force]
+
+ Example:
+ shipyard commit configdocs
+
+\--force
+ Force the commit to occur, even if validations fail.
+
+Sample
+^^^^^^
+
+::
+
+ TBD
+
+Control commands
+----------------
+
+pause, unpause, stop
+~~~~~~~~~~~~~~~~~~~~
+
+Three separate commands with a common format that allow for controlling
+the processing of actions created in Shipyard.
+
+pause
+ pause something in progress e.g. an executing action
+unpause
+ unpause something paused e.g. a paused action
+stop
+ stops an executing or paused item e.g. an action
+
+::
+
+ shipyard pause
+
+
+
+ shipyard unpause
+
+
+
+ shipyard stop
+
+
+
+ shipyard
+ pause|unpause|stop
+
+
+ Example:
+
+ shipyard pause action 01BTG32JW87G0YKA1K29TKNAFX
+
+ shipyard unpause action 01BTG32JW87G0YKA1K29TKNAFX
+
+ shipyard stop action 01BTG32JW87G0YKA1K29TKNAFX
+
+ shipyard pause action/01BTG32JW87G0YKA1K29TKNAFX
+
+
+ The type of entity to take action upon. Currently supports: action
+
+ The id of the entity to take action upon.
+
+ The qualified name of the item to take the specified action upon
+
+Sample
+^^^^^^
+
+::
+
+ $ shipyard pause action/01BZZMEXAVYGG7BT0BMA3RHYY7
+ pause successfully submitted for action 01BZZMEXAVYGG7BT0BMA3RHYY7
+
+A failed command:
+
+::
+
+ $ shipyard pause action/01BZZK07NF04XPC5F4SCTHNPKN
+ Error: Unable to pause action
+ Reason: dag_run state must be running, but is failed
+ - Error: dag_run state must be running, but is failed
+
+Create Commands
+---------------
+
+create action
+~~~~~~~~~~~~~
+
+Invokes the specified workflow through Shipyard. Returns the
+id of the action invoked so that it can be queried subsequently.
+
+::
+
+ shipyard create action
+
+ --param= (repeatable)
+
+ Example:
+ shipyard create action redeploy_server --param="server-name=mcp"
+
+
+ The action to invoke.
+
+\--param=
+ A parameter to be provided to the action being invoked. (repeatable)
+
+Sample
+^^^^^^
+
+::
+
+ $ shipyard create action deploy_site
+ Name Action Lifecycle
+ deploy_site action/01BZZK07NF04XPC5F4SCTHNPKN None
+
+
+create configdocs
+~~~~~~~~~~~~~~~~~
+Load documents into the Shipyard Buffer. The use of one or more filename
+or a single directory option must be specified.
+
+::
+
+ shipyard create configdocs
+
+ [--append | --replace]
+ --filename= (repeatable)
+ |
+ --directory=
+
+ Example:
+ shipyard create configdocs design --append --filename=site_design.yaml
+
+.. note::
+
+ If neither append nor replace are specified, the Shipyard API default value
+ of rejectoncontents will be used.
+
+.. note::
+
+ Either --filename or --directory must be specified, but both may not be
+ specified for the same invocation of shipyard.
+
+
+ The collection to load.
+
+\--append
+ Add the collection to the Shipyard Buffer. This will fail if the collection
+ already exists.
+
+\--replace
+ Clear the shipyard buffer and replace it with the specified contents.
+
+\--filename=
+ The file name to use as the contents of the collection. (repeatable) If
+ any documents specified fail basic validation, all of the documents will
+ be rejected. Use of filename parameters may not be used in conjunction
+ with the directory parameter.
+
+\--directory=
+ A directory containing documents that will be joined and loaded as a
+ collection. Any documents that fail basic validation will reject the
+ whole set. Use of the directory parameter may not be used with the
+ filename parameter.
+
+Sample
+^^^^^^
+
+::
+
+ $ shipyard create configdocs coll1 --filename=/home/ubuntu/yaml/coll1.yaml
+ Configuration documents added.
+ Status: Validations succeeded
+ Reason: Validation
+
+Attempting to load the same collection into the uncommitted buffer.
+
+::
+
+ $ shipyard create configdocs coll1 --filename=/home/ubuntu/yaml/coll1.yaml
+ Error: Invalid collection specified for buffer
+ Reason: Buffermode : rejectoncontents
+ - Error: Buffer is either not empty or the collection already exists in buffer. Setting a different buffermode may provide the desired functionality
+
+Replace the buffer with --replace
+
+::
+
+ $ shipyard create configdocs coll1 --replace --filename=/home/ubuntu/yaml/coll1.yaml
+ Configuration documents added.
+ Status: Validations succeeded
+ Reason: Validation
+
+Describe Commands
+-----------------
+
+describe
+~~~~~~~~
+
+Retrieves the detailed information about the supplied namespaced item
+
+::
+
+ shipyard describe
+
+
+ Example:
+ shipyard describe action/01BTG32JW87G0YKA1K29TKNAFX
+ Equivalent to:
+ shipyard describe action 01BTG32JW87G0YKA1K29TKNAFX
+
+ shipyard describe step/01BTG32JW87G0YKA1K29TKNAFX/preflight
+ Equivalent to:
+ shipyard describe step preflight --action=01BTG32JW87G0YKA1K29TKNAFX
+
+ shipyard describe validation/01BTG32JW87G0YKA1K29TKNAFX/01BTG3PKBS15KCKFZ56XXXBGF2
+ Equivalent to:
+ shipyard describe validation 01BTG3PKBS15KCKFZ56XXXBGF2 \
+ --action=01BTG32JW87G0YKA1K29TKNAFX
+
+ shipyard describe workflow/deploy_site__2017-01-01T12:34:56.123456
+ Equivalent to:
+ shipyard describe workflow deploy_site__2017-01-01T12:34:56.123456
+
+
+describe action
+~~~~~~~~~~~~~~~
+
+Retrieves the detailed information about the supplied action id.
+
+::
+
+ shipyard describe action
+
+
+ Example:
+ shipyard describe action 01BTG32JW87G0YKA1K29TKNAFX
+
+Sample
+^^^^^^
+
+
+::
+
+ $ shipyard describe action/01BZZK07NF04XPC5F4SCTHNPKN
+ Name: deploy_site
+ Action: action/01BZZK07NF04XPC5F4SCTHNPKN
+ Lifecycle: Failed
+ Parameters: {}
+ Datetime: 2017-11-27 20:34:24.610604+00:00
+ Dag Status: failed
+ Context Marker: 71d4112e-8b6d-44e8-9617-d9587231ffba
+ User: shipyard
+
+ Steps Index State
+ step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom 1 success
+ step/01BZZK07NF04XPC5F4SCTHNPKN/dag_concurrency_check 2 success
+ step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 3 failed
+ step/01BZZK07NF04XPC5F4SCTHNPKN/validate_site_design 4 None
+ step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 5 failed
+ step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 6 failed
+ step/01BZZK07NF04XPC5F4SCTHNPKN/drydock_build 7 None
+
+ Commands User Datetime
+ invoke shipyard 2017-11-27 20:34:34.443053+00:00
+
+ Validations: None
+
+
+describe step
+~~~~~~~~~~~~~
+Retrieves the step details associated with an action and step.
+
+::
+
+ shipyard describe step
+
+ --action=
+
+ Example:
+ shipyard describe step preflight --action=01BTG32JW87G0YKA1K29TKNAFX
+
+
+ The id of the step found in the describe action response.
+
+\--action=
+ The action id that provides the context for this step.
+
+Sample
+^^^^^^
+
+
+::
+
+ $ shipyard describe step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom
+ Name: action_xcom
+ Task ID: step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom
+ Index: 1
+ State: success
+ Start Date: 2017-11-27 20:34:45.604109
+ End Date: 2017-11-27 20:34:45.818946
+ Duration: 0.214837
+ Try Number: 1
+ Operator: PythonOperator
+
+describe validation
+~~~~~~~~~~~~~~~~~~~
+
+Retrieves the validation details associated with an action and
+validation id
+
+::
+
+ shipyard describe validation
+
+ --action=
+
+ Example:
+ shipyard describe validation 01BTG3PKBS15KCKFZ56XXXBGF2 \
+ --action=01BTG32JW87G0YKA1K29TKNAFX
+
+
+ The id of the validation found in the describe action response.
+
+\--action=
+ The action id that provides the context for this validation.
+
+Sample
+^^^^^^
+
+
+::
+
+ TBD
+
+describe workflow
+~~~~~~~~~~~~~~~~~
+
+Retrieves the details for a workflow that is running or has run in the
+workflow engine.
+
+::
+
+ shipyard describe workflow
+
+
+ Example:
+ shipyard describe workflow deploy_site__2017-01-01T12:34:56.123456
+
+
+ The id of the workflow found in the get workflows response.
+
+Sample
+^^^^^^
+
+
+::
+
+ $ shipyard describe workflow deploy_site__2017-11-27T20:34:33.000000
+ Workflow: deploy_site__2017-11-27T20:34:33.000000
+ State: failed
+ Dag ID: deploy_site
+ Execution Date: 2017-11-27 20:34:33
+ Start Date: 2017-11-27 20:34:33.979594
+ End Date: None
+ External Trigger: True
+
+ Steps State
+ action_xcom success
+ dag_concurrency_check success
+ deckhand_get_design_version failed
+ validate_site_design None
+ deckhand_get_design_version failed
+ deckhand_get_design_version failed
+ drydock_build None
+
+ Subworkflows:
+ Workflow: deploy_site.deckhand_get_design_version__2017-11-27T20:34:33.000000
+ State: failed
+ Dag ID: deploy_site.deckhand_get_design_version
+ Execution Date: 2017-11-27 20:34:33
+ Start Date: 2017-11-27 20:35:06.281825
+ End Date: None
+ External Trigger: False
+
+ Workflow: deploy_site.deckhand_get_design_version.deckhand_get_design_version__2017-11-27T20:34:33.000000
+ State: failed
+ Dag ID: deploy_site.deckhand_get_design_version.deckhand_get_design_version
+ Execution Date: 2017-11-27 20:34:33
+ Start Date: 2017-11-27 20:35:20.725506
+ End Date: None
+ External Trigger: False
+
+Get Commands
+------------
+
+get actions
+~~~~~~~~~~~
+
+Lists the actions that have been invoked.
+
+::
+
+ shipyard get actions
+
+
+Sample
+^^^^^^
+
+::
+
+ $ shipyard get actions
+ Name Action Lifecycle
+ deploy_site action/01BZZK07NF04XPC5F4SCTHNPKN Failed
+ update_site action/01BZZKMW60DV2CJZ858QZ93HRS Processing
+
+get configdocs
+~~~~~~~~~~~~~~
+
+Retrieve documents loaded into Shipyard, either committed or from the
+Shipyard Buffer.
+
+::
+
+ shipyard get configdocs
+
+ [--committed | --buffer]
+
+ Example:
+ shipyard get configdocs design
+
+
+ The collection to retrieve for viewing.
+
+\--committed
+ Retrieve the documents that have last been committed for this collection
+
+\--buffer
+ Retrive the documents that have been loaded into Shipyard since the
+ prior commit. If no documents have been loaded into the buffer for this
+ collection, this will return an empty response (default)
+
+Sample
+^^^^^^
+
+::
+
+ $ shipyard get configdocs coll1
+ data:
+ chart_groups: [kubernetes-proxy, container-networking, dns, kubernetes, kubernetes-rbac]
+ release_prefix: ucp
+ id: 1
+ metadata:
+ layeringDefinition: {abstract: false, layer: site}
+ name: cluster-bootstrap-1
+ schema: metadata/Document/v1.0
+ storagePolicy: cleartext
+ schema: armada/Manifest/v1.0
+ status: {bucket: coll1, revision: 1}
+
+get renderedconfigdocs
+~~~~~~~~~~~~~~~~~~~~~~
+Retrieve the rendered version of documents loaded into Shipyard.
+Rendered documents are the "final" version of the documents after
+applying Deckhand layering and substitution.
+
+::
+
+ shipyard get renderedconfigdocs
+ [--committed | --buffer]
+
+ Example:
+ shipyard get renderedconfigdocs
+
+\--committed
+ Retrieve the documents that have last been committed.
+
+\--buffer
+ Retrieve the documents that have been loaded into Shipyard since the
+ prior commit. (default)
+
+Sample
+^^^^^^
+
+::
+
+ $ shipyard get renderedconfigdocs
+ data:
+ chart_groups: [kubernetes-proxy, container-networking, dns, kubernetes, kubernetes-rbac]
+ release_prefix: ucp
+ id: 1
+ metadata:
+ layeringDefinition: {abstract: false, layer: site}
+ name: cluster-bootstrap-1
+ schema: metadata/Document/v1.0
+ storagePolicy: cleartext
+ schema: armada/Manifest/v1.0
+ status: {bucket: coll1, revision: 1}
+
+get workflows
+~~~~~~~~~~~~~
+Retrieve workflows that are running or have run in the workflow engine.
+This includes processses that may not have been started as an action
+(e.g. scheduled tasks).
+
+::
+
+ shipyard get workflows
+ [--since=]
+
+ Example:
+ shipyard get workflows
+
+ shipyard get workflows --since=2017-01-01T12:34:56.123456
+
+\--since=
+ The historical cutoff date to limit the results of of this response.
+
+Sample
+^^^^^^
+
+::
+
+ $ shipyard get workflows
+ Workflows State
+ deploy_site__2017-11-27T20:34:33.000000 failed
+ update_site__2017-11-27T20:45:47.000000 running
+
+
+Help Commands
+-------------
+
+help
+~~~~
+Provides topical help for shipyard.
+
+.. note::
+
+ --help will provide more specific command help.
+
+::
+
+ shipyard help
+ []
+
+ Example:
+ shipyard help configdocs
+
+
+ The topic of the help to be displayed. If this parameter is not
+ specified the list of available topics will be displayed.
+
+Sample
+^^^^^^
+
+
+::
+
+ $ shipyard help
+ THE SHIPYARD COMMAND
+ The base shipyard command supports options that determine cross-CLI behaviors.
+
+ FORMAT
+ shipyard [--context-marker=] [--os_{various}=]
+ [--debug/--no-debug] [--output-format]
+
+ Please Note: --os_auth_url is required for every command except shipyard help
+ .
+
+ TOPICS
+ For information of the following topics, run shipyard help
+ actions
+ configdocs
diff --git a/docs/source/docutils.conf b/docs/source/docutils.conf
new file mode 100644
index 00000000..b49cd480
--- /dev/null
+++ b/docs/source/docutils.conf
@@ -0,0 +1,2 @@
+[general]
+smart_quotes=no
\ No newline at end of file
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 42795635..4657791b 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -32,3 +32,6 @@ Shipyard Configuration Guide
sampleconf
policy-enforcement
+ API
+ API_action_commands
+ CLI