Move most Shipyard docs to .rst

This patchset moves most of the documentation for
shipyard from markdown to reStructuredText.
README.md remains as it is nice for Github consumption
and serves as a gateway to the rest of the generated
documentation.

Change-Id: I8cec38fe0e30e9a545cb6e79ae6de1934f7dfc4e
This commit is contained in:
Bryan Strassner 2017-12-01 16:01:45 -06:00
parent 2af8f143bd
commit fde16f218e
10 changed files with 1678 additions and 1486 deletions

3
.gitignore vendored
View File

@ -107,4 +107,5 @@ AUTHORS
# build/lint artifacts
/charts/shipyard/charts
/charts/shipyard/requirements.lock
/charts/shipyard/requirements.lock
.DS_Store

View File

@ -1,755 +0,0 @@
# Shipyard API
Logically, the API has three parts to handle the three areas of functionality
in Shipyard.
1. Document Staging
2. Action Handling
3. Airflow Monitoring
## Standards used by the API
See [UCP API conventions](https://github.com/att-comdev/ucp-integration/blob/master/docs/api-conventions.md)
## Notes on examples
Examples assume the following environment variables are set before issuing
the curl commands shown:
```
$TOKEN={a valid keystone token}
$URL={the url and port of the shipyard api}
```
* Examples will use json formatted by the jq command for sake of presentation.
* Actual responses will not formatted.
* The use of ellipsis indicate a repeated structure in the case of lists, or
prior/subsequent structure unimportant to the example (or considered
understood).
* The content-length response headers have been removed so as to not cause
confusion with the listed output.
### Example response for an invalid token:
```
HTTP/1.1 401 Unauthorized
content-type: application/json
x-shipyard-req: a8194b97-8973-4b04-a3b3-2bd319024c5d
WWW-Authenticate: Keystone uri='http://keystone-api.ucp.svc.cluster.local:80/v3'
{
"apiVersion": "v1.0",
"status": "Failure",
"metadata": {},
"message": "Unauthenticated",
"code": "401 Unauthorized",
"details": {
"errorList": [
{
"message": "Credentials are not established"
}
],
"errorCount": 1,
"errorType": "ApiError"
},
"kind": "status",
"reason": "Credentials are not established"
}
```
---
## <a name="DocumentStagingAPI"></a> Document Staging API
>Shipyard will serve as the entrypoint for documents (designs, secrets,
configurations, etc...) into a site. Documents are posted to Shipyard in
collections, rather than individually. At any point in time, there will be two
represented versions of documents in a site that are accessible via this API:
>
>* The "Committed Documents" version, which represents the last version of
documents that were successfully commited with a commit_configdocs action.
>* The "Shipyard Buffer" version, which represents the collection of documents
that have been ingested by this API since the last commited version. Note that
only one set of documents maybe posted to the buffer at a time by default.
(This behavior can be overridden by query parameters issued by the user of
Shipyard)
>
> All versions of documents rely upon Deckhand for storage. Shipyard uses the
tagging features of Deckhand of to find the appropriate Committed Documents
and Shipyard Buffer version.
---
### /v1.0/configdocs/{collection_id}
Represents the site configuration documents
#### Entity Structure
The documents as noted above (commonly yaml), in a format understood by
Deckhand
#### POST /v1.0/configdocs/{collection_id}
Ingests a collection of documents. Synchronous. POSTing an empty body
indicates that the specified collection should be deleted when the Shipyard
Buffer is committed. If a POST to the commitconfigdocs is in progress, this
POST should be rejected with a 409 error.
Important:
The expected input type for this request is 'Content-Type: application/x-yaml'
##### Query Parameters
* bufferMode=append|replace|**rejectOnContents**
Indicates how the existing Shipyard Buffer should be handled. By default,
Shipyard will reject the POST if contents already exist in the Shipyard
Buffer.
* append: Add the collection to the Shipyard Buffer, only if that
collection doesn't already exist in the Shipyard Buffer. If the collection
is already present, the request will be rejected and a 409 Conflict will be
returned.
* replace: Clear the Shipyard Buffer before adding the specified collection.
##### Responses
* 201 Created
If the documents are successfully ingested, even with validation failures.
Response message includes:
* a list of validation results
* The response headers will include a Location indicating the GET endpoint
to retrieve the configDocs
* 409 Conflict
* If a commitconfigdocs POST is in progress.
* If any collections exist in the Shipyard Buffer unless bufferMode=replace
or bufferMode=append.
* If bufferMode=append, and the collection being posted is already in the
Shipyard Buffer
#### GET /v1.0/configdocs/{collection_id}
Returns the source documents for a collection of documents
##### Query Parameters
* version=committed|**buffer**
Return the documents for the version specified - buffer by default.
Important:
The output type for this request is 'Content-Type: application/x-yaml'
##### Responses
* 200 OK
If documents can be retrieved.
* If the response is 200 with an empty response body, this indicates that
the buffer version is attempting to 'delete' the collection when it is
committed. An empty response body will only be possible for version=buffer.
* 404 Not Found
If the collection is not represented
* When version=buffer, this indicates that no representations of this
collection have been POSTed since the last committed version.
* When version=committed, this indicates that either the collection has
never existed or has been deleted by a prior commit
---
### /v1.0/renderedconfigdocs
Represents the site configuration documents, as a whole set - does not
consider collections in any way.
#### GET /v1.0/renderedconfigdocs
Returns the full set of configdocs in their rendered form.
Important:
The output type for this request is 'Content-Type: application/x-yaml'
##### Query Parameters
* version=committed|**buffer**
Return the documents for the version specified - buffer by default.
##### Responses
* 200 OK
If documents can be retrieved.
---
### /v1.0/commitconfigdocs
A RPC style command to trigger a commit of the configuration documents from the
Shipyard Buffer to the Committed Documents. This resource will support POST
only.
#### Entity Structure
The response will be the list of validations from all downstream systems that
perform validation during the commit process. The structure will match the
error response object described in the [UCP API conventions](https://github.com/att-comdev/ucp-integration/blob/master/docs/api-conventions.md)
and will be an aggregation of each UCP component's responses.
#### POST /v1.0/commitconfigdocs
Performs the commit of the Shipyard Buffer to the Committed Documents.
Synchronous. This invokes each of the UCP components to examine the Shipyard
Buffer version of the configuration documents and aggregate the responses.
While performing this commit, further POSTing of configdocs, or other commits
may not be invoked (Shipyard will block those requests with a 409 response).
If there are any failures to validate, the Shipyard Buffer and Commited
Documents will remain unchanged. If successful, the Shipyard Buffer will be
cleared, and the Committed documents will be updated.
**Note**: if there are unhandled runtime errors during the commitconfigdocs POST,
a deadlock situation may be possible. Future enhancements may improve this
handling.
##### Query Parameters
* force=true|**false**
By default, false, if there are validation failures the POST will fail with
a 400 response. With force=true, allows for the commit to succeed (with a 200
response) even if there are validation failures from downstream components. The
aggregate response of validation failures will be returned in this case, but
the invalid documents will still be moved from the Shipyard Buffer to the
Committed Documents.
##### Responses
* 200 OK
If the validations are successful. Returns an "empty" structure as as response
indicating no errors. A 200 may also be returned if there are validation
failures, but the force=true query parameter was specified. In this case, the
response will contain the list of validations.
* 400 Bad Request
If the validations fail. Returns a populated response structure containing
the aggregation of the failed validations.
* 409 Conflict
If the there is a POST to commitconfigdocs in progress.
---
## <a name="ActionAPI"></a> Action API
>The Shipyard Action API is a resource that allows for creation, control and
investigation of triggered workflows. These actions encapsulate a command
interface for the Undercloud Platform.
See [Action Commands](API_action_commands.md) for supported actions
---
### /v1.0/actions
#### Entity Structure
A list of actions that have been executed through shipyard's action API.
```
[
{ Action objects summarized, See below},
...
]
```
#### GET /v1.0/actions
Returns the list of actions in the system that have been posted, and are
accessible to the current user.
##### Responses
* 200 OK
If the actions can be retrieved.
##### Example
```
$ curl -X GET $URL/api/v1.0/actions -H "X-Auth-Token:$TOKEN"
HTTP/1.1 200 OK
x-shipyard-req: 0804d13e-08fc-4e60-a819-3b7532cac4ec
content-type: application/json; charset=UTF-8
[
{
"dag_status": "failed",
"parameters": {},
"steps": [
{
"id": "action_xcom",
"url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/action_xcom",
"index": 1,
"state": "success"
},
{
"id": "dag_concurrency_check",
"url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/dag_concurrency_check",
"index": 2,
"state": "success"
},
{
"id": "preflight",
"url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/preflight",
"index": 3,
"state": "failed"
},
...
],
"action_lifecycle": "Failed",
"dag_execution_date": "2017-09-23T02:42:12",
"id": "01BTP9T2WCE1PAJR2DWYXG805V",
"dag_id": "deploy_site",
"datetime": "2017-09-23 02:42:06.860597+00:00",
"user": "shipyard",
"context_marker": "416dec4b-82f9-4339-8886-3a0c4982aec3",
"name": "deploy_site"
},
...
]
```
#### POST /v1.0/actions
Creates an action in the system. This will cause some action to start.
The input body to this post will represent an action object that has at least
these fields:
* name
The name of the action to invoke, as noted in
[Action Commands](API_action_commands.md)
* parameters
A dictionary of parameters to use for the trigger invocation. The supported
parameters will vary for the action invoked.
```
{
"name" : "action name",
"parameters" : { varies by action }
}
```
The POST will synchronously create the action (a shell object that represents
a DAG invocation), perform any checks to validate the preconditions to run the
DAG, and trigger the invocation of the DAG. The DAG will run asynchronously in
airflow.
##### Responses
* 201 Created
If the action is created successfully, and all preconditions to run the DAG
are successful. The response body is the action entity created.
* 400 Bad Request
If the action name doesn't exist, or the input entity is otherwise malformed.
* 409 Conflict
For any failed pre-run validations. The response body is the action entity
created, with the failed validations. The DAG will not begin execution in this
case.
##### Example
```
$ curl -D - -d '{"name":"deploy_site"}' -X POST $URL/api/v1.0/actions \
-H "X-Auth-Token:$TOKEN" -H "content-type:application/json"
HTTP/1.1 201 Created
location: {$URL}/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN
x-shipyard-req: 629f2ea2-c59d-46b9-8641-7367a91a7016
content-type: application/json; charset=UTF-8
{
"dag_status": "SCHEDULED",
"parameters": {},
"dag_execution_date": "2017-09-24T19:05:49",
"id": "01BTTMFVDKZFRJM80FGD7J1AKN",
"dag_id": "deploy_site",
"name": "deploy_site",
"user": "shipyard",
"context_marker": "629f2ea2-c59d-46b9-8641-7367a91a7016",
"timestamp": "2017-09-24 19:05:43.603591"
}
```
---
### /v1.0/actions/{action_id}
Each action will be assigned an unique id that can be used to get details for
the action, including the execution status.
#### Entity Structure
All actions will include fields that indicate the following data:
<dl>
<dt>action_lifecycle</dt>
<dd>A summarized value indicating the status or lifecycle phase of the
action.</dd>
<ul>
<li>Pending - The action is scheduled or preparing for execution.</li>
<li>Processing - The action is underway.</li>
<li>Complete - The action has completed successfully.</li>
<li>Failed - The action has encountered an error, and has failed.</li>
<li>Paused - The action has been paused by a user.</li>
</ul>
</dd>
<dt>command audit</dt>
<dd>A list of commands that have been issued against the action. Initially,
the action listed will be "invoke", but may include "pause", "unpause",
or "stop" if those commands are issued.</dd>
<dt>context_marker</dt>
<dd>The user supplied or system assigned context marker associated with the
action</dd>
<dt>dag_execution_date</dt>
<dd>The execution date assigned by the workflow system during action
creation.</dd>
<dt>dag_status</dt>
<dd>Represents the status that airflow provides for an executing DAG.</dd>
<dt>datetime</dt>
<dd>The time at which the action was invoked.</dd>
<dt>id</dt>
<dd>The identifier for the action, a 26 character ULID assigned during the
creation of the action.</dd>
<dt>name</dt>
<dd>The name of the action, e.g.: deploy_site.</dd>
<dt>parameters</dt>
<dd>The parameters configuring the action that were supplied by the
user during action creation.</dd>
<dt>steps</dt>
<dd>The list of steps for the action, including the status for that
step.</dd>
<dt>user</dt>
<dd>The user who has invoked this action, as acquired from the authorization
token.</dd>
<dt>validations</dt>
<dd>A list of validations that have been done, including any status
information for those validations. During the lifecycle of the action,
this list of validations may continue to grow.</dd>
#### GET /v1.0/actions/{action_id}
Returns the action entity for the specified id.
##### Responses
* 200 OK
##### Example
```
$ curl -D - -X GET $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN \
-H "X-Auth-Token:$TOKEN"
HTTP/1.1 200 OK
x-shipyard-req: eb3eacb3-4206-40df-bd91-2a3a6d81cd02
content-type: application/json; charset=UTF-8
{
"name": "deploy_site",
"dag_execution_date": "2017-09-24T19:05:49",
"validations": [],
"id": "01BTTMFVDKZFRJM80FGD7J1AKN",
"dag_id": "deploy_site",
"command_audit": [
{
"id": "01BTTMG16R9H3Z4JVQNBMRV1MZ",
"action_id": "01BTTMFVDKZFRJM80FGD7J1AKN",
"datetime": "2017-09-24 19:05:49.530223+00:00",
"user": "shipyard",
"command": "invoke"
}
],
"user": "shipyard",
"context_marker": "629f2ea2-c59d-46b9-8641-7367a91a7016",
"datetime": "2017-09-24 19:05:43.603591+00:00",
"dag_status": "failed",
"parameters": {},
"steps": [
{
"id": "action_xcom",
"url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/action_xcom",
"index": 1,
"state": "success"
},
{
"id": "dag_concurrency_check",
"url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/dag_concurrency_check",
"index": 2,
"state": "success"
},
{
"id": "preflight",
"url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/preflight",
"index": 3,
"state": "failed"
},
{
"id": "deckhand_get_design_version",
"url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/deckhand_get_design_version",
"index": 4,
"state": null
},
...
],
"action_lifecycle": "Failed"
}
```
---
### /v1.0/actions/{action_id}/validationdetails/{validation_id}
Allows for drilldown to validation detailed info.
#### Entity Structure
The detailed information for a validation
```
{ TBD }
```
#### GET /v1.0/actions/{action_id}/validationdetails/{validation_id}
Returns the validation detail by Id for the supplied action Id.
##### Responses
* 200 OK
---
### /v1.0/actions/{action_id}/steps/{step_id}
Allow for drilldown to step information. The step information includes details
of the steps excution, successful or not, and enough to facilitate
troubleshooting in as easy a fashion as possible.
#### Entity Structure
A step entity represents detailed information representing a single step of
execution as part of an action. Not all fields are necessarily represented in
every returned entity.
<dl>
<dt>dag_id</dt>
<dd>The name/id of the workflow DAG that contains this step.</dd>
<dt>duration</dt>
<dd>The duration (seconds) for the step.</dd>
<dt>end_date</dt>
<dd>The timestamp of the completion of the step.</dd>
<dt>execution_date</dt>
<dd>The execution date of the workflow that contains this step.</dd>
<dt>index</dt>
<dd>The ordinal value representing the position of this step in the sequence
of steps associated with this step.</dd>
<dt>operator</dt>
<dd>The name of the processing facility used by the workflow system.</dd>
<dt>queued_dttm</dt>
<dd>The timestamp when the step was enqueued by the workflow system.</dd>
<dt>start_date</dt>
<dd>The timestamp for the beginning of execution for this step.</dd>
<dt>state</dt>
<dd>The execution state of the step.</dd>
<dt>task_id</dt>
<dd>The name of the task used by the workflow system (and also representing
this step name queried in the reqeust.</dd>
<dt>try_number</dt>
<dd>A number of retries taken in the case of failure. Some workflow steps may
be configured to retry before considering the step truly failed.</dd>
</dl>
#### GET /v1.0/actions/{action_id}/steps/{step_id}
Returns the details for a step by id for the given action by Id.
##### Responses
* 200 OK
##### Example
```
$ curl -D - \
-X GET $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/action_xcom \
-H "X-Auth-Token:$TOKEN"
HTTP/1.1 200 OK
x-shipyard-req: 72daca4d-1f79-4e08-825f-2ad181912a47
content-type: application/json; charset=UTF-8
{
"end_date": "2017-09-24 19:05:59.446213",
"duration": 0.165181,
"queued_dttm": "2017-09-24 19:05:52.993983",
"operator": "PythonOperator",
"try_number": 1,
"task_id": "action_xcom",
"state": "success",
"execution_date": "2017-09-24 19:05:49",
"dag_id": "deploy_site",
"index": 1,
"start_date": "2017-09-24 19:05:59.281032"
}
```
---
### /v1.0/actions/{action_id}/control/{control_verb}
Allows for issuing DAG controls against an action.
#### Entity Structure
None, there is no associated response entity for this resource
#### POST /v1.0/actions/{action_id}/{control_verb}
Trigger a control action against an activity.- this includes: pause, unpause
##### Responses
* 202 Accepted
##### Example
Failure case - command is invalid for the execution state of the action.
```
$ curl -D - \
-X POST $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/control/pause \
-H "X-Auth-Token:$TOKEN"
HTTP/1.1 409 Conflict
content-type: application/json
x-shipyard-req: 9c9551e0-335c-4297-af93-8440cc6b324f
{
"apiVersion": "v1.0",
"status": "Failure",
"metadata": {},
"message": "Unable to pause action",
"code": "409 Conflict",
"details": {
"errorList": [
{
"message": "dag_run state must be running, but is failed"
}
],
"errorCount": 1,
"errorType": "ApiError"
},
"kind": "status",
"reason": "dag_run state must be running, but is failed"
}
```
Success case
```
$ curl -D - \
-X POST $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/control/pause \
-H "X-Auth-Token:$TOKEN"
HTTP/1.1 202 Accepted
content-length: 0
x-shipyard-req: 019fae1c-03b0-4af1-b57d-451ae6ddac77
content-type: application/json; charset=UTF-8
```
---
## <a name="AirflowMonitoringAPI"></a> Airflow Monitoring API
>Airflow has a primary function of scheduling DAGs, as opposed to Shipyard's
primary case of triggering DAGs.
Shipyard provides functionality to allow for an operator to monitor and review
these scheduled workflows (DAGs) in addition to the ones triggered by Shipyard.
This API will allow for accessing Airflow DAGs of any type -- providing a peek
into the totality of what is happening in Airflow.
---
### /v1.0/workflows
The resource that represents DAGs (workflows) in airflow
#### Entity Structure
A list of objects representing the DAGs that have run in airflow.
#### GET /v1.0/workflows
Queries airflow for DAGs that are running or have run (successfully or
unsuccessfully) and provides a summary of those things.
##### Query parameters
* since={iso8601 date (past) or duration}
optional, a boundary in the past within which to retrieve results. Default is
30 days in the past.
##### Responses
* 200 OK
##### Example
Note the workflow_id values, these can be used for drilldown.
```
curl -D - -X GET $URL/api/v1.0/workflows -H "X-Auth-Token:$TOKEN"
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
x-shipyard-req: 3ab4ccc6-b956-4c7a-9ae6-183c562d8297
[
{
"execution_date": "2017-10-09 21:18:56",
"end_date": null,
"workflow_id": "deploy_site__2017-10-09T21:18:56.000000",
"start_date": "2017-10-09 21:18:56.685999",
"external_trigger": true,
"dag_id": "deploy_site",
"state": "failed",
"run_id": "manual__2017-10-09T21:18:56"
},
{
"execution_date": "2017-10-09 21:19:03",
"end_date": null,
"workflow_id": "deploy_site__2017-10-09T21:19:03.000000",
"start_date": "2017-10-09 21:19:03.361522",
"external_trigger": true,
"dag_id": "deploy_site",
"state": "failed",
"run_id": "manual__2017-10-09T21:19:03"
}
...
]
```
---
### /v1.0/workflows/{workflow_id}
#### Entity Structure
An object representing the information available from airflow regarding a DAG's
execution
#### GET /v1.0/workflows/{id}
Further details of a particular workflow's steps. All steps of all sub-dags
will be included in the list of steps, as well as section indicating the
sub-dags for this parent workflow.
##### Responses
* 200 OK
##### Example
Note that sub_dags can be queried to restrict to only that sub-dag's steps.
e.g. using this as {workflow_id}:
deploy_site.preflight.armada_preflight_check__2017-10-09T21:19:03.000000
```
curl -D - \
-X GET $URL/api/v1.0/workflows/deploy_site__2017-10-09T21:19:03.000000 \
-H "X-Auth-Token:$TOKEN"
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
x-shipyard-req: 98d71530-816a-4692-9df2-68f22c057467
{
"execution_date": "2017-10-09 21:19:03",
"end_date": null,
"workflow_id": "deploy_site__2017-10-09T21:19:03.000000",
"start_date": "2017-10-09 21:19:03.361522",
"external_trigger": true,
"steps": [
{
"end_date": "2017-10-09 21:19:14.916220",
"task_id": "action_xcom",
"start_date": "2017-10-09 21:19:14.798053",
"duration": 0.118167,
"queued_dttm": "2017-10-09 21:19:08.432582",
"try_number": 1,
"state": "success",
"operator": "PythonOperator",
"dag_id": "deploy_site",
"execution_date": "2017-10-09 21:19:03"
},
{
"end_date": "2017-10-09 21:19:25.283785",
"task_id": "dag_concurrency_check",
"start_date": "2017-10-09 21:19:25.181492",
"duration": 0.102293,
"queued_dttm": "2017-10-09 21:19:19.283132",
"try_number": 1,
"state": "success",
"operator": "ConcurrencyCheckOperator",
"dag_id": "deploy_site",
"execution_date": "2017-10-09 21:19:03"
},
{
"end_date": "2017-10-09 21:20:05.394677",
"task_id": "preflight",
"start_date": "2017-10-09 21:19:34.994775",
"duration": 30.399902,
"queued_dttm": "2017-10-09 21:19:28.449848",
"try_number": 1,
"state": "failed",
"operator": "SubDagOperator",
"dag_id": "deploy_site",
"execution_date": "2017-10-09 21:19:03"
},
...
],
"dag_id": "deploy_site",
"state": "failed",
"run_id": "manual__2017-10-09T21:19:03",
"sub_dags": [
{
"execution_date": "2017-10-09 21:19:03",
"end_date": null,
"workflow_id": "deploy_site.preflight__2017-10-09T21:19:03.000000",
"start_date": "2017-10-09 21:19:35.082479",
"external_trigger": false,
"dag_id": "deploy_site.preflight",
"state": "failed",
"run_id": "backfill_2017-10-09T21:19:03"
},
...,
{
"execution_date": "2017-10-09 21:19:03",
"end_date": null,
"workflow_id": "deploy_site.preflight.armada_preflight_check__2017-10-09T21:19:03.000000",
"start_date": "2017-10-09 21:19:48.265023",
"external_trigger": false,
"dag_id": "deploy_site.preflight.armada_preflight_check",
"state": "failed",
"run_id": "backfill_2017-10-09T21:19:03"
}
]
}
```

View File

@ -1,49 +0,0 @@
# Action API supported commands
## Supported actions
These actions are currently supported using the [Action API](API.md#ActionAPI)
None at this time.
----
## Actions under development
These actions are under active development
* deploy_site
Triggers the initial deployment of a site, using the latest committed
configuration documents. Steps:
1) Concurrency check - to prevent concurrent site modifications by
conflicting actions/workflows.
2) Preflight checks - ensures all UCP components are in a responsive state.
3) Get design version - uses Deckhand to discover the latest committed
version of design for the site.
4) Validate design - asks each involved UCP component to validate the design.
This ensures that the design, which was previously committed is valid at the
present time.
5) Drydock build - orchestrates the Drydock component to configure hardware
and the Kubernetes environment (Drydock -> Promenade)
6) Check deployed node status - checks that the deployment of nodes is
successful.
7) Armada build - orchestrates Armada to configure software on the nodes as
designed.
* update_site
Triggers the initial deployment of a site, using the latest committed
configuration documents. Steps: (same as deploy_site)
* redeploy_server
Using parameters to indicate which server(s), triggers a redeployment of server
to the last known good design and secrets
---
## Future actions
These actions are anticipated for development
* test region
Invoke site validation testing - perhaps baseline is a invocation of all
components regular "component" tests. This test would be used as a
preflight-style test to ensure all components are in a working state.
* test component
Invoke a particular platform component to test it. This test would be used to
interrogate a particular platform component to ensure it is in a working state,
and that its own downstream dependencies are also operational

View File

@ -1,644 +0,0 @@
# Supported Environment Variables
All commands will utilize the following environment variables to determine
necessary information for execution, unless otherwise noted.
<dl>
<dt>Openstack Keystone Authorization environment variables</dt>
<dd>
The Shipyard CLI/API Client will check for the presence of appropriate
environment setup to do authentication on behalf of the user.
The openrc variables that will be used are as follows:<br />
OS_PROJECT_DOMAIN_NAME ("default" if not specified)<br />
OS_USER_DOMAIN_NAME ("default" if not specified)<br />
OS_PROJECT_NAME<br />
OS_USERNAME<br />
OS_PASSWORD<br />
OS_AUTH_URL - The fully qualified identity endpoint.
E.g. http://keystone.ucp.fully.qualified.name:80/v3<br />
</dd>
<dt>Openstack Keystone Authorization environment variables not used</dt>
<dd>
OS_IDENTITY_API_VERSION -- this value will be ignored as Shipyard only
supports version 3 at this time<br />
</dd>
</dl>
# Shipyard command options
The base shipyard command supports options that determine cross-CLI behaviors.
These options are positionally immediately following the shipyard command as
shown here:
```
shipyard <--these options> subcommands...
shipyard
[--context-marker=<uuid>]
[--debug/--no-debug]
[--os_{various}=<value>]
[--output-format=[format | raw | cli]] (default = cli)
<subcommands, as noted in this document>
```
<dl>
<dt>--context-marker=&lt;uuid&gt;</dt>
<dd>
Specifies a UUID (8-4-4-4-12 format) that will be used to correlate logs,
transactions, etc... in downstream activities triggered by this
interaction. If not specified, Shipyard will supply a new UUID to serve
as this marker. (optional)
</dd>
<dt>--debug | --no-debug (default)</dt>
<dd>
Enable/disable debugging of this CLI and API client.
</dd>
<dt>--os_{various}=&lt;value&gt;</dt>
<dd>
See supported Openstack Keystone Authorization Environment variables above
for the list of supported names, converting to a downcase version of the
environment variable.<br />
E.g.: --os_auth_url=http://keystone.ucp:80/v3<br />
If not specified, the environment variables matching these actions will be
used instead. The Keystone os_auth_url should reference the exposed
keystone:port for the target Shipyard environment, as this Keystone will
be used to discover the instance of Shipyard. For most invocations other
than help, a valid combination of values must be resolved to authenticate
and authorize the user's invocation.
</dd>
<dt>--output-format=[format | raw | cli] (default = cli)</dt>
<dd>
Specifies the desired output formating such that:<br />
<b>format</b>: Display the raw ouptut from the invoked Shipyard API in a
column restricted mode.<br />
<b>raw</b>: Display the result from the invoked Shipyard API as-is, without
modification.<br />
<b>cli</b>: (default) Display results in a plain text interpretation of the
response from the invoked Shipyard API.
</dd>
</dl>
# Commit Commands
## commit configdocs
Attempts to commit the Shipyard Buffer documents, first invoking validation
by downstream components.
```
shipyard commit configdocs
[--force]
Example:
shipyard commit configdocs
```
<dl>
<dt>--force</dt>
<dd>
Force the commit to occur, even if validations fail. Note that while this
allows for bypassing validations, it does not bypass a failure to
communicate with the systems that provide the validations.
</dd>
</dl>
### Sample
```
TBD
```
---
# Control commands
## pause, unpause, stop
Three separate commands with a common format that allow for controlling
the processing of actions created in Shipyard.
<dl>
<dt>pause</dt>
<dd>pause something in progress e.g. an executing action</dd>
<dt>unpause</dt>
<dd>unpause something paused e.g. a paused action</dd>
<dt>stop</dt>
<dd>stops an executing or paused item e.g. an action</dd>
</dl>
```
shipyard pause
<type>
<id>
shipyard unpause
<type>
<id>
shipyard stop
<type>
<id>
shipyard
pause|unpause|stop
<qualified name>
Example:
shipyard pause action 01BTG32JW87G0YKA1K29TKNAFX
shipyard unpause action 01BTG32JW87G0YKA1K29TKNAFX
shipyard stop action 01BTG32JW87G0YKA1K29TKNAFX
shipyard pause action/01BTG32JW87G0YKA1K29TKNAFX
```
<dl>
<dt>&lt;type&gt;</dt>
<dd>
The type of entity to take action upon. Currently supports: action
</dd>
<dt>&lt;id&gt;</dt>
<dd>
The id of the entity to take action upon.
</dd>
<dt>&lt;qualified name&gt;</dt>
<dd>
The qualified name of the item to take the specified action upon
</dd>
</dl>
### Sample
```
$ shipyard pause action/01BZZMEXAVYGG7BT0BMA3RHYY7
pause successfully submitted for action 01BZZMEXAVYGG7BT0BMA3RHYY7
```
A failed command:
```
$ shipyard pause action/01BZZK07NF04XPC5F4SCTHNPKN
Error: Unable to pause action
Reason: dag_run state must be running, but is failed
- Error: dag_run state must be running, but is failed
```
---
# Create Commands
## create action
Invokes the specified workflow through Shipyard.
See [Action Commands](API_action_commands.md) for supported actions
Returns the id of the action invoked so that it can be queried subsequently.
```
shipyard create action
<action command>
--param=<parameter> (repeatable)
Example:
shipyard create action redeploy_server --param="server-name=mcp"
```
<dl>
<dt>&lt;action command&gt;</dt>
<dd>
The action to invoke.
</dd>
<dt>--param=&lt;parameter&gt;</dt>
<dd>
A parameter to be provided to the action being invoked. (repeatable)
</dd>
</dl>
### Sample
```
$ shipyard create action deploy_site
Name Action Lifecycle
deploy_site action/01BZZK07NF04XPC5F4SCTHNPKN None
```
---
## create configdocs
Load documents into the Shipyard Buffer. The use of one or more filename or
a single directory option must be specified.
```
shipyard create configdocs
<collection>
[--append | --replace]
--filename=<filename> (repeatable)
|
--directory=<directory>
Example:
shipyard create configdocs design --append --filename=site_design.yaml
```
Note: If neither append or replace are specified, the Shipyard API default
value of rejectoncontents will be used.
Note: Either --filename or --directory must be specified, but both may not be
specified for the same invocation of shipyard.
<dl>
<dt>&lt;collection&gt;</dt>
<dd>
The collection to load.
</dd>
<dt>--append</dt>
<dd>
Add the collection to the Shipyard Buffer. This will fail if the
collection already exists.
</dd>
<dt>--replace</dt>
<dd>
Clear the shipyard buffer and replace it with the specified contents.
</dd>
<dt>--filename=&lt;filename&gt;</dt>
<dd>
The file name to use as the contents of the collection. (repeatable)
If any documents specified fail basic validation, all of the documents
will be rejected. Use of filename parameters may not be used in
conjunction with the directory parameter.
</dd>
<dt>--directory=&lt;directory&gt;</dt>
<dd>
A directory containing documents that will be joined and loaded as a
collection. Any documents that fail basic validation will reject the
whole set. Use of the directory parameter may not be used with the filename
parameter.
</dd>
</dl>
### Sample
```
$ shipyard create configdocs coll1 --filename=/home/ubuntu/yaml/coll1.yaml
Configuration documents added.
Status: Validations succeeded
Reason: Validation
```
Attempting to load the same collection into the uncommitted buffer.
```
$ shipyard create configdocs coll1 --filename=/home/ubuntu/yaml/coll1.yaml
Error: Invalid collection specified for buffer
Reason: Buffermode : rejectoncontents
- Error: Buffer is either not empty or the collection already exists in buffer. Setting a different buffermode may provide the desired functionality
```
Replace the buffer with --replace
```
$ shipyard create configdocs coll1 --replace --filename=/home/ubuntu/yaml/coll1.yaml
Configuration documents added.
Status: Validations succeeded
Reason: Validation
```
---
# Describe Commands
## describe
Retrieves the detailed information about the supplied namespaced item
```
shipyard describe
<namespaced item>
Example:
shipyard describe action/01BTG32JW87G0YKA1K29TKNAFX
Equivalent to:
shipyard describe action 01BTG32JW87G0YKA1K29TKNAFX
shipyard describe step/01BTG32JW87G0YKA1K29TKNAFX/preflight
Equivalent to:
shipyard describe step preflight --action=01BTG32JW87G0YKA1K29TKNAFX
shipyard describe validation/01BTG32JW87G0YKA1K29TKNAFX/01BTG3PKBS15KCKFZ56XXXBGF2
Equivalent to:
shipyard describe validation 01BTG3PKBS15KCKFZ56XXXBGF2 \
--action=01BTG32JW87G0YKA1K29TKNAFX
shipyard describe workflow/deploy_site__2017-01-01T12:34:56.123456
Equivalent to:
shipyard describe workflow deploy_site__2017-01-01T12:34:56.123456
```
---
## describe action
Retrieves the detailed information about the supplied action id.
```
shipyard describe action
<action id>
Example:
shipyard describe action 01BTG32JW87G0YKA1K29TKNAFX
```
### Sample
```
$ shipyard describe action/01BZZK07NF04XPC5F4SCTHNPKN
Name: deploy_site
Action: action/01BZZK07NF04XPC5F4SCTHNPKN
Lifecycle: Failed
Parameters: {}
Datetime: 2017-11-27 20:34:24.610604+00:00
Dag Status: failed
Context Marker: 71d4112e-8b6d-44e8-9617-d9587231ffba
User: shipyard
Steps Index State
step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom 1 success
step/01BZZK07NF04XPC5F4SCTHNPKN/dag_concurrency_check 2 success
step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 3 failed
step/01BZZK07NF04XPC5F4SCTHNPKN/validate_site_design 4 None
step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 5 failed
step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 6 failed
step/01BZZK07NF04XPC5F4SCTHNPKN/drydock_build 7 None
Commands User Datetime
invoke shipyard 2017-11-27 20:34:34.443053+00:00
Validations: None
```
---
## describe step
Retrieves the step details associated with an action and step.
```
shipyard describe step
<step id>
--action=<action id>
Example:
shipyard describe step preflight --action=01BTG32JW87G0YKA1K29TKNAFX
```
<dl>
<dt>&lt;step id&gt;</dt>
<dd>
The id of the step found in the describe action response.
</dd>
<dt>--action=&lt;action id&gt;</dt>
<dd>
The action id that provides the context for this step.
</dd>
</dl>
### Sample
```
$ shipyard describe step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom
Name: action_xcom
Task ID: step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom
Index: 1
State: success
Start Date: 2017-11-27 20:34:45.604109
End Date: 2017-11-27 20:34:45.818946
Duration: 0.214837
Try Number: 1
Operator: PythonOperator
```
---
## describe validation
Retrieves the validation details associated with an action and validation id
```
shipyard describe validation
<validation id>
--action=<action id>
Example:
shipyard describe validation 01BTG3PKBS15KCKFZ56XXXBGF2 \
--action=01BTG32JW87G0YKA1K29TKNAFX
```
<dl>
<dt>&lt;validation id&gt;</dt>
<dd>
The id of the validation found in the describe action response.
</dd>
<dt>--action=&lt;action id&gt;</dt>
<dd>
The action id that provides the context for this validation.
</dd>
</dl>
### Sample
```
TBD
```
---
## describe workflow
Retrieves the details for a workflow that is running or has run in the workflow
engine.
```
shipyard describe workflow
<workflow id>
Example:
shipyard describe workflow deploy_site__2017-01-01T12:34:56.123456
```
<dl>
<dt>&lt;workflow id&gt;</dt>
<dd>
The id of the workflow found in the get workflows response.
</dd>
</dl>
### Sample
```
$ shipyard describe workflow deploy_site__2017-11-27T20:34:33.000000
Workflow: deploy_site__2017-11-27T20:34:33.000000
State: failed
Dag ID: deploy_site
Execution Date: 2017-11-27 20:34:33
Start Date: 2017-11-27 20:34:33.979594
End Date: None
External Trigger: True
Steps State
action_xcom success
dag_concurrency_check success
deckhand_get_design_version failed
validate_site_design None
deckhand_get_design_version failed
deckhand_get_design_version failed
drydock_build None
Subworkflows:
Workflow: deploy_site.deckhand_get_design_version__2017-11-27T20:34:33.000000
State: failed
Dag ID: deploy_site.deckhand_get_design_version
Execution Date: 2017-11-27 20:34:33
Start Date: 2017-11-27 20:35:06.281825
End Date: None
External Trigger: False
Workflow: deploy_site.deckhand_get_design_version.deckhand_get_design_version__2017-11-27T20:34:33.000000
State: failed
Dag ID: deploy_site.deckhand_get_design_version.deckhand_get_design_version
Execution Date: 2017-11-27 20:34:33
Start Date: 2017-11-27 20:35:20.725506
End Date: None
External Trigger: False
```
---
# Get Commands
## get actions
Lists the actions that have been invoked.
```
shipyard get actions
```
### Sample
```
$ shipyard get actions
Name Action Lifecycle
deploy_site action/01BZZK07NF04XPC5F4SCTHNPKN Failed
update_site action/01BZZKMW60DV2CJZ858QZ93HRS Processing
```
---
## get configdocs
Retrieve documents loaded into Shipyard, either committed or from the
Shipyard Buffer.
```
shipyard get configdocs
<collection>
[--committed | --buffer]
Example:
shipyard get configdocs design
```
<dl>
<dt>&lt;collection&gt;</dt>
<dd>
The collection to retrieve for viewing.
</dd>
<dt>--committed</dt>
<dd>
Retrieve the documents that have last been committed for this
collection
</dd>
<dt>--buffer</dt>
<dd>
Retrive the documents that have been loaded into Shipyard
since the prior commit. If no documents have been loaded
into the buffer for this collection, this will return an empty
response (default)
</dd>
</dl>
### Sample
```
$ shipyard get configdocs coll1
data:
chart_groups: [kubernetes-proxy, container-networking, dns, kubernetes, kubernetes-rbac]
release_prefix: ucp
id: 1
metadata:
layeringDefinition: {abstract: false, layer: site}
name: cluster-bootstrap-1
schema: metadata/Document/v1.0
storagePolicy: cleartext
schema: armada/Manifest/v1.0
status: {bucket: coll1, revision: 1}
```
---
## get renderedconfigdocs
Retrieve the rendered version of documents loaded into Shipyard. Rendered
documents are the "final" version of the documents after applying Deckhand
layering and substitution.
```
shipyard get renderedconfigdocs
[--committed | --buffer]
Example:
shipyard get renderedconfigdocs
```
<dl>
<dt>--committed</dt>
<dd>
Retrieve the documents that have last been committed.
</dd>
<dt>--buffer</dt>
<dd>
Retrieve the documents that have been loaded into Shipyard
since the prior commit. (default)
</dd>
</dl>
### Sample
```
$ shipyard get renderedconfigdocs
data:
chart_groups: [kubernetes-proxy, container-networking, dns, kubernetes, kubernetes-rbac]
release_prefix: ucp
id: 1
metadata:
layeringDefinition: {abstract: false, layer: site}
name: cluster-bootstrap-1
schema: metadata/Document/v1.0
storagePolicy: cleartext
schema: armada/Manifest/v1.0
status: {bucket: coll1, revision: 1}
```
---
## get workflows
Retrieve workflows that are running or have run in the workflow engine. This
includes processses that may not have been started as an action
(e.g. scheduled tasks).
```
shipyard get workflows
[--since=<date>]
Example:
shipyard get workflows
shipyard get workflows --since=2017-01-01T12:34:56.123456
```
<dl>
<dt>--since=&lt;date&gt;</dt>
<dd>
The historical cutoff date to limit the results of of this response.
</dd>
</dl>
### Sample
```
$ shipyard get workflows
Workflows State
deploy_site__2017-11-27T20:34:33.000000 failed
update_site__2017-11-27T20:45:47.000000 running
```
---
# help commands
Provides topical help for shipyard. Note that --help will provide more
specific command help.
```
shipyard help
[<topic>]
Example:
shipyard help configdocs
```
<dl>
<dt>&lt;topic&gt;</dt>
<dd>
The topic of the help to be displayed. If this parameter is not specified
the list of avaialable topics will be displayed.
</dd>
</dl>
### Sample
```
$ shipyard help
THE SHIPYARD COMMAND
The base shipyard command supports options that determine cross-CLI behaviors.
FORMAT
shipyard [--context-marker=<uuid>] [--os_{various}=<value>]
[--debug/--no-debug] [--output-format] <subcommands>
Please Note: --os_auth_url is required for every command except shipyard help
<topic>.
TOPICS
For information of the following topics, run shipyard help <topic>
actions
configdocs
```

View File

@ -1,49 +1,55 @@
## Shipyard ##
## Shipyard
Shipyard adopts the Falcon web framework and uses Apache Airflow as the backend engine to programmatically
author, schedule and monitor workflows.
Shipyard adopts the Falcon web framework and uses Apache Airflow as the backend
engine to programmatically author, schedule and monitor workflows.
The current workflow is as follows:
1. Inital region/site data will be passed to Shipyard from either a human operator or Jenkins
2. The data (in YAML format) will be sent to [DeckHand](https://github.com/att-comdev/deckhand) for processing
3. Shipyard will make use of the post-processed data from DeckHand to interact with [DryDock](https://github.com/att-comdev/drydock)
4. DryDock will interact with [Promenade](https://github.com/att-comdev/promenade) to provision and deploy
bare metal nodes using Ubuntu MAAS and a resilient Kubernetes cluster will be created at the end of the
process
5. Once the Kubernetes clusters are up and validated to be working properly, Shipyard will interact with
[Armada](https://github.com/att-comdev/armada) to deploy OpenStack using [OpenStack Helm](https://github.com/openstack/openstack-helm)
6. Once the OpenStack cluster is deployed, Shipyard will trigger a workflow to perform basic sanity health
checks on the cluster
1. Inital region/site data will be passed to Shipyard from either a human
operator or Jenkins
2. The data (in YAML format) will be sent to
[DeckHand](https://github.com/att-comdev/deckhand) for validation and
storage
3. Shipyard will make use of the post-processed data from DeckHand to interact
with [DryDock](https://github.com/att-comdev/drydock)
4. DryDock will interact with
[Promenade](https://github.com/att-comdev/promenade) to provision and deploy
bare metal nodes using Ubuntu MAAS and a resilient Kubernetes cluster will
be created at the end of the process
5. Once the Kubernetes clusters are up and validated to be working properly,
Shipyard will interact with [Armada](https://github.com/att-comdev/armada)
to deploy OpenStack using
[OpenStack Helm](https://github.com/openstack/openstack-helm)
6. Once the OpenStack cluster is deployed, Shipyard will trigger a workflow to
perform basic sanity health checks on the cluster
Note: This project, along with the tools used within are community-based and
open sourced.
### Mission
The goal for Shipyard is to provide a customizable *framework* for operators
and developers alike. This framework will enable end-users to orchestrate and
deploy a fully functional container-based Cloud.
Note: This project, along with the tools used within are community-based and open sourced.
### Roadmap
The detailed Roadmap can be viewed on the
[LCOO JIRA](https://openstack-lcoo.atlassian.net/projects/SHIPYARD/issues/)
### Mission ###
The goal for Shipyard is to provide a customizable *framework* for operators and developers alike. This
framework will enable end-users to orchestrate and deploy a fully functional container-based Cloud
### Roadmap ###
The detailed Roadmap can be viewed on the [LCOO JIRA](https://openstack-lcoo.atlassian.net/projects/SHIPYARD/issues/)
- Create Helm Charts for Airflow
- Create Helm Charts for Shipyard
- Create Falcon API Framework
- Integrate with DeckHand, DryDock/Promenade, Armada
- Expand funcitionality to provide visibility into and options for operation of
a UCP installation
### Getting Started
This project is under development at the moment. We encourage anyone who is
interested in Shipyard to review our
[documentation](http://shipyard.readthedocs.io/en/latest/)
### Getting Started ###
This project is under development at the moment. We encourage anyone who is interested in Shipyard to review
our [Installation](https://github.com/att-comdev/shipyard/blob/master/docs/deployment_guide.md) documentation
### Bugs ###
Bugs are traced in [LCOO JIRA](https://openstack-lcoo.atlassian.net/projects/SHIPYARD/issues/). If you find
a bug, please feel free to create a [GitHub issue](https://github.com/att-comdev/shipyard_issues) and it will be synced to JIRA.
### Bugs
If you find a bug, please feel free to create a
[GitHub issue](https://github.com/att-comdev/shipyard/issues)

894
docs/source/API.rst Normal file
View File

@ -0,0 +1,894 @@
.. _shipyard_api:
Shipyard API
============
Logically, the API has three parts to handle the three areas of
functionality in Shipyard.
1. Document Staging
2. Action Handling
3. Airflow Monitoring
Standards used by the API
-------------------------
See `UCP API
conventions <https://github.com/att-comdev/ucp-integration/blob/master/docs>`__
Notes on examples
-----------------
Examples assume the following environment variables are set before
issuing the curl commands shown:
::
$TOKEN={a valid keystone token}
$URL={the url and port of the shipyard api}
- Examples will use json formatted by the jq command for sake of
presentation.
- Actual responses will not formatted.
- The use of ellipsis indicate a repeated structure in the case of
lists, or prior/subsequent structure unimportant to the example (or
considered understood).
- The content-length response headers have been removed so as to not
cause confusion with the listed output.
Example response for an invalid token:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
HTTP/1.1 401 Unauthorized
content-type: application/json
x-shipyard-req: a8194b97-8973-4b04-a3b3-2bd319024c5d
WWW-Authenticate: Keystone uri='http://keystone-api.ucp.svc.cluster.local:80/v3'
{
"apiVersion": "v1.0",
"status": "Failure",
"metadata": {},
"message": "Unauthenticated",
"code": "401 Unauthorized",
"details": {
"errorList": [
{
"message": "Credentials are not established"
}
],
"errorCount": 1,
"errorType": "ApiError"
},
"kind": "status",
"reason": "Credentials are not established"
}
Document Staging API
--------------------
Shipyard will serve as the entrypoint for documents (designs, secrets,
configurations, etc...) into a site. Documents are posted to Shipyard in
collections, rather than individually. At any point in time, there will be two
represented versions of documents in a site that are accessible via this API:
- The "Committed Documents" version, which represents the last version of
documents that were successfully commited with a commit_configdocs action.
- The "Shipyard Buffer" version, which represents the collection of documents
that have been ingested by this API since the last commited version. Note
that only one set of documents maybe posted to the buffer at a time by
default. (This behavior can be overridden by query parameters issued by the
user of Shipyard)
All versions of documents rely upon Deckhand for storage. Shipyard uses the
tagging features of Deckhand of to find the appropriate Committed Documents
and Shipyard Buffer version.
/v1.0/configdocs/{collection_id}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Represents the site configuration documents
Entity Structure
^^^^^^^^^^^^^^^^
The documents as noted above (commonly yaml), in a format understood by
Deckhand
POST /v1.0/configdocs/{collection_id}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Ingests a collection of documents. Synchronous. POSTing an empty body
indicates that the specified collection should be deleted when the
Shipyard Buffer is committed. If a POST to the commitconfigdocs is in
progress, this POST should be rejected with a 409 error.
.. note::
The expected input type for this request is Content-Type: application/x-yaml
Query Parameters
''''''''''''''''
- bufferMode=append|replace\|\ **rejectOnContents**
Indicates how the existing Shipyard Buffer should be handled. By default,
Shipyard will reject the POST if contents already exist in the Shipyard
Buffer.
- append: Add the collection to the Shipyard Buffer, only if that
collection doesnt already exist in the Shipyard Buffer. If the
collection is already present, the request will be rejected and a 409
Conflict will be returned.
- replace: Clear the Shipyard Buffer before adding the specified
collection.
Responses
'''''''''
201 Created
If the documents are successfully ingested, even with validation failures.
Response message includes:
- a list of validation results
- The response headers will include a Location indicating the GET
endpoint to retrieve the configDocs
409 Conflict
A condition in the system is blocking this document ingestion
- If a commitconfigdocs POST is in progress.
- If any collections exist in the Shipyard Buffer unless bufferMode=replace
or bufferMode=append.
- If bufferMode=append, and the collection being posted is already in the
Shipyard Buffer
GET /v1.0/configdocs/{collection_id}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Returns the source documents for a collection of documents
.. note::
The output type for this request is Content-Type: application/x-yaml
Query Parameters
''''''''''''''''
version=committed | **buffer**
Return the documents for the version specified - buffer by default.
Responses
'''''''''
200 OK
If documents can be retrieved.
- If the response is 200 with an empty response body, this indicates
that the buffer version is attempting to delete the collection
when it is committed. An empty response body will only be possible
for version=buffer.
404 Not Found
If the collection is not represented
- When version=buffer, this indicates that no representations of this
collection have been POSTed since the last committed version.
- When version=committed, this indicates that either the collection has
never existed or has been deleted by a prior commit.
/v1.0/renderedconfigdocs
~~~~~~~~~~~~~~~~~~~~~~~~
Represents the site configuration documents, as a whole set - does not
consider collections in any way.
GET /v1.0/renderedconfigdocs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Returns the full set of configdocs in their rendered form.
.. note::
The output type for this request is 'Content-Type: application/x-yaml'
Query Parameters
''''''''''''''''
version=committed|**buffer**
Return the documents for the version specified - buffer by default.
Responses
'''''''''
200 OK
If documents can be retrieved.
/v1.0/commitconfigdocs
~~~~~~~~~~~~~~~~~~~~~~
An RPC style command to trigger a commit of the configuration documents from
the Shipyard Buffer to the Committed Documents. This resource will support POST
only.
Entity Structure
^^^^^^^^^^^^^^^^
The response will be the list of validations from all downstream systems that
perform validation during the commit process. The structure will match the
error response object described in the `UCP API
conventions <https://github.com/att-comdev/ucp-integration/blob/master/docs>`__
and will be an aggregation of each UCP components responses.
POST /v1.0/commitconfigdocs
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Synchronous. Performs the commit of the Shipyard Buffer to the Committed
Documents. This invokes each of the UCP components to examine the Shipyard
Buffer version of the configuration documents and aggregate the responses.
While performing this commit, further POSTing of configdocs, or other commits
may not be invoked (Shipyard will block those requests with a 409 response). If
there are any failures to validate, the Shipyard Buffer and Commited Documents
will remain unchanged. If successful, the Shipyard Buffer will be cleared, and
the Committed documents will be updated.
.. note::
If there are unhandled runtime errors during the commitconfigdocs POST, a
deadlock situation may be possible. Future enhancements may improve this
handling.
Query Parameters
''''''''''''''''
force=true | **false**
By default, false, if there are validation failures the POST will
fail with a 400 response. With force=true, allows for the commit to
succeed (with a 200 response) even if there are validation failures
from downstream components. The aggregate response of validation
failures will be returned in this case, but the invalid documents
will still be moved from the Shipyard Buffer to the Committed
Documents.
Responses
'''''''''
200 OK
If the validations are successful. Returns an “empty” structure as as
response indicating no errors. A 200 may also be returned if there
are validation failures, but the force=true query parameter was
specified. In this case, the response will contain the list of
validations.
400 Bad Request
If the validations fail. Returns a populated response structure
containing the aggregation of the failed validations.
409 Conflict
If the there is a POST to commitconfigdocs in progress.
Example
'''''''
::
{
"apiVersion": "v1",
"code": "400 Bad Request",
"details": {
"errorCount": 2,
"messageList": [
{
"error": true,
"message": "Error loading effective site: 'NoneType' object is not iterable",
"name": "Drydock"
},
{
"error": true,
"message": "Armada unable to validate configdocs",
"name": "Armada"
}
]
},
"kind": "Status",
"message": "Validations failed",
"metadata": {},
"reason": "Validation",
"status": "Invalid"
}
Action API
----------
The Shipyard Action API is a resource that allows for creation, control and
investigation of triggered workflows. These actions encapsulate a command
interface for the Undercloud Platform. See :ref:`shipyard_action_commands` for
supported actions
/v1.0/actions
~~~~~~~~~~~~~
Entity Structure
^^^^^^^^^^^^^^^^
A list of actions that have been executed through shipyard's action API.
::
[
{ Action objects summarized, See below},
...
]
GET /v1.0/actions
^^^^^^^^^^^^^^^^^
Returns the list of actions in the system that have been posted, and are
accessible to the current user.
Responses
'''''''''
200 OK
If the actions can be retrieved.
Example
'''''''
::
$ curl -X GET $URL/api/v1.0/actions -H "X-Auth-Token:$TOKEN"
HTTP/1.1 200 OK
x-shipyard-req: 0804d13e-08fc-4e60-a819-3b7532cac4ec
content-type: application/json; charset=UTF-8
[
{
"dag_status": "failed",
"parameters": {},
"steps": [
{
"id": "action_xcom",
"url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/action_xcom",
"index": 1,
"state": "success"
},
{
"id": "dag_concurrency_check",
"url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/dag_concurrency_check",
"index": 2,
"state": "success"
},
{
"id": "preflight",
"url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/preflight",
"index": 3,
"state": "failed"
},
...
],
"action_lifecycle": "Failed",
"dag_execution_date": "2017-09-23T02:42:12",
"id": "01BTP9T2WCE1PAJR2DWYXG805V",
"dag_id": "deploy_site",
"datetime": "2017-09-23 02:42:06.860597+00:00",
"user": "shipyard",
"context_marker": "416dec4b-82f9-4339-8886-3a0c4982aec3",
"name": "deploy_site"
},
...
]
POST /v1.0/actions
^^^^^^^^^^^^^^^^^^
Creates an action in the system. This will cause some action to start. The
input body to this post will represent an action object that has at least these
fields:
name
The name of the action to invoke, as noted in :ref:`shipyard_action_commands`
parameters
A dictionary of parameters to use for the trigger invocation. The supported
parameters will vary for the action invoked.
::
{
"name" : "action name",
"parameters" : { varies by action }
}
The POST will synchronously create the action (a shell object that represents
a DAG invocation), perform any checks to validate the preconditions to run the
DAG, and trigger the invocation of the DAG. The DAG will run asynchronously in
airflow.
Responses
'''''''''
201 Created
If the action is created successfully, and all preconditions to run the DAG
are successful. The response body is the action entity created.
400 Bad Request
If the action name doesn't exist, or the input entity is otherwise malformed.
409 Conflict
For any failed pre-run validations. The response body is the action entity
created, with the failed validations. The DAG will not begin execution in
this case.
Example
'''''''
::
$ curl -D - -d '{"name":"deploy_site"}' -X POST $URL/api/v1.0/actions \
-H "X-Auth-Token:$TOKEN" -H "content-type:application/json"
HTTP/1.1 201 Created
location: {$URL}/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN
x-shipyard-req: 629f2ea2-c59d-46b9-8641-7367a91a7016
content-type: application/json; charset=UTF-8
{
"dag_status": "SCHEDULED",
"parameters": {},
"dag_execution_date": "2017-09-24T19:05:49",
"id": "01BTTMFVDKZFRJM80FGD7J1AKN",
"dag_id": "deploy_site",
"name": "deploy_site",
"user": "shipyard",
"context_marker": "629f2ea2-c59d-46b9-8641-7367a91a7016",
"timestamp": "2017-09-24 19:05:43.603591"
}
/v1.0/actions/{action_id}
~~~~~~~~~~~~~~~~~~~~~~~~~
Each action will be assigned an unique id that can be used to get
details for the action, including the execution status.
Entity Structure
^^^^^^^^^^^^^^^^
All actions will include fields that indicate the following data:
action_lifecycle
A summarized value indicating the status or lifecycle phase of the action.
- Pending - The action is scheduled or preparing for execution.
- Processing - The action is underway.
- Complete - The action has completed successfully.
- Failed - The action has encountered an error, and has failed.
- Paused - The action has been paused by a user.
command audit
A list of commands that have been issued against the action. Initially,
the action listed will be “invoke”, but may include “pause”, “unpause”,
or “stop” if those commands are issued.
context_marker
The user supplied or system assigned context marker associated with the
action
dag_execution_date
The execution date assigned by the workflow system during action
creation.
dag_status
Represents the status that airflow provides for an executing DAG.
datetime
The time at which the action was invoked.
id
The identifier for the action, a 26 character ULID assigned during the
creation of the action.
name
The name of the action, e.g.: deploy_site.
parameters
The parameters configuring the action that were supplied by the user
during action creation.
steps
The list of steps for the action, including the status for that step.
user
The user who has invoked this action, as acquired from the authorization
token.
validations
A list of validations that have been done, including any status
information for those validations. During the lifecycle of the action,
this list of validations may continue to grow.
GET /v1.0/actions/{action_id}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Returns the action entity for the specified id.
Responses
'''''''''
200 OK
Example
'''''''
::
$ curl -D - -X GET $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN \
-H "X-Auth-Token:$TOKEN"
HTTP/1.1 200 OK
x-shipyard-req: eb3eacb3-4206-40df-bd91-2a3a6d81cd02
content-type: application/json; charset=UTF-8
{
"name": "deploy_site",
"dag_execution_date": "2017-09-24T19:05:49",
"validations": [],
"id": "01BTTMFVDKZFRJM80FGD7J1AKN",
"dag_id": "deploy_site",
"command_audit": [
{
"id": "01BTTMG16R9H3Z4JVQNBMRV1MZ",
"action_id": "01BTTMFVDKZFRJM80FGD7J1AKN",
"datetime": "2017-09-24 19:05:49.530223+00:00",
"user": "shipyard",
"command": "invoke"
}
],
"user": "shipyard",
"context_marker": "629f2ea2-c59d-46b9-8641-7367a91a7016",
"datetime": "2017-09-24 19:05:43.603591+00:00",
"dag_status": "failed",
"parameters": {},
"steps": [
{
"id": "action_xcom",
"url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/action_xcom",
"index": 1,
"state": "success"
},
{
"id": "dag_concurrency_check",
"url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/dag_concurrency_check",
"index": 2,
"state": "success"
},
{
"id": "preflight",
"url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/preflight",
"index": 3,
"state": "failed"
},
{
"id": "deckhand_get_design_version",
"url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/deckhand_get_design_version",
"index": 4,
"state": null
},
...
],
"action_lifecycle": "Failed"
}
/v1.0/actions/{action_id}/validationdetails/{validation_id}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Allows for drilldown to validation detailed info.
Entity Structure
^^^^^^^^^^^^^^^^
The detailed information for a validation
::
{ TBD }
GET /v1.0/actions/{action_id}/validationdetails/{validation_id}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Returns the validation detail by Id for the supplied action Id.
Responses
'''''''''
200 OK
/v1.0/actions/{action_id}/steps/{step_id}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Allow for drilldown to step information. The step information includes
details of the steps excution, successful or not, and enough to
facilitate troubleshooting in as easy a fashion as possible.
Entity Structure
^^^^^^^^^^^^^^^^
A step entity represents detailed information representing a single step
of execution as part of an action. Not all fields are necessarily
represented in every returned entity.
dag_id
The name/id of the workflow DAG that contains this step.
duration
The duration (seconds) for the step.
end_date
The timestamp of the completion of the step.
execution_date
The execution date of the workflow that contains this step.
index
The numeric value representing the position of this step in the sequence
of steps associated with this step.
operator
The name of the processing facility used by the workflow system.
queued_dttm
The timestamp when the step was enqueued by the workflow system.
start_date
The timestamp for the beginning of execution for this step.
state
The execution state of the step.
task_id
The name of the task used by the workflow system (and also representing
this step name queried in the reqeust.
try_number
A number of retries taken in the case of failure. Some workflow steps
may be configured to retry before considering the step truly failed.
GET /v1.0/actions/{action_id}/steps/{step_id}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Returns the details for a step by id for the given action by Id. #####
Responses
'''''''''
200 OK
Example
'''''''
::
$ curl -D - \
-X GET $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/action_xcom \
-H "X-Auth-Token:$TOKEN"
HTTP/1.1 200 OK
x-shipyard-req: 72daca4d-1f79-4e08-825f-2ad181912a47
content-type: application/json; charset=UTF-8
{
"end_date": "2017-09-24 19:05:59.446213",
"duration": 0.165181,
"queued_dttm": "2017-09-24 19:05:52.993983",
"operator": "PythonOperator",
"try_number": 1,
"task_id": "action_xcom",
"state": "success",
"execution_date": "2017-09-24 19:05:49",
"dag_id": "deploy_site",
"index": 1,
"start_date": "2017-09-24 19:05:59.281032"
}
/v1.0/actions/{action_id}/control/{control_verb}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Allows for issuing DAG controls against an action.
Entity Structure
^^^^^^^^^^^^^^^^
None, there is no associated response entity for this resource
POST /v1.0/actions/{action_id}/{control_verb}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Trigger a control action against an activity.- this includes: pause, unpause
Responses
'''''''''
202 Accepted
Example
'''''''
Failure case - command is invalid for the execution state of the action.
::
$ curl -D - \
-X POST $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/control/pause \
-H "X-Auth-Token:$TOKEN"
HTTP/1.1 409 Conflict
content-type: application/json
x-shipyard-req: 9c9551e0-335c-4297-af93-8440cc6b324f
{
"apiVersion": "v1.0",
"status": "Failure",
"metadata": {},
"message": "Unable to pause action",
"code": "409 Conflict",
"details": {
"errorList": [
{
"message": "dag_run state must be running, but is failed"
}
],
"errorCount": 1,
"errorType": "ApiError"
},
"kind": "status",
"reason": "dag_run state must be running, but is failed"
}
Success case
::
$ curl -D - \
-X POST $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/control/pause \
-H "X-Auth-Token:$TOKEN"
HTTP/1.1 202 Accepted
content-length: 0
x-shipyard-req: 019fae1c-03b0-4af1-b57d-451ae6ddac77
content-type: application/json; charset=UTF-8
Airflow Monitoring API
----------------------
Airflow has a primary function of scheduling DAGs, as opposed to Shipyards
primary case of triggering DAGs. Shipyard provides functionality to allow for
an operator to monitor and review these scheduled workflows (DAGs) in addition
to the ones triggered by Shipyard. This API will allow for accessing Airflow
DAGs of any type providing a peek into the totality of what is happening in
Airflow.
/v1.0/workflows
~~~~~~~~~~~~~~~
The resource that represents DAGs (workflows) in airflow
Entity Structure
^^^^^^^^^^^^^^^^
A list of objects representing the DAGs that have run in airflow.
GET /v1.0/workflows
^^^^^^^^^^^^^^^^^^^
Queries airflow for DAGs that are running or have run (successfully or
unsuccessfully) and provides a summary of those things.
Query parameters
''''''''''''''''
since={iso8601 date (past) or duration}
optional, a boundary in the past within which to retrieve results. Default is
30 days in the past.
Responses
'''''''''
200 OK
Example
'''''''
Notice the workflow_id values, these can be used for drilldown.
::
curl -D - -X GET $URL/api/v1.0/workflows -H "X-Auth-Token:$TOKEN"
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
x-shipyard-req: 3ab4ccc6-b956-4c7a-9ae6-183c562d8297
[
{
"execution_date": "2017-10-09 21:18:56",
"end_date": null,
"workflow_id": "deploy_site__2017-10-09T21:18:56.000000",
"start_date": "2017-10-09 21:18:56.685999",
"external_trigger": true,
"dag_id": "deploy_site",
"state": "failed",
"run_id": "manual__2017-10-09T21:18:56"
},
{
"execution_date": "2017-10-09 21:19:03",
"end_date": null,
"workflow_id": "deploy_site__2017-10-09T21:19:03.000000",
"start_date": "2017-10-09 21:19:03.361522",
"external_trigger": true,
"dag_id": "deploy_site",
"state": "failed",
"run_id": "manual__2017-10-09T21:19:03"
}
...
]
/v1.0/workflows/{workflow_id}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Entity Structure
^^^^^^^^^^^^^^^^
An object representing the information available from airflow regarding
a DAGs execution
GET /v1.0/workflows/{id}
^^^^^^^^^^^^^^^^^^^^^^^^
Further details of a particular workflows steps. All steps of all
sub-dags will be included in the list of steps, as well as section
indicating the sub-dags for this parent workflow.
Responses
'''''''''
200 OK
Example
'''''''
.. note::
Sub_dags can be queried to restrict to only that sub-dags steps. e.g. using
this as {workflow_id}:
deploy_site.preflight.armada_preflight_check__2017-10-09T21:19:03.000000
::
curl -D - \
-X GET $URL/api/v1.0/workflows/deploy_site__2017-10-09T21:19:03.000000 \
-H "X-Auth-Token:$TOKEN"
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
x-shipyard-req: 98d71530-816a-4692-9df2-68f22c057467
{
"execution_date": "2017-10-09 21:19:03",
"end_date": null,
"workflow_id": "deploy_site__2017-10-09T21:19:03.000000",
"start_date": "2017-10-09 21:19:03.361522",
"external_trigger": true,
"steps": [
{
"end_date": "2017-10-09 21:19:14.916220",
"task_id": "action_xcom",
"start_date": "2017-10-09 21:19:14.798053",
"duration": 0.118167,
"queued_dttm": "2017-10-09 21:19:08.432582",
"try_number": 1,
"state": "success",
"operator": "PythonOperator",
"dag_id": "deploy_site",
"execution_date": "2017-10-09 21:19:03"
},
{
"end_date": "2017-10-09 21:19:25.283785",
"task_id": "dag_concurrency_check",
"start_date": "2017-10-09 21:19:25.181492",
"duration": 0.102293,
"queued_dttm": "2017-10-09 21:19:19.283132",
"try_number": 1,
"state": "success",
"operator": "ConcurrencyCheckOperator",
"dag_id": "deploy_site",
"execution_date": "2017-10-09 21:19:03"
},
{
"end_date": "2017-10-09 21:20:05.394677",
"task_id": "preflight",
"start_date": "2017-10-09 21:19:34.994775",
"duration": 30.399902,
"queued_dttm": "2017-10-09 21:19:28.449848",
"try_number": 1,
"state": "failed",
"operator": "SubDagOperator",
"dag_id": "deploy_site",
"execution_date": "2017-10-09 21:19:03"
},
...
],
"dag_id": "deploy_site",
"state": "failed",
"run_id": "manual__2017-10-09T21:19:03",
"sub_dags": [
{
"execution_date": "2017-10-09 21:19:03",
"end_date": null,
"workflow_id": "deploy_site.preflight__2017-10-09T21:19:03.000000",
"start_date": "2017-10-09 21:19:35.082479",
"external_trigger": false,
"dag_id": "deploy_site.preflight",
"state": "failed",
"run_id": "backfill_2017-10-09T21:19:03"
},
...,
{
"execution_date": "2017-10-09 21:19:03",
"end_date": null,
"workflow_id": "deploy_site.preflight.armada_preflight_check__2017-10-09T21:19:03.000000",
"start_date": "2017-10-09 21:19:48.265023",
"external_trigger": false,
"dag_id": "deploy_site.preflight.armada_preflight_check",
"state": "failed",
"run_id": "backfill_2017-10-09T21:19:03"
}
]
}

View File

@ -0,0 +1,68 @@
.. _shipyard_action_commands:
Action Commands
===============
Supported actions
-----------------
These actions are currently supported using the Action API
deploy_site
~~~~~~~~~~~
Triggers the initial deployment of a site, using the latest committed
configuration documents. Steps:
#. Concurrency check
Prevents concurrent site modifications by conflicting
actions/workflows.
#. Preflight checks
Ensures all UCP components are in a responsive state.
#. Get design version
Uses Deckhand to discover the latest committed version of design for
the site.
#. Validate design
Asks each involved UCP component to validate the design. This ensures
that the previously committed design is valid at the present time.
#. Drydock build
Orchestrates the Drydock component to configure hardware and the
Kubernetes environment (Drydock -> Promenade)
#. Check deployed node status
Checks that the deployment of nodes is successful.
#. Armada build
Orchestrates Armada to configure software on the nodes as designed.
Actions under development
~~~~~~~~~~~~~~~~~~~~~~~~~
These actions are under active development
- update_site
Triggers the initial deployment of a site, using the latest committed
configuration documents. Steps: (a superset of deploy_site)
- redeploy_server
Using parameters to indicate which server(s), triggers a redeployment of
server to the last known good design and secrets
Future actions
~~~~~~~~~~~~~~
These actions are anticipated for development
- test region
Invoke site validation testing - perhaps baseline is a invocation of all
components regular “component” tests. This test would be used as a
preflight-style test to ensure all components are in a working state.
- test component
Invoke a particular platform component to test it. This test would be
used to interrogate a particular platform component to ensure it is in a
working state, and that its own downstream dependencies are also
operational

666
docs/source/CLI.rst Normal file
View File

@ -0,0 +1,666 @@
.. _shipyard_cli:
Shipyard CLI
============
Environment Variables
---------------------
All commands will utilize the following environment variables to
determine necessary information for execution, unless otherwise noted.
Openstack Keystone Authorization environment variables
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The Shipyard CLI/API Client will check for the presence of appropriate
environment setup to do authentication on behalf of the user. The openrc
variables that will be used are as follows:
- OS_PROJECT_DOMAIN_NAME ("default" if not specified)
- OS_USER_DOMAIN_NAME ("default" if not specified)
- OS_PROJECT_NAME
- OS_USERNAME
- OS_PASSWORD
- OS_AUTH_URL The fully qualified identity endpoint. E.g. http://keystone.ucp.fully.qualified.name:80/v3
Openstack Keystone Authorization environment variables *not* used
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
These openstack identity variables are not supported by shipyard.
- OS_IDENTITY_API_VERSION
This value will be ignored as Shipyard only supports version 3 at this time
Shipyard command options
------------------------
The base shipyard command supports options that determine cross-CLI behaviors.
These options are positionally immediately following the shipyard command as
shown here:
::
shipyard <--these options> subcommands...
shipyard
[--context-marker=<uuid>]
[--debug/--no-debug]
[--os-{various}=<value>]
[--output-format=[format | raw | cli]] (default = cli)
<subcommands, as noted in this document>
\--context-marker=<uuid>
Specifies a UUID (8-4-4-4-12 format) that will be used to correlate logs,
transactions, etc... in downstream activities triggered by this interaction.
If not specified, Shipyard will supply a new UUID to serve as this marker.
(optional)
\--debug | --no-debug
Enable/disable debugging of this CLI and API client. Defaults to no debug
\--os-<various>=<value>
See supported Openstack Keystone Authorization Environment variables above
for the list of supported names, converting to a downcase version of the
environment variable. E.g.: --os-auth-url=http://keystone.ucp:80/v3
If not specified, the environment variables matching these options will be
used instead. The Keystone os-auth-url should reference the exposed
keystone:port for the target Shipyard environment, as this Keystone will be
used to discover the instance of Shipyard. For most invocations other than
help, a valid combination of values must be resolved to authenticate and
authorize the user's invocation.
\--output-format=<format | raw | cli>
Specifies the desired output formating such that:
- format
Display the raw ouptut from the invoked Shipyard API in a column
restricted mode.
- raw
Display the result from the invoked Shipyard API as-is, without
modification.
- cli (default)
Display results in a plain text interpretation of the response from the
invoked Shipyard API.
Commit Commands
---------------
commit configdocs
~~~~~~~~~~~~~~~~~
Attempts to commit the Shipyard Buffer documents, first invoking validation by
downstream components.
::
shipyard commit configdocs
[--force]
Example:
shipyard commit configdocs
\--force
Force the commit to occur, even if validations fail.
Sample
^^^^^^
::
TBD
Control commands
----------------
pause, unpause, stop
~~~~~~~~~~~~~~~~~~~~
Three separate commands with a common format that allow for controlling
the processing of actions created in Shipyard.
pause
pause something in progress e.g. an executing action
unpause
unpause something paused e.g. a paused action
stop
stops an executing or paused item e.g. an action
::
shipyard pause
<type>
<id>
shipyard unpause
<type>
<id>
shipyard stop
<type>
<id>
shipyard
pause|unpause|stop
<qualified name>
Example:
shipyard pause action 01BTG32JW87G0YKA1K29TKNAFX
shipyard unpause action 01BTG32JW87G0YKA1K29TKNAFX
shipyard stop action 01BTG32JW87G0YKA1K29TKNAFX
shipyard pause action/01BTG32JW87G0YKA1K29TKNAFX
<type>
The type of entity to take action upon. Currently supports: action
<id>
The id of the entity to take action upon.
<qualified name>
The qualified name of the item to take the specified action upon
Sample
^^^^^^
::
$ shipyard pause action/01BZZMEXAVYGG7BT0BMA3RHYY7
pause successfully submitted for action 01BZZMEXAVYGG7BT0BMA3RHYY7
A failed command:
::
$ shipyard pause action/01BZZK07NF04XPC5F4SCTHNPKN
Error: Unable to pause action
Reason: dag_run state must be running, but is failed
- Error: dag_run state must be running, but is failed
Create Commands
---------------
create action
~~~~~~~~~~~~~
Invokes the specified workflow through Shipyard. Returns the
id of the action invoked so that it can be queried subsequently.
::
shipyard create action
<action_command>
--param=<parameter> (repeatable)
Example:
shipyard create action redeploy_server --param="server-name=mcp"
<action_command>
The action to invoke.
\--param=<parameter>
A parameter to be provided to the action being invoked. (repeatable)
Sample
^^^^^^
::
$ shipyard create action deploy_site
Name Action Lifecycle
deploy_site action/01BZZK07NF04XPC5F4SCTHNPKN None
create configdocs
~~~~~~~~~~~~~~~~~
Load documents into the Shipyard Buffer. The use of one or more filename
or a single directory option must be specified.
::
shipyard create configdocs
<collection>
[--append | --replace]
--filename=<filename> (repeatable)
|
--directory=<directory>
Example:
shipyard create configdocs design --append --filename=site_design.yaml
.. note::
If neither append nor replace are specified, the Shipyard API default value
of rejectoncontents will be used.
.. note::
Either --filename or --directory must be specified, but both may not be
specified for the same invocation of shipyard.
<collection>
The collection to load.
\--append
Add the collection to the Shipyard Buffer. This will fail if the collection
already exists.
\--replace
Clear the shipyard buffer and replace it with the specified contents.
\--filename=<filename>
The file name to use as the contents of the collection. (repeatable) If
any documents specified fail basic validation, all of the documents will
be rejected. Use of filename parameters may not be used in conjunction
with the directory parameter.
\--directory=<directory>
A directory containing documents that will be joined and loaded as a
collection. Any documents that fail basic validation will reject the
whole set. Use of the directory parameter may not be used with the
filename parameter.
Sample
^^^^^^
::
$ shipyard create configdocs coll1 --filename=/home/ubuntu/yaml/coll1.yaml
Configuration documents added.
Status: Validations succeeded
Reason: Validation
Attempting to load the same collection into the uncommitted buffer.
::
$ shipyard create configdocs coll1 --filename=/home/ubuntu/yaml/coll1.yaml
Error: Invalid collection specified for buffer
Reason: Buffermode : rejectoncontents
- Error: Buffer is either not empty or the collection already exists in buffer. Setting a different buffermode may provide the desired functionality
Replace the buffer with --replace
::
$ shipyard create configdocs coll1 --replace --filename=/home/ubuntu/yaml/coll1.yaml
Configuration documents added.
Status: Validations succeeded
Reason: Validation
Describe Commands
-----------------
describe
~~~~~~~~
Retrieves the detailed information about the supplied namespaced item
::
shipyard describe
<namespaced_item>
Example:
shipyard describe action/01BTG32JW87G0YKA1K29TKNAFX
Equivalent to:
shipyard describe action 01BTG32JW87G0YKA1K29TKNAFX
shipyard describe step/01BTG32JW87G0YKA1K29TKNAFX/preflight
Equivalent to:
shipyard describe step preflight --action=01BTG32JW87G0YKA1K29TKNAFX
shipyard describe validation/01BTG32JW87G0YKA1K29TKNAFX/01BTG3PKBS15KCKFZ56XXXBGF2
Equivalent to:
shipyard describe validation 01BTG3PKBS15KCKFZ56XXXBGF2 \
--action=01BTG32JW87G0YKA1K29TKNAFX
shipyard describe workflow/deploy_site__2017-01-01T12:34:56.123456
Equivalent to:
shipyard describe workflow deploy_site__2017-01-01T12:34:56.123456
describe action
~~~~~~~~~~~~~~~
Retrieves the detailed information about the supplied action id.
::
shipyard describe action
<action_id>
Example:
shipyard describe action 01BTG32JW87G0YKA1K29TKNAFX
Sample
^^^^^^
::
$ shipyard describe action/01BZZK07NF04XPC5F4SCTHNPKN
Name: deploy_site
Action: action/01BZZK07NF04XPC5F4SCTHNPKN
Lifecycle: Failed
Parameters: {}
Datetime: 2017-11-27 20:34:24.610604+00:00
Dag Status: failed
Context Marker: 71d4112e-8b6d-44e8-9617-d9587231ffba
User: shipyard
Steps Index State
step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom 1 success
step/01BZZK07NF04XPC5F4SCTHNPKN/dag_concurrency_check 2 success
step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 3 failed
step/01BZZK07NF04XPC5F4SCTHNPKN/validate_site_design 4 None
step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 5 failed
step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 6 failed
step/01BZZK07NF04XPC5F4SCTHNPKN/drydock_build 7 None
Commands User Datetime
invoke shipyard 2017-11-27 20:34:34.443053+00:00
Validations: None
describe step
~~~~~~~~~~~~~
Retrieves the step details associated with an action and step.
::
shipyard describe step
<step_id>
--action=<action id>
Example:
shipyard describe step preflight --action=01BTG32JW87G0YKA1K29TKNAFX
<step id>
The id of the step found in the describe action response.
\--action=<action id>
The action id that provides the context for this step.
Sample
^^^^^^
::
$ shipyard describe step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom
Name: action_xcom
Task ID: step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom
Index: 1
State: success
Start Date: 2017-11-27 20:34:45.604109
End Date: 2017-11-27 20:34:45.818946
Duration: 0.214837
Try Number: 1
Operator: PythonOperator
describe validation
~~~~~~~~~~~~~~~~~~~
Retrieves the validation details associated with an action and
validation id
::
shipyard describe validation
<validation_id>
--action=<action_id>
Example:
shipyard describe validation 01BTG3PKBS15KCKFZ56XXXBGF2 \
--action=01BTG32JW87G0YKA1K29TKNAFX
<validation_id>
The id of the validation found in the describe action response.
\--action=<action_id>
The action id that provides the context for this validation.
Sample
^^^^^^
::
TBD
describe workflow
~~~~~~~~~~~~~~~~~
Retrieves the details for a workflow that is running or has run in the
workflow engine.
::
shipyard describe workflow
<workflow_id>
Example:
shipyard describe workflow deploy_site__2017-01-01T12:34:56.123456
<workflow_id>
The id of the workflow found in the get workflows response.
Sample
^^^^^^
::
$ shipyard describe workflow deploy_site__2017-11-27T20:34:33.000000
Workflow: deploy_site__2017-11-27T20:34:33.000000
State: failed
Dag ID: deploy_site
Execution Date: 2017-11-27 20:34:33
Start Date: 2017-11-27 20:34:33.979594
End Date: None
External Trigger: True
Steps State
action_xcom success
dag_concurrency_check success
deckhand_get_design_version failed
validate_site_design None
deckhand_get_design_version failed
deckhand_get_design_version failed
drydock_build None
Subworkflows:
Workflow: deploy_site.deckhand_get_design_version__2017-11-27T20:34:33.000000
State: failed
Dag ID: deploy_site.deckhand_get_design_version
Execution Date: 2017-11-27 20:34:33
Start Date: 2017-11-27 20:35:06.281825
End Date: None
External Trigger: False
Workflow: deploy_site.deckhand_get_design_version.deckhand_get_design_version__2017-11-27T20:34:33.000000
State: failed
Dag ID: deploy_site.deckhand_get_design_version.deckhand_get_design_version
Execution Date: 2017-11-27 20:34:33
Start Date: 2017-11-27 20:35:20.725506
End Date: None
External Trigger: False
Get Commands
------------
get actions
~~~~~~~~~~~
Lists the actions that have been invoked.
::
shipyard get actions
Sample
^^^^^^
::
$ shipyard get actions
Name Action Lifecycle
deploy_site action/01BZZK07NF04XPC5F4SCTHNPKN Failed
update_site action/01BZZKMW60DV2CJZ858QZ93HRS Processing
get configdocs
~~~~~~~~~~~~~~
Retrieve documents loaded into Shipyard, either committed or from the
Shipyard Buffer.
::
shipyard get configdocs
<collection>
[--committed | --buffer]
Example:
shipyard get configdocs design
<collection>
The collection to retrieve for viewing.
\--committed
Retrieve the documents that have last been committed for this collection
\--buffer
Retrive the documents that have been loaded into Shipyard since the
prior commit. If no documents have been loaded into the buffer for this
collection, this will return an empty response (default)
Sample
^^^^^^
::
$ shipyard get configdocs coll1
data:
chart_groups: [kubernetes-proxy, container-networking, dns, kubernetes, kubernetes-rbac]
release_prefix: ucp
id: 1
metadata:
layeringDefinition: {abstract: false, layer: site}
name: cluster-bootstrap-1
schema: metadata/Document/v1.0
storagePolicy: cleartext
schema: armada/Manifest/v1.0
status: {bucket: coll1, revision: 1}
get renderedconfigdocs
~~~~~~~~~~~~~~~~~~~~~~
Retrieve the rendered version of documents loaded into Shipyard.
Rendered documents are the "final" version of the documents after
applying Deckhand layering and substitution.
::
shipyard get renderedconfigdocs
[--committed | --buffer]
Example:
shipyard get renderedconfigdocs
\--committed
Retrieve the documents that have last been committed.
\--buffer
Retrieve the documents that have been loaded into Shipyard since the
prior commit. (default)
Sample
^^^^^^
::
$ shipyard get renderedconfigdocs
data:
chart_groups: [kubernetes-proxy, container-networking, dns, kubernetes, kubernetes-rbac]
release_prefix: ucp
id: 1
metadata:
layeringDefinition: {abstract: false, layer: site}
name: cluster-bootstrap-1
schema: metadata/Document/v1.0
storagePolicy: cleartext
schema: armada/Manifest/v1.0
status: {bucket: coll1, revision: 1}
get workflows
~~~~~~~~~~~~~
Retrieve workflows that are running or have run in the workflow engine.
This includes processses that may not have been started as an action
(e.g. scheduled tasks).
::
shipyard get workflows
[--since=<date>]
Example:
shipyard get workflows
shipyard get workflows --since=2017-01-01T12:34:56.123456
\--since=<date>
The historical cutoff date to limit the results of of this response.
Sample
^^^^^^
::
$ shipyard get workflows
Workflows State
deploy_site__2017-11-27T20:34:33.000000 failed
update_site__2017-11-27T20:45:47.000000 running
Help Commands
-------------
help
~~~~
Provides topical help for shipyard.
.. note::
--help will provide more specific command help.
::
shipyard help
[<topic>]
Example:
shipyard help configdocs
<topic>
The topic of the help to be displayed. If this parameter is not
specified the list of available topics will be displayed.
Sample
^^^^^^
::
$ shipyard help
THE SHIPYARD COMMAND
The base shipyard command supports options that determine cross-CLI behaviors.
FORMAT
shipyard [--context-marker=<uuid>] [--os_{various}=<value>]
[--debug/--no-debug] [--output-format] <subcommands>
Please Note: --os_auth_url is required for every command except shipyard help
<topic>.
TOPICS
For information of the following topics, run shipyard help <topic>
actions
configdocs

View File

@ -0,0 +1,2 @@
[general]
smart_quotes=no

View File

@ -32,3 +32,6 @@ Shipyard Configuration Guide
sampleconf
policy-enforcement
API
API_action_commands
CLI