From fde16f218eea62a473001049ca9422aa432e204f Mon Sep 17 00:00:00 2001 From: Bryan Strassner Date: Fri, 1 Dec 2017 16:01:45 -0600 Subject: [PATCH] Move most Shipyard docs to .rst This patchset moves most of the documentation for shipyard from markdown to reStructuredText. README.md remains as it is nice for Github consumption and serves as a gateway to the rest of the generated documentation. Change-Id: I8cec38fe0e30e9a545cb6e79ae6de1934f7dfc4e --- .gitignore | 3 +- docs/API.md | 755 ----------------------- docs/API_action_commands.md | 49 -- docs/CLI.md | 644 -------------------- docs/README.md | 80 +-- docs/source/API.rst | 894 ++++++++++++++++++++++++++++ docs/source/API_action_commands.rst | 68 +++ docs/source/CLI.rst | 666 +++++++++++++++++++++ docs/source/docutils.conf | 2 + docs/source/index.rst | 3 + 10 files changed, 1678 insertions(+), 1486 deletions(-) delete mode 100644 docs/API.md delete mode 100644 docs/API_action_commands.md delete mode 100644 docs/CLI.md create mode 100644 docs/source/API.rst create mode 100644 docs/source/API_action_commands.rst create mode 100644 docs/source/CLI.rst create mode 100644 docs/source/docutils.conf diff --git a/.gitignore b/.gitignore index c487c53e..7a17c73d 100644 --- a/.gitignore +++ b/.gitignore @@ -107,4 +107,5 @@ AUTHORS # build/lint artifacts /charts/shipyard/charts -/charts/shipyard/requirements.lock \ No newline at end of file +/charts/shipyard/requirements.lock +.DS_Store \ No newline at end of file diff --git a/docs/API.md b/docs/API.md deleted file mode 100644 index 353a9f42..00000000 --- a/docs/API.md +++ /dev/null @@ -1,755 +0,0 @@ -# Shipyard API - -Logically, the API has three parts to handle the three areas of functionality -in Shipyard. - -1. Document Staging -2. Action Handling -3. Airflow Monitoring - - -## Standards used by the API -See [UCP API conventions](https://github.com/att-comdev/ucp-integration/blob/master/docs/api-conventions.md) - - -## Notes on examples -Examples assume the following environment variables are set before issuing -the curl commands shown: -``` -$TOKEN={a valid keystone token} -$URL={the url and port of the shipyard api} -``` -* Examples will use json formatted by the jq command for sake of presentation. -* Actual responses will not formatted. -* The use of ellipsis indicate a repeated structure in the case of lists, or -prior/subsequent structure unimportant to the example (or considered -understood). -* The content-length response headers have been removed so as to not cause -confusion with the listed output. - -### Example response for an invalid token: - -``` -HTTP/1.1 401 Unauthorized -content-type: application/json -x-shipyard-req: a8194b97-8973-4b04-a3b3-2bd319024c5d -WWW-Authenticate: Keystone uri='http://keystone-api.ucp.svc.cluster.local:80/v3' - -{ - "apiVersion": "v1.0", - "status": "Failure", - "metadata": {}, - "message": "Unauthenticated", - "code": "401 Unauthorized", - "details": { - "errorList": [ - { - "message": "Credentials are not established" - } - ], - "errorCount": 1, - "errorType": "ApiError" - }, - "kind": "status", - "reason": "Credentials are not established" -} -``` - - ---- -## Document Staging API ->Shipyard will serve as the entrypoint for documents (designs, secrets, -configurations, etc...) into a site. Documents are posted to Shipyard in -collections, rather than individually. At any point in time, there will be two -represented versions of documents in a site that are accessible via this API: -> ->* The "Committed Documents" version, which represents the last version of -documents that were successfully commited with a commit_configdocs action. ->* The "Shipyard Buffer" version, which represents the collection of documents -that have been ingested by this API since the last commited version. Note that -only one set of documents maybe posted to the buffer at a time by default. -(This behavior can be overridden by query parameters issued by the user of -Shipyard) -> -> All versions of documents rely upon Deckhand for storage. Shipyard uses the -tagging features of Deckhand of to find the appropriate Committed Documents -and Shipyard Buffer version. - ---- -### /v1.0/configdocs/{collection_id} -Represents the site configuration documents - -#### Entity Structure -The documents as noted above (commonly yaml), in a format understood by -Deckhand - - -#### POST /v1.0/configdocs/{collection_id} -Ingests a collection of documents. Synchronous. POSTing an empty body -indicates that the specified collection should be deleted when the Shipyard -Buffer is committed. If a POST to the commitconfigdocs is in progress, this -POST should be rejected with a 409 error. - -Important: -The expected input type for this request is 'Content-Type: application/x-yaml' - -##### Query Parameters -* bufferMode=append|replace|**rejectOnContents** -Indicates how the existing Shipyard Buffer should be handled. By default, -Shipyard will reject the POST if contents already exist in the Shipyard -Buffer. - * append: Add the collection to the Shipyard Buffer, only if that - collection doesn't already exist in the Shipyard Buffer. If the collection - is already present, the request will be rejected and a 409 Conflict will be - returned. - * replace: Clear the Shipyard Buffer before adding the specified collection. -##### Responses -* 201 Created -If the documents are successfully ingested, even with validation failures. -Response message includes: - * a list of validation results - * The response headers will include a Location indicating the GET endpoint - to retrieve the configDocs - -* 409 Conflict - * If a commitconfigdocs POST is in progress. - * If any collections exist in the Shipyard Buffer unless bufferMode=replace - or bufferMode=append. - * If bufferMode=append, and the collection being posted is already in the - Shipyard Buffer - -#### GET /v1.0/configdocs/{collection_id} -Returns the source documents for a collection of documents -##### Query Parameters -* version=committed|**buffer** -Return the documents for the version specified - buffer by default. - -Important: -The output type for this request is 'Content-Type: application/x-yaml' - -##### Responses -* 200 OK -If documents can be retrieved. - * If the response is 200 with an empty response body, this indicates that -the buffer version is attempting to 'delete' the collection when it is -committed. An empty response body will only be possible for version=buffer. -* 404 Not Found -If the collection is not represented - * When version=buffer, this indicates that no representations of this -collection have been POSTed since the last committed version. - * When version=committed, this indicates that either the collection has -never existed or has been deleted by a prior commit - ---- -### /v1.0/renderedconfigdocs -Represents the site configuration documents, as a whole set - does not -consider collections in any way. - -#### GET /v1.0/renderedconfigdocs -Returns the full set of configdocs in their rendered form. - -Important: -The output type for this request is 'Content-Type: application/x-yaml' - -##### Query Parameters -* version=committed|**buffer** -Return the documents for the version specified - buffer by default. -##### Responses -* 200 OK -If documents can be retrieved. - ---- -### /v1.0/commitconfigdocs -A RPC style command to trigger a commit of the configuration documents from the -Shipyard Buffer to the Committed Documents. This resource will support POST -only. -#### Entity Structure -The response will be the list of validations from all downstream systems that -perform validation during the commit process. The structure will match the -error response object described in the [UCP API conventions](https://github.com/att-comdev/ucp-integration/blob/master/docs/api-conventions.md) -and will be an aggregation of each UCP component's responses. - -#### POST /v1.0/commitconfigdocs -Performs the commit of the Shipyard Buffer to the Committed Documents. -Synchronous. This invokes each of the UCP components to examine the Shipyard -Buffer version of the configuration documents and aggregate the responses. -While performing this commit, further POSTing of configdocs, or other commits -may not be invoked (Shipyard will block those requests with a 409 response). -If there are any failures to validate, the Shipyard Buffer and Commited -Documents will remain unchanged. If successful, the Shipyard Buffer will be -cleared, and the Committed documents will be updated. - -**Note**: if there are unhandled runtime errors during the commitconfigdocs POST, -a deadlock situation may be possible. Future enhancements may improve this -handling. - -##### Query Parameters -* force=true|**false** -By default, false, if there are validation failures the POST will fail with -a 400 response. With force=true, allows for the commit to succeed (with a 200 -response) even if there are validation failures from downstream components. The -aggregate response of validation failures will be returned in this case, but -the invalid documents will still be moved from the Shipyard Buffer to the -Committed Documents. - -##### Responses -* 200 OK -If the validations are successful. Returns an "empty" structure as as response -indicating no errors. A 200 may also be returned if there are validation -failures, but the force=true query parameter was specified. In this case, the -response will contain the list of validations. -* 400 Bad Request -If the validations fail. Returns a populated response structure containing -the aggregation of the failed validations. -* 409 Conflict -If the there is a POST to commitconfigdocs in progress. ---- -## Action API ->The Shipyard Action API is a resource that allows for creation, control and -investigation of triggered workflows. These actions encapsulate a command -interface for the Undercloud Platform. -See [Action Commands](API_action_commands.md) for supported actions - ---- -### /v1.0/actions - -#### Entity Structure -A list of actions that have been executed through shipyard's action API. -``` -[ - { Action objects summarized, See below}, - ... -] -``` - -#### GET /v1.0/actions -Returns the list of actions in the system that have been posted, and are -accessible to the current user. - -##### Responses -* 200 OK -If the actions can be retrieved. - -##### Example -``` -$ curl -X GET $URL/api/v1.0/actions -H "X-Auth-Token:$TOKEN" - -HTTP/1.1 200 OK -x-shipyard-req: 0804d13e-08fc-4e60-a819-3b7532cac4ec -content-type: application/json; charset=UTF-8 - -[ - { - "dag_status": "failed", - "parameters": {}, - "steps": [ - { - "id": "action_xcom", - "url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/action_xcom", - "index": 1, - "state": "success" - }, - { - "id": "dag_concurrency_check", - "url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/dag_concurrency_check", - "index": 2, - "state": "success" - }, - { - "id": "preflight", - "url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/preflight", - "index": 3, - "state": "failed" - }, - ... - ], - "action_lifecycle": "Failed", - "dag_execution_date": "2017-09-23T02:42:12", - "id": "01BTP9T2WCE1PAJR2DWYXG805V", - "dag_id": "deploy_site", - "datetime": "2017-09-23 02:42:06.860597+00:00", - "user": "shipyard", - "context_marker": "416dec4b-82f9-4339-8886-3a0c4982aec3", - "name": "deploy_site" - }, - ... -] -``` - -#### POST /v1.0/actions -Creates an action in the system. This will cause some action to start. -The input body to this post will represent an action object that has at least -these fields: -* name -The name of the action to invoke, as noted in -[Action Commands](API_action_commands.md) -* parameters -A dictionary of parameters to use for the trigger invocation. The supported -parameters will vary for the action invoked. -``` -{ - "name" : "action name", - "parameters" : { varies by action } -} -``` - -The POST will synchronously create the action (a shell object that represents -a DAG invocation), perform any checks to validate the preconditions to run the -DAG, and trigger the invocation of the DAG. The DAG will run asynchronously in -airflow. -##### Responses -* 201 Created -If the action is created successfully, and all preconditions to run the DAG -are successful. The response body is the action entity created. -* 400 Bad Request -If the action name doesn't exist, or the input entity is otherwise malformed. -* 409 Conflict -For any failed pre-run validations. The response body is the action entity -created, with the failed validations. The DAG will not begin execution in this -case. - -##### Example -``` -$ curl -D - -d '{"name":"deploy_site"}' -X POST $URL/api/v1.0/actions \ - -H "X-Auth-Token:$TOKEN" -H "content-type:application/json" - -HTTP/1.1 201 Created -location: {$URL}/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN -x-shipyard-req: 629f2ea2-c59d-46b9-8641-7367a91a7016 -content-type: application/json; charset=UTF-8 - -{ - "dag_status": "SCHEDULED", - "parameters": {}, - "dag_execution_date": "2017-09-24T19:05:49", - "id": "01BTTMFVDKZFRJM80FGD7J1AKN", - "dag_id": "deploy_site", - "name": "deploy_site", - "user": "shipyard", - "context_marker": "629f2ea2-c59d-46b9-8641-7367a91a7016", - "timestamp": "2017-09-24 19:05:43.603591" -} -``` ---- -### /v1.0/actions/{action_id} -Each action will be assigned an unique id that can be used to get details for -the action, including the execution status. - -#### Entity Structure -All actions will include fields that indicate the following data: -
-
action_lifecycle
-
A summarized value indicating the status or lifecycle phase of the - action.
- - -
command audit
-
A list of commands that have been issued against the action. Initially, - the action listed will be "invoke", but may include "pause", "unpause", - or "stop" if those commands are issued.
-
context_marker
-
The user supplied or system assigned context marker associated with the - action
-
dag_execution_date
-
The execution date assigned by the workflow system during action - creation.
-
dag_status
-
Represents the status that airflow provides for an executing DAG.
-
datetime
-
The time at which the action was invoked.
-
id
-
The identifier for the action, a 26 character ULID assigned during the -creation of the action.
-
name
-
The name of the action, e.g.: deploy_site.
-
parameters
-
The parameters configuring the action that were supplied by the -user during action creation.
-
steps
-
The list of steps for the action, including the status for that - step.
-
user
-
The user who has invoked this action, as acquired from the authorization - token.
-
validations
-
A list of validations that have been done, including any status - information for those validations. During the lifecycle of the action, - this list of validations may continue to grow.
- - -#### GET /v1.0/actions/{action_id} -Returns the action entity for the specified id. -##### Responses -* 200 OK - -##### Example -``` -$ curl -D - -X GET $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN \ - -H "X-Auth-Token:$TOKEN" - -HTTP/1.1 200 OK -x-shipyard-req: eb3eacb3-4206-40df-bd91-2a3a6d81cd02 -content-type: application/json; charset=UTF-8 - -{ - "name": "deploy_site", - "dag_execution_date": "2017-09-24T19:05:49", - "validations": [], - "id": "01BTTMFVDKZFRJM80FGD7J1AKN", - "dag_id": "deploy_site", - "command_audit": [ - { - "id": "01BTTMG16R9H3Z4JVQNBMRV1MZ", - "action_id": "01BTTMFVDKZFRJM80FGD7J1AKN", - "datetime": "2017-09-24 19:05:49.530223+00:00", - "user": "shipyard", - "command": "invoke" - } - ], - "user": "shipyard", - "context_marker": "629f2ea2-c59d-46b9-8641-7367a91a7016", - "datetime": "2017-09-24 19:05:43.603591+00:00", - "dag_status": "failed", - "parameters": {}, - "steps": [ - { - "id": "action_xcom", - "url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/action_xcom", - "index": 1, - "state": "success" - }, - { - "id": "dag_concurrency_check", - "url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/dag_concurrency_check", - "index": 2, - "state": "success" - }, - { - "id": "preflight", - "url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/preflight", - "index": 3, - "state": "failed" - }, - { - "id": "deckhand_get_design_version", - "url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/deckhand_get_design_version", - "index": 4, - "state": null - }, - ... - ], - "action_lifecycle": "Failed" -} - -``` - ---- -### /v1.0/actions/{action_id}/validationdetails/{validation_id} -Allows for drilldown to validation detailed info. - -#### Entity Structure -The detailed information for a validation -``` -{ TBD } -``` - -#### GET /v1.0/actions/{action_id}/validationdetails/{validation_id} -Returns the validation detail by Id for the supplied action Id. -##### Responses -* 200 OK - ---- -### /v1.0/actions/{action_id}/steps/{step_id} -Allow for drilldown to step information. The step information includes details -of the steps excution, successful or not, and enough to facilitate -troubleshooting in as easy a fashion as possible. - -#### Entity Structure -A step entity represents detailed information representing a single step of -execution as part of an action. Not all fields are necessarily represented in -every returned entity. - -
-
dag_id
-
The name/id of the workflow DAG that contains this step.
-
duration
-
The duration (seconds) for the step.
-
end_date
-
The timestamp of the completion of the step.
-
execution_date
-
The execution date of the workflow that contains this step.
-
index
-
The ordinal value representing the position of this step in the sequence - of steps associated with this step.
-
operator
-
The name of the processing facility used by the workflow system.
-
queued_dttm
-
The timestamp when the step was enqueued by the workflow system.
-
start_date
-
The timestamp for the beginning of execution for this step.
-
state
-
The execution state of the step.
-
task_id
-
The name of the task used by the workflow system (and also representing - this step name queried in the reqeust.
-
try_number
-
A number of retries taken in the case of failure. Some workflow steps may - be configured to retry before considering the step truly failed.
-
- -#### GET /v1.0/actions/{action_id}/steps/{step_id} -Returns the details for a step by id for the given action by Id. -##### Responses -* 200 OK - -##### Example -``` -$ curl -D - \ - -X GET $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/action_xcom \ - -H "X-Auth-Token:$TOKEN" - -HTTP/1.1 200 OK -x-shipyard-req: 72daca4d-1f79-4e08-825f-2ad181912a47 -content-type: application/json; charset=UTF-8 - -{ - "end_date": "2017-09-24 19:05:59.446213", - "duration": 0.165181, - "queued_dttm": "2017-09-24 19:05:52.993983", - "operator": "PythonOperator", - "try_number": 1, - "task_id": "action_xcom", - "state": "success", - "execution_date": "2017-09-24 19:05:49", - "dag_id": "deploy_site", - "index": 1, - "start_date": "2017-09-24 19:05:59.281032" -} - -``` ---- -### /v1.0/actions/{action_id}/control/{control_verb} -Allows for issuing DAG controls against an action. - -#### Entity Structure -None, there is no associated response entity for this resource - -#### POST /v1.0/actions/{action_id}/{control_verb} -Trigger a control action against an activity.- this includes: pause, unpause -##### Responses -* 202 Accepted -##### Example -Failure case - command is invalid for the execution state of the action. -``` -$ curl -D - \ - -X POST $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/control/pause \ - -H "X-Auth-Token:$TOKEN" - -HTTP/1.1 409 Conflict -content-type: application/json -x-shipyard-req: 9c9551e0-335c-4297-af93-8440cc6b324f - -{ - "apiVersion": "v1.0", - "status": "Failure", - "metadata": {}, - "message": "Unable to pause action", - "code": "409 Conflict", - "details": { - "errorList": [ - { - "message": "dag_run state must be running, but is failed" - } - ], - "errorCount": 1, - "errorType": "ApiError" - }, - "kind": "status", - "reason": "dag_run state must be running, but is failed" -} -``` - -Success case -``` -$ curl -D - \ - -X POST $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/control/pause \ - -H "X-Auth-Token:$TOKEN" - -HTTP/1.1 202 Accepted -content-length: 0 -x-shipyard-req: 019fae1c-03b0-4af1-b57d-451ae6ddac77 -content-type: application/json; charset=UTF-8 -``` - - - ---- -## Airflow Monitoring API ->Airflow has a primary function of scheduling DAGs, as opposed to Shipyard's -primary case of triggering DAGs. -Shipyard provides functionality to allow for an operator to monitor and review -these scheduled workflows (DAGs) in addition to the ones triggered by Shipyard. -This API will allow for accessing Airflow DAGs of any type -- providing a peek -into the totality of what is happening in Airflow. - ---- -### /v1.0/workflows -The resource that represents DAGs (workflows) in airflow - -#### Entity Structure -A list of objects representing the DAGs that have run in airflow. - -#### GET /v1.0/workflows -Queries airflow for DAGs that are running or have run (successfully or -unsuccessfully) and provides a summary of those things. -##### Query parameters -* since={iso8601 date (past) or duration} -optional, a boundary in the past within which to retrieve results. Default is -30 days in the past. -##### Responses -* 200 OK - -##### Example - -Note the workflow_id values, these can be used for drilldown. - -``` -curl -D - -X GET $URL/api/v1.0/workflows -H "X-Auth-Token:$TOKEN" - -HTTP/1.1 200 OK -content-type: application/json; charset=UTF-8 -x-shipyard-req: 3ab4ccc6-b956-4c7a-9ae6-183c562d8297 - -[ - { - "execution_date": "2017-10-09 21:18:56", - "end_date": null, - "workflow_id": "deploy_site__2017-10-09T21:18:56.000000", - "start_date": "2017-10-09 21:18:56.685999", - "external_trigger": true, - "dag_id": "deploy_site", - "state": "failed", - "run_id": "manual__2017-10-09T21:18:56" - }, - { - "execution_date": "2017-10-09 21:19:03", - "end_date": null, - "workflow_id": "deploy_site__2017-10-09T21:19:03.000000", - "start_date": "2017-10-09 21:19:03.361522", - "external_trigger": true, - "dag_id": "deploy_site", - "state": "failed", - "run_id": "manual__2017-10-09T21:19:03" - } - ... -] -``` - ---- -### /v1.0/workflows/{workflow_id} - -#### Entity Structure -An object representing the information available from airflow regarding a DAG's -execution - -#### GET /v1.0/workflows/{id} -Further details of a particular workflow's steps. All steps of all sub-dags -will be included in the list of steps, as well as section indicating the -sub-dags for this parent workflow. -##### Responses -* 200 OK -##### Example - -Note that sub_dags can be queried to restrict to only that sub-dag's steps. -e.g. using this as {workflow_id}: -deploy_site.preflight.armada_preflight_check__2017-10-09T21:19:03.000000 - - -``` -curl -D - \ - -X GET $URL/api/v1.0/workflows/deploy_site__2017-10-09T21:19:03.000000 \ - -H "X-Auth-Token:$TOKEN" - -HTTP/1.1 200 OK -content-type: application/json; charset=UTF-8 -x-shipyard-req: 98d71530-816a-4692-9df2-68f22c057467 - -{ - "execution_date": "2017-10-09 21:19:03", - "end_date": null, - "workflow_id": "deploy_site__2017-10-09T21:19:03.000000", - "start_date": "2017-10-09 21:19:03.361522", - "external_trigger": true, - "steps": [ - { - "end_date": "2017-10-09 21:19:14.916220", - "task_id": "action_xcom", - "start_date": "2017-10-09 21:19:14.798053", - "duration": 0.118167, - "queued_dttm": "2017-10-09 21:19:08.432582", - "try_number": 1, - "state": "success", - "operator": "PythonOperator", - "dag_id": "deploy_site", - "execution_date": "2017-10-09 21:19:03" - }, - { - "end_date": "2017-10-09 21:19:25.283785", - "task_id": "dag_concurrency_check", - "start_date": "2017-10-09 21:19:25.181492", - "duration": 0.102293, - "queued_dttm": "2017-10-09 21:19:19.283132", - "try_number": 1, - "state": "success", - "operator": "ConcurrencyCheckOperator", - "dag_id": "deploy_site", - "execution_date": "2017-10-09 21:19:03" - }, - { - "end_date": "2017-10-09 21:20:05.394677", - "task_id": "preflight", - "start_date": "2017-10-09 21:19:34.994775", - "duration": 30.399902, - "queued_dttm": "2017-10-09 21:19:28.449848", - "try_number": 1, - "state": "failed", - "operator": "SubDagOperator", - "dag_id": "deploy_site", - "execution_date": "2017-10-09 21:19:03" - }, - ... - ], - "dag_id": "deploy_site", - "state": "failed", - "run_id": "manual__2017-10-09T21:19:03", - "sub_dags": [ - { - "execution_date": "2017-10-09 21:19:03", - "end_date": null, - "workflow_id": "deploy_site.preflight__2017-10-09T21:19:03.000000", - "start_date": "2017-10-09 21:19:35.082479", - "external_trigger": false, - "dag_id": "deploy_site.preflight", - "state": "failed", - "run_id": "backfill_2017-10-09T21:19:03" - }, - ..., - { - "execution_date": "2017-10-09 21:19:03", - "end_date": null, - "workflow_id": "deploy_site.preflight.armada_preflight_check__2017-10-09T21:19:03.000000", - "start_date": "2017-10-09 21:19:48.265023", - "external_trigger": false, - "dag_id": "deploy_site.preflight.armada_preflight_check", - "state": "failed", - "run_id": "backfill_2017-10-09T21:19:03" - } - ] -} -``` diff --git a/docs/API_action_commands.md b/docs/API_action_commands.md deleted file mode 100644 index 64a2410e..00000000 --- a/docs/API_action_commands.md +++ /dev/null @@ -1,49 +0,0 @@ -# Action API supported commands - -## Supported actions -These actions are currently supported using the [Action API](API.md#ActionAPI) - -None at this time. - ----- -## Actions under development -These actions are under active development - -* deploy_site -Triggers the initial deployment of a site, using the latest committed -configuration documents. Steps: - 1) Concurrency check - to prevent concurrent site modifications by - conflicting actions/workflows. - 2) Preflight checks - ensures all UCP components are in a responsive state. - 3) Get design version - uses Deckhand to discover the latest committed - version of design for the site. - 4) Validate design - asks each involved UCP component to validate the design. - This ensures that the design, which was previously committed is valid at the - present time. - 5) Drydock build - orchestrates the Drydock component to configure hardware - and the Kubernetes environment (Drydock -> Promenade) - 6) Check deployed node status - checks that the deployment of nodes is - successful. - 7) Armada build - orchestrates Armada to configure software on the nodes as - designed. - -* update_site -Triggers the initial deployment of a site, using the latest committed -configuration documents. Steps: (same as deploy_site) - -* redeploy_server -Using parameters to indicate which server(s), triggers a redeployment of server -to the last known good design and secrets - ---- -## Future actions -These actions are anticipated for development -* test region -Invoke site validation testing - perhaps baseline is a invocation of all -components regular "component" tests. This test would be used as a -preflight-style test to ensure all components are in a working state. - -* test component -Invoke a particular platform component to test it. This test would be used to -interrogate a particular platform component to ensure it is in a working state, -and that its own downstream dependencies are also operational diff --git a/docs/CLI.md b/docs/CLI.md deleted file mode 100644 index f008107b..00000000 --- a/docs/CLI.md +++ /dev/null @@ -1,644 +0,0 @@ -# Supported Environment Variables - -All commands will utilize the following environment variables to determine -necessary information for execution, unless otherwise noted. - -
-
Openstack Keystone Authorization environment variables
-
- The Shipyard CLI/API Client will check for the presence of appropriate - environment setup to do authentication on behalf of the user. - The openrc variables that will be used are as follows:
- OS_PROJECT_DOMAIN_NAME ("default" if not specified)
- OS_USER_DOMAIN_NAME ("default" if not specified)
- OS_PROJECT_NAME
- OS_USERNAME
- OS_PASSWORD
- OS_AUTH_URL - The fully qualified identity endpoint. - E.g. http://keystone.ucp.fully.qualified.name:80/v3
-
-
Openstack Keystone Authorization environment variables not used
-
- OS_IDENTITY_API_VERSION -- this value will be ignored as Shipyard only - supports version 3 at this time
-
-
- -# Shipyard command options -The base shipyard command supports options that determine cross-CLI behaviors. -These options are positionally immediately following the shipyard command as -shown here: -``` -shipyard <--these options> subcommands... - -shipyard - [--context-marker=] - [--debug/--no-debug] - [--os_{various}=] - [--output-format=[format | raw | cli]] (default = cli) - -``` -
-
--context-marker=<uuid>
-
- Specifies a UUID (8-4-4-4-12 format) that will be used to correlate logs, - transactions, etc... in downstream activities triggered by this - interaction. If not specified, Shipyard will supply a new UUID to serve - as this marker. (optional) -
-
--debug | --no-debug (default)
-
- Enable/disable debugging of this CLI and API client. -
-
--os_{various}=<value>
-
- See supported Openstack Keystone Authorization Environment variables above - for the list of supported names, converting to a downcase version of the - environment variable.
- E.g.: --os_auth_url=http://keystone.ucp:80/v3
- If not specified, the environment variables matching these actions will be - used instead. The Keystone os_auth_url should reference the exposed - keystone:port for the target Shipyard environment, as this Keystone will - be used to discover the instance of Shipyard. For most invocations other - than help, a valid combination of values must be resolved to authenticate - and authorize the user's invocation. -
-
--output-format=[format | raw | cli] (default = cli)
-
- Specifies the desired output formating such that:
- format: Display the raw ouptut from the invoked Shipyard API in a - column restricted mode.
- raw: Display the result from the invoked Shipyard API as-is, without - modification.
- cli: (default) Display results in a plain text interpretation of the - response from the invoked Shipyard API. -
-
- -# Commit Commands -## commit configdocs -Attempts to commit the Shipyard Buffer documents, first invoking validation -by downstream components. -``` -shipyard commit configdocs - [--force] - -Example: - shipyard commit configdocs -``` -
-
--force
-
- Force the commit to occur, even if validations fail. Note that while this - allows for bypassing validations, it does not bypass a failure to - communicate with the systems that provide the validations. -
-
- -### Sample -``` -TBD -``` - ---- - -# Control commands -## pause, unpause, stop -Three separate commands with a common format that allow for controlling -the processing of actions created in Shipyard. -
-
pause
-
pause something in progress e.g. an executing action
-
unpause
-
unpause something paused e.g. a paused action
-
stop
-
stops an executing or paused item e.g. an action
-
- -``` -shipyard pause - - - -shipyard unpause - - - -shipyard stop - - - -shipyard - pause|unpause|stop - - -Example: - - shipyard pause action 01BTG32JW87G0YKA1K29TKNAFX - - shipyard unpause action 01BTG32JW87G0YKA1K29TKNAFX - - shipyard stop action 01BTG32JW87G0YKA1K29TKNAFX - - shipyard pause action/01BTG32JW87G0YKA1K29TKNAFX -``` -
-
<type>
-
- The type of entity to take action upon. Currently supports: action -
-
<id>
-
- The id of the entity to take action upon. -
-
<qualified name>
-
- The qualified name of the item to take the specified action upon -
-
- -### Sample -``` -$ shipyard pause action/01BZZMEXAVYGG7BT0BMA3RHYY7 -pause successfully submitted for action 01BZZMEXAVYGG7BT0BMA3RHYY7 -``` - -A failed command: -``` -$ shipyard pause action/01BZZK07NF04XPC5F4SCTHNPKN -Error: Unable to pause action -Reason: dag_run state must be running, but is failed -- Error: dag_run state must be running, but is failed -``` - ---- - -# Create Commands -## create action -Invokes the specified workflow through Shipyard. -See [Action Commands](API_action_commands.md) for supported actions -Returns the id of the action invoked so that it can be queried subsequently. -``` -shipyard create action - - --param= (repeatable) - -Example: - shipyard create action redeploy_server --param="server-name=mcp" -``` -
-
<action command>
-
- The action to invoke. -
-
--param=<parameter>
-
- A parameter to be provided to the action being invoked. (repeatable) -
-
- -### Sample -``` -$ shipyard create action deploy_site -Name Action Lifecycle -deploy_site action/01BZZK07NF04XPC5F4SCTHNPKN None -``` - ---- - -## create configdocs -Load documents into the Shipyard Buffer. The use of one or more filename or -a single directory option must be specified. - -``` -shipyard create configdocs - - [--append | --replace] - --filename= (repeatable) - | - --directory= - -Example: - shipyard create configdocs design --append --filename=site_design.yaml -``` -Note: If neither append or replace are specified, the Shipyard API default -value of rejectoncontents will be used. - -Note: Either --filename or --directory must be specified, but both may not be -specified for the same invocation of shipyard. -
-
<collection>
-
- The collection to load. -
-
--append
-
- Add the collection to the Shipyard Buffer. This will fail if the - collection already exists. -
-
--replace
-
- Clear the shipyard buffer and replace it with the specified contents. -
-
--filename=<filename>
-
- The file name to use as the contents of the collection. (repeatable) - If any documents specified fail basic validation, all of the documents - will be rejected. Use of filename parameters may not be used in - conjunction with the directory parameter. -
-
--directory=<directory>
-
- A directory containing documents that will be joined and loaded as a - collection. Any documents that fail basic validation will reject the - whole set. Use of the directory parameter may not be used with the filename - parameter. -
-
- -### Sample -``` -$ shipyard create configdocs coll1 --filename=/home/ubuntu/yaml/coll1.yaml -Configuration documents added. -Status: Validations succeeded -Reason: Validation -``` -Attempting to load the same collection into the uncommitted buffer. -``` -$ shipyard create configdocs coll1 --filename=/home/ubuntu/yaml/coll1.yaml -Error: Invalid collection specified for buffer -Reason: Buffermode : rejectoncontents -- Error: Buffer is either not empty or the collection already exists in buffer. Setting a different buffermode may provide the desired functionality -``` -Replace the buffer with --replace -``` -$ shipyard create configdocs coll1 --replace --filename=/home/ubuntu/yaml/coll1.yaml -Configuration documents added. -Status: Validations succeeded -Reason: Validation -``` - ---- - -# Describe Commands -## describe -Retrieves the detailed information about the supplied namespaced item -``` -shipyard describe - - -Example: - shipyard describe action/01BTG32JW87G0YKA1K29TKNAFX - Equivalent to: - shipyard describe action 01BTG32JW87G0YKA1K29TKNAFX - - shipyard describe step/01BTG32JW87G0YKA1K29TKNAFX/preflight - Equivalent to: - shipyard describe step preflight --action=01BTG32JW87G0YKA1K29TKNAFX - - shipyard describe validation/01BTG32JW87G0YKA1K29TKNAFX/01BTG3PKBS15KCKFZ56XXXBGF2 - Equivalent to: - shipyard describe validation 01BTG3PKBS15KCKFZ56XXXBGF2 \ - --action=01BTG32JW87G0YKA1K29TKNAFX - - shipyard describe workflow/deploy_site__2017-01-01T12:34:56.123456 - Equivalent to: - shipyard describe workflow deploy_site__2017-01-01T12:34:56.123456 -``` - ---- - -## describe action -Retrieves the detailed information about the supplied action id. -``` -shipyard describe action - - -Example: - shipyard describe action 01BTG32JW87G0YKA1K29TKNAFX -``` - -### Sample -``` -$ shipyard describe action/01BZZK07NF04XPC5F4SCTHNPKN -Name: deploy_site -Action: action/01BZZK07NF04XPC5F4SCTHNPKN -Lifecycle: Failed -Parameters: {} -Datetime: 2017-11-27 20:34:24.610604+00:00 -Dag Status: failed -Context Marker: 71d4112e-8b6d-44e8-9617-d9587231ffba -User: shipyard - -Steps Index State -step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom 1 success -step/01BZZK07NF04XPC5F4SCTHNPKN/dag_concurrency_check 2 success -step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 3 failed -step/01BZZK07NF04XPC5F4SCTHNPKN/validate_site_design 4 None -step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 5 failed -step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 6 failed -step/01BZZK07NF04XPC5F4SCTHNPKN/drydock_build 7 None - -Commands User Datetime -invoke shipyard 2017-11-27 20:34:34.443053+00:00 - -Validations: None -``` - ---- - -## describe step -Retrieves the step details associated with an action and step. -``` -shipyard describe step - - --action= - -Example: - shipyard describe step preflight --action=01BTG32JW87G0YKA1K29TKNAFX -``` -
-
<step id>
-
- The id of the step found in the describe action response. -
-
--action=<action id>
-
- The action id that provides the context for this step. -
-
- -### Sample -``` -$ shipyard describe step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom -Name: action_xcom -Task ID: step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom -Index: 1 -State: success -Start Date: 2017-11-27 20:34:45.604109 -End Date: 2017-11-27 20:34:45.818946 -Duration: 0.214837 -Try Number: 1 -Operator: PythonOperator -``` - ---- - -## describe validation -Retrieves the validation details associated with an action and validation id -``` -shipyard describe validation - - --action= - -Example: - shipyard describe validation 01BTG3PKBS15KCKFZ56XXXBGF2 \ - --action=01BTG32JW87G0YKA1K29TKNAFX -``` -
-
<validation id>
-
- The id of the validation found in the describe action response. -
-
--action=<action id>
-
- The action id that provides the context for this validation. -
-
- -### Sample -``` -TBD -``` - ---- - -## describe workflow -Retrieves the details for a workflow that is running or has run in the workflow -engine. -``` -shipyard describe workflow - - -Example: - shipyard describe workflow deploy_site__2017-01-01T12:34:56.123456 -``` -
-
<workflow id>
-
- The id of the workflow found in the get workflows response. -
-
- -### Sample -``` -$ shipyard describe workflow deploy_site__2017-11-27T20:34:33.000000 -Workflow: deploy_site__2017-11-27T20:34:33.000000 -State: failed -Dag ID: deploy_site -Execution Date: 2017-11-27 20:34:33 -Start Date: 2017-11-27 20:34:33.979594 -End Date: None -External Trigger: True - -Steps State -action_xcom success -dag_concurrency_check success -deckhand_get_design_version failed -validate_site_design None -deckhand_get_design_version failed -deckhand_get_design_version failed -drydock_build None - -Subworkflows: -Workflow: deploy_site.deckhand_get_design_version__2017-11-27T20:34:33.000000 -State: failed -Dag ID: deploy_site.deckhand_get_design_version -Execution Date: 2017-11-27 20:34:33 -Start Date: 2017-11-27 20:35:06.281825 -End Date: None -External Trigger: False - -Workflow: deploy_site.deckhand_get_design_version.deckhand_get_design_version__2017-11-27T20:34:33.000000 -State: failed -Dag ID: deploy_site.deckhand_get_design_version.deckhand_get_design_version -Execution Date: 2017-11-27 20:34:33 -Start Date: 2017-11-27 20:35:20.725506 -End Date: None -External Trigger: False -``` - ---- - -# Get Commands -## get actions -Lists the actions that have been invoked. -``` -shipyard get actions -``` - -### Sample -``` -$ shipyard get actions -Name Action Lifecycle -deploy_site action/01BZZK07NF04XPC5F4SCTHNPKN Failed -update_site action/01BZZKMW60DV2CJZ858QZ93HRS Processing -``` - ---- - -## get configdocs -Retrieve documents loaded into Shipyard, either committed or from the -Shipyard Buffer. -``` -shipyard get configdocs - - [--committed | --buffer] - -Example: - shipyard get configdocs design -``` -
-
<collection>
-
- The collection to retrieve for viewing. -
-
--committed
-
- Retrieve the documents that have last been committed for this - collection -
-
--buffer
-
- Retrive the documents that have been loaded into Shipyard - since the prior commit. If no documents have been loaded - into the buffer for this collection, this will return an empty - response (default) -
-
- -### Sample -``` -$ shipyard get configdocs coll1 -data: - chart_groups: [kubernetes-proxy, container-networking, dns, kubernetes, kubernetes-rbac] - release_prefix: ucp -id: 1 -metadata: - layeringDefinition: {abstract: false, layer: site} - name: cluster-bootstrap-1 - schema: metadata/Document/v1.0 - storagePolicy: cleartext -schema: armada/Manifest/v1.0 -status: {bucket: coll1, revision: 1} -``` - ---- - -## get renderedconfigdocs -Retrieve the rendered version of documents loaded into Shipyard. Rendered -documents are the "final" version of the documents after applying Deckhand -layering and substitution. -``` -shipyard get renderedconfigdocs - [--committed | --buffer] - -Example: - shipyard get renderedconfigdocs -``` -
-
--committed
-
- Retrieve the documents that have last been committed. -
-
--buffer
-
- Retrieve the documents that have been loaded into Shipyard - since the prior commit. (default) -
-
- -### Sample -``` -$ shipyard get renderedconfigdocs -data: - chart_groups: [kubernetes-proxy, container-networking, dns, kubernetes, kubernetes-rbac] - release_prefix: ucp -id: 1 -metadata: - layeringDefinition: {abstract: false, layer: site} - name: cluster-bootstrap-1 - schema: metadata/Document/v1.0 - storagePolicy: cleartext -schema: armada/Manifest/v1.0 -status: {bucket: coll1, revision: 1} -``` - ---- - -## get workflows -Retrieve workflows that are running or have run in the workflow engine. This -includes processses that may not have been started as an action -(e.g. scheduled tasks). -``` -shipyard get workflows - [--since=] - -Example: - shipyard get workflows - - shipyard get workflows --since=2017-01-01T12:34:56.123456 -``` -
-
--since=<date>
-
- The historical cutoff date to limit the results of of this response. -
-
- -### Sample -``` -$ shipyard get workflows -Workflows State -deploy_site__2017-11-27T20:34:33.000000 failed -update_site__2017-11-27T20:45:47.000000 running -``` - ---- - -# help commands -Provides topical help for shipyard. Note that --help will provide more -specific command help. -``` -shipyard help - [] - -Example: - shipyard help configdocs -``` -
-
<topic>
-
- The topic of the help to be displayed. If this parameter is not specified - the list of avaialable topics will be displayed. -
-
- -### Sample -``` -$ shipyard help -THE SHIPYARD COMMAND -The base shipyard command supports options that determine cross-CLI behaviors. - -FORMAT -shipyard [--context-marker=] [--os_{various}=] - [--debug/--no-debug] [--output-format] - -Please Note: --os_auth_url is required for every command except shipyard help - . - -TOPICS -For information of the following topics, run shipyard help - actions - configdocs -``` \ No newline at end of file diff --git a/docs/README.md b/docs/README.md index 811779a5..59b9a3e2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,49 +1,55 @@ -## Shipyard ## +## Shipyard -Shipyard adopts the Falcon web framework and uses Apache Airflow as the backend engine to programmatically -author, schedule and monitor workflows. +Shipyard adopts the Falcon web framework and uses Apache Airflow as the backend +engine to programmatically author, schedule and monitor workflows. The current workflow is as follows: -1. Inital region/site data will be passed to Shipyard from either a human operator or Jenkins -2. The data (in YAML format) will be sent to [DeckHand](https://github.com/att-comdev/deckhand) for processing -3. Shipyard will make use of the post-processed data from DeckHand to interact with [DryDock](https://github.com/att-comdev/drydock) -4. DryDock will interact with [Promenade](https://github.com/att-comdev/promenade) to provision and deploy - bare metal nodes using Ubuntu MAAS and a resilient Kubernetes cluster will be created at the end of the - process -5. Once the Kubernetes clusters are up and validated to be working properly, Shipyard will interact with - [Armada](https://github.com/att-comdev/armada) to deploy OpenStack using [OpenStack Helm](https://github.com/openstack/openstack-helm) -6. Once the OpenStack cluster is deployed, Shipyard will trigger a workflow to perform basic sanity health - checks on the cluster +1. Inital region/site data will be passed to Shipyard from either a human +operator or Jenkins +2. The data (in YAML format) will be sent to + [DeckHand](https://github.com/att-comdev/deckhand) for validation and + storage +3. Shipyard will make use of the post-processed data from DeckHand to interact + with [DryDock](https://github.com/att-comdev/drydock) +4. DryDock will interact with + [Promenade](https://github.com/att-comdev/promenade) to provision and deploy + bare metal nodes using Ubuntu MAAS and a resilient Kubernetes cluster will + be created at the end of the process +5. Once the Kubernetes clusters are up and validated to be working properly, + Shipyard will interact with [Armada](https://github.com/att-comdev/armada) + to deploy OpenStack using + [OpenStack Helm](https://github.com/openstack/openstack-helm) +6. Once the OpenStack cluster is deployed, Shipyard will trigger a workflow to + perform basic sanity health checks on the cluster + +Note: This project, along with the tools used within are community-based and +open sourced. + +### Mission + +The goal for Shipyard is to provide a customizable *framework* for operators +and developers alike. This framework will enable end-users to orchestrate and +deploy a fully functional container-based Cloud. -Note: This project, along with the tools used within are community-based and open sourced. +### Roadmap +The detailed Roadmap can be viewed on the +[LCOO JIRA](https://openstack-lcoo.atlassian.net/projects/SHIPYARD/issues/) -### Mission ### - -The goal for Shipyard is to provide a customizable *framework* for operators and developers alike. This -framework will enable end-users to orchestrate and deploy a fully functional container-based Cloud - - -### Roadmap ### - -The detailed Roadmap can be viewed on the [LCOO JIRA](https://openstack-lcoo.atlassian.net/projects/SHIPYARD/issues/) - -- Create Helm Charts for Airflow -- Create Helm Charts for Shipyard -- Create Falcon API Framework - Integrate with DeckHand, DryDock/Promenade, Armada +- Expand funcitionality to provide visibility into and options for operation of + a UCP installation + +### Getting Started + +This project is under development at the moment. We encourage anyone who is +interested in Shipyard to review our +[documentation](http://shipyard.readthedocs.io/en/latest/) -### Getting Started ### - -This project is under development at the moment. We encourage anyone who is interested in Shipyard to review -our [Installation](https://github.com/att-comdev/shipyard/blob/master/docs/deployment_guide.md) documentation - - -### Bugs ### - -Bugs are traced in [LCOO JIRA](https://openstack-lcoo.atlassian.net/projects/SHIPYARD/issues/). If you find -a bug, please feel free to create a [GitHub issue](https://github.com/att-comdev/shipyard_issues) and it will be synced to JIRA. +### Bugs +If you find a bug, please feel free to create a +[GitHub issue](https://github.com/att-comdev/shipyard/issues) diff --git a/docs/source/API.rst b/docs/source/API.rst new file mode 100644 index 00000000..d75286c4 --- /dev/null +++ b/docs/source/API.rst @@ -0,0 +1,894 @@ +.. _shipyard_api: + +Shipyard API +============ +Logically, the API has three parts to handle the three areas of +functionality in Shipyard. + +1. Document Staging +2. Action Handling +3. Airflow Monitoring + +Standards used by the API +------------------------- +See `UCP API +conventions `__ + +Notes on examples +----------------- +Examples assume the following environment variables are set before +issuing the curl commands shown: + +:: + + $TOKEN={a valid keystone token} + $URL={the url and port of the shipyard api} + +- Examples will use json formatted by the jq command for sake of + presentation. +- Actual responses will not formatted. +- The use of ellipsis indicate a repeated structure in the case of + lists, or prior/subsequent structure unimportant to the example (or + considered understood). +- The content-length response headers have been removed so as to not + cause confusion with the listed output. + +Example response for an invalid token: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + HTTP/1.1 401 Unauthorized + content-type: application/json + x-shipyard-req: a8194b97-8973-4b04-a3b3-2bd319024c5d + WWW-Authenticate: Keystone uri='http://keystone-api.ucp.svc.cluster.local:80/v3' + + { + "apiVersion": "v1.0", + "status": "Failure", + "metadata": {}, + "message": "Unauthenticated", + "code": "401 Unauthorized", + "details": { + "errorList": [ + { + "message": "Credentials are not established" + } + ], + "errorCount": 1, + "errorType": "ApiError" + }, + "kind": "status", + "reason": "Credentials are not established" + } + +Document Staging API +-------------------- +Shipyard will serve as the entrypoint for documents (designs, secrets, +configurations, etc...) into a site. Documents are posted to Shipyard in +collections, rather than individually. At any point in time, there will be two +represented versions of documents in a site that are accessible via this API: + +- The "Committed Documents" version, which represents the last version of + documents that were successfully commited with a commit_configdocs action. +- The "Shipyard Buffer" version, which represents the collection of documents + that have been ingested by this API since the last commited version. Note + that only one set of documents maybe posted to the buffer at a time by + default. (This behavior can be overridden by query parameters issued by the + user of Shipyard) + +All versions of documents rely upon Deckhand for storage. Shipyard uses the +tagging features of Deckhand of to find the appropriate Committed Documents +and Shipyard Buffer version. + +/v1.0/configdocs/{collection_id} +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Represents the site configuration documents + +Entity Structure +^^^^^^^^^^^^^^^^ +The documents as noted above (commonly yaml), in a format understood by +Deckhand + +POST /v1.0/configdocs/{collection_id} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Ingests a collection of documents. Synchronous. POSTing an empty body +indicates that the specified collection should be deleted when the +Shipyard Buffer is committed. If a POST to the commitconfigdocs is in +progress, this POST should be rejected with a 409 error. + +.. note:: + + The expected input type for this request is ‘Content-Type: application/x-yaml’ + + +Query Parameters +'''''''''''''''' + +- bufferMode=append|replace\|\ **rejectOnContents** + Indicates how the existing Shipyard Buffer should be handled. By default, + Shipyard will reject the POST if contents already exist in the Shipyard + Buffer. + + - append: Add the collection to the Shipyard Buffer, only if that + collection doesn’t already exist in the Shipyard Buffer. If the + collection is already present, the request will be rejected and a 409 + Conflict will be returned. + - replace: Clear the Shipyard Buffer before adding the specified + collection. + +Responses +''''''''' +201 Created + If the documents are successfully ingested, even with validation failures. + Response message includes: + + - a list of validation results + - The response headers will include a Location indicating the GET + endpoint to retrieve the configDocs + +409 Conflict + A condition in the system is blocking this document ingestion + + - If a commitconfigdocs POST is in progress. + - If any collections exist in the Shipyard Buffer unless bufferMode=replace + or bufferMode=append. + - If bufferMode=append, and the collection being posted is already in the + Shipyard Buffer + +GET /v1.0/configdocs/{collection_id} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Returns the source documents for a collection of documents + +.. note:: + + The output type for this request is ‘Content-Type: application/x-yaml’ + +Query Parameters +'''''''''''''''' +version=committed | **buffer** + Return the documents for the version specified - buffer by default. + +Responses +''''''''' +200 OK + If documents can be retrieved. + + - If the response is 200 with an empty response body, this indicates + that the buffer version is attempting to ‘delete’ the collection + when it is committed. An empty response body will only be possible + for version=buffer. + +404 Not Found + If the collection is not represented + + - When version=buffer, this indicates that no representations of this + collection have been POSTed since the last committed version. + - When version=committed, this indicates that either the collection has + never existed or has been deleted by a prior commit. + +/v1.0/renderedconfigdocs +~~~~~~~~~~~~~~~~~~~~~~~~ +Represents the site configuration documents, as a whole set - does not +consider collections in any way. + +GET /v1.0/renderedconfigdocs +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Returns the full set of configdocs in their rendered form. + +.. note:: + + The output type for this request is 'Content-Type: application/x-yaml' + +Query Parameters +'''''''''''''''' +version=committed|**buffer** + Return the documents for the version specified - buffer by default. + +Responses +''''''''' +200 OK + If documents can be retrieved. + + +/v1.0/commitconfigdocs +~~~~~~~~~~~~~~~~~~~~~~ +An RPC style command to trigger a commit of the configuration documents from +the Shipyard Buffer to the Committed Documents. This resource will support POST +only. + +Entity Structure +^^^^^^^^^^^^^^^^ +The response will be the list of validations from all downstream systems that +perform validation during the commit process. The structure will match the +error response object described in the `UCP API +conventions `__ +and will be an aggregation of each UCP component’s responses. + +POST /v1.0/commitconfigdocs +^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Synchronous. Performs the commit of the Shipyard Buffer to the Committed +Documents. This invokes each of the UCP components to examine the Shipyard +Buffer version of the configuration documents and aggregate the responses. +While performing this commit, further POSTing of configdocs, or other commits +may not be invoked (Shipyard will block those requests with a 409 response). If +there are any failures to validate, the Shipyard Buffer and Commited Documents +will remain unchanged. If successful, the Shipyard Buffer will be cleared, and +the Committed documents will be updated. + +.. note:: + + If there are unhandled runtime errors during the commitconfigdocs POST, a + deadlock situation may be possible. Future enhancements may improve this + handling. + +Query Parameters +'''''''''''''''' +force=true | **false** + By default, false, if there are validation failures the POST will + fail with a 400 response. With force=true, allows for the commit to + succeed (with a 200 response) even if there are validation failures + from downstream components. The aggregate response of validation + failures will be returned in this case, but the invalid documents + will still be moved from the Shipyard Buffer to the Committed + Documents. + +Responses +''''''''' +200 OK + If the validations are successful. Returns an “empty” structure as as + response indicating no errors. A 200 may also be returned if there + are validation failures, but the force=true query parameter was + specified. In this case, the response will contain the list of + validations. +400 Bad Request + If the validations fail. Returns a populated response structure + containing the aggregation of the failed validations. +409 Conflict + If the there is a POST to commitconfigdocs in progress. + +Example +''''''' + +:: + + { + "apiVersion": "v1", + "code": "400 Bad Request", + "details": { + "errorCount": 2, + "messageList": [ + { + "error": true, + "message": "Error loading effective site: 'NoneType' object is not iterable", + "name": "Drydock" + }, + { + "error": true, + "message": "Armada unable to validate configdocs", + "name": "Armada" + } + ] + }, + "kind": "Status", + "message": "Validations failed", + "metadata": {}, + "reason": "Validation", + "status": "Invalid" + } + +Action API +---------- +The Shipyard Action API is a resource that allows for creation, control and +investigation of triggered workflows. These actions encapsulate a command +interface for the Undercloud Platform. See :ref:`shipyard_action_commands` for +supported actions + +/v1.0/actions +~~~~~~~~~~~~~ + +Entity Structure +^^^^^^^^^^^^^^^^ +A list of actions that have been executed through shipyard's action API. + +:: + + [ + { Action objects summarized, See below}, + ... + ] + + +GET /v1.0/actions +^^^^^^^^^^^^^^^^^ +Returns the list of actions in the system that have been posted, and are +accessible to the current user. + +Responses +''''''''' +200 OK + If the actions can be retrieved. + +Example +''''''' + +:: + + $ curl -X GET $URL/api/v1.0/actions -H "X-Auth-Token:$TOKEN" + + HTTP/1.1 200 OK + x-shipyard-req: 0804d13e-08fc-4e60-a819-3b7532cac4ec + content-type: application/json; charset=UTF-8 + + [ + { + "dag_status": "failed", + "parameters": {}, + "steps": [ + { + "id": "action_xcom", + "url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/action_xcom", + "index": 1, + "state": "success" + }, + { + "id": "dag_concurrency_check", + "url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/dag_concurrency_check", + "index": 2, + "state": "success" + }, + { + "id": "preflight", + "url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/preflight", + "index": 3, + "state": "failed" + }, + ... + ], + "action_lifecycle": "Failed", + "dag_execution_date": "2017-09-23T02:42:12", + "id": "01BTP9T2WCE1PAJR2DWYXG805V", + "dag_id": "deploy_site", + "datetime": "2017-09-23 02:42:06.860597+00:00", + "user": "shipyard", + "context_marker": "416dec4b-82f9-4339-8886-3a0c4982aec3", + "name": "deploy_site" + }, + ... + ] + +POST /v1.0/actions +^^^^^^^^^^^^^^^^^^ +Creates an action in the system. This will cause some action to start. The +input body to this post will represent an action object that has at least these +fields: + +name + The name of the action to invoke, as noted in :ref:`shipyard_action_commands` + +parameters + A dictionary of parameters to use for the trigger invocation. The supported + parameters will vary for the action invoked. + + :: + + { + "name" : "action name", + "parameters" : { varies by action } + } + +The POST will synchronously create the action (a shell object that represents +a DAG invocation), perform any checks to validate the preconditions to run the +DAG, and trigger the invocation of the DAG. The DAG will run asynchronously in +airflow. + +Responses +''''''''' +201 Created + If the action is created successfully, and all preconditions to run the DAG + are successful. The response body is the action entity created. +400 Bad Request + If the action name doesn't exist, or the input entity is otherwise malformed. +409 Conflict + For any failed pre-run validations. The response body is the action entity + created, with the failed validations. The DAG will not begin execution in + this case. + +Example +''''''' + +:: + + $ curl -D - -d '{"name":"deploy_site"}' -X POST $URL/api/v1.0/actions \ + -H "X-Auth-Token:$TOKEN" -H "content-type:application/json" + + HTTP/1.1 201 Created + location: {$URL}/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN + x-shipyard-req: 629f2ea2-c59d-46b9-8641-7367a91a7016 + content-type: application/json; charset=UTF-8 + + { + "dag_status": "SCHEDULED", + "parameters": {}, + "dag_execution_date": "2017-09-24T19:05:49", + "id": "01BTTMFVDKZFRJM80FGD7J1AKN", + "dag_id": "deploy_site", + "name": "deploy_site", + "user": "shipyard", + "context_marker": "629f2ea2-c59d-46b9-8641-7367a91a7016", + "timestamp": "2017-09-24 19:05:43.603591" + } + +/v1.0/actions/{action_id} +~~~~~~~~~~~~~~~~~~~~~~~~~ +Each action will be assigned an unique id that can be used to get +details for the action, including the execution status. + +Entity Structure +^^^^^^^^^^^^^^^^ +All actions will include fields that indicate the following data: + +action_lifecycle + A summarized value indicating the status or lifecycle phase of the action. + + - Pending - The action is scheduled or preparing for execution. + - Processing - The action is underway. + - Complete - The action has completed successfully. + - Failed - The action has encountered an error, and has failed. + - Paused - The action has been paused by a user. + +command audit + A list of commands that have been issued against the action. Initially, + the action listed will be “invoke”, but may include “pause”, “unpause”, + or “stop” if those commands are issued. + +context_marker + The user supplied or system assigned context marker associated with the + action + +dag_execution_date + The execution date assigned by the workflow system during action + creation. + +dag_status + Represents the status that airflow provides for an executing DAG. + +datetime + The time at which the action was invoked. + +id + The identifier for the action, a 26 character ULID assigned during the + creation of the action. + +name + The name of the action, e.g.: deploy_site. + +parameters + The parameters configuring the action that were supplied by the user + during action creation. + +steps + The list of steps for the action, including the status for that step. + +user + The user who has invoked this action, as acquired from the authorization + token. + +validations + A list of validations that have been done, including any status + information for those validations. During the lifecycle of the action, + this list of validations may continue to grow. + +GET /v1.0/actions/{action_id} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Returns the action entity for the specified id. + +Responses +''''''''' +200 OK + +Example +''''''' + +:: + + $ curl -D - -X GET $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN \ + -H "X-Auth-Token:$TOKEN" + + HTTP/1.1 200 OK + x-shipyard-req: eb3eacb3-4206-40df-bd91-2a3a6d81cd02 + content-type: application/json; charset=UTF-8 + + { + "name": "deploy_site", + "dag_execution_date": "2017-09-24T19:05:49", + "validations": [], + "id": "01BTTMFVDKZFRJM80FGD7J1AKN", + "dag_id": "deploy_site", + "command_audit": [ + { + "id": "01BTTMG16R9H3Z4JVQNBMRV1MZ", + "action_id": "01BTTMFVDKZFRJM80FGD7J1AKN", + "datetime": "2017-09-24 19:05:49.530223+00:00", + "user": "shipyard", + "command": "invoke" + } + ], + "user": "shipyard", + "context_marker": "629f2ea2-c59d-46b9-8641-7367a91a7016", + "datetime": "2017-09-24 19:05:43.603591+00:00", + "dag_status": "failed", + "parameters": {}, + "steps": [ + { + "id": "action_xcom", + "url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/action_xcom", + "index": 1, + "state": "success" + }, + { + "id": "dag_concurrency_check", + "url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/dag_concurrency_check", + "index": 2, + "state": "success" + }, + { + "id": "preflight", + "url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/preflight", + "index": 3, + "state": "failed" + }, + { + "id": "deckhand_get_design_version", + "url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/deckhand_get_design_version", + "index": 4, + "state": null + }, + ... + ], + "action_lifecycle": "Failed" + } + +/v1.0/actions/{action_id}/validationdetails/{validation_id} +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Allows for drilldown to validation detailed info. + +Entity Structure +^^^^^^^^^^^^^^^^ +The detailed information for a validation + +:: + + { TBD } + +GET /v1.0/actions/{action_id}/validationdetails/{validation_id} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Returns the validation detail by Id for the supplied action Id. + +Responses +''''''''' +200 OK + +/v1.0/actions/{action_id}/steps/{step_id} +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Allow for drilldown to step information. The step information includes +details of the steps excution, successful or not, and enough to +facilitate troubleshooting in as easy a fashion as possible. + +Entity Structure +^^^^^^^^^^^^^^^^ +A step entity represents detailed information representing a single step +of execution as part of an action. Not all fields are necessarily +represented in every returned entity. + +dag_id + The name/id of the workflow DAG that contains this step. + +duration + The duration (seconds) for the step. + +end_date + The timestamp of the completion of the step. + +execution_date + The execution date of the workflow that contains this step. + +index + The numeric value representing the position of this step in the sequence + of steps associated with this step. + +operator + The name of the processing facility used by the workflow system. + +queued_dttm + The timestamp when the step was enqueued by the workflow system. + +start_date + The timestamp for the beginning of execution for this step. + +state + The execution state of the step. + +task_id + The name of the task used by the workflow system (and also representing + this step name queried in the reqeust. + +try_number + A number of retries taken in the case of failure. Some workflow steps + may be configured to retry before considering the step truly failed. + + +GET /v1.0/actions/{action_id}/steps/{step_id} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Returns the details for a step by id for the given action by Id. ##### + +Responses +''''''''' +200 OK + +Example +''''''' + +:: + + $ curl -D - \ + -X GET $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/action_xcom \ + -H "X-Auth-Token:$TOKEN" + + HTTP/1.1 200 OK + x-shipyard-req: 72daca4d-1f79-4e08-825f-2ad181912a47 + content-type: application/json; charset=UTF-8 + + { + "end_date": "2017-09-24 19:05:59.446213", + "duration": 0.165181, + "queued_dttm": "2017-09-24 19:05:52.993983", + "operator": "PythonOperator", + "try_number": 1, + "task_id": "action_xcom", + "state": "success", + "execution_date": "2017-09-24 19:05:49", + "dag_id": "deploy_site", + "index": 1, + "start_date": "2017-09-24 19:05:59.281032" + } + +/v1.0/actions/{action_id}/control/{control_verb} +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Allows for issuing DAG controls against an action. + +Entity Structure +^^^^^^^^^^^^^^^^ +None, there is no associated response entity for this resource + +POST /v1.0/actions/{action_id}/{control_verb} +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Trigger a control action against an activity.- this includes: pause, unpause + +Responses +''''''''' +202 Accepted + +Example +''''''' +Failure case - command is invalid for the execution state of the action. + +:: + + $ curl -D - \ + -X POST $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/control/pause \ + -H "X-Auth-Token:$TOKEN" + + HTTP/1.1 409 Conflict + content-type: application/json + x-shipyard-req: 9c9551e0-335c-4297-af93-8440cc6b324f + + { + "apiVersion": "v1.0", + "status": "Failure", + "metadata": {}, + "message": "Unable to pause action", + "code": "409 Conflict", + "details": { + "errorList": [ + { + "message": "dag_run state must be running, but is failed" + } + ], + "errorCount": 1, + "errorType": "ApiError" + }, + "kind": "status", + "reason": "dag_run state must be running, but is failed" + } + +Success case + +:: + + $ curl -D - \ + -X POST $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/control/pause \ + -H "X-Auth-Token:$TOKEN" + + HTTP/1.1 202 Accepted + content-length: 0 + x-shipyard-req: 019fae1c-03b0-4af1-b57d-451ae6ddac77 + content-type: application/json; charset=UTF-8 + + +Airflow Monitoring API +---------------------- +Airflow has a primary function of scheduling DAGs, as opposed to Shipyard’s +primary case of triggering DAGs. Shipyard provides functionality to allow for +an operator to monitor and review these scheduled workflows (DAGs) in addition +to the ones triggered by Shipyard. This API will allow for accessing Airflow +DAGs of any type – providing a peek into the totality of what is happening in +Airflow. + +/v1.0/workflows +~~~~~~~~~~~~~~~ +The resource that represents DAGs (workflows) in airflow + +Entity Structure +^^^^^^^^^^^^^^^^ +A list of objects representing the DAGs that have run in airflow. + +GET /v1.0/workflows +^^^^^^^^^^^^^^^^^^^ +Queries airflow for DAGs that are running or have run (successfully or +unsuccessfully) and provides a summary of those things. + +Query parameters +'''''''''''''''' +since={iso8601 date (past) or duration} + optional, a boundary in the past within which to retrieve results. Default is + 30 days in the past. + +Responses +''''''''' +200 OK + +Example +''''''' +Notice the workflow_id values, these can be used for drilldown. + +:: + + curl -D - -X GET $URL/api/v1.0/workflows -H "X-Auth-Token:$TOKEN" + + HTTP/1.1 200 OK + content-type: application/json; charset=UTF-8 + x-shipyard-req: 3ab4ccc6-b956-4c7a-9ae6-183c562d8297 + + [ + { + "execution_date": "2017-10-09 21:18:56", + "end_date": null, + "workflow_id": "deploy_site__2017-10-09T21:18:56.000000", + "start_date": "2017-10-09 21:18:56.685999", + "external_trigger": true, + "dag_id": "deploy_site", + "state": "failed", + "run_id": "manual__2017-10-09T21:18:56" + }, + { + "execution_date": "2017-10-09 21:19:03", + "end_date": null, + "workflow_id": "deploy_site__2017-10-09T21:19:03.000000", + "start_date": "2017-10-09 21:19:03.361522", + "external_trigger": true, + "dag_id": "deploy_site", + "state": "failed", + "run_id": "manual__2017-10-09T21:19:03" + } + ... + ] + +/v1.0/workflows/{workflow_id} +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Entity Structure +^^^^^^^^^^^^^^^^ +An object representing the information available from airflow regarding +a DAG’s execution + +GET /v1.0/workflows/{id} +^^^^^^^^^^^^^^^^^^^^^^^^ +Further details of a particular workflow’s steps. All steps of all +sub-dags will be included in the list of steps, as well as section +indicating the sub-dags for this parent workflow. + +Responses +''''''''' +200 OK + +Example +''''''' +.. note:: + + Sub_dags can be queried to restrict to only that sub-dag’s steps. e.g. using + this as {workflow_id}: + deploy_site.preflight.armada_preflight_check__2017-10-09T21:19:03.000000 + +:: + + curl -D - \ + -X GET $URL/api/v1.0/workflows/deploy_site__2017-10-09T21:19:03.000000 \ + -H "X-Auth-Token:$TOKEN" + + HTTP/1.1 200 OK + content-type: application/json; charset=UTF-8 + x-shipyard-req: 98d71530-816a-4692-9df2-68f22c057467 + + { + "execution_date": "2017-10-09 21:19:03", + "end_date": null, + "workflow_id": "deploy_site__2017-10-09T21:19:03.000000", + "start_date": "2017-10-09 21:19:03.361522", + "external_trigger": true, + "steps": [ + { + "end_date": "2017-10-09 21:19:14.916220", + "task_id": "action_xcom", + "start_date": "2017-10-09 21:19:14.798053", + "duration": 0.118167, + "queued_dttm": "2017-10-09 21:19:08.432582", + "try_number": 1, + "state": "success", + "operator": "PythonOperator", + "dag_id": "deploy_site", + "execution_date": "2017-10-09 21:19:03" + }, + { + "end_date": "2017-10-09 21:19:25.283785", + "task_id": "dag_concurrency_check", + "start_date": "2017-10-09 21:19:25.181492", + "duration": 0.102293, + "queued_dttm": "2017-10-09 21:19:19.283132", + "try_number": 1, + "state": "success", + "operator": "ConcurrencyCheckOperator", + "dag_id": "deploy_site", + "execution_date": "2017-10-09 21:19:03" + }, + { + "end_date": "2017-10-09 21:20:05.394677", + "task_id": "preflight", + "start_date": "2017-10-09 21:19:34.994775", + "duration": 30.399902, + "queued_dttm": "2017-10-09 21:19:28.449848", + "try_number": 1, + "state": "failed", + "operator": "SubDagOperator", + "dag_id": "deploy_site", + "execution_date": "2017-10-09 21:19:03" + }, + ... + ], + "dag_id": "deploy_site", + "state": "failed", + "run_id": "manual__2017-10-09T21:19:03", + "sub_dags": [ + { + "execution_date": "2017-10-09 21:19:03", + "end_date": null, + "workflow_id": "deploy_site.preflight__2017-10-09T21:19:03.000000", + "start_date": "2017-10-09 21:19:35.082479", + "external_trigger": false, + "dag_id": "deploy_site.preflight", + "state": "failed", + "run_id": "backfill_2017-10-09T21:19:03" + }, + ..., + { + "execution_date": "2017-10-09 21:19:03", + "end_date": null, + "workflow_id": "deploy_site.preflight.armada_preflight_check__2017-10-09T21:19:03.000000", + "start_date": "2017-10-09 21:19:48.265023", + "external_trigger": false, + "dag_id": "deploy_site.preflight.armada_preflight_check", + "state": "failed", + "run_id": "backfill_2017-10-09T21:19:03" + } + ] + } diff --git a/docs/source/API_action_commands.rst b/docs/source/API_action_commands.rst new file mode 100644 index 00000000..1faaaffc --- /dev/null +++ b/docs/source/API_action_commands.rst @@ -0,0 +1,68 @@ +.. _shipyard_action_commands: + +Action Commands +=============== + +Supported actions +----------------- + +These actions are currently supported using the Action API + +deploy_site +~~~~~~~~~~~ + +Triggers the initial deployment of a site, using the latest committed +configuration documents. Steps: + +#. Concurrency check + Prevents concurrent site modifications by conflicting + actions/workflows. +#. Preflight checks + Ensures all UCP components are in a responsive state. +#. Get design version + Uses Deckhand to discover the latest committed version of design for + the site. +#. Validate design + Asks each involved UCP component to validate the design. This ensures + that the previously committed design is valid at the present time. +#. Drydock build + Orchestrates the Drydock component to configure hardware and the + Kubernetes environment (Drydock -> Promenade) +#. Check deployed node status + Checks that the deployment of nodes is successful. +#. Armada build + Orchestrates Armada to configure software on the nodes as designed. + + +Actions under development +~~~~~~~~~~~~~~~~~~~~~~~~~ + +These actions are under active development + +- update_site + + Triggers the initial deployment of a site, using the latest committed + configuration documents. Steps: (a superset of deploy_site) + +- redeploy_server + + Using parameters to indicate which server(s), triggers a redeployment of + server to the last known good design and secrets + +Future actions +~~~~~~~~~~~~~~ + +These actions are anticipated for development + +- test region + + Invoke site validation testing - perhaps baseline is a invocation of all + components regular “component” tests. This test would be used as a + preflight-style test to ensure all components are in a working state. + +- test component + + Invoke a particular platform component to test it. This test would be + used to interrogate a particular platform component to ensure it is in a + working state, and that its own downstream dependencies are also + operational diff --git a/docs/source/CLI.rst b/docs/source/CLI.rst new file mode 100644 index 00000000..c76291fe --- /dev/null +++ b/docs/source/CLI.rst @@ -0,0 +1,666 @@ +.. _shipyard_cli: + +Shipyard CLI +============ + +Environment Variables +--------------------- +All commands will utilize the following environment variables to +determine necessary information for execution, unless otherwise noted. + +Openstack Keystone Authorization environment variables +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The Shipyard CLI/API Client will check for the presence of appropriate +environment setup to do authentication on behalf of the user. The openrc +variables that will be used are as follows: + +- OS_PROJECT_DOMAIN_NAME ("default" if not specified) +- OS_USER_DOMAIN_NAME ("default" if not specified) +- OS_PROJECT_NAME +- OS_USERNAME +- OS_PASSWORD +- OS_AUTH_URL The fully qualified identity endpoint. E.g. http://keystone.ucp.fully.qualified.name:80/v3 + +Openstack Keystone Authorization environment variables *not* used +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +These openstack identity variables are not supported by shipyard. + +- OS_IDENTITY_API_VERSION + This value will be ignored as Shipyard only supports version 3 at this time + +Shipyard command options +------------------------ +The base shipyard command supports options that determine cross-CLI behaviors. +These options are positionally immediately following the shipyard command as +shown here: + +:: + + shipyard <--these options> subcommands... + + shipyard + [--context-marker=] + [--debug/--no-debug] + [--os-{various}=] + [--output-format=[format | raw | cli]] (default = cli) + + + +\--context-marker= + Specifies a UUID (8-4-4-4-12 format) that will be used to correlate logs, + transactions, etc... in downstream activities triggered by this interaction. + If not specified, Shipyard will supply a new UUID to serve as this marker. + (optional) + +\--debug | --no-debug + Enable/disable debugging of this CLI and API client. Defaults to no debug + +\--os-= + See supported Openstack Keystone Authorization Environment variables above + for the list of supported names, converting to a downcase version of the + environment variable. E.g.: --os-auth-url=http://keystone.ucp:80/v3 + If not specified, the environment variables matching these options will be + used instead. The Keystone os-auth-url should reference the exposed + keystone:port for the target Shipyard environment, as this Keystone will be + used to discover the instance of Shipyard. For most invocations other than + help, a valid combination of values must be resolved to authenticate and + authorize the user's invocation. + +\--output-format= + Specifies the desired output formating such that: + + - format + Display the raw ouptut from the invoked Shipyard API in a column + restricted mode. + - raw + Display the result from the invoked Shipyard API as-is, without + modification. + - cli (default) + Display results in a plain text interpretation of the response from the + invoked Shipyard API. + +Commit Commands +--------------- + +commit configdocs +~~~~~~~~~~~~~~~~~ +Attempts to commit the Shipyard Buffer documents, first invoking validation by +downstream components. + +:: + + shipyard commit configdocs + [--force] + + Example: + shipyard commit configdocs + +\--force + Force the commit to occur, even if validations fail. + +Sample +^^^^^^ + +:: + + TBD + +Control commands +---------------- + +pause, unpause, stop +~~~~~~~~~~~~~~~~~~~~ + +Three separate commands with a common format that allow for controlling +the processing of actions created in Shipyard. + +pause + pause something in progress e.g. an executing action +unpause + unpause something paused e.g. a paused action +stop + stops an executing or paused item e.g. an action + +:: + + shipyard pause + + + + shipyard unpause + + + + shipyard stop + + + + shipyard + pause|unpause|stop + + + Example: + + shipyard pause action 01BTG32JW87G0YKA1K29TKNAFX + + shipyard unpause action 01BTG32JW87G0YKA1K29TKNAFX + + shipyard stop action 01BTG32JW87G0YKA1K29TKNAFX + + shipyard pause action/01BTG32JW87G0YKA1K29TKNAFX + + + The type of entity to take action upon. Currently supports: action + + The id of the entity to take action upon. + + The qualified name of the item to take the specified action upon + +Sample +^^^^^^ + +:: + + $ shipyard pause action/01BZZMEXAVYGG7BT0BMA3RHYY7 + pause successfully submitted for action 01BZZMEXAVYGG7BT0BMA3RHYY7 + +A failed command: + +:: + + $ shipyard pause action/01BZZK07NF04XPC5F4SCTHNPKN + Error: Unable to pause action + Reason: dag_run state must be running, but is failed + - Error: dag_run state must be running, but is failed + +Create Commands +--------------- + +create action +~~~~~~~~~~~~~ + +Invokes the specified workflow through Shipyard. Returns the +id of the action invoked so that it can be queried subsequently. + +:: + + shipyard create action + + --param= (repeatable) + + Example: + shipyard create action redeploy_server --param="server-name=mcp" + + + The action to invoke. + +\--param= + A parameter to be provided to the action being invoked. (repeatable) + +Sample +^^^^^^ + +:: + + $ shipyard create action deploy_site + Name Action Lifecycle + deploy_site action/01BZZK07NF04XPC5F4SCTHNPKN None + + +create configdocs +~~~~~~~~~~~~~~~~~ +Load documents into the Shipyard Buffer. The use of one or more filename +or a single directory option must be specified. + +:: + + shipyard create configdocs + + [--append | --replace] + --filename= (repeatable) + | + --directory= + + Example: + shipyard create configdocs design --append --filename=site_design.yaml + +.. note:: + + If neither append nor replace are specified, the Shipyard API default value + of rejectoncontents will be used. + +.. note:: + + Either --filename or --directory must be specified, but both may not be + specified for the same invocation of shipyard. + + + The collection to load. + +\--append + Add the collection to the Shipyard Buffer. This will fail if the collection + already exists. + +\--replace + Clear the shipyard buffer and replace it with the specified contents. + +\--filename= + The file name to use as the contents of the collection. (repeatable) If + any documents specified fail basic validation, all of the documents will + be rejected. Use of filename parameters may not be used in conjunction + with the directory parameter. + +\--directory= + A directory containing documents that will be joined and loaded as a + collection. Any documents that fail basic validation will reject the + whole set. Use of the directory parameter may not be used with the + filename parameter. + +Sample +^^^^^^ + +:: + + $ shipyard create configdocs coll1 --filename=/home/ubuntu/yaml/coll1.yaml + Configuration documents added. + Status: Validations succeeded + Reason: Validation + +Attempting to load the same collection into the uncommitted buffer. + +:: + + $ shipyard create configdocs coll1 --filename=/home/ubuntu/yaml/coll1.yaml + Error: Invalid collection specified for buffer + Reason: Buffermode : rejectoncontents + - Error: Buffer is either not empty or the collection already exists in buffer. Setting a different buffermode may provide the desired functionality + +Replace the buffer with --replace + +:: + + $ shipyard create configdocs coll1 --replace --filename=/home/ubuntu/yaml/coll1.yaml + Configuration documents added. + Status: Validations succeeded + Reason: Validation + +Describe Commands +----------------- + +describe +~~~~~~~~ + +Retrieves the detailed information about the supplied namespaced item + +:: + + shipyard describe + + + Example: + shipyard describe action/01BTG32JW87G0YKA1K29TKNAFX + Equivalent to: + shipyard describe action 01BTG32JW87G0YKA1K29TKNAFX + + shipyard describe step/01BTG32JW87G0YKA1K29TKNAFX/preflight + Equivalent to: + shipyard describe step preflight --action=01BTG32JW87G0YKA1K29TKNAFX + + shipyard describe validation/01BTG32JW87G0YKA1K29TKNAFX/01BTG3PKBS15KCKFZ56XXXBGF2 + Equivalent to: + shipyard describe validation 01BTG3PKBS15KCKFZ56XXXBGF2 \ + --action=01BTG32JW87G0YKA1K29TKNAFX + + shipyard describe workflow/deploy_site__2017-01-01T12:34:56.123456 + Equivalent to: + shipyard describe workflow deploy_site__2017-01-01T12:34:56.123456 + + +describe action +~~~~~~~~~~~~~~~ + +Retrieves the detailed information about the supplied action id. + +:: + + shipyard describe action + + + Example: + shipyard describe action 01BTG32JW87G0YKA1K29TKNAFX + +Sample +^^^^^^ + + +:: + + $ shipyard describe action/01BZZK07NF04XPC5F4SCTHNPKN + Name: deploy_site + Action: action/01BZZK07NF04XPC5F4SCTHNPKN + Lifecycle: Failed + Parameters: {} + Datetime: 2017-11-27 20:34:24.610604+00:00 + Dag Status: failed + Context Marker: 71d4112e-8b6d-44e8-9617-d9587231ffba + User: shipyard + + Steps Index State + step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom 1 success + step/01BZZK07NF04XPC5F4SCTHNPKN/dag_concurrency_check 2 success + step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 3 failed + step/01BZZK07NF04XPC5F4SCTHNPKN/validate_site_design 4 None + step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 5 failed + step/01BZZK07NF04XPC5F4SCTHNPKN/deckhand_get_design_version 6 failed + step/01BZZK07NF04XPC5F4SCTHNPKN/drydock_build 7 None + + Commands User Datetime + invoke shipyard 2017-11-27 20:34:34.443053+00:00 + + Validations: None + + +describe step +~~~~~~~~~~~~~ +Retrieves the step details associated with an action and step. + +:: + + shipyard describe step + + --action= + + Example: + shipyard describe step preflight --action=01BTG32JW87G0YKA1K29TKNAFX + + + The id of the step found in the describe action response. + +\--action= + The action id that provides the context for this step. + +Sample +^^^^^^ + + +:: + + $ shipyard describe step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom + Name: action_xcom + Task ID: step/01BZZK07NF04XPC5F4SCTHNPKN/action_xcom + Index: 1 + State: success + Start Date: 2017-11-27 20:34:45.604109 + End Date: 2017-11-27 20:34:45.818946 + Duration: 0.214837 + Try Number: 1 + Operator: PythonOperator + +describe validation +~~~~~~~~~~~~~~~~~~~ + +Retrieves the validation details associated with an action and +validation id + +:: + + shipyard describe validation + + --action= + + Example: + shipyard describe validation 01BTG3PKBS15KCKFZ56XXXBGF2 \ + --action=01BTG32JW87G0YKA1K29TKNAFX + + + The id of the validation found in the describe action response. + +\--action= + The action id that provides the context for this validation. + +Sample +^^^^^^ + + +:: + + TBD + +describe workflow +~~~~~~~~~~~~~~~~~ + +Retrieves the details for a workflow that is running or has run in the +workflow engine. + +:: + + shipyard describe workflow + + + Example: + shipyard describe workflow deploy_site__2017-01-01T12:34:56.123456 + + + The id of the workflow found in the get workflows response. + +Sample +^^^^^^ + + +:: + + $ shipyard describe workflow deploy_site__2017-11-27T20:34:33.000000 + Workflow: deploy_site__2017-11-27T20:34:33.000000 + State: failed + Dag ID: deploy_site + Execution Date: 2017-11-27 20:34:33 + Start Date: 2017-11-27 20:34:33.979594 + End Date: None + External Trigger: True + + Steps State + action_xcom success + dag_concurrency_check success + deckhand_get_design_version failed + validate_site_design None + deckhand_get_design_version failed + deckhand_get_design_version failed + drydock_build None + + Subworkflows: + Workflow: deploy_site.deckhand_get_design_version__2017-11-27T20:34:33.000000 + State: failed + Dag ID: deploy_site.deckhand_get_design_version + Execution Date: 2017-11-27 20:34:33 + Start Date: 2017-11-27 20:35:06.281825 + End Date: None + External Trigger: False + + Workflow: deploy_site.deckhand_get_design_version.deckhand_get_design_version__2017-11-27T20:34:33.000000 + State: failed + Dag ID: deploy_site.deckhand_get_design_version.deckhand_get_design_version + Execution Date: 2017-11-27 20:34:33 + Start Date: 2017-11-27 20:35:20.725506 + End Date: None + External Trigger: False + +Get Commands +------------ + +get actions +~~~~~~~~~~~ + +Lists the actions that have been invoked. + +:: + + shipyard get actions + + +Sample +^^^^^^ + +:: + + $ shipyard get actions + Name Action Lifecycle + deploy_site action/01BZZK07NF04XPC5F4SCTHNPKN Failed + update_site action/01BZZKMW60DV2CJZ858QZ93HRS Processing + +get configdocs +~~~~~~~~~~~~~~ + +Retrieve documents loaded into Shipyard, either committed or from the +Shipyard Buffer. + +:: + + shipyard get configdocs + + [--committed | --buffer] + + Example: + shipyard get configdocs design + + + The collection to retrieve for viewing. + +\--committed + Retrieve the documents that have last been committed for this collection + +\--buffer + Retrive the documents that have been loaded into Shipyard since the + prior commit. If no documents have been loaded into the buffer for this + collection, this will return an empty response (default) + +Sample +^^^^^^ + +:: + + $ shipyard get configdocs coll1 + data: + chart_groups: [kubernetes-proxy, container-networking, dns, kubernetes, kubernetes-rbac] + release_prefix: ucp + id: 1 + metadata: + layeringDefinition: {abstract: false, layer: site} + name: cluster-bootstrap-1 + schema: metadata/Document/v1.0 + storagePolicy: cleartext + schema: armada/Manifest/v1.0 + status: {bucket: coll1, revision: 1} + +get renderedconfigdocs +~~~~~~~~~~~~~~~~~~~~~~ +Retrieve the rendered version of documents loaded into Shipyard. +Rendered documents are the "final" version of the documents after +applying Deckhand layering and substitution. + +:: + + shipyard get renderedconfigdocs + [--committed | --buffer] + + Example: + shipyard get renderedconfigdocs + +\--committed + Retrieve the documents that have last been committed. + +\--buffer + Retrieve the documents that have been loaded into Shipyard since the + prior commit. (default) + +Sample +^^^^^^ + +:: + + $ shipyard get renderedconfigdocs + data: + chart_groups: [kubernetes-proxy, container-networking, dns, kubernetes, kubernetes-rbac] + release_prefix: ucp + id: 1 + metadata: + layeringDefinition: {abstract: false, layer: site} + name: cluster-bootstrap-1 + schema: metadata/Document/v1.0 + storagePolicy: cleartext + schema: armada/Manifest/v1.0 + status: {bucket: coll1, revision: 1} + +get workflows +~~~~~~~~~~~~~ +Retrieve workflows that are running or have run in the workflow engine. +This includes processses that may not have been started as an action +(e.g. scheduled tasks). + +:: + + shipyard get workflows + [--since=] + + Example: + shipyard get workflows + + shipyard get workflows --since=2017-01-01T12:34:56.123456 + +\--since= + The historical cutoff date to limit the results of of this response. + +Sample +^^^^^^ + +:: + + $ shipyard get workflows + Workflows State + deploy_site__2017-11-27T20:34:33.000000 failed + update_site__2017-11-27T20:45:47.000000 running + + +Help Commands +------------- + +help +~~~~ +Provides topical help for shipyard. + +.. note:: + + --help will provide more specific command help. + +:: + + shipyard help + [] + + Example: + shipyard help configdocs + + + The topic of the help to be displayed. If this parameter is not + specified the list of available topics will be displayed. + +Sample +^^^^^^ + + +:: + + $ shipyard help + THE SHIPYARD COMMAND + The base shipyard command supports options that determine cross-CLI behaviors. + + FORMAT + shipyard [--context-marker=] [--os_{various}=] + [--debug/--no-debug] [--output-format] + + Please Note: --os_auth_url is required for every command except shipyard help + . + + TOPICS + For information of the following topics, run shipyard help + actions + configdocs diff --git a/docs/source/docutils.conf b/docs/source/docutils.conf new file mode 100644 index 00000000..b49cd480 --- /dev/null +++ b/docs/source/docutils.conf @@ -0,0 +1,2 @@ +[general] +smart_quotes=no \ No newline at end of file diff --git a/docs/source/index.rst b/docs/source/index.rst index 42795635..4657791b 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -32,3 +32,6 @@ Shipyard Configuration Guide sampleconf policy-enforcement + API + API_action_commands + CLI