airship
/
shipyard
Code Issues Proposed changes

Update Action API documentation 

This patchset aims to update the documentation for the action api of
shipyard to match the functionality being developed. Request and
response examples have been included for those known entities. The
representation of validations has not been finalized, and is therefore
missing from this patchset.

Other changes made include adding some notation explanation for the examples,
Duplicating the name of the API to the HTTP verb associated to make for
better readability, and some cleanup of existing verbiage to use more
precise terms.

Change-Id: Ibf80bd1f5782f346bdd628337167b5e64670d978
This commit is contained in:
Bryan Strassner 2017-09-24 15:07:52 -05:00
parent e7c27d615c
commit 25c8adbe75
2 changed files with 343 additions and 201 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

394
docs/API.md
View File

@ -11,6 +11,51 @@ in Shipyard.
## Standards used by the API
See [UCP API conventions](https://github.com/att-comdev/ucp-integration/blob/master/docs/api-conventions.md)
## Notes on examples
Examples assume the following environment variables are set before issuing
the curl commands shown:
```
$TOKEN={a valid keystone token}
$URL={the url and port of the shipyard api}
```
* Examples will use json formatted by the jq command for sake of presentation.
* Actual responses will not formatted.
* The use of ellipsis indicate a repeated structure in the case of lists, or
prior/subsequent structure unimportant to the example (or considered
understood).
* The content-length response headers have been removed so as to not cause
confusion with the listed output.
### Example response for an invalid token:
```
HTTP/1.1 401 Unauthorized
content-type: application/json
x-shipyard-req: a8194b97-8973-4b04-a3b3-2bd319024c5d
WWW-Authenticate: Keystone uri='http://keystone-api.ucp.svc.cluster.local:80/v3'
{
"apiVersion": "v1.0",
"status": "Failure",
"metadata": {},
"message": "Unauthenticated",
"code": "401 Unauthorized",
"details": {
"errorList": [
{
"message": "Credentials are not established"
}
],
"errorCount": 1,
"errorType": "ApiError"
},
"kind": "status",
"reason": "Credentials are not established"
}
```
---
## <a name="DocumentStagingAPI"></a> Document Staging API
>Shipyard will serve as the entrypoint for documents (designs, secrets,
@ -34,12 +79,12 @@ and Shipyard Buffer version.
### /v1.0/configdocs/{collection_id}
Represents the site configuration documents
#### Payload Structure
#### Entity Structure
The documents as noted above (commonly yaml), in a format understood by
Deckhand
#### POST
#### POST /v1.0/configdocs/{collection_id}
Ingests a collection of documents. Synchronous. POSTing an empty body
indicates that the specified collection should be deleted when the Shipyard
Buffer is committed. If a POST to the commitconfigdocs is in progress, this
@ -70,7 +115,7 @@ Response message includes:
* If bufferMode=append, and the collection being posted is already in the
Shipyard Buffer
#### GET
#### GET /v1.0/configdocs/{collection_id}
Returns the source documents for a collection of documents
##### Query Parameters
* version=committed|**buffer**
@ -94,7 +139,7 @@ never existed or has been deleted by a prior commit
Represents the site configuration documents, as a whole set - does not
consider collections in any way.
#### GET
#### GET /v1.0/renderedconfigdocs
Returns the full set of configdocs in their rendered form.
##### Query Parameters
* version=committed|**buffer**
@ -108,13 +153,13 @@ If documents can be retrieved.
A RPC style command to trigger a commit of the configuration documents from the
Shipyard Buffer to the Committed Documents. This resource will support POST
only.
#### Payload Structure
#### Entity Structure
The response will be the list of validations from all downstream systems that
perform validation during the commit process. The structure will match the
error response object described in the [UCP API conventions](https://github.com/att-comdev/ucp-integration/blob/master/docs/api-conventions.md)
and will be an aggregation of each UCP component's responses.
#### POST
#### POST /v1.0/commitconfigdocs
Performs the commit of the Shipyard Buffer to the Committed Documents.
Synchronous. This invokes each of the UCP components to examine the Shipyard
Buffer version of the configuration documents and aggregate the responses.
@ -158,16 +203,16 @@ See [Action Commands](API_action_commands.md) for supported actions
---
### /v1.0/actions
#### Payload Structure
#### Entity Structure
A list of actions that have been executed through shipyard's action API.
```
[
{ Action objects summarized, TBD },
{ Action objects summarized, See below},
...
]
```
#### GET
#### GET /v1.0/actions
Returns the list of actions in the system that have been posted, and are
accessible to the current user.
@ -175,7 +220,53 @@ accessible to the current user.
* 200 OK
If the actions can be retrieved.
#### POST
##### Example
```
$ curl -X GET $URL/api/v1.0/actions -H "X-Auth-Token:$TOKEN"
HTTP/1.1 200 OK
x-shipyard-req: 0804d13e-08fc-4e60-a819-3b7532cac4ec
content-type: application/json; charset=UTF-8
[
{
"dag_status": "failed",
"parameters": {},
"steps": [
{
"id": "action_xcom",
"url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/action_xcom",
"index": 1,
"state": "success"
},
{
"id": "dag_concurrency_check",
"url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/dag_concurrency_check",
"index": 2,
"state": "success"
},
{
"id": "preflight",
"url": "/actions/01BTP9T2WCE1PAJR2DWYXG805V/steps/preflight",
"index": 3,
"state": "failed"
},
...
],
"action_lifecycle": "Failed",
"dag_execution_date": "2017-09-23T02:42:12",
"id": "01BTP9T2WCE1PAJR2DWYXG805V",
"dag_id": "deploy_site",
"datetime": "2017-09-23 02:42:06.860597+00:00",
"user": "shipyard",
"context_marker": "416dec4b-82f9-4339-8886-3a0c4982aec3",
"name": "deploy_site"
},
...
]
```
#### POST /v1.0/actions
Creates an action in the system. This will cause some action to start.
The input body to this post will represent an action object that has at least
these fields:
@ -192,7 +283,7 @@ parameters will vary for the action invoked.
}
```
The POST will synchrounously create the action (a shell object that represents
The POST will synchronously create the action (a shell object that represents
a DAG invocation), perform any checks to validate the preconditions to run the
DAG, and trigger the invocation of the DAG. The DAG will run asynchronously in
airflow.
@ -201,58 +292,164 @@ airflow.
If the action is created successfully, and all preconditions to run the DAG
are successful. The response body is the action entity created.
* 400 Bad Request
If the action name doesn't exist, or the payload is otherwise malformed.
If the action name doesn't exist, or the input entity is otherwise malformed.
* 409 Conflict
For any failed pre-run validations. The response body is the action entity
created, with the failed validations. The DAG will not begin execution in this
case.
##### Example
```
$ curl -D - -d '{"name":"deploy_site"}' -X POST $URL/api/v1.0/actions \
-H "X-Auth-Token:$TOKEN" -H "content-type:application/json"
HTTP/1.1 201 Created
location: {$URL}/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN
x-shipyard-req: 629f2ea2-c59d-46b9-8641-7367a91a7016
content-type: application/json; charset=UTF-8
{
"dag_status": "SCHEDULED",
"parameters": {},
"dag_execution_date": "2017-09-24T19:05:49",
"id": "01BTTMFVDKZFRJM80FGD7J1AKN",
"dag_id": "deploy_site",
"name": "deploy_site",
"user": "shipyard",
"context_marker": "629f2ea2-c59d-46b9-8641-7367a91a7016",
"timestamp": "2017-09-24 19:05:43.603591"
}
```
---
### /v1.0/actions/{action_id}
Each action will be assigned an unique id that can be used to get details for
the action, including the execution status.
#### Payload Structure
#### Entity Structure
All actions will include fields that indicate the following data:
* id (assigned during POST)
* name - the name of the action - likely to be the DAG Names, but may be mapped
for "nicer" values.
* parameters - a dictionary of the parameters configuring the action.
* tracking info - (user, time, etc)
* action_lifecycle value:
* Pending
* Validation Failed
* Processing
* Complete
* Failed
* DAG_status - representing the status that airflow provides for an executing
DAG
* validations - a list of validations that have been done, including any status
information for those validations. During the lifecycle of the DAG, this list
of validations may continue to grow.
* steps - the list of steps for this action, including the status for that
step.
<dl>
<dt>action_lifecycle</dt>
<dd>A summarized value indicating the status or lifecycle phase of the
action.</dd>
<ul>
<li>Pending - The action is scheduled or preparing for execution.</li>
<li>Processing - The action is underway.</li>
<li>Complete - The action has completed successfully.</li>
<li>Failed - The action has encountered an error, and has failed.</li>
<li>Paused - The action has been paused by a user.</li>
</ul>
</dd>
<dt>command audit</dt>
<dd>A list of commands that have been issued against the action. Initially,
the action listed will be "invoke", but may include "pause", "unpause",
or "stop" if those commands are issued.</dd>
<dt>context_marker</dt>
<dd>The user supplied or system assigned context marker associated with the
action</dd>
<dt>dag_execution_date</dt>
<dd>The execution date assigned by the workflow system during action
creation.</dd>
<dt>dag_status</dt>
<dd>Represents the status that airflow provides for an executing DAG.</dd>
<dt>datetime</dt>
<dd>The time at which the action was invoked.</dd>
<dt>id</dt>
<dd>The identifier for the action, a 26 character ULID assigned during the
creation of the action.</dd>
<dt>name</dt>
<dd>The name of the action, e.g.: deploy_site.</dd>
<dt>parameters</dt>
<dd>The parameters configuring the action that were supplied by the
user during action creation.</dd>
<dt>steps</dt>
<dd>The list of steps for the action, including the status for that
step.</dd>
<dt>user</dt>
<dd>The user who has invoked this action, as acquired from the authorization
token.</dd>
<dt>validations</dt>
<dd>A list of validations that have been done, including any status
information for those validations. During the lifecycle of the action,
this list of validations may continue to grow.</dd>
```
{ TBD }
```
#### GET
#### GET /v1.0/actions/{action_id}
Returns the action entity for the specified id.
##### Responses
* 200 OK
##### Example
```
$ curl -D - -X GET $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN \
-H "X-Auth-Token:$TOKEN"
HTTP/1.1 200 OK
x-shipyard-req: eb3eacb3-4206-40df-bd91-2a3a6d81cd02
content-type: application/json; charset=UTF-8
{
"name": "deploy_site",
"dag_execution_date": "2017-09-24T19:05:49",
"validations": [],
"id": "01BTTMFVDKZFRJM80FGD7J1AKN",
"dag_id": "deploy_site",
"command_audit": [
{
"id": "01BTTMG16R9H3Z4JVQNBMRV1MZ",
"action_id": "01BTTMFVDKZFRJM80FGD7J1AKN",
"datetime": "2017-09-24 19:05:49.530223+00:00",
"user": "shipyard",
"command": "invoke"
}
],
"user": "shipyard",
"context_marker": "629f2ea2-c59d-46b9-8641-7367a91a7016",
"datetime": "2017-09-24 19:05:43.603591+00:00",
"dag_status": "failed",
"parameters": {},
"steps": [
{
"id": "action_xcom",
"url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/action_xcom",
"index": 1,
"state": "success"
},
{
"id": "dag_concurrency_check",
"url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/dag_concurrency_check",
"index": 2,
"state": "success"
},
{
"id": "preflight",
"url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/preflight",
"index": 3,
"state": "failed"
},
{
"id": "deckhand_get_design_version",
"url": "/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/deckhand_get_design_version",
"index": 4,
"state": null
},
...
],
"action_lifecycle": "Failed"
}
```
---
### /v1.0/actions/{action_id}/validationdetails/{validation_id}
Allows for drilldown to validation detailed info.
#### Payload Structure
#### Entity Structure
The detailed information for a validation
```
{ TBD }
```
#### GET
#### GET /v1.0/actions/{action_id}/validationdetails/{validation_id}
Returns the validation detail by Id for the supplied action Id.
##### Responses
* 200 OK
@ -263,29 +460,124 @@ Allow for drilldown to step information. The step information includes details
of the steps excution, successful or not, and enough to facilitate
troubleshooting in as easy a fashion as possible.
#### Payload Structure
The detailed information representing a single step of execution as part of an
action.
```
{ TBD }
```
#### Entity Structure
A step entity represents detailed information representing a single step of
execution as part of an action. Not all fields are necessarily represented in
every returned entity.
#### GET
<dl>
<dt>dag_id</dt>
<dd>The name/id of the workflow DAG that contains this step.</dd>
<dt>duration</dt>
<dd>The duration (seconds) for the step.</dd>
<dt>end_date</dt>
<dd>The timestamp of the completion of the step.</dd>
<dt>execution_date</dt>
<dd>The execution date of the workflow that contains this step.</dd>
<dt>index</dt>
<dd>The ordinal value representing the position of this step in the sequence
of steps associated with this step.</dd>
<dt>operator</dt>
<dd>The name of the processing facility used by the workflow system.</dd>
<dt>queued_dttm</dt>
<dd>The timestamp when the step was enqueued by the workflow system.</dd>
<dt>start_date</dt>
<dd>The timestamp for the beginning of execution for this step.</dd>
<dt>state</dt>
<dd>The execution state of the step.</dd>
<dt>task_id</dt>
<dd>The name of the task used by the workflow system (and also representing
this step name queried in the reqeust.</dd>
<dt>try_number</dt>
<dd>A number of retries taken in the case of failure. Some workflow steps may
be configured to retry before considering the step truly failed.</dd>
</dl>
#### GET /v1.0/actions/{action_id}/steps/{step_id}
Returns the details for a step by id for the given action by Id.
##### Responses
* 200 OK
##### Example
```
$ curl -D - \
-X GET $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/steps/action_xcom \
-H "X-Auth-Token:$TOKEN"
HTTP/1.1 200 OK
x-shipyard-req: 72daca4d-1f79-4e08-825f-2ad181912a47
content-type: application/json; charset=UTF-8
{
"end_date": "2017-09-24 19:05:59.446213",
"duration": 0.165181,
"queued_dttm": "2017-09-24 19:05:52.993983",
"operator": "PythonOperator",
"try_number": 1,
"task_id": "action_xcom",
"state": "success",
"execution_date": "2017-09-24 19:05:49",
"dag_id": "deploy_site",
"index": 1,
"start_date": "2017-09-24 19:05:59.281032"
}
```
---
### /v1.0/actions/{action_id}/control/{control_verb}
Allows for issuing DAG controls against an action.
#### Payload Structure
None, there is no associated body for this resource
#### Entity Structure
None, there is no associated response entity for this resource
#### POST
#### POST /v1.0/actions/{action_id}/{control_verb}
Trigger a control action against an activity.- this includes: pause, unpause
##### Responses
* 202 Accepted
##### Example
Failure case - command is invalid for the execution state of the action.
```
$ curl -D - \
-X POST $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/control/pause \
-H "X-Auth-Token:$TOKEN"
HTTP/1.1 409 Conflict
content-type: application/json
x-shipyard-req: 9c9551e0-335c-4297-af93-8440cc6b324f
{
"apiVersion": "v1.0",
"status": "Failure",
"metadata": {},
"message": "Unable to pause action",
"code": "409 Conflict",
"details": {
"errorList": [
{
"message": "dag_run state must be running, but is failed"
}
],
"errorCount": 1,
"errorType": "ApiError"
},
"kind": "status",
"reason": "dag_run state must be running, but is failed"
}
```
Success case
```
$ curl -D - \
-X POST $URL/api/v1.0/actions/01BTTMFVDKZFRJM80FGD7J1AKN/control/pause \
-H "X-Auth-Token:$TOKEN"
HTTP/1.1 202 Accepted
content-length: 0
x-shipyard-req: 019fae1c-03b0-4af1-b57d-451ae6ddac77
content-type: application/json; charset=UTF-8
```
---
## <a name="AirflowMonitoringAPI"></a> Airflow Monitoring API
@ -300,7 +592,7 @@ into the totality of what is happening in Airflow.
### /v1.0/workflows
The resource that represents DAGs (workflows) in airflow
#### Payload Structure
#### Entity Structure
A list of objects representing the DAGs that have run in airflow.
```
[
@ -309,7 +601,7 @@ A list of objects representing the DAGs that have run in airflow.
]
```
#### GET
#### GET /v1.0/workflows
Queries airflow for DAGs that are running or have run (successfully or
unsuccessfully) and provides a summary of those things.
##### Query parameters
@ -322,7 +614,7 @@ optional, a boundary in the past within which to retrieve results. Default is
---
### /v1.0/workflows/{id}
#### Payload Structure
#### Entity Structure
An object representing the information available from airflow regarding a DAG's
execution
@ -330,7 +622,7 @@ execution
{ TBD }
```
#### GET
#### GET /v1.0/workflows/{id}
Further details of a particular scheduled DAG's output
##### Responses
* 200 OK

150
docs/REST_API.md
View File

@ -1,150 +0,0 @@
Please note that this API information is being superseded by a new client facing API (under development) This file will stay in place for now, but needs to be removed when the newer API is in place. Please see API.md for the newer information.
## REST API ##
The default deployment will build an environment where the Falcon API server listens on port 31901
The Airflow Web UI can be accessed on the localhost via port 32080
At the present moment, Shipyard expects authentication token to be passed, i.e. *10* for the shipyard
user and *admin* for the admin user. An authentication error like the one below will be thrown if the
token is not passed in or if an invalid token is used (note that keystone will be used to perform
authentication in the future).
```
$ http localhost:31901/api/experimental/dags/airflow_task_state/tasks/airflow_task_state
HTTP/1.1 401 Unauthorized
content-length: 97
content-type: application/json; charset=UTF-8
vary: Accept
x-shipyard-req: 1a35a1cf-29c4-4642-aba6-9c12e3fb7f04
{
"description": "This resource requires an authorized role.",
"title": "Authentication required"
}
```
Note: An optional X-Context-Marker can also be passed with the REST call
The same Airflow REST API [endpoints](https://airflow.incubator.apache.org/api.html) are used in
Shipyard to perform API calls to airflow
Note: The following examples make use of [HTTPie](https://httpie.org/) to perform RESTful API calls
### Example 1 ###
Retrieve information of a particular task
```
$ http localhost:31901/api/experimental/dags/airflow_task_state/tasks/airflow_task_state X-Auth-Token:admin
HTTP/1.1 200 OK
content-length: 1153
content-type: application/json; charset=UTF-8
x-shipyard-req: e3afc941-a31a-42ab-a1b5-8717777fe155
{
"adhoc": "False",
"airflow_command": "airflow task_state airflow_task_state airflow_task_state 2016-06-26T16:35:45.068576",
"airflow_dag_id": "airflow_task_state",
"airflow_execution_date": "2016-06-26T16:35:45.068576",
"airflow_task_id": "airflow_task_state",
"depends_on_past": "False",
"email": "['airflow@example.com']",
"email_on_failure": "False",
"email_on_retry": "False",
"end_date": "None",
"execution_timeout": "None",
"max_retry_delay": "None",
"on_failure_callback": "None",
"on_retry_callback": "None",
"on_success_callback": "None",
"owner": "airflow",
"params": "{}",
"pool": "None",
"priority_weight": "1",
"queue": "default",
"resources": "{'disk': {'_qty': 512, '_units_str': 'MB', '_name': 'Disk'}, 'gpus': {'_qty': 0, '_units_str': 'gpu(s)', '_name': 'GPU'}, 'ram': {'_qty': 512, '_units_str': 'MB', '_name': 'RAM'}, 'cpus': {'_qty': 1, '_units_str': 'core(s)', '_name': 'CPU'}}",
"retries": "1",
"retry_delay": "0:01:00",
"retry_exponential_backoff": "False",
"run_as_user": "None",
"sla": "None",
"start_date": "2017-06-25 00:00:00",
"task_id": "airflow_task_state",
"trigger_rule": "all_success",
"wait_for_downstream": "False"
}
```
Note that we will get an error response if we try and retrieve an invalid task:
```
$ http localhost:31901/api/experimental/dags/airflow_task_state/tasks/test123 X-Auth-Token:10
HTTP/1.1 400 Bad Request
content-length: 61
content-type: application/json; charset=UTF-8
x-shipyard-req: ec150a0c-30bf-40ba-a40e-ad4627d8770b
{
"error": "Task test123 not found in dag airflow_task_state"
}
```
Retrieval of invalid dag will end up with error as well:
```
$ http localhost:31901/api/experimental/dags/test123/tasks/airflow_task_state X-Auth-Token:10
HTTP/1.1 400 Bad Request
content-length: 34
content-type: application/json; charset=UTF-8
x-shipyard-req: d718820b-7f74-4893-affb-92e783ec4541
{
"error": "Dag test123 not found"
}
```
### Example 2 ###
Execute a Dag
```
$ http POST localhost:31901/api/experimental/dags/airflow_task_state/dag_runs X-Auth-Token:10 X-Context-Marker:airflow_task_state_testing
HTTP/1.1 200 OK
content-length: 0
content-type: application/json; charset=UTF-8
x-shipyard-req: a2f2a49c-82c8-4131-9764-fc15495b520c
```
In this example, the *airflow_task_state* dag will get executed and the information of the task instances
can be retrieved from the Airflow Web UI which is usually located at this [link](http://localhost:32080/admin/taskinstance/)
Note that we will get an error response if we try and execute an invalid dag:
```
$ http POST localhost:31901/api/experimental/dags/test123/dag_runs X-Auth-Token:10
HTTP/1.1 400 Bad Request
content-length: 67
content-type: application/json
x-shipyard-req: ad978d6c-eb5b-4392-a8cb-461dd42a9b00
{
"message": "Fail to Execute Dag",
"retry": false,
"type": "error"
}
```