shipyard/docs/REST_API.md

Please note that this API information is being superseded by a new client facing API (under development) This file will stay in place for now, but needs to be removed when the newer API is in place. Please see API.md for the newer information.

REST API

The default deployment will build an environment where the Falcon API server listens on port 31901

The Airflow Web UI can be accessed on the localhost via port 32080

At the present moment, Shipyard expects authentication token to be passed, i.e. 10 for the shipyard user and admin for the admin user. An authentication error like the one below will be thrown if the token is not passed in or if an invalid token is used (note that keystone will be used to perform authentication in the future).

$ http localhost:31901/api/experimental/dags/airflow_task_state/tasks/airflow_task_state
HTTP/1.1 401 Unauthorized
content-length: 97
content-type: application/json; charset=UTF-8
vary: Accept
x-shipyard-req: 1a35a1cf-29c4-4642-aba6-9c12e3fb7f04

{
    "description": "This resource requires an authorized role.",
    "title": "Authentication required"
}

Note: An optional X-Context-Marker can also be passed with the REST call

The same Airflow REST API endpoints are used in Shipyard to perform API calls to airflow

Note: The following examples make use of HTTPie to perform RESTful API calls

Example 1

Retrieve information of a particular task

$ http localhost:31901/api/experimental/dags/airflow_task_state/tasks/airflow_task_state X-Auth-Token:admin
HTTP/1.1 200 OK
content-length: 1153
content-type: application/json; charset=UTF-8
x-shipyard-req: e3afc941-a31a-42ab-a1b5-8717777fe155

{
    "adhoc": "False",
    "airflow_command": "airflow task_state airflow_task_state airflow_task_state 2016-06-26T16:35:45.068576",
    "airflow_dag_id": "airflow_task_state",
    "airflow_execution_date": "2016-06-26T16:35:45.068576",
    "airflow_task_id": "airflow_task_state",
    "depends_on_past": "False",
    "email": "['airflow@example.com']",
    "email_on_failure": "False",
    "email_on_retry": "False",
    "end_date": "None",
    "execution_timeout": "None",
    "max_retry_delay": "None",
    "on_failure_callback": "None",
    "on_retry_callback": "None",
    "on_success_callback": "None",
    "owner": "airflow",
    "params": "{}",
    "pool": "None",
    "priority_weight": "1",
    "queue": "default",
    "resources": "{'disk': {'_qty': 512, '_units_str': 'MB', '_name': 'Disk'}, 'gpus': {'_qty': 0, '_units_str': 'gpu(s)', '_name': 'GPU'}, 'ram': {'_qty': 512, '_units_str': 'MB', '_name': 'RAM'}, 'cpus': {'_qty': 1, '_units_str': 'core(s)', '_name': 'CPU'}}",
    "retries": "1",
    "retry_delay": "0:01:00",
    "retry_exponential_backoff": "False",
    "run_as_user": "None",
    "sla": "None",
    "start_date": "2017-06-25 00:00:00",
    "task_id": "airflow_task_state",
    "trigger_rule": "all_success",
    "wait_for_downstream": "False"
}

Note that we will get an error response if we try and retrieve an invalid task:

$ http localhost:31901/api/experimental/dags/airflow_task_state/tasks/test123 X-Auth-Token:10
HTTP/1.1 400 Bad Request
content-length: 61
content-type: application/json; charset=UTF-8
x-shipyard-req: ec150a0c-30bf-40ba-a40e-ad4627d8770b

{
    "error": "Task test123 not found in dag airflow_task_state"
}

Retrieval of invalid dag will end up with error as well:

$ http localhost:31901/api/experimental/dags/test123/tasks/airflow_task_state X-Auth-Token:10
HTTP/1.1 400 Bad Request
content-length: 34
content-type: application/json; charset=UTF-8
x-shipyard-req: d718820b-7f74-4893-affb-92e783ec4541

{
    "error": "Dag test123 not found"
}

Example 2

Execute a Dag

$ http POST localhost:31901/api/experimental/dags/airflow_task_state/dag_runs X-Auth-Token:10 X-Context-Marker:airflow_task_state_testing
HTTP/1.1 200 OK
content-length: 0
content-type: application/json; charset=UTF-8
x-shipyard-req: a2f2a49c-82c8-4131-9764-fc15495b520c

In this example, the airflow_task_state dag will get executed and the information of the task instances can be retrieved from the Airflow Web UI which is usually located at this link

Note that we will get an error response if we try and execute an invalid dag:

$ http POST localhost:31901/api/experimental/dags/test123/dag_runs X-Auth-Token:10
HTTP/1.1 400 Bad Request
content-length: 67
content-type: application/json
x-shipyard-req: ad978d6c-eb5b-4392-a8cb-461dd42a9b00

{
    "message": "Fail to Execute Dag",
    "retry": false,
    "type": "error"
}