airship
/
shipyard
Code Issues Proposed changes

Update Deployment Guide 

- Change Namespace to shipyard instead of airflow
- Add steps to add custom dags and operators to mount point on host machine
- Update steps to retrieve shipyard chart

Change-Id: Ia54201a7d1241b0aa6da092bb60705a15b5d9c71
This commit is contained in:
Anthony Lin 2017-08-10 01:05:53 +00:00
parent 2d7f7dc57c
commit 065bf9259a
1 changed files with 62 additions and 48 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

48
docs/deployment_guide.md
View File

@ -35,10 +35,10 @@ infrastructure and helm charts to deploy the containerized applications.
Note: The airflow chart requires a postgresql instance and rabbitmq to be running
Create airflow namespace:
Create shipyard namespace:
```
kubectl create namespace airflow
kubectl create namespace shipyard
```
@ -48,13 +48,13 @@ infrastructure and helm charts to deploy the containerized applications.
get the postgresql helm chart. Bring up the postgresql container using the postgresql helm chart:
```
helm install --name=airflow-postgresql postgresql/ --namespace=airflow
helm install --name=shipyard-postgresql postgresql/ --namespace=shipyard
```
Note: Postgresql may take a short time to reach the 'Running' state. Verify that postgresql is running:
```
# kubectl get pods -n airflow
# kubectl get pods -n shipyard
NAME READY STATUS RESTARTS AGE
postgresql-0 1/1 Running 0 1m
```
@ -70,14 +70,14 @@ infrastructure and helm charts to deploy the containerized applications.
Execute the following commands:
```
helm install --name=airflow-etcd-rabbitmq etcd/ --namespace=airflow
helm install --name=airflow-rabbitmq rabbitmq/ --namespace=airflow
helm install --name=shipyard-etcd-rabbitmq etcd/ --namespace=shipyard
helm install --name=shipyard-rabbitmq rabbitmq/ --namespace=shipyard
```
Note: We need to make sure that the etcd chart is executed before the rabbitmq chart due to dependencies
```
# kubectl get pods -n airflow
# kubectl get pods -n shipyard
NAME READY STATUS RESTARTS AGE
etcd-2810752095-054xb 1/1 Running 0 2m
postgresql-0 1/1 Running 0 3m
@ -89,17 +89,30 @@ infrastructure and helm charts to deploy the containerized applications.
Airflow Helm Chart Installation:
Clone the [AIC-Helm](https://github.com/att-comdev/aic-helm.git) repository to get the Airflow Helm Charts.
Create the following directories on the target host machine:
```
$ mkdir -p /home/ubuntu/workbench/dags
$ mkdir -p /home/ubuntu/workbench/plugins
$ mkdir -p /home/ubuntu/workbench/logs
```
Copy the [rest_api_plugin](https://github.com/att-comdev/shipyard/blob/master/shipyard_airflow/plugins/rest_api_plugin.py)
into the newly created plugins directory, i.e. `/home/ubuntu/workbench/plugins` so that it can be loaded by Airflow during
startup. Note that other custom operators can also be added to the directory as required.
Note: Custom dags should be added into the newly created dags directory, i.e. `/home/ubuntu/workbench/dags`
Next, proceed to clone the [AIC-Helm](https://github.com/att-comdev/aic-helm.git) repository to get the Airflow Helm Charts.
The charts can be found under the airflow directory.
```
helm install --name=airflow airflow/ --namespace=airflow
helm install --name=airflow airflow/ --namespace=shipyard
```
Verify that the airflow helm charts were successful deployed:
```
# kubectl get pods -n airflow
# kubectl get pods -n shipyard
NAME READY STATUS RESTARTS AGE
etcd-2810752095-054xb 1/1 Running 0 1h
flower-57424757-xqzls 1/1 Running 0 1m
@ -117,7 +130,7 @@ infrastructure and helm charts to deploy the containerized applications.
To check that all resources are working as intended:
```
$ kubectl get all --namespace=airflow
$ kubectl get all --namespace=shipyard
NAME READY STATUS RESTARTS AGE
po/etcd-2810752095-054xb 1/1 Running 0 1h
po/flower-57424757-xqzls 1/1 Running 0 1m
@ -164,23 +177,24 @@ infrastructure and helm charts to deploy the containerized applications.
3) Deploy Shipyard Helm Chart
The Shipyard helm chart is still under review at the moment and can be retrieved by cloning the *shipyard_chart*
branch with the following command:
The Shipyard helm chart is still under review at the moment and can be retrieved by performing the
the following steps:
```
$ git clone https://github.com/eanylin/aic-helm.git --branch shipyard_chart
$ git clone http://review.gerrithub.io/att-comdev/aic-helm
$ git fetch https://review.gerrithub.io/att-comdev/aic-helm refs/changes/91/373591/1 && git checkout FETCH_HEAD
```
Shipyard Helm Chart Installation:
```
$ helm install --name=airflow-shipyard shipyard/ --namespace=airflow
$ helm install --name=shipyard shipyard/ --namespace=shipyard
```
Check that all the helm charts have been properly deployed and that all services are up and running:
```
# kubectl get pods -n airflow
# kubectl get pods -n shipyard
NAME READY STATUS RESTARTS AGE
etcd-2810752095-7xbj2 1/1 Running 0 14m
flower-3408049844-ntpt3 1/1 Running 0 6m
@ -194,7 +208,7 @@ infrastructure and helm charts to deploy the containerized applications.
worker-2534280303-fcb49 1/1 Running 0 6m
# kubectl get service -n airflow
# kubectl get service -n shipyard
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
etcd 10.97.149.59 <none> 2379/TCP 13m
flower 10.108.197.84 <nodes> 5555:32081/TCP 5m