d6c72d19e6
Updates to Airflow 1.10.1; See (1), (2) for some notes Related, and additionally: configures Airflow to restore logging of workflow steps to a console/sdtout logger, supporting the desired ability to attach logging and monitoring to standard container mechanisms. This does not change the behavior of also logging to the airflow-arranged log files for steps and DAG runs. A side effect of updating to 1.10.1 includes a major decrease in resource usage by the Airflow scheudler process (reducing from ~ 1 core fully consumed to less than 5% of a core consumed YMMV, but significant) Additional adjustment downward of resources allocated, threads produced, and frequency of polling leads to an overall significant reduction in resource usage. Airship note: Because Airflow 1.10.0 and 1.10.1 use compatible versions of celery and dag_run information, updating from 1.10.0 - 1.10.1 in place is possible if airflow-worker pods are allowed to continue to run. (1) https://github.com/apache/incubator-airflow/blob/master/UPDATING.md (2) https://github.com/apache/incubator-airflow/releases/tag/1.10.1 Change-Id: I9b024e3996c528c7b74e2888191d48c7a45a1f04 |
||
---|---|---|
charts/shipyard | ||
doc | ||
etc/shipyard | ||
images | ||
src/bin | ||
tools | ||
.dockerignore | ||
.editorconfig | ||
.gitignore | ||
.gitreview | ||
.zuul.yaml | ||
LICENSE | ||
Makefile | ||
README.rst | ||
requirements.readthedocs.txt | ||
tox.ini |
README.rst
Shipyard
Shipyard adopts the Falcon web framework and uses Apache Airflow as the backend engine to programmatically author, schedule and monitor workflows.
Find more documentation for Shipyard on Read the Docs.
The current workflow is as follows:
- Initial region/site data will be passed to Shipyard from either a human operator or Jenkins
- The data (in YAML format) will be sent to Deckhand for validation and storage
- Shipyard will make use of the post-processed data from DeckHand to interact with Drydock.
- Drydock will interact with Promenade to provision and deploy bare metal nodes using Ubuntu MAAS and a resilient Kubernetes cluster will be created at the end of the process
- Once the Kubernetes clusters are up and validated to be working properly, Shipyard will interact with Armada to deploy OpenStack using OpenStack Helm
- Once the OpenStack cluster is deployed, Shipyard will trigger a workflow to perform basic sanity health checks on the cluster
Note: This project, along with the tools used within are community-based and open sourced.
Mission
The goal for Shipyard is to provide a customizable framework for operators and developers alike. This framework will enable end-users to orchestrate and deploy a fully functional container-based Cloud.
Getting Started
This project is under development at the moment. We encourage anyone who is interested in Shipyard to review our documentation.
Bugs
If you find a bug, please feel free to create a Storyboard issue.