d6c72d19e6
Updates to Airflow 1.10.1; See (1), (2) for some notes Related, and additionally: configures Airflow to restore logging of workflow steps to a console/sdtout logger, supporting the desired ability to attach logging and monitoring to standard container mechanisms. This does not change the behavior of also logging to the airflow-arranged log files for steps and DAG runs. A side effect of updating to 1.10.1 includes a major decrease in resource usage by the Airflow scheudler process (reducing from ~ 1 core fully consumed to less than 5% of a core consumed YMMV, but significant) Additional adjustment downward of resources allocated, threads produced, and frequency of polling leads to an overall significant reduction in resource usage. Airship note: Because Airflow 1.10.0 and 1.10.1 use compatible versions of celery and dag_run information, updating from 1.10.0 - 1.10.1 in place is possible if airflow-worker pods are allowed to continue to run. (1) https://github.com/apache/incubator-airflow/blob/master/UPDATING.md (2) https://github.com/apache/incubator-airflow/releases/tag/1.10.1 Change-Id: I9b024e3996c528c7b74e2888191d48c7a45a1f04 |
||
|---|---|---|
| .. | ||
| alembic | ||
| etc/shipyard | ||
| generator | ||
| shipyard_airflow | ||
| tests | ||
| .coveragerc | ||
| README.md | ||
| alembic.ini | ||
| entrypoint.sh | ||
| requirements.txt | ||
| setup.cfg | ||
| setup.py | ||
| test-requirements.txt | ||
| tox.ini | ||
README.md
Shipyard
Shipyard is the directed acyclic graph controller for Kubernetes and OpenStack control plane life cycle management, and a component of the Airship Undercloud Platform (UCP).
Shipyard provides the entrypoint for the following aspects of the control plane established by the Airship:
- Designs and Secrets
-
Site designs, including the configuration of bare metal host
nodes, network design, operating systems, Kubernetes nodes,
Armada manifests, Helm charts, and any other descriptors that
define the build out of a group of servers enter the Airship via
Shipyard. Secrets, such as passwords and certificates use the
same mechanism.
The designs and secrets are stored in Airship's Deckhand, providing for version history and secure storage among other document-based conveniences. - Actions
- Interaction with the site's control plane is done via invocation of actions in Shipyard. Each action is backed by a workflow implemented as a directed acyclic graph (DAG) that runs using Apache Airflow. Shipyard provides a mechanism to monitor and control the execution of the workflow.
Find more documentation for Shipyard on Read the Docs
Integration Points:
OpenStack Identity (Keystone)
provides authentication and support for role based authorization
Apache Airflow provides the
framework and automation of workflows provided by Shipyard
PostgreSQL is used to persist
information to correlate workflows with users and history of workflow
commands
Deckhand supplies storage
and management of site designs and secrets
Drydock is orchestrated by
Shipyard to perform bare metal node provisioning
Promenade is indirectly
orchestrated by Shipyard to configure and join Kubernetes nodes
Armada is orchestrated by
Shipyard to deploy and test Kubernetes workloads
Getting Started:
Shipyard @ Openstack Gerrit
Helm chart