This PS bumps up Airflow version to the latest
2.8.2 and also bumps up openstack dependences to
Antelope 2023.1
Change-Id: I7bbcbc8f9e6334100f47702f4546f0963d11b2ca
This PS enable TLS connection from celery to rabbitmq
when TLS connection is enabled
Change-Id: I49ccf159ca73e0764703a6d3c686c108143f12e2
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
This PS adds deployment with Airflow webserver UI in viewer mode
protected by base http autorization and exposed via ingress.
Change-Id: I5692eecf5a9af2930f8cce98b7a1e430f26b5a1b
Signed-off-by: Sergiy Markin <smarkin@mirantis.com>
This PS is mainly fixing SubDAGs timing issues when they
started along with main DAG and not at the time the main
DAG needs them. In Airflow 2.6.2 SubDAGs are deprecated
in favor of TaskGroups. So in this PS all SubDAGS were
replaced with TaskGroups.
Also task level logging config was extended by adding
py-console to obtain logs from tasks like it was
configured in Airflow 1.10.5.
Change-Id: I3f6d3961b1511e3b7cd2f7aab9810d033cfc14a3
This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2
Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
This PS adds default values for chart values and resolves some issues
in python code that utilizes these values:
validation_connect_timeout: 20
validation_read_timeout: 300
deckhand_client_connect_timeout: 20
deckhand_client_read_timeout: 300
drydock_client_connect_timeout: 20
drydock_client_read_timeout: 300
Change-Id: Ic5b1920257859239613a3ce77134e6b05bd7e9dd
This PS is created to :
- roll back apache-airflow back to 1.10.5
- change default chart values from bionic to ubuntu_focal
- save freshly generated config and policy files samples
- in focal Dockerfile postgres client version is pubped up to v15
- change airflow docker image requirements from limited to fully frozen for shipyard-airflow project
- adjusted requirements-direct.txt for shipyard_airflow
- adjusted requirements-direct.txt for shipyard_client
- regenerated requirements-frozen for both projects
- fixed unit tests after upgrading click module
- gen_all tox profile processing has been moved over to py8 gate because it requires focal zuul node to run
- upgraded airskiff gate playbook to include latest treasuremap patchset with updated airskiff site admada manifests
Change-Id: I47e44f5cfa19b2649697e7cc5a31557a6f4fcfea
- Overrode uwsgi default configs to improve stability and performance.
- Increased mas number of worker processes to increase capacity and
performance.
- Enabled uwsgi cheaper subsystem to scale worker processes dynamically.
- Uplifted uwsgi to the latest release to bring bug fixes and
improvements since 2018.
Upgraded uwsgi to bring in bug fixes since 2018.
For background information for this change please see:
https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html
Change-Id: If067e9786e9dbbd39ef832dea6f51aa5523af4d7
This updates the airflow chart to include the pod
security context on the pod template.
This also adds the container security context to set
readOnlyRootFilesystem flag
Change-Id: I84cd4581d6ae915e9caf5c50d407dfcc34b962b3
This updates the shipyard chart to include the pod
security context on the pod template.
This also adds the container security context to set
readOnlyRootFilesystem flag to true
Change-Id: I2ffe17fc7d42aa5544e606f3a354496a64005640
Remove OSH Authors copyright
The current copyright refers to a non-existent group
"openstack helm authors" with often out-of-date references that
are confusing when adding a new file to the repo.
This change removes all references to this copyright by the
non-existent group and any blank lines underneath.
Change-Id: Ic8de1678a754ba466dbd8d12c4f078151a78a091
This adds the container security context to set
readOnlyRootFilesystem to true
Depends-on: https://review.opendev.org/#/c/708948/2
Change-Id: I4c7e7dba26d6bdfd0032a31469fd1777ae06cfec
Added support to buid shipyard and airflow images using either a xenial
or Ubuntu bionic base image.
The default base image is set to bionic.
Change-Id: I6ad4d42dede081586b3ccea87a42e250979ac106
The patch introduces network policy configuration similar
to openstack-helm services. It allows users to configure
policies depending on the environment.
* Network policies are disabled by default.
* When enabled default policies allow all ingress and
egress traffic (i.e. policy set to {}), this may be
changed in future patch-sets.
Change-Id: Ic0b44eb142445d45d81e3e546d394e1c7b451238
Updated airflow config and Dockerfiles for apache airflow 1.10.4
For details see: https://github.com/apache/airflow/blob/master/UPDATING.md
Also updated the kubectl to 1.16.2, as part of kubernetes 1.16 uplift.
Change-Id: Ib24ff0304b6279ff0be749141854d6a604473597
This patch makes Celery to connect to RabbitMQ directly instead
of using LB. It also brings a forked version of a transport url
template, the reason for this is the format for Kombu/Celery
broker url is different from oslo_messaging transport url:
1. URLs need to be separated with semicolons vs commas.
2. Every item in Kombu broker url needs to be a complete url
that includes schema, vhost, and all credentials.
This format is specific to Airflow and is not used in upstream
OSH projects, hence it is included here and not in htk.
Depends-On: I5150a64bd29fa062e30496c1f2127de138322863
Change-Id: I0b4ae6a9538f2f6988ed42c8f5cf0a54e7a7ad2e
Since ':master' and ':latest' Shipyard and Airflow images are outdated,
set defaults to Ubuntu Xenial -based images.
Change-Id: I40978747f31c6a8c5cc8689a9768f8c4836ac1a1
The current logrotate logic deletes logs that are
more than X days old in the Airflow log path, however
the Airflow log archive may still reach 100%
usage and cause the airflow-worker to crashloop.
This PS adds logic to logrotate.sh to delete the oldest
logs and empty dirs when the Airflow log archive
reaches the max usage specified in values.yaml.
Change-Id: I3dcb80901d7dd36da6812850a1f54e7ebf3b1cf2
This PS adds pod anti-affinity to airflow/shipyard pods,
so that the scheduler can constrain pods against labels on other pods
running on the node. The default soft rule is in place so that if
the scheduler can’t satisfy the the requirement, the pod will still
be scheduled, and is overridable.
Change-Id: I67d0792a1f624044f8975c9540ab691f4e638b3f
This change adds a new Shipyard Operator that creates/updates a
ConfigMap with information on the version and status of the current
running deployment. This ConfigMap will be created at the start of the
deployments, and will be updated at the end even if the previous steps
fail.
This operator has been added to the deploy_site, update_site, and
update_software DAGs.
Change-Id: Iab9ea84d5e1edd6a8635cc4e4fa93647ee485194
This PS adds funtionality to Shipyard to validate the existence of
the Pegleg-generated "deployment-version" document (Pegleg change id:
I7919b02d70c9797f689cdad85066d3953b978901) when a user runs create
configdocs. This validation only checks the presence of the document
(by name and schema) and does not care about the document's other
contents.
The severity of a failed validation is configurable through the
"validations" config section in shipyard.conf. The default severity
is "Skip", meaning the validation is not ran at all.
Note that with the default configuration of new validation, Shipyard
functionality should be unchanged.
Change-Id: I754617de81f628a24232e890b12b157ba6731c25
This PS adds funtionality to Shipyard to validate the existence of
the Pegleg-generated "deployment-version" document (Pegleg change id:
I7919b02d70c9797f689cdad85066d3953b978901). As implied, this new
validation only checks for the presence of the document (by name and
schema) and currently does not care about any of the document's
contents under "data".
The severity of a failed validation is configurable through the new
"validations" configuration section in shipyard.conf, and is
defaulted to skip the validation altogether. This means that by
default, this patch set does not alter the functionality of Shipyard
Note that with the default configuration of this new validation,
Shipyard functionality should be unchanged.
Change-Id: I5e7269066f769804710a0fd1f2c8d0aece0d3314
Currently, any document name or schema referenced in the Shipyard
code base is a hard-coded string. Often times, these strings are
repeated throughout the code. This patch set adds a new configuration
section to shipyard.conf to define document names and schemas so they
can then be referenced in the Shipyard code via the oslo
configuration object. This functionality will be important for
upcoming Shipyard features which will call for more documents to be
validated as well as some new Shipyard-created docs.
Change-Id: I34ae8cd578bab730d004c3d176e3817b5a45c89e
Updates the definition for queue mirroring to use the same vhost as the
queue itself is defined to be.
Change-Id: Ibb631d7adb237fa17c5e853a9e9e35297a525782
- deployment-shipyard
This updates the shipyard chart to include the pod
security context on the pod template.
This also adds the container security context to set
allowPrivilegeEscalation to false and readOnlyRootFilesystem to true
Change-Id: Idb1b848847eaec2b6e24389c063b7ece2973c4dc
Moves the airflow web server container from its own pod into the
Shipyard pod. This removes exposed network surface area from the
Shipyard suite of software. Shipyard, after this change accesses the
Airflow API using localhost in the same k8s pod.
Change-Id: Ied4bd415a8d78c393b7256ead27a6a2176f4a2d6
Changes to make the docker image build to include the workflows from
Shipyard, rather than adding them to the container during Helm install
of Shipyard. This also removes the "prod" switch, as it is now always
built the same way, with the workflows in place.
Change-Id: I4acd6195cbec32193e15621e75ccaeb9879455f5
Moves the airflow sceduler to a container in the airflow-worker
statefulset so that its version lifecycle matches that of the worker.
Leaves the stand-alone scheduler in place to support upgradability from
prior installations that included a standalone scheduler. New
installations are advised to turn off the scheduler template from
rendering using the values.yaml flag.
This is an attempt to make disruptive upgrades to airflow less impactful
to a "update_site" action from Shipyard.
Additionally this removes the template for airflow-flower, which is not in use.
Change-Id: I0608793ee6aba1eb3ce0f5e9567655287014a0ca
Sergiy Markin