This PS enable TLS connection from celery to rabbitmq
when TLS connection is enabled
Change-Id: I49ccf159ca73e0764703a6d3c686c108143f12e2
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2
Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
Remove OSH Authors copyright
The current copyright refers to a non-existent group
"openstack helm authors" with often out-of-date references that
are confusing when adding a new file to the repo.
This change removes all references to this copyright by the
non-existent group and any blank lines underneath.
Change-Id: Ic8de1678a754ba466dbd8d12c4f078151a78a091
This adds the container security context to set
readOnlyRootFilesystem to true
Depends-on: https://review.opendev.org/#/c/708948/2
Change-Id: I4c7e7dba26d6bdfd0032a31469fd1777ae06cfec
Update apiversion for ClusterRole, ClusterRoleBinding to rbac.authorization.k8s.io/v1
Update apiversion for deployment to apps/v1
Update apiversion for statefulset to apps/v1
Add selector match labels to deployment
This patch is similar to https://review.opendev.org/#/c/638276/
These changes are required to install shipyard helm chart on k8s 1.16.0
Change-Id: I7ac6fc060fbd6a5feea747ebbe8121c5a2eb4b6f
airflow.cfg file was mounted as a dir, not a file,
so airflow service doesn't want to start.
This reverts commit 6794903558.
Change-Id: I6db528ac91fc5cb6719831eb2915467105f4c491
Recently, the airflow config mounts were changed to projected volumes to
workaround a K8s bug [0]; however, a subpath prevents the configs from
being properly mounted. This change removes the subpath.
[0] https://review.opendev.org/671944
Change-Id: I9bbe91d3e27b293a6fd27c00545329bc8a36f926
Signed-off-by: Drew Walters <andrew.walters@att.com>
Because of a kubernetes bug [0] when a container which
is mounted with the subpath option, the configmap is
changed and then the container restarts the mounting of
the configmap fails.
This PS uses the projected key for volume definitions
as a workaround.
[0] https://github.com/kubernetes/kubernetes/issues/68211
Change-Id: I6820a0f963c5b28e1674ea58214ffc86009db4dd
The current logrotate logic deletes logs that are
more than X days old in the Airflow log path, however
the Airflow log archive may still reach 100%
usage and cause the airflow-worker to crashloop.
This PS adds logic to logrotate.sh to delete the oldest
logs and empty dirs when the Airflow log archive
reaches the max usage specified in values.yaml.
Change-Id: I3dcb80901d7dd36da6812850a1f54e7ebf3b1cf2
This PS adds pod anti-affinity to airflow/shipyard pods,
so that the scheduler can constrain pods against labels on other pods
running on the node. The default soft rule is in place so that if
the scheduler can’t satisfy the the requirement, the pod will still
be scheduled, and is overridable.
Change-Id: I67d0792a1f624044f8975c9540ab691f4e638b3f
This change adds a new Shipyard Operator that creates/updates a
ConfigMap with information on the version and status of the current
running deployment. This ConfigMap will be created at the start of the
deployments, and will be updated at the end even if the previous steps
fail.
This operator has been added to the deploy_site, update_site, and
update_software DAGs.
Change-Id: Iab9ea84d5e1edd6a8635cc4e4fa93647ee485194
Moves the airflow web server container from its own pod into the
Shipyard pod. This removes exposed network surface area from the
Shipyard suite of software. Shipyard, after this change accesses the
Airflow API using localhost in the same k8s pod.
Change-Id: Ied4bd415a8d78c393b7256ead27a6a2176f4a2d6
Changes to make the docker image build to include the workflows from
Shipyard, rather than adding them to the container during Helm install
of Shipyard. This also removes the "prod" switch, as it is now always
built the same way, with the workflows in place.
Change-Id: I4acd6195cbec32193e15621e75ccaeb9879455f5
Moves the airflow sceduler to a container in the airflow-worker
statefulset so that its version lifecycle matches that of the worker.
Leaves the stand-alone scheduler in place to support upgradability from
prior installations that included a standalone scheduler. New
installations are advised to turn off the scheduler template from
rendering using the values.yaml flag.
This is an attempt to make disruptive upgrades to airflow less impactful
to a "update_site" action from Shipyard.
Additionally this removes the template for airflow-flower, which is not in use.
Change-Id: I0608793ee6aba1eb3ce0f5e9567655287014a0ca
The 'airflow-worker' and 'airflow-logrotate' containers do not need to
run as privileged containers to perform their jobs. Shipyard deploy_site
action was used to test the 'airflow-worker' as a deploy_site invokes
'airflow-worker'. When performing deploy_site action, all steps succeeded
and the 'airflow-worker' shows no errors when 'airflow-worker' is
non-privileged.
When 'airflow-logrotate' runs as non-privileged, the 'airflow-logrotate'
container still logs correctly and is able to delete/rotate logs without
problems.
Note: Making airflow-worker run with non-privileged containers means that
these containers will use the docker-default apparmor profile by default.
Change-Id: I26eda3eb8b7a36e67c2e7b593326f1d063600fc3
This PS adds the ability to attach a release uuid to pods and rc
objects as desired. This can be used, for example, to force an
artificial manifest change in CICD scenarios, for upgradability
testing purposes.
Change-Id: I75fc7516e0d32e7e0df0fecf1f9bec0e234adfcc
Provides different fields in the values.yaml to use for the node
selector keys/values used in deployments vs jobs.
Change-Id: I12d7c6257aea0ac00cd77cd3f6331a2b7380b589
Updated configurations to point to openstack-helm-infra
for reference to helm-toolkit as helm-toolkit has been
removed from the openstack-helm repo [0]
Also aligned with changes to the keystone user set up in
OSH using Helm ToolKit so as to get pass Helm Lint.
Updated Makefile targets to install helm dynamically
[0] https://review.openstack.org/#/c/558065/
Change-Id: I0a0813516f9ad176ff005b4693e6b933013a99fd
This patch set is meant to create a workflow that will allow us
to upgrade the airflow worker without causing disruption to the
current running workflow.
Note that we will set the update strategy for airflow worker
to 'OnDelete'. The 'OnDelete' update strategy implements the legacy
(1.6 and prior) behavior. When we select this update strategy, the
statefulSet controller will not automatically update Pods when a
modification is made to the StatefulSet’s '.spec.template field'.
This strategy can be selected by setting the '.spec.template.updateStrategy.type'
to 'OnDelete'. Refer to [0] for more information.
[0] https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#creating-a-statefulset
Change-Id: I1f6c3564b7fba6abe422b86e36818eb2cd3454ea
We need a side car container to perform log rotation
on the log files. Logs shall be retained for 30 days.
This is the default setting and can be changed by updating
values.yaml
Also cleaned up README.md
Change-Id: I39a7797e96abd349160d753f8917f7f78f7d8797
This patch set removes the (pre)generated config ini file from
airflow. The configuration will now be pulled directly from
values.yaml which will be inline with OpenStack-Helm's approach.
This will do away with the need to maintain the verbose .conf.tpl
in the repository as mentioned by Tin in his comments for [0].
[0] https://review.gerrithub.io/#/c/400925/
Change-Id: I5a9766e52536ac9b143b397faa3563e69dfb6bf3
There is a need to make the airflow worker a stateful set
so that the name of the pod will be consistent. This will
allow us to properly extract and correlate logs in the
database.
We are also adding pvc for the airflow worker pods so that
the logs persist.
Change-Id: I79917aa02b38672cac13d6148c4ed44007a78d32
Anselme, Schubert (sa246v)