Commit Graph

79 Commits

Author SHA1 Message Date
Sergiy Markin bd67c08136 Disable airflow webserver UI by default
This PS disables Airflow Webserver UI by default.

Change-Id: Ic4734e8159db8b44675fcbced2abb1858f1813fc
2024-03-19 18:01:29 +00:00
Sergiy Markin 20593bc746 Airflow stable 2.8.1
Change-Id: Iac800d9bc6e53a5a60e11c649d05dc663e8dd8e9
2024-02-23 19:43:03 +00:00
Anselme, Schubert (sa246v) 2a6c028a41
Enable TLS for celery
This PS enable TLS connection from celery to rabbitmq
when TLS connection is enabled

Change-Id: I49ccf159ca73e0764703a6d3c686c108143f12e2
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
2023-12-06 15:48:29 -05:00
Anselme, Schubert (sa246v) f571611f3c
Enable TLS connection to rabbitmq
Change-Id: Ia4d65393ad0112ae63433a7f67a8c8706a15b216
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
2023-11-20 10:42:30 -05:00
Sergiy Markin 181da83d34 Airflow webserver UI
This PS adds deployment with Airflow webserver UI in viewer mode
protected by base http autorization and exposed via ingress.

Change-Id: I5692eecf5a9af2930f8cce98b7a1e430f26b5a1b
Signed-off-by: Sergiy Markin <smarkin@mirantis.com>
2023-09-25 22:01:00 +00:00
Sergiy Markin 81066ae98f Airflow stable 2.6.2
This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:

- bionic py36 gates  were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
  airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
  items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
  schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
  new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2

Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
2023-08-30 16:04:47 +00:00
Ritchie, Frank (fr801x) 3fcc6e0d65 Use helm-toolkit for worker readiness probes
Use helm-toolkit for worker readiness probes.

Change-Id: Ice6c2ff11e007c28093d051c118d733c4e757788
2023-08-25 15:57:48 +00:00
Chris Wedgwood 034b906dd6 [airflow] fix ordering of affinity/psp in charts
Change-Id: I08afee40ac3965adc2bcff7f1a2a21ebb39fb87c
2021-11-03 14:08:25 -05:00
Sean Eagan 27b4dc952c Helm 3: Fix Job labels
See the dependency below for details.

Depends-On: https://review.opendev.org/c/openstack/openstack-helm-infra/+/811826
Change-Id: I100a68eb4cf457fba0783e41779f9fdc2c8daf78
2021-10-01 11:21:52 -05:00
Mahmoudi, Ahmad (am495p) a5e57879ab Override uwsgi default config
- Overrode uwsgi default configs to improve stability and performance.
- Increased mas number of worker processes to increase capacity and
  performance.
- Enabled uwsgi cheaper subsystem to scale worker processes dynamically.
- Uplifted uwsgi to the latest release to bring bug fixes and
  improvements since 2018.

Upgraded uwsgi to bring in bug fixes since 2018.

For background information for this change please see:
https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html

Change-Id: If067e9786e9dbbd39ef832dea6f51aa5523af4d7
2020-08-06 02:14:07 +00:00
KHIYANI, RAHUL (rk0850) db37122336 Implement helm-toolkit snippet to airflow pods/containers
This updates the airflow chart to include the pod
security context on the pod template.

This also adds the container security context to set
readOnlyRootFilesystem flag

Change-Id: I84cd4581d6ae915e9caf5c50d407dfcc34b962b3
2020-07-09 20:36:39 +00:00
Zuul 66d410779c Merge "Add configmap-hash annotations for Shipyard & Airflow" 2020-07-06 21:48:15 +00:00
DODDA, PRATEEK REDDY 5247fed4ba Add configmap-hash annotations for Shipyard & Airflow
Adds configmap-hash annotations to the job-db-init and job-db-sync
for configmap-bin and configmap-etc.

These annotations ensure that if configmaps change, the pods
are redeployed according to their upgrade strategy.

Change-Id: I59eb516086c4fd41f7c18923f86f135101656af8
2020-07-06 12:09:18 -05:00
KHIYANI, RAHUL (rk0850) 02929cfc44 Implement helm-toolkit snippet to shipyard pods/containers
This updates the shipyard chart to include the pod
security context on the pod template.

This also adds the container security context to set
readOnlyRootFilesystem flag to true

Change-Id: I2ffe17fc7d42aa5544e606f3a354496a64005640
2020-07-02 09:25:11 -05:00
DODDA, PRATEEK 9831e545c9 Enabling Apparmor profile to shipyard init containers
Remove OSH Authors copyright

The current copyright refers to a non-existent group
"openstack helm authors" with often out-of-date references that
are confusing when adding a new file to the repo.

This change removes all references to this copyright by the
non-existent group and any blank lines underneath.

Change-Id: Ic8de1678a754ba466dbd8d12c4f078151a78a091
2020-06-26 09:11:41 -05:00
Prateek Dodda e066274b8b Implement Security Context for Airflow_Scheduler
This adds the container security context to set readOnlyRootFilesystem
to true

Change-Id: Ia9cad50decfcf9638e8fc1cf5d652ee72d978a40
2020-04-24 17:03:24 -05:00
Zuul 8b6bd94410 Merge "Implement Security Context for Airflow_Worker" 2020-04-02 13:32:14 +00:00
Prateek Dodda cc0bfac0c2 Implement Security Context for Airflow_Worker
This adds the container security context to set
readOnlyRootFilesystem to true

Depends-on: https://review.opendev.org/#/c/708948/2
Change-Id: I4c7e7dba26d6bdfd0032a31469fd1777ae06cfec
2020-03-31 14:14:03 +00:00
KHIYANI, RAHUL (rk0850) 29824c78b4 [Fix] Fixing shipyard endpoints path name
fixing path helm-toolkit to shipyard

PS: https://review.opendev.org/#/c/671575/16

Change-Id: I73bf30e0c27365802b730d27e6ecfd28092de24e
2020-03-18 23:08:33 -05:00
NarlaSandeepNarlaSaibaba 5aa0cde5f0 Adding default apparmor profile to shipyard components
Change-Id: Idfc103c85bc95c8cd0a48aa0c18a17a4b1d12d3f
2020-02-17 09:52:37 -06:00
Evgeny L a9fc62e31f Allow to configure service network policy
The patch introduces network policy configuration similar
to openstack-helm services. It allows users to configure
policies depending on the environment.

* Network policies are disabled by default.
* When enabled default policies allow all ingress and
  egress traffic (i.e. policy set to {}), this may be
  changed in future patch-sets.

Change-Id: Ic0b44eb142445d45d81e3e546d394e1c7b451238
2019-11-04 18:49:11 +00:00
Hemanth Nakkina 0de9209ca1 Use apps/v1 k8s controllers and add labels
Update apiversion for ClusterRole, ClusterRoleBinding to rbac.authorization.k8s.io/v1
Update apiversion for deployment to apps/v1
Update apiversion for statefulset to apps/v1
Add selector match labels to deployment

This patch is similar to https://review.opendev.org/#/c/638276/
These changes are required to install shipyard helm chart on k8s 1.16.0

Change-Id: I7ac6fc060fbd6a5feea747ebbe8121c5a2eb4b6f
2019-10-02 14:25:06 +00:00
Evgeny L c21555fce0 Use a direct connectivity to RabbitMQ
This patch makes Celery to connect to RabbitMQ directly instead
of using LB. It also brings a forked version of a transport url
template, the reason for this is the format for Kombu/Celery
broker url is different from oslo_messaging transport url:
1. URLs need to be separated with semicolons vs commas.
2. Every item in Kombu broker url needs to be a complete url
   that includes schema, vhost, and all credentials.

This format is specific to Airflow and is not used in upstream
OSH projects, hence it is included here and not in htk.

Depends-On: I5150a64bd29fa062e30496c1f2127de138322863
Change-Id: I0b4ae6a9538f2f6988ed42c8f5cf0a54e7a7ad2e
2019-08-28 23:38:59 +00:00
Daniel Pawlik fe03770031 Revert "charts: Remove subpath from airflow volumes"
airflow.cfg file was mounted as a dir, not a file,
so airflow service doesn't want to start.

This reverts commit 6794903558.

Change-Id: I6db528ac91fc5cb6719831eb2915467105f4c491
2019-07-25 12:03:45 -05:00
Drew Walters 6794903558 charts: Remove subpath from airflow volumes
Recently, the airflow config mounts were changed to projected volumes to
workaround a K8s bug [0]; however, a subpath prevents the configs from
being properly mounted. This change removes the subpath.

[0] https://review.opendev.org/671944

Change-Id: I9bbe91d3e27b293a6fd27c00545329bc8a36f926
Signed-off-by: Drew Walters <andrew.walters@att.com>
2019-07-22 15:17:51 +00:00
anthony.bellino e29d826c4a [fix] Airflow Modified subpath configmap mount fails
Because of a kubernetes bug [0] when a container which
is mounted with the subpath option, the configmap is
changed and then the container restarts the mounting of
the configmap fails.

This PS uses the projected key for volume definitions
as a workaround.

[0] https://github.com/kubernetes/kubernetes/issues/68211

Change-Id: I6820a0f963c5b28e1674ea58214ffc86009db4dd
2019-07-21 21:04:39 +00:00
Zuul 8a99439453 Merge "Add realease uuid annotation to POD spec" 2019-06-26 13:54:14 +00:00
Kumar, Nishant(nk613n) 8fba639411 Add realease uuid annotation to POD spec
Change-Id: Id81fbc239a641bed5a1486f647a46bb9c3408584
2019-06-25 14:49:07 +00:00
Dejaeger, Darren (dd118r) 4c33f54fea Add node selector to test pod
This PS looks to add a node selector into the test pod's spec.

Change-Id: I11989d5106363e8c3c7d9950da1b4247cfef7539
2019-06-20 08:29:56 -04:00
Zuul 078388c224 Merge "Update Airflow logrotate logic" 2019-06-10 21:57:44 +00:00
Zuul 0b29f89d3f Merge "Store status of deployment in a ConfigMap" 2019-06-10 15:25:32 +00:00
anthony.bellino 5f92be2f07 Update Airflow logrotate logic
The current logrotate logic deletes logs that are
more than X days old in the Airflow log path, however
the Airflow log archive may still reach 100%
usage and cause the airflow-worker to crashloop.

This PS adds logic to logrotate.sh to delete the oldest
logs and empty dirs when the Airflow log archive
reaches the max usage specified in values.yaml.

Change-Id: I3dcb80901d7dd36da6812850a1f54e7ebf3b1cf2
2019-06-07 19:59:57 +00:00
anthony.bellino 89a8eda43e Add pod affinity to Shipyard and Airflow
This PS adds pod anti-affinity to airflow/shipyard pods,
so that the scheduler can constrain pods against labels on other pods
running on the node.   The default soft rule is in place so that if
the scheduler can’t satisfy the the requirement, the pod will still
be scheduled, and is overridable.


Change-Id: I67d0792a1f624044f8975c9540ab691f4e638b3f
2019-05-29 14:02:21 +00:00
Michael Beaver 53e863954b Store status of deployment in a ConfigMap
This change adds a new Shipyard Operator that creates/updates a
ConfigMap with information on the version and status of the current
running deployment. This ConfigMap will be created at the start of the
deployments, and will be updated at the end even if the previous steps
fail.

This operator has been added to the deploy_site, update_site, and
update_software DAGs.

Change-Id: Iab9ea84d5e1edd6a8635cc4e4fa93647ee485194
2019-05-28 08:32:07 -05:00
Rahul Khiyani 25defd8ca7 Shipyard: Add pod/container security context
- deployment-shipyard

This updates the shipyard chart to include the pod
security context on the pod template.

This also adds the container security context to set
allowPrivilegeEscalation to false and readOnlyRootFilesystem to true

Change-Id: Idb1b848847eaec2b6e24389c063b7ece2973c4dc
2019-04-02 15:37:59 +00:00
Nishant Kumar d9f145e2a6 [Database] Shipyard DB changes
- Use helm-toolkit for DB initialization [0]
- Create DB auxiliary Job for shipyard specific
  additonal DB operations
- Refactor Job dependencies

[0] https://review.openstack.org/#/c/635348/

Depends-On: https://review.openstack.org/#/c/635348/

Change-Id: I093671f9bce747b491f22dd8f38f597bd9dae9af
2019-03-07 15:16:39 +00:00
Bryan Strassner a11e962eef Move Airflow web container into Shipyard pod
Moves the airflow web server container from its own pod into the
Shipyard pod. This removes exposed network surface area from the
Shipyard suite of software. Shipyard, after this change accesses the
Airflow API using localhost in the same k8s pod.

Change-Id: Ied4bd415a8d78c393b7256ead27a6a2176f4a2d6
2019-01-29 09:41:16 -06:00
Bryan Strassner 9725b0f337 Build workflows into Airflow image
Changes to make the docker image build to include the workflows from
Shipyard, rather than adding them to the container during Helm install
of Shipyard. This also removes the "prod" switch, as it is now always
built the same way, with the workflows in place.

Change-Id: I4acd6195cbec32193e15621e75ccaeb9879455f5
2019-01-29 09:41:16 -06:00
Bryan Strassner 6b75c7119a Move airflow scheduler to worker statefulset
Moves the airflow sceduler to a container in the airflow-worker
statefulset so that its version lifecycle matches that of the worker.
Leaves the stand-alone scheduler in place to support upgradability from
prior installations that included a standalone scheduler. New
installations are advised to turn off the scheduler template from
rendering using the values.yaml flag.

This is an attempt to make disruptive upgrades to airflow less impactful
to a "update_site" action from Shipyard.

Additionally this removes the template for airflow-flower, which is not in use.

Change-Id: I0608793ee6aba1eb3ce0f5e9567655287014a0ca
2019-01-29 09:41:16 -06:00
Vladyslav Drok 2134a87875 Setup mirrored queues for celery in rabbit
This change starts setting up mirrored queues by default.
If there is only one rabbitmq pod present this will still
work, though will cause some performance overhead.

Depends-On: https://review.openstack.org/617812
Depends-On: https://review.openstack.org/617817
Change-Id: I8982aed699185f9b7fb4962e108eb76377643f25
2018-12-17 19:37:28 +00:00
Drew Walters 54224ea98d Add Shipyard profiler
This commit adds the Werkzeug ProfilerMiddleware to Shipyard API
requests. This option can be enabled using the
`conf.shipyard.base.profiler` option and should not be used in
production.

Change-Id: I293840d78baf670478047faad87fdcfe2f8af70e
2018-10-05 13:55:58 +00:00
Rick Bartra 9eb430566b Make airflow-worker containers non-privileged
The 'airflow-worker' and 'airflow-logrotate' containers do not need to
run as privileged containers to perform their jobs. Shipyard deploy_site
action was used to test the 'airflow-worker' as a deploy_site invokes
'airflow-worker'. When performing deploy_site action, all steps succeeded
and the 'airflow-worker' shows no errors when 'airflow-worker' is
non-privileged.

When 'airflow-logrotate' runs as non-privileged, the 'airflow-logrotate'
container still logs correctly and is able to delete/rotate logs without
problems.

Note: Making airflow-worker run with non-privileged containers means that
these containers will use the docker-default apparmor profile by default.

Change-Id: I26eda3eb8b7a36e67c2e7b593326f1d063600fc3
2018-09-27 15:14:20 -04:00
Bryan Strassner 44c526af96 Update to Airflow 1.10
Updates the image building to Airflow 1.10, including necessary
configuration changes and a general update of dependencies.

Airflow 1.10 includes many enhancements and bugfixes since 1.9 [0]

This change introduces many "unused" configuration parameters to satsify
Airflow's expectations[1].  An ugly, but likely harmless change to the
log output with interleaved newline characters from Airflow steps[2].

Changes to the chart and other dependendencies have also been introduced
to match this update.

[0] https://github.com/apache/incubator-airflow/blob/master/CHANGELOG.txt
[1] https://issues.apache.org/jira/browse/AIRFLOW-3099
[2] https://issues.apache.org/jira/browse/AIRFLOW-1917

Change-Id: I179dcf1f0369650b8c4519f704abb7fb495f4248
2018-09-24 15:32:31 -05:00
Matt McEuen ae688e7fb4 Add release uuid to pods and rc objects (shipyard)
This PS adds the ability to attach a release uuid to pods and rc
objects as desired.  This can be used, for example, to force an
artificial manifest change in CICD scenarios, for upgradability
testing purposes.

Change-Id: I75fc7516e0d32e7e0df0fecf1f9bec0e234adfcc
2018-09-14 15:55:04 -05:00
Bryan Strassner e59fb314c1 Set ULID of action on DAG request
Sets the run_id for a DAG invoked in Airflow to the same ULID assigned
to it in Shipyard. While this was already happening as a parameter to
the DAG being invoked, by making it the run_id, further correlation is
possible, at a level that both Shipyard and the Airflow framework are
aware.

As part of making this change, fragility was uncovered in the
rest_api_plugin that expedited the need to switch to the built-in, but
experimental airflow API to trigger a dag (one of two API endpoints
provided - this is important later in this story). In any case, the 3rd
party rest_api_plugin was removed.

As a result of the rest_api_plugin being removed:
1) the simpleton helm test to check the api of airflow was also removed
(it used the version endpoint of this plugin). As the built-in api
provides no version endpoint or similarly accessible-without-being-stateful
endpoint, the helm test had no new place to look for something to call.
2) Some clean up of exclusions and documentation was possible - test
coverage, security exclusions, left over documentation remnants

Change-Id: I0b68496a8500408b776b4acc12888aa017c4c7d2
2018-08-10 10:23:30 -05:00
Aaron Sheffield 16cc15f856 Add test pods labels.
- Uses helm toolkit to add labels to test pods.

Change-Id: I8796379b0370fb41c1a519023b49139b5401810e
2018-07-11 08:49:49 -05:00
Bryan Strassner 2651f6e831 Separate source of node_selector_key and values
Provides different fields in the values.yaml to use for the node
selector keys/values used in deployments vs jobs.

Change-Id: I12d7c6257aea0ac00cd77cd3f6331a2b7380b589
2018-07-06 04:22:38 +00:00
Pete Birley a7b6d184a5 Update chart to support TLS for Shipyard
Adds the secret to support TLS for the Shipyard API

Change-Id: I34d753bc0c65b00df54aeb32ff66eef5bf2c4c6e
Co-Authored-By: Pete Birley <pete@port.direct>
Signed-off-by: Pete Birley <pete@port.direct>
2018-06-27 18:21:54 -05:00
anthony.lin c7a9c65c88 Update Shipyard Chart - HTK OSH Infra
Updated configurations to point to openstack-helm-infra
for reference to helm-toolkit as helm-toolkit has been
removed from the openstack-helm repo [0]

Also aligned with changes to the keystone user set up in
OSH using Helm ToolKit so as to get pass Helm Lint.

Updated Makefile targets to install helm dynamically

[0] https://review.openstack.org/#/c/558065/

Change-Id: I0a0813516f9ad176ff005b4693e6b933013a99fd
2018-06-25 17:14:28 -05:00
Anthony Lin 14d66afb01 Update Shipyard API Pod Labels
As part of ongoing effort to update the "application" and
"component" labels for the UCP components, there is a need
to align with the convention. We will update the label for
the shipyard API pod in this case.

Also updated helm_tk.sh to point to openstack-helm-infra for
reference to helm-toolkit as helm-toolkit has been removed
from the openstack-helm repo [0]

[0] https://review.openstack.org/#/c/558065/

Change-Id: I0b2acda47d87f8dda35fbf054e1c8d906b495061
2018-05-15 14:40:38 +00:00