Commit Graph

147 Commits

Author SHA1 Message Date
Sergiy Markin bd67c08136 Disable airflow webserver UI by default
This PS disables Airflow Webserver UI by default.

Change-Id: Ic4734e8159db8b44675fcbced2abb1858f1813fc
2024-03-19 18:01:29 +00:00
Sergiy Markin 580c42e0ac Airflow stable 2.8.2
This PS bumps up Airflow version to the latest
2.8.2 and also bumps up openstack dependences to
Antelope 2023.1

Change-Id: I7bbcbc8f9e6334100f47702f4546f0963d11b2ca
2024-02-27 19:04:31 +00:00
Sergiy Markin 20593bc746 Airflow stable 2.8.1
Change-Id: Iac800d9bc6e53a5a60e11c649d05dc663e8dd8e9
2024-02-23 19:43:03 +00:00
Anselme, Schubert (sa246v) 2a6c028a41
Enable TLS for celery
This PS enable TLS connection from celery to rabbitmq
when TLS connection is enabled

Change-Id: I49ccf159ca73e0764703a6d3c686c108143f12e2
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
2023-12-06 15:48:29 -05:00
Anselme, Schubert (sa246v) f571611f3c
Enable TLS connection to rabbitmq
Change-Id: Ia4d65393ad0112ae63433a7f67a8c8706a15b216
Signed-off-by: Anselme, Schubert (sa246v) <sa246v@att.com>
2023-11-20 10:42:30 -05:00
Sergiy Markin 181da83d34 Airflow webserver UI
This PS adds deployment with Airflow webserver UI in viewer mode
protected by base http autorization and exposed via ingress.

Change-Id: I5692eecf5a9af2930f8cce98b7a1e430f26b5a1b
Signed-off-by: Sergiy Markin <smarkin@mirantis.com>
2023-09-25 22:01:00 +00:00
Sergiy Markin 99c2da745a Airflow fix
This PS is mainly fixing SubDAGs timing issues when they
started along with main DAG and not at the time the main
DAG needs them. In Airflow 2.6.2 SubDAGs are deprecated
in favor of TaskGroups. So in this PS all SubDAGS were
replaced with TaskGroups.

Also task level logging config was extended by adding
py-console to obtain logs from tasks like it was
configured in Airflow 1.10.5.

Change-Id: I3f6d3961b1511e3b7cd2f7aab9810d033cfc14a3
2023-09-15 19:14:39 +00:00
Sergiy Markin 81066ae98f Airflow stable 2.6.2
This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:

- bionic py36 gates  were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
  airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
  items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
  schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
  new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2

Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
2023-08-30 16:04:47 +00:00
Ritchie, Frank (fr801x) 3fcc6e0d65 Use helm-toolkit for worker readiness probes
Use helm-toolkit for worker readiness probes.

Change-Id: Ice6c2ff11e007c28093d051c118d733c4e757788
2023-08-25 15:57:48 +00:00
Sergiy Markin 9c28c832dd Shipyard timeout issue
This PS  adds default values for chart values and resolves some issues
in python code that utilizes these values:

      validation_connect_timeout: 20
      validation_read_timeout: 300
      deckhand_client_connect_timeout: 20
      deckhand_client_read_timeout: 300
      drydock_client_connect_timeout: 20
      drydock_client_read_timeout: 300

Change-Id: Ic5b1920257859239613a3ce77134e6b05bd7e9dd
2023-05-16 20:37:29 +00:00
Sergiy Markin b7e7d25308 Rollback apache-airflow back to 1.10.5
This PS is created to :
- roll back apache-airflow back to 1.10.5
- change default chart values from bionic to ubuntu_focal
- save freshly generated config and policy files samples
- in focal Dockerfile postgres client version is pubped up to v15
- change airflow docker image requirements from limited to fully frozen for shipyard-airflow project
- adjusted requirements-direct.txt for shipyard_airflow
- adjusted requirements-direct.txt for shipyard_client
- regenerated requirements-frozen for both projects
- fixed unit tests after upgrading click module
- gen_all tox profile processing has been moved over to py8 gate because it requires focal zuul node to run
- upgraded airskiff gate playbook to include latest treasuremap patchset with updated airskiff site admada manifests

Change-Id: I47e44f5cfa19b2649697e7cc5a31557a6f4fcfea
2023-05-08 21:58:34 +00:00
Markin, Sergiy (sm515x) ed5775c0ed [shipyard] Postgres upgrade to 14.5
This PS upgrades shipyard postgres to 14.5

Change-Id: I1524cca569015aa7c4068fbaa6cf9b3221821941
2022-09-23 13:14:12 +00:00
Chris Wedgwood 034b906dd6 [airflow] fix ordering of affinity/psp in charts
Change-Id: I08afee40ac3965adc2bcff7f1a2a21ebb39fb87c
2021-11-03 14:08:25 -05:00
Sean Eagan a0da68409a Use helm 3 in chart build
`helm serve` is removed in helm 3 so this moves
to using local `file://` dependencies [0] instead.

[0]: https://helm.sh/docs/chart_best_practices/dependencies/#repository-urls

Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: Ibdea9f1ccda043259620add022643e5c9800259a
2021-10-01 11:48:37 -05:00
Sean Eagan 27b4dc952c Helm 3: Fix Job labels
See the dependency below for details.

Depends-On: https://review.opendev.org/c/openstack/openstack-helm-infra/+/811826
Change-Id: I100a68eb4cf457fba0783e41779f9fdc2c8daf78
2021-10-01 11:21:52 -05:00
Andrii Ostapenko af63252e8d
Change helm-toolkit dependency version to ">= 0.1.0"
Since we introduced chart version check in gates, requirements are not
satisfied with strict check of 0.1.0

Change-Id: I0b52182baa9a0541b4e79a66e64829f2619e91b5
2020-09-24 19:43:13 -05:00
Mahmoudi, Ahmad (am495p) a5e57879ab Override uwsgi default config
- Overrode uwsgi default configs to improve stability and performance.
- Increased mas number of worker processes to increase capacity and
  performance.
- Enabled uwsgi cheaper subsystem to scale worker processes dynamically.
- Uplifted uwsgi to the latest release to bring bug fixes and
  improvements since 2018.

Upgraded uwsgi to bring in bug fixes since 2018.

For background information for this change please see:
https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html

Change-Id: If067e9786e9dbbd39ef832dea6f51aa5523af4d7
2020-08-06 02:14:07 +00:00
KHIYANI, RAHUL (rk0850) 841d0ee2c8 Add the missing readOnly-fs flag for airflow-web containers
Change-Id: Ic31288d2f88c9610621ef1b74d8de813ea985ca8
2020-07-17 16:00:26 +00:00
KHIYANI, RAHUL (rk0850) db37122336 Implement helm-toolkit snippet to airflow pods/containers
This updates the airflow chart to include the pod
security context on the pod template.

This also adds the container security context to set
readOnlyRootFilesystem flag

Change-Id: I84cd4581d6ae915e9caf5c50d407dfcc34b962b3
2020-07-09 20:36:39 +00:00
Zuul 66d410779c Merge "Add configmap-hash annotations for Shipyard & Airflow" 2020-07-06 21:48:15 +00:00
DODDA, PRATEEK REDDY 5247fed4ba Add configmap-hash annotations for Shipyard & Airflow
Adds configmap-hash annotations to the job-db-init and job-db-sync
for configmap-bin and configmap-etc.

These annotations ensure that if configmaps change, the pods
are redeployed according to their upgrade strategy.

Change-Id: I59eb516086c4fd41f7c18923f86f135101656af8
2020-07-06 12:09:18 -05:00
KHIYANI, RAHUL (rk0850) 02929cfc44 Implement helm-toolkit snippet to shipyard pods/containers
This updates the shipyard chart to include the pod
security context on the pod template.

This also adds the container security context to set
readOnlyRootFilesystem flag to true

Change-Id: I2ffe17fc7d42aa5544e606f3a354496a64005640
2020-07-02 09:25:11 -05:00
DODDA, PRATEEK 9831e545c9 Enabling Apparmor profile to shipyard init containers
Remove OSH Authors copyright

The current copyright refers to a non-existent group
"openstack helm authors" with often out-of-date references that
are confusing when adding a new file to the repo.

This change removes all references to this copyright by the
non-existent group and any blank lines underneath.

Change-Id: Ic8de1678a754ba466dbd8d12c4f078151a78a091
2020-06-26 09:11:41 -05:00
Prateek Dodda e066274b8b Implement Security Context for Airflow_Scheduler
This adds the container security context to set readOnlyRootFilesystem
to true

Change-Id: Ia9cad50decfcf9638e8fc1cf5d652ee72d978a40
2020-04-24 17:03:24 -05:00
Zuul 8b6bd94410 Merge "Implement Security Context for Airflow_Worker" 2020-04-02 13:32:14 +00:00
Prateek Dodda cc0bfac0c2 Implement Security Context for Airflow_Worker
This adds the container security context to set
readOnlyRootFilesystem to true

Depends-on: https://review.opendev.org/#/c/708948/2
Change-Id: I4c7e7dba26d6bdfd0032a31469fd1777ae06cfec
2020-03-31 14:14:03 +00:00
KHIYANI, RAHUL (rk0850) 29824c78b4 [Fix] Fixing shipyard endpoints path name
fixing path helm-toolkit to shipyard

PS: https://review.opendev.org/#/c/671575/16

Change-Id: I73bf30e0c27365802b730d27e6ecfd28092de24e
2020-03-18 23:08:33 -05:00
NarlaSandeepNarlaSaibaba 5aa0cde5f0 Adding default apparmor profile to shipyard components
Change-Id: Idfc103c85bc95c8cd0a48aa0c18a17a4b1d12d3f
2020-02-17 09:52:37 -06:00
Ahmad Mahmoudi 24f6a01e0b Add support for Ubuntu bionic base image
Added support to buid shipyard and airflow images using either a xenial
or Ubuntu bionic base image.
The default base image is set to bionic.

Change-Id: I6ad4d42dede081586b3ccea87a42e250979ac106
2020-02-04 13:38:39 -06:00
Evgeny L a9fc62e31f Allow to configure service network policy
The patch introduces network policy configuration similar
to openstack-helm services. It allows users to configure
policies depending on the environment.

* Network policies are disabled by default.
* When enabled default policies allow all ingress and
  egress traffic (i.e. policy set to {}), this may be
  changed in future patch-sets.

Change-Id: Ic0b44eb142445d45d81e3e546d394e1c7b451238
2019-11-04 18:49:11 +00:00
Daniel Pawlik 1fc6829de8 Fix airskiff deployment
Change-Id: I800200e851a8c896696dbc97b56a8c997ff716ea
2019-10-25 11:49:25 +00:00
Ahmad Mahmoudi f0f876afef (airflow) - Apache airflow uplift to 1.10.4
Updated airflow config and Dockerfiles for apache airflow 1.10.4
For details see: https://github.com/apache/airflow/blob/master/UPDATING.md

Also updated the kubectl to 1.16.2, as part of kubernetes 1.16 uplift.

Change-Id: Ib24ff0304b6279ff0be749141854d6a604473597
2019-10-22 14:46:57 +00:00
Zuul 0cbb5cf9b7 Merge "Use apps/v1 k8s controllers and add labels" 2019-10-08 20:43:24 +00:00
Roman Gorshunov c65a93a895 Change URLs to opendev.org due to migration
Change URLs from git.openstack.org & github.com to opendev.org due to the
migration; wrap multiple LABELs into one.

Change-Id: I240fa6f746bd1e424e5a2b7fd381903c46059ae5
2019-10-07 18:59:45 +02:00
Hemanth Nakkina 0de9209ca1 Use apps/v1 k8s controllers and add labels
Update apiversion for ClusterRole, ClusterRoleBinding to rbac.authorization.k8s.io/v1
Update apiversion for deployment to apps/v1
Update apiversion for statefulset to apps/v1
Add selector match labels to deployment

This patch is similar to https://review.opendev.org/#/c/638276/
These changes are required to install shipyard helm chart on k8s 1.16.0

Change-Id: I7ac6fc060fbd6a5feea747ebbe8121c5a2eb4b6f
2019-10-02 14:25:06 +00:00
Evgeny L c21555fce0 Use a direct connectivity to RabbitMQ
This patch makes Celery to connect to RabbitMQ directly instead
of using LB. It also brings a forked version of a transport url
template, the reason for this is the format for Kombu/Celery
broker url is different from oslo_messaging transport url:
1. URLs need to be separated with semicolons vs commas.
2. Every item in Kombu broker url needs to be a complete url
   that includes schema, vhost, and all credentials.

This format is specific to Airflow and is not used in upstream
OSH projects, hence it is included here and not in htk.

Depends-On: I5150a64bd29fa062e30496c1f2127de138322863
Change-Id: I0b4ae6a9538f2f6988ed42c8f5cf0a54e7a7ad2e
2019-08-28 23:38:59 +00:00
Daniel Pawlik fe03770031 Revert "charts: Remove subpath from airflow volumes"
airflow.cfg file was mounted as a dir, not a file,
so airflow service doesn't want to start.

This reverts commit 6794903558.

Change-Id: I6db528ac91fc5cb6719831eb2915467105f4c491
2019-07-25 12:03:45 -05:00
Drew Walters 6794903558 charts: Remove subpath from airflow volumes
Recently, the airflow config mounts were changed to projected volumes to
workaround a K8s bug [0]; however, a subpath prevents the configs from
being properly mounted. This change removes the subpath.

[0] https://review.opendev.org/671944

Change-Id: I9bbe91d3e27b293a6fd27c00545329bc8a36f926
Signed-off-by: Drew Walters <andrew.walters@att.com>
2019-07-22 15:17:51 +00:00
anthony.bellino e29d826c4a [fix] Airflow Modified subpath configmap mount fails
Because of a kubernetes bug [0] when a container which
is mounted with the subpath option, the configmap is
changed and then the container restarts the mounting of
the configmap fails.

This PS uses the projected key for volume definitions
as a workaround.

[0] https://github.com/kubernetes/kubernetes/issues/68211

Change-Id: I6820a0f963c5b28e1674ea58214ffc86009db4dd
2019-07-21 21:04:39 +00:00
Roman Gorshunov da4426d173 Set default images to Ubuntu Xenial -based images
Since ':master' and ':latest' Shipyard and Airflow images are outdated,
set defaults to Ubuntu Xenial -based images.

Change-Id: I40978747f31c6a8c5cc8689a9768f8c4836ac1a1
2019-07-12 21:21:58 +02:00
Zuul 8a99439453 Merge "Add realease uuid annotation to POD spec" 2019-06-26 13:54:14 +00:00
Kumar, Nishant(nk613n) 8fba639411 Add realease uuid annotation to POD spec
Change-Id: Id81fbc239a641bed5a1486f647a46bb9c3408584
2019-06-25 14:49:07 +00:00
Dejaeger, Darren (dd118r) 4c33f54fea Add node selector to test pod
This PS looks to add a node selector into the test pod's spec.

Change-Id: I11989d5106363e8c3c7d9950da1b4247cfef7539
2019-06-20 08:29:56 -04:00
Zuul 078388c224 Merge "Update Airflow logrotate logic" 2019-06-10 21:57:44 +00:00
Zuul 0b29f89d3f Merge "Store status of deployment in a ConfigMap" 2019-06-10 15:25:32 +00:00
anthony.bellino 5f92be2f07 Update Airflow logrotate logic
The current logrotate logic deletes logs that are
more than X days old in the Airflow log path, however
the Airflow log archive may still reach 100%
usage and cause the airflow-worker to crashloop.

This PS adds logic to logrotate.sh to delete the oldest
logs and empty dirs when the Airflow log archive
reaches the max usage specified in values.yaml.

Change-Id: I3dcb80901d7dd36da6812850a1f54e7ebf3b1cf2
2019-06-07 19:59:57 +00:00
anthony.bellino 89a8eda43e Add pod affinity to Shipyard and Airflow
This PS adds pod anti-affinity to airflow/shipyard pods,
so that the scheduler can constrain pods against labels on other pods
running on the node.   The default soft rule is in place so that if
the scheduler can’t satisfy the the requirement, the pod will still
be scheduled, and is overridable.


Change-Id: I67d0792a1f624044f8975c9540ab691f4e638b3f
2019-05-29 14:02:21 +00:00
Michael Beaver 53e863954b Store status of deployment in a ConfigMap
This change adds a new Shipyard Operator that creates/updates a
ConfigMap with information on the version and status of the current
running deployment. This ConfigMap will be created at the start of the
deployments, and will be updated at the end even if the previous steps
fail.

This operator has been added to the deploy_site, update_site, and
update_software DAGs.

Change-Id: Iab9ea84d5e1edd6a8635cc4e4fa93647ee485194
2019-05-28 08:32:07 -05:00
Carter, Matthew (mc981n) 0761099337 Validate existence of "deployment-version" during create configdocs
This PS adds funtionality to Shipyard to validate the existence of
the Pegleg-generated "deployment-version" document (Pegleg change id:
I7919b02d70c9797f689cdad85066d3953b978901) when a user runs create
configdocs. This validation only checks the presence of the document
(by name and schema) and does not care about the document's other
contents.
The severity of a failed validation is configurable through the
"validations" config section in shipyard.conf. The default severity
is "Skip", meaning the validation is not ran at all.

Note that with the default configuration of new validation, Shipyard
functionality should be unchanged.

Change-Id: I754617de81f628a24232e890b12b157ba6731c25
2019-05-09 08:33:52 -05:00
Carter, Matthew (mc981n) 64171aacf4 Validate existence of "deployment-version" doc on configdocs commit
This PS adds funtionality to Shipyard to validate the existence of
the Pegleg-generated "deployment-version" document (Pegleg change id:
I7919b02d70c9797f689cdad85066d3953b978901). As implied, this new
validation only checks for the presence of the document (by name and
schema) and currently does not care about any of the document's
contents under "data".
The severity of a failed validation is configurable through the new
"validations" configuration section in shipyard.conf, and is
defaulted to skip the validation altogether. This means that by
default, this patch set does not alter the functionality of Shipyard

Note that with the default configuration of this new validation,
Shipyard functionality should be unchanged.

Change-Id: I5e7269066f769804710a0fd1f2c8d0aece0d3314
2019-05-09 08:25:37 -05:00