Commit Graph

203 Commits

Author SHA1 Message Date
Sergiy Markin 580c42e0ac Airflow stable 2.8.2
This PS bumps up Airflow version to the latest
2.8.2 and also bumps up openstack dependences to
Antelope 2023.1

Change-Id: I7bbcbc8f9e6334100f47702f4546f0963d11b2ca
2024-02-27 19:04:31 +00:00
Sergiy Markin 20593bc746 Airflow stable 2.8.1
Change-Id: Iac800d9bc6e53a5a60e11c649d05dc663e8dd8e9
2024-02-23 19:43:03 +00:00
Sergiy Markin 1177707fae Airflow fix #2
This PS fixes one more SubDAG, makes sure that decide_airflow_upgrade
task works correctly after we previously replaced armada_build SubDAG
with corresponding TaskGroup. Also a related unit test has been fixed.

Change-Id: Ic8304cc1985f8197d363861c588b8b48e13fe5d4
2023-09-15 22:37:22 +00:00
Sergiy Markin 99c2da745a Airflow fix
This PS is mainly fixing SubDAGs timing issues when they
started along with main DAG and not at the time the main
DAG needs them. In Airflow 2.6.2 SubDAGs are deprecated
in favor of TaskGroups. So in this PS all SubDAGS were
replaced with TaskGroups.

Also task level logging config was extended by adding
py-console to obtain logs from tasks like it was
configured in Airflow 1.10.5.

Change-Id: I3f6d3961b1511e3b7cd2f7aab9810d033cfc14a3
2023-09-15 19:14:39 +00:00
Sergiy Markin 1ba0e72628 Shipyard fix
This PS has to fix select tasks by id action.

Change-Id: I6503a48fb43793b2aae91dc7e5549f6bc3818d4a
2023-08-31 19:40:13 +00:00
Sergiy Markin 81066ae98f Airflow stable 2.6.2
This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:

- bionic py36 gates  were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
  airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
  items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
  schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
  new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2

Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
2023-08-30 16:04:47 +00:00
Sergiy Markin 7ea83c25ef Restored ubuntu_bionic image build
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.

Change-Id: I4c4c7186e67c771f8249b2017b2507aac17760b7
2023-05-30 22:39:59 +00:00
Sergiy Markin 9c28c832dd Shipyard timeout issue
This PS  adds default values for chart values and resolves some issues
in python code that utilizes these values:

      validation_connect_timeout: 20
      validation_read_timeout: 300
      deckhand_client_connect_timeout: 20
      deckhand_client_read_timeout: 300
      drydock_client_connect_timeout: 20
      drydock_client_read_timeout: 300

Change-Id: Ic5b1920257859239613a3ce77134e6b05bd7e9dd
2023-05-16 20:37:29 +00:00
Sergiy Markin 296853f159 Uplift airship modules
This PS uplifts airship dependent modules.

Change-Id: Ib17cd12c6b88fbb618af68a2112e315c2cc9c0fd
2023-05-10 13:19:44 +00:00
Sergiy Markin b7e7d25308 Rollback apache-airflow back to 1.10.5
This PS is created to :
- roll back apache-airflow back to 1.10.5
- change default chart values from bionic to ubuntu_focal
- save freshly generated config and policy files samples
- in focal Dockerfile postgres client version is pubped up to v15
- change airflow docker image requirements from limited to fully frozen for shipyard-airflow project
- adjusted requirements-direct.txt for shipyard_airflow
- adjusted requirements-direct.txt for shipyard_client
- regenerated requirements-frozen for both projects
- fixed unit tests after upgrading click module
- gen_all tox profile processing has been moved over to py8 gate because it requires focal zuul node to run
- upgraded airskiff gate playbook to include latest treasuremap patchset with updated airskiff site admada manifests

Change-Id: I47e44f5cfa19b2649697e7cc5a31557a6f4fcfea
2023-05-08 21:58:34 +00:00
Sergiy Markin 154a099b28 Shipyard upgrade for focal
- upgraded Airflow to 1.10.15  -
  https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html
- disabled xenial, bionic and opensuse images build  gates
- added focal image build  gate
- added focal zuul build node
- adjusted Makefile for focal
- added bindep.txt to utilize bindep zuul base role for zuul build node
  pre-setup
- added focal Dockerfile
- implemented freeze requirements.txt approach like in other Airship
  projects
- removed specific requirements.txt for airflow in favor of using
  requirements-frozen.txt from shipyard_airflow project when building
  airflow docker image
- fixed docker image publishing to Quay
- replaces deprecated LOG.warn with new LOG.warning call
- replaced deprecated body attribute in responce wiht responce.text
  attribute
- update of falcon module deprecated .API call - replaced wiht
  falcon.App call
- deprecated routing.create_http_method_map method replaced with
  routing.map_http_methods
- re-formatted code tabulations based on yapf recommendations
- replaced deprecated protocol attribute in Pytest create_environ() with
  http_version attribute
- replaced deprecated app attribute in Pytest create_environ() with
  root_path attribute
- fixed airflow CLI commands to match 1.10.15 version
- updated zuul gates to work on focal nodes and added focal specific
  node setup items by adding appriate ansible tasks and roles
- uplifted Helm to 3.9.4
- uplifted stable HTK commit id
- updated tox.in to with with tox v4
- uplifted dependences references to other Airship projects
- common python dependences were syncronized with other Airship
  projects(Promenade, Deckhand, Armada, Drydock)
- fixed airskiff deployment gate
- fixed genconfig* profiles in shipyard-airflow tox.ini responsible for
  maintanance of policy.yaml.sample and shipyard.conf.sample

Change-Id: I0c85187dc9bacf0849382563dd5ff7e9b2814c59
2023-04-28 20:40:50 +00:00
Ruslan Aliev e3e71f7d16 Bump airship dependencies
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I3bc287af7a89410b7e7e248673a854477360f9ac
2022-09-28 09:51:39 -05:00
Ruslan Aliev 8240608fe5 Loose pyyaml requirements
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I2a3cf7b3af74db3331329931761b5686988f0ac0
2022-07-14 15:12:47 -05:00
Ruslan Aliev 9a8218aa25 Bump airship dependencies
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I61d65874864f233372df72787cdb25a54b2edbc3
2022-07-07 14:14:51 -05:00
Phil Sphicas 7dcada80f1 Uplift Armada client, enable socket keepalives
Connections from airflow-worker can be long-lived (e.g. when applying
large Armada manifests), and are prone to disconnection due to network
inactivity.

This change uplifts the Armada client to enable socket keepalives in
order to avoid this issue. (Promenade is also uplifted to address a
requirements conflict.)

Change-Id: I067a5edbbdff06f45394e772ebb16d5012987b6c
2022-03-21 13:54:38 -07:00
Phil Sphicas 0b985d60a9 Gate fixes: remove py35, make xenial non-voting
Update Zuul config and tox settings to not run python3.5 jobs, and to
make the xenial image jobs non-voting.

Change-Id: Icc9f71e60e6b48254917ea1d41ce9bc823cce512
2021-10-18 12:43:13 -07:00
Sean Eagan e64f17b91b Remove Helm 2 based Armada integrations
- Tiller status check
- Test cleanup arg removed

Depends-On: https://review.opendev.org/c/airship/armada/+/812047
Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: I77ef3fb8e952ad28132e3476138d34bbb5a6fd3d
2021-10-04 15:03:04 -05:00
Maximilian Weiss 88286bc690 Gate fixes for Shipyard
* Fixed typo for jsonschema version

* Setuptools version forced to <58 to allow usage of use_2to3, which
  some dependencies require

* Fixed scripting error that allowed Airflow installation to silently
  fail

Change-Id: I237801488795df07f2d4de1cde6cd4ec41182e31
2021-09-09 20:23:05 +00:00
anthony.bellino 00200ef98d Gate fixes for Shipyard
* Install older version of pip<21.0 for ubuntu_xenial images
* Install setuptools via pip for ubuntu_xenial images
* Pin typing-extensions to 3.7.2 and apache-airflow to 1.10.5
* Move promenade Dependencies under UCP components in requirements.txt
* With apache-airflow=1.10.5, strip ANSI escape sequences
  in test_deployment_group_manager.py
* Update tox.ini to support apache-airflow=1.10.5
* airskiff gate fixes
  - Pin treasuremap to v1.9 branch
  - Pin openstack-helm-infra to master
  - Remove openstack client setup as it's not used

Change-Id: Iee4ce59fdceacb165120a69d11c44e6e47feaea8
2021-06-25 13:15:31 -07:00
Rick Bartra 35e5a7f796 Update pip package versions in preparation of pip 20.3
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].

These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.

Change-Id: Idd9ea0d57b5be063b133036cfc9ebaa69956f4fc
2020-09-30 01:19:51 +00:00
Phil Sphicas 4e7ff997a6 Accelerate YAML operations with LibYAML
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.

https://pypi.org/project/pylibyaml/

Change-Id: Ic3f7ba1ce6c4db1a4bc18fe1aef8e0675f7cd69e
2020-09-24 05:09:46 +00:00
Zuul 8b69cebeea Merge "fix: Gate fix for Shipyard docs and pep8 gates" 2020-09-15 19:52:36 +00:00
Rick Bartra d38078cce3 fix: Gate fix for Shipyard docs and pep8 gates
Now that the Shipyard gates are running on Focal(20.04), flake8, grpcio,
and pyflakes need to use a version compatible with python3.8 and Ubuntu
20.04 which is the default python version in Ubuntu focal. Also unpinning
setuptools so that it is compatible with python3.8.

Additionally, address pep8 violations that arise from using a newer
version of flake8

Change-Id: Idc3c5d66b48fc9e4497a71d1b640bcd2872c22eb
2020-09-15 18:16:45 +00:00
Bartra, Rick (rb560u) 1ae9a7c4be Improve logging of Shipyard _execute_task method during failure or time out
In the case where a Drydock task either fails or times out, it is not very
clear which Drydock task is being referred to. This commit updates the log
messages to include the Drydock task-id that has either failed or timed out.
Additionally, debug logging is added to log the Drydock task status. The reason
for this is because there are situations where a Drydock task is still running,
but on the Shipyard side the task has timed out and therefore, Shipyard shows
the task (i.e. prepare_nodes or deploy_nodes) as failed. It is good to get a point
in time reference to the task state for a couple of reasons:
1) To verify via logging that Shipyard is reporting Drydock tasks (success,
   failures, partial_successes, etc) accurately
2) Later on if Drydock shows a node as deployed, then the task can be
   queried and the state can be checked to see if the task was indeed a success.

Change-Id: I7050338e2f92ad548e639e2ea4059a520c27e686
2020-08-28 14:00:13 -04:00
Mahmoudi, Ahmad (am495p) a5e57879ab Override uwsgi default config
- Overrode uwsgi default configs to improve stability and performance.
- Increased mas number of worker processes to increase capacity and
  performance.
- Enabled uwsgi cheaper subsystem to scale worker processes dynamically.
- Uplifted uwsgi to the latest release to bring bug fixes and
  improvements since 2018.

Upgraded uwsgi to bring in bug fixes since 2018.

For background information for this change please see:
https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html

Change-Id: If067e9786e9dbbd39ef832dea6f51aa5523af4d7
2020-08-06 02:14:07 +00:00
Ahmad Mahmoudi e2a3d13875 Uplift celery to address memory leak
Uplifted celery to 4.4.2 to address memory leak issue raised
in:  https://github.com/celery/celery/issues/4843.
This issue is addressed in celery 4.4.2

Change-Id: I26c403513ba48308044d69130d33561b314fd209
2020-06-25 21:04:22 +00:00
DODDA, PRATEEK REDDY (pd2839) 0957ede4a5 (fix) Update amqp
Locked amgp to 2.6.0 as per the latest release on 06/01/2020.
  https://pypi.org/project/amqp/

Change-Id: I2edc5eb1959d71454236eaf14560fd38c4d1140b
2020-06-08 18:21:10 +00:00
Phil Sphicas c8c8934c77 Armada DAG: increase retries from 3 to 5
The current Armada DAG allows for 3 retries, reduced from 10 here:
bef8eecac1

This is sometimes insufficient, especially in cases where chart updates
of underlying Airship or Kubernetes components are upgraded and pods are
restarted underneath the airflow workers. The chart installation may be
successful, but an Armada retry may still be consumed.

This change increases the number of retries to 5. This will allow Armada
to progress further through the manifest if there is a disruption after
a chart is successfully installed. The tradeoff is that Armada may try
to repeatedly install a chart that keeps failing in the same way,
delaying the ultimate failure of the deployment.

Change-Id: I1fad7b1d95af061595680a76d24c6d323b365a67
2020-04-23 20:33:58 +00:00
Ahmad Mahmoudi 0091670583 (fix) Updated WTForms and cleaned psycopg2
1. Locked the WTForms to 2.2.1 to address the import issue with
   wtforms.widgets.HTMLString. WTForms 2.3.0 was released on
   April 21/2020. This release causes shipyard gate fail with
   import error for wtforms.widgets.HTMLString.
2. Deleted psycopg2==2.7.7, which is installed as a dependency of
   apache-airflow extra package postgres, and resoted the newer
   release psycopg2-binary==2.8.4, to be used instead.

Change-Id: I303a2c94ec409e97af1192ae892b8148fcdbb8d5
2020-04-22 17:06:08 +00:00
Ahmad Mahmoudi df751e302c Upgrade apache-airflow to 1.10.5
Upgraded apache-airflow to 1.10.5 to address the database migration issue
https://issues.apache.org/jira/browse/AIRFLOW-6046.

Change-Id: Ib8678cddb2deb4f5ef944d026e0d2633a326d9cc
2020-03-30 05:21:31 +00:00
Ahmad Mahmoudi 70410cc478 (fix) Address image build issues, bionic
- With bionic image based shipyard docker images, uwsgi crashes
  with segmentation fault, when it tries to load the psycopg2 library,
  causing the api become unreachable on both shipyard docker images.
  This happens because psycopg2 2.7.x and uwsgi binary wheels are built
  with incompatible ssl libraries. This patch upgrades psycopg2 to the
  latest release to address this issue.

- The existing image build script cannot run in a docker or a pod,
  based pipeline because of two reasons:
  - The build script runs a docker (docker-in-docker) and mounts a
    volume.
    In a dind case, volume bind mounts will not work, because the nested
    container will need the host file system's path for the source path.
  - The shipyard service listens to its exposed service port in the
    nested docker network namespace, which is not reachable from the host
    pod/container.
This patch address both of the above issues. It first creates the
container, copies needed config files to the container and then starts
it. Also it execs into the nested docker to access the shipyard services
in a dind (docker-in-dcoker) case.

Change-Id: Ifdfed539babab01608bfaef37001bb79cd3a080d
2020-03-10 03:23:05 +00:00
Ahmad Mahmoudi 3d987309b8 Locked Werkzeug package release to 0.16.1
The latest Werkzeug package release 1.0.0, released on February 8,
is not backwards compatible with the earler releases of this package,
which is used in Flask, used by apache airflow.

This causes shipyard make image job fail, with missing import errors
from the Werkzeug library.

This change locks the Werkzeuz package release to the last compatible
release with the apache airflow in shipyard.

Change-Id: I54dad4ccc1858f4d5986c6e8e9fbf8f5d9847158
2020-02-12 01:23:15 +00:00
Ahmad Mahmoudi 24f6a01e0b Add support for Ubuntu bionic base image
Added support to buid shipyard and airflow images using either a xenial
or Ubuntu bionic base image.
The default base image is set to bionic.

Change-Id: I6ad4d42dede081586b3ccea87a42e250979ac106
2020-02-04 13:38:39 -06:00
Roman Gorshunov c65a93a895 Change URLs to opendev.org due to migration
Change URLs from git.openstack.org & github.com to opendev.org due to the
migration; wrap multiple LABELs into one.

Change-Id: I240fa6f746bd1e424e5a2b7fd381903c46059ae5
2019-10-07 18:59:45 +02:00
Ahmad Mahmoudi e6c7a6e1a5 (airflow) - Requirements update for apache airflow
Updated airflow requirements list needed for apache airflow 1.10.4 and later.

Change-Id: I142e46e4fa202d216287ea2f663bd4811e47f0ba
2019-10-03 09:11:51 -05:00
Roman Gorshunov 7430cc5869 Update pytest and pin amqp version to previous one
- Pytest reference [0]
- amqp reference [1]

[0] https://github.com/pytest-dev/pytest/issues/3280
[1] b51d1d678e

Change-Id: I676f54abf27e27646ae9ded4f1c2648824b047c9
2019-10-02 15:22:00 +02:00
Bryan Strassner fe87c64f97 Update help docs for actions in Shipyard Client
- Adds the information related to the test_site action.
- Reformats, slightly, the output from 'shipyard help actions'
- Adds tests that use an externalized list of actions to keep the help
documentation in alignment with the actions supported in the API.

Change-Id: I2efd473da0dbf6c8cbadfc9fae575c303996c43b
2019-09-17 16:48:56 +00:00
Zuul f1c193a232 Merge "(airflow) - Updated deckhand reference" 2019-09-16 19:28:16 +00:00
Ahmad Mahmoudi 815eef7f5d (airflow) - Updated deckhand reference
Updated deckhand commit id, because apache airflow release 1.10.3 and
later use flask >=1.1.0, which needs Werkzeug library version >=0.15.0.
see: https://issues.apache.org/jira/browse/AIRFLOW-4900

The updated deckhand commit lets the Werkzeug package float to >=0.15.0.

Change-Id: I62d7f4e5eecb2f05035a1c9552544aa65e70ee3f
2019-09-16 14:55:44 +00:00
Carter, Matt (mc981n) 1e1fc065b8 Enhance/fix logging in document validator
Change-Id: Ie600986597143f51908005ec655ba78496ca4cd9
2019-09-14 06:58:56 +00:00
Scott Hussey 185dc9823c (airflow) Configure source of Airflow pkg
- Allow a Docker build-arg to specify the source of the Airflow
  python package in any format supported by pip

Change-Id: Ifa2dd62d91570035cff91ff07868b0fcf659add8
2019-09-13 11:09:18 -05:00
Svetlana Shturm d8332bcebb Add support domain-scoped token for CLI
Change-Id: I4c0b4a6de9598e234e0b49b9cfc0b4dc5daaf7be
2019-08-02 10:15:20 -05:00
Zuul 1472a22dce Merge "Update packages related to requests" 2019-07-31 19:30:53 +00:00
Zuul fa55f5697e Merge "Change message when no nodes found during update" 2019-07-31 16:26:29 +00:00
HUGHES, ALEXANDER (ah8742) bfcd24f63a Update packages related to requests
Pegleg Promenade and Shipyard should all use same versions of
packages when able.  Requests currently is giving a warning
in Pegleg:
ERROR: deckhand 0.0.1.dev657 has requirement urllib3==1.24, but
you'll have urllib3 1.24.3 which is incompatible.

Change-Id: I9f21203b4109e7542a952d68c7a6ffcdb2653026
2019-07-31 15:20:35 +00:00
Roman Gorshunov 8706ad7e65 Fix: use python3 for shipyard_airflow
Change-Id: I39f87aa34d48ddb8936da82362e1ea2b05d7fde5
2019-07-25 16:35:49 +02:00
Ian H Pittwood 9bf298fea7 Remove setuptools from requirements
Setuptools is currently listed as a requirement for shipyard-client, but
it is not used. Pegging setuptools forces downgrading when installing
shipyard-client, which can cause issues for packages that attempt to
utilize features found in newer versions of setuptools. Removing the
setuptools requirements will allow installing users to choose their own
version of setuptools to use.

Change-Id: I88570b69346ec7019e753bdd8681fd63d55dc8c8
2019-07-16 11:59:23 -05:00
HUGHES, ALEXANDER (ah8742) 4594b8a719 Allow user to specify token as env var
This change is going to be done in conjunction with Pegleg's shipyard
helper code [0] such that an auth token can be set as an environment
variable.  If present it will be used rather than authenticating
against Keystone.  This is a requested change by Darren Dejaeger so
that when using automation such as Jenkins pipelines the user is not
forced to provide their credentials to have shipyard automatically
obtain the token - instead they can obtain their token separately and
provide that token for shipyard to use.

[0]: https://opendev.org/airship/pegleg/src/branch/master/pegleg/engine/util/shipyard_helper.py

Change-Id: I859bff978069e8cc3502b952273237e84de83456
2019-06-25 12:46:48 +00:00
Ian H. Pittwood e45f28057d Update PyYAML to match other projects
Shipyard Client's version of PyYAML (3.13) conflicts with the version
used by Pegleg (5.1). This conflict can cause issues when trying to
install Pegleg as it uses Shipyard Client as a dependency. This update
to Shipyard Client's PyYAML version will fix the dependency conflict
with Pegleg and will also fix a possible security vulnerability
associated with older versions of PyYAML.

Upgrades PyYAML to version 5.1, which makes several changes:
- Deprecates yaml.load in favor of FullLoader and UnsafeLoader due to
yaml.load's vulnerability with code injection [0]
- Drops support for py33 and py26
- Changes to default_flow_style=False
- Several other bug fixes and additional changes

[0] https://nvd.nist.gov/vuln/detail/CVE-2017-18342

Change-Id: I8eb4bb02dcc99eab3d983640d6c769cadb3b97e8
2019-06-14 15:38:21 -05:00
Ian H. Pittwood efb21f2958 Update dependencies to match other projects
Deckhand's requirement for keystoneauth1 (3.11.1) does not match with
the version used by Shipyard Client (3.11.0). In addition, Shipyard
Client uses requests-2.19.1 which is an insecure version of requests.
Both of these requirements conflict with other Airship project
dependencies. These conflicts become apparent in Pegleg, which uses both
Shipyard Client and Deckhand. The updates made in this change make
Shipyard Client compatible with Pegleg's own dependencies and other
projects it imports from.

Upticks keystoneauth1 to 3.11.1, a bugfix which should not impact
Shipyard Client behavior.

Upticks requests to 2.20.0, which contains several bug fixes and
deprecates support for py26.

Change-Id: I8716c56351e90bca8e7dbdf672638a45f15c655a
2019-06-12 18:57:50 +00:00