This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2
Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.
Change-Id: I4c4c7186e67c771f8249b2017b2507aac17760b7
This PS is created to :
- roll back apache-airflow back to 1.10.5
- change default chart values from bionic to ubuntu_focal
- save freshly generated config and policy files samples
- in focal Dockerfile postgres client version is pubped up to v15
- change airflow docker image requirements from limited to fully frozen for shipyard-airflow project
- adjusted requirements-direct.txt for shipyard_airflow
- adjusted requirements-direct.txt for shipyard_client
- regenerated requirements-frozen for both projects
- fixed unit tests after upgrading click module
- gen_all tox profile processing has been moved over to py8 gate because it requires focal zuul node to run
- upgraded airskiff gate playbook to include latest treasuremap patchset with updated airskiff site admada manifests
Change-Id: I47e44f5cfa19b2649697e7cc5a31557a6f4fcfea
- upgraded Airflow to 1.10.15 -
https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html
- disabled xenial, bionic and opensuse images build gates
- added focal image build gate
- added focal zuul build node
- adjusted Makefile for focal
- added bindep.txt to utilize bindep zuul base role for zuul build node
pre-setup
- added focal Dockerfile
- implemented freeze requirements.txt approach like in other Airship
projects
- removed specific requirements.txt for airflow in favor of using
requirements-frozen.txt from shipyard_airflow project when building
airflow docker image
- fixed docker image publishing to Quay
- replaces deprecated LOG.warn with new LOG.warning call
- replaced deprecated body attribute in responce wiht responce.text
attribute
- update of falcon module deprecated .API call - replaced wiht
falcon.App call
- deprecated routing.create_http_method_map method replaced with
routing.map_http_methods
- re-formatted code tabulations based on yapf recommendations
- replaced deprecated protocol attribute in Pytest create_environ() with
http_version attribute
- replaced deprecated app attribute in Pytest create_environ() with
root_path attribute
- fixed airflow CLI commands to match 1.10.15 version
- updated zuul gates to work on focal nodes and added focal specific
node setup items by adding appriate ansible tasks and roles
- uplifted Helm to 3.9.4
- uplifted stable HTK commit id
- updated tox.in to with with tox v4
- uplifted dependences references to other Airship projects
- common python dependences were syncronized with other Airship
projects(Promenade, Deckhand, Armada, Drydock)
- fixed airskiff deployment gate
- fixed genconfig* profiles in shipyard-airflow tox.ini responsible for
maintanance of policy.yaml.sample and shipyard.conf.sample
Change-Id: I0c85187dc9bacf0849382563dd5ff7e9b2814c59
Update Zuul config and tox settings to not run python3.5 jobs, and to
make the xenial image jobs non-voting.
Change-Id: Icc9f71e60e6b48254917ea1d41ce9bc823cce512
We want to default to running all tox environments under python 3, so
set the basepython value in each environment.
We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.
We do not want to set the override once in testenv, because that
breaks the more specific versions used in default environments like
py35 and py36.
Change-Id: If0f86ecb4c18216daf158101dbd3663bda06956f
This patch set adds tox tests to .zuul.yaml
Prometheus scrapers are not running in the airship-shipyard
and hence removed to prevent airship-shipyard-ubuntu job
from failing.
Change-Id: Ib805946ea76664b7448e02b4c7e4adb40c4dbaf2
Refactor Shipyard to be better able to leverage common
packages and conform with the target UCP standard layout.
This change supports the same tox entrypoints at
the root level, but the preferred approach is to use make
targets defined in the Makefile such as 'make tests' and
'make lint'
The previous tox.ini has moved and been
tailored to the specifics of each subproject at
src/bin/*/tox.ini
Autotmatic generation of the policy and configuration
files has been removed from the sphinx build for now
but these files will be automatically generated locally
into the docs source by using a 'make docs' command.
This may need to be revisited later to re-enable the
automatic generation of these files such that readthedocs
would still support the project layout.
Change-Id: Ifdc1cd4cf35fb3c5923414c677b781a60a9bae42
Puts into place the DeploymentConfiguration yaml that
provides the options that should be configured by the site
design to the deployment (and update) workflows.
This change additionally refactors reused parts to common
modules as related to info passing (xcom)
Change-Id: Ib6470899b204dbc18d2a9a2e4f95540b3b0032b0
test_workflow_api.py contains the unit tests for the WorkflowResource and
WorkflowIdResource
.coveragerc was modified to omit the unit test files from test coverage
tox.ini was modified to generate an HTML report
Change-Id: I86c81fbd65739fc8cc2fd089898aad6d87f7fdc0
This PS adds basepython=python3.5 to avoid errors that pop up in
environments where python2.7 may be installed as well, e.g.:
pep8 runtests: commands[0] | flake8
./shipyard_client/cli/action.py:34:25: E999 SyntaxError: invalid syntax
./shipyard_client/api_client/base_client.py:28:27: E999 SyntaxError: invalid syntax
./shipyard_airflow/plugins/deckhand_operator.py:122:25: E225 missing whitespace around operator
./shipyard_airflow/control/ucp_logging.py💯28: E999 SyntaxError: invalid syntax
Change-Id: I0cb28110d7cb3dcdff93c6010adb5c009a90bb34
test_base_resource.py contains the unit tests for the BaseResource
test_shipyard_request_context.py contains the unit tests for the ShipyarRequestContext
Change-Id: Idad82c06828e5ce2ef4e2ceb9fe2f2805e5746a1
Introduces code coverage and expands the use of bandit to include
the shipyard_client. Triggers the run of bandit, coverage, and docs
to the base tox execution. Addresses several bandit issues by either
fixing or supplying # nosec to known "ok" uses of code that would
otherwise trigger a bandit error being reported.
This change also removes an artifact from refactoring (deckhand_client.py)
which moved to a subdirectory.
Change-Id: I47b4a6b945ade7ba42bc73a850848cd45c0b2390
This patch cleans up docs/*.py and update tox.ini's exclude target
as well as specifying pep8 to scan *.py files only.
Change-Id: Ic81cc81edd9da682581d9deb0a7a5404bb7aa4d9
This PS adds tooling and automation to automatically generate
Shipyard's documentation into feature-rich HTML pages that can
be hosted.
To run the documentation job, simply execute:
tox -e docs
Shipyard's .md documentation will have to be changed to .rst
in a future PS for sphinx to be able to render them into HTML.
Change-Id: Iba8cfd68e4905d49cd5a1dfd39d556044efdabbd
This PS makes shipyard mostly pep8 compliant, with the exeption of:
./shipyard_airflow/plugins/rest_api_plugin.py which will require some
rework to make compliant.
Change-Id: I913f7ac696eb5558c1fed7c4f4104211bf4b5876
This change introduces a large section of the API for the next major
version of Shipyard - the action api. By interfacing with Airflow,
Shipyard will invoke workflows and allow for controlling and querying
status of those workflows. Foundationally, this patchset introduces
a lot of framework code for other apis, including error handling
to a common output format, database interaction for persistence of
action information, and use of oslo_config for configuration support.
Add GET all actions primary code - db connection not yet impl
Update base classes to have more structure
Add POST actions framework
Add GET action by id
Add GET of validations and steps
Add control api
Add unit tests of action api methods
Re-Removed duplicate deps from test reqs
Add routes for API
Removed a lot of code better handled by falcon directly
Cleaned up error flows- handlers and defaults
Refactored existing airflow tests to match standard output format
Updated json validation to be more specific
Added basic start for alembic
Added alembic upgrade at startup
Added table creation definitions
Added base revision for alembic upgrade
Bug fixes - DB queries, airflow comm, logic issues, logging issues
Bug fixes - date formats and alignment of keys between systems
Exclusions to bandit / tox.ini
Resolved merge conflicts with integration of auth
Update to use oslo config and PBR
Update the context middleware to check uuid in a less contentious way
Removed routes and resources for regions endpoint - not used
Add auth policies for action api
Restructure execptions to be consistent class hierarchy and common handler
Add generation of config and policy examples
Update tests to init configs
Update database configs to not use env. vars
Removed examples directory, it was no longer accurate
Addressed/removed several TODOs - left some behind as well
Aligned input to DAGs with action: header
Retrieved all sub-steps for dags
Expanded step information
Refactored auth handling for better logging
rename create_actions policy to create_action
removed some templated file comments in env.py generated by alembic
updated inconsistent exception parameters
updated to use ulid instead of uuid for action ids
added action control audit code per review suggestion
Fixed correlation date betwen dags/actions by more string parsing
Change-Id: I2f9ea5250923f45456aa86826e344fc055bba762
This patch set adds [testenv:bandit] target to the shipyard project
to allow CICD to perform necessary static code analysis on the
shipyard project.
Change-Id: I748c52bc90ce667c891f15d018df2c01ea9ec351
Added a file exclusion to tox.ini, and also excluded rule for
unused named variables.
excluded more things that are system or tox related.
Change-Id: I022b72cbe048e3fe1f70e6017038248e6a2e9538
Sergiy Markin