Commit Graph

19 Commits

Author SHA1 Message Date
Sergiy Markin 181da83d34 Airflow webserver UI
This PS adds deployment with Airflow webserver UI in viewer mode
protected by base http autorization and exposed via ingress.

Change-Id: I5692eecf5a9af2930f8cce98b7a1e430f26b5a1b
Signed-off-by: Sergiy Markin <smarkin@mirantis.com>
2023-09-25 22:01:00 +00:00
Sergiy Markin 99c2da745a Airflow fix
This PS is mainly fixing SubDAGs timing issues when they
started along with main DAG and not at the time the main
DAG needs them. In Airflow 2.6.2 SubDAGs are deprecated
in favor of TaskGroups. So in this PS all SubDAGS were
replaced with TaskGroups.

Also task level logging config was extended by adding
py-console to obtain logs from tasks like it was
configured in Airflow 1.10.5.

Change-Id: I3f6d3961b1511e3b7cd2f7aab9810d033cfc14a3
2023-09-15 19:14:39 +00:00
Sergiy Markin 81066ae98f Airflow stable 2.6.2
This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:

- bionic py36 gates  were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
  airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
  items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
  schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
  new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2

Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
2023-08-30 16:04:47 +00:00
Sergiy Markin f0a24b11a0 Shipyard updates
This PS makes the following changes:

- makes sure the airskiff-deploy playbook is using 80Gb partition if
  available
- adds available security updates to docker images

Change-Id: Ibcb8fcddb319c3032483fdb73b55b725ea7737d4
2023-08-24 14:47:04 +00:00
Sean Eagan a0da68409a Use helm 3 in chart build
`helm serve` is removed in helm 3 so this moves
to using local `file://` dependencies [0] instead.

[0]: https://helm.sh/docs/chart_best_practices/dependencies/#repository-urls

Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: Ibdea9f1ccda043259620add022643e5c9800259a
2021-10-01 11:48:37 -05:00
Roman Gorshunov 392f75e9a4 Minor: docs location fix
Change-Id: Ifdb02451ee9feada4a18e8e67c1ac64a77efe973
2018-09-14 23:38:29 +02:00
Bryan Strassner 04906cce68 Workflow to support deployment groups
Updates the Shipyard/Airflow workflow for deploy_site and
update_site to use the deployment group/deployment strategy
information from the design.

This allows for baremetal nodes to be deployed in a design-
specified order, with criticality and success criteria driving
the success and failure of deployment.

Includes refactoring of service endpoints to reduce the need
for so much data passing.

Change-Id: Ib5e9fca535ca74d1819fe46959695acfed5b65c2
2018-06-20 09:55:15 -05:00
Bryan Strassner 7708ecbe64 Add Keystone options to registration of config
When registering options, the list_options was not returning the
appropriate options for keystone. This adds those options. Aslo changes
to use a more direct/generic adding of keystone options.

Related to this, a change was required to the service_endpoints module
to generate an auth object used to get a session.

In addition, the zuul gates for py35 were resolving to a non-existent
target. The changes to tox.ini files addresses making it so the unit
test gates are restored to functionality.

This change also uncovered a misconfigured unit test, and corrects that
test.

Change-Id: Ibc1c6c217ac7396390e7a881afbb9ef1c5ab7091
2018-06-06 14:50:54 -05:00
Bryan Strassner 977bdb3b84 [refactor] logging refactor + redaction filter
Refactors the uwsgi-based logging filter to encapsulate
better and adds a redaction filter for api logging.

This change now also includes bringing a stray test back into
the correct location.

Change-Id: I53de7da6bd4182e824366fdbb5221c7c9ae5be09
2018-06-01 17:14:15 -05:00
Bryan Strassner 769d0ded47 Refactor shipyard to UCP target layout
Refactor Shipyard to be better able to leverage common
packages and conform with the target UCP standard layout.

This change supports the same tox entrypoints at
the root level, but the preferred approach is to use make
targets defined in the Makefile such as 'make tests' and
'make lint'

The previous tox.ini has moved and been
tailored to the specifics of each subproject at
src/bin/*/tox.ini

Autotmatic generation of the policy and configuration
files has been removed from the sphinx build for now
but these files will be automatically generated locally
into the docs source by using a 'make docs' command.
This may need to be revisited later to re-enable the
automatic generation of these files such that readthedocs
would still support the project layout.

Change-Id: Ifdc1cd4cf35fb3c5923414c677b781a60a9bae42
2018-04-24 16:47:13 -05:00
Pete Birley fb7548230d Git: update gitignore to exclude built charts
This PS updates the gitignore to exlude chart build artifacts.

Change-Id: Icf8cfe496c18615441b1b118e7da12531d65eee3
2018-04-11 23:11:18 -05:00
Bryan Strassner 6cc7211630 Cleanup dockerfile and add editorconfig
Updates dockerfile to use python 3.5 base image and
reorders to take advantage of image layering caching better
Removes some outdated and unused files
adds .editorconfig file
adds .vscode to gitignore

Change-Id: I08571b7176f0cb9fab42f072326c6c14d86d518a
2018-02-16 13:44:15 -06:00
One-Fine-Day 2cd77d393d ActionsValidationsResource and VersionsResource Unit Tests
test_actions_validations_id_api.py contains the unit tests for the
ActionsValidationsResource

test_api.py contains the unit tests for the VersionsResource

.gitignore was modified in follow up to the additon of --cov-report=html
in the tox file from the 'Unit Test for WorkflowResource and
WorkflowIdResource' commit

Change-Id: Ic830ade260591e7145b1199a2b4be1c6f9cdf8a6
2017-12-06 13:27:18 -06:00
Bryan Strassner fde16f218e Move most Shipyard docs to .rst
This patchset moves most of the documentation for
shipyard from markdown to reStructuredText.
README.md remains as it is nice for Github consumption
and serves as a gateway to the rest of the generated
documentation.

Change-Id: I8cec38fe0e30e9a545cb6e79ae6de1934f7dfc4e
2017-12-04 13:36:26 -06:00
Bryan Strassner ccebbeb5ef Action lifecycle mapping include unknown
When a dag_status is not in the known list of statuses, indicate
unknown, and include the returned status parenthetically.

Change-Id: I42a083433e26c6f48aa4ef8195f0df03d1567d71
2017-11-07 10:45:24 -06:00
Bryan Strassner b6212a3f1b Update makefile with additional functionality
Adds linting and build of charts

Change-Id: I8ccb9978e419a84ed73081f8e27b31c9dbf9982c
2017-10-27 11:09:21 -05:00
Felipe Monteiro 467358dcc2 Shipyard documentation via build_sphinx
This PS adds tooling and automation to automatically generate
Shipyard's documentation into feature-rich HTML pages that can
be hosted.

To run the documentation job, simply execute:

    tox -e docs

Shipyard's .md documentation will have to be changed to .rst
in a future PS for sphinx to be able to render them into HTML.

Change-Id: Iba8cfd68e4905d49cd5a1dfd39d556044efdabbd
2017-10-06 15:51:35 -04:00
Bryan Strassner 38e58cfd30 Add Action API
This change introduces a large section of the API for the next major
version of Shipyard - the action api.  By interfacing with Airflow,
Shipyard will invoke workflows and allow for controlling and querying
status of those workflows. Foundationally, this patchset introduces
a lot of framework code for other apis, including error handling
to a common output format, database interaction for persistence of
action information, and use of oslo_config for configuration support.

Add GET all actions primary code - db connection not yet impl
Update base classes to have more structure
Add POST actions framework
Add GET action by id
Add GET of validations and steps
Add control api
Add unit tests of action api methods
Re-Removed duplicate deps from test reqs
Add routes for API
Removed a lot of code better handled by falcon directly
Cleaned up error flows- handlers and defaults
Refactored existing airflow tests to match standard output format
Updated json validation to be more specific
Added basic start for alembic
Added alembic upgrade at startup
Added table creation definitions
Added base revision for alembic upgrade
Bug fixes - DB queries, airflow comm, logic issues, logging issues
Bug fixes - date formats and alignment of keys between systems
Exclusions to bandit / tox.ini
Resolved merge conflicts with integration of auth
Update to use oslo config and PBR
Update the context middleware to check uuid in a less contentious way
Removed routes and resources for regions endpoint - not used
Add auth policies for action api
Restructure execptions to be consistent class hierarchy and common handler
Add generation of config and policy examples
Update tests to init configs
Update database configs to not use env. vars
Removed examples directory, it was no longer accurate
Addressed/removed several TODOs - left some behind as well
Aligned input to DAGs with action: header
Retrieved all sub-steps for dags
Expanded step information
Refactored auth handling for better logging
rename create_actions policy to create_action
removed some templated file comments in env.py generated by alembic
updated inconsistent exception parameters
updated to use ulid instead of uuid for action ids
added action control audit code per review suggestion
Fixed correlation date betwen dags/actions by more string parsing

Change-Id: I2f9ea5250923f45456aa86826e344fc055bba762
2017-09-22 21:47:13 -05:00
Alan Meadows 535f263c7e Initial commit 2017-05-17 13:20:48 -07:00