Refactor Shipyard to be better able to leverage common
packages and conform with the target UCP standard layout.
This change supports the same tox entrypoints at
the root level, but the preferred approach is to use make
targets defined in the Makefile such as 'make tests' and
'make lint'
The previous tox.ini has moved and been
tailored to the specifics of each subproject at
src/bin/*/tox.ini
Autotmatic generation of the policy and configuration
files has been removed from the sphinx build for now
but these files will be automatically generated locally
into the docs source by using a 'make docs' command.
This may need to be revisited later to re-enable the
automatic generation of these files such that readthedocs
would still support the project layout.
Change-Id: Ifdc1cd4cf35fb3c5923414c677b781a60a9bae42
1) Refactor Drydock Base Operator to make use of the
UCP Base Operator instead
2) Dump logs from Drydock Pods when there are Exceptions
Change-Id: I3fbe03d13b5fc89a503cfb2c3c25751076718554
Introduce a new endpoint to retrieve Airflow logs
- API path:
GET /actions/{action_id}/steps/{step_id}/logs?try=2
Change-Id: I6a16cdab148a8a7a9f1bc5fb98a18bce1406cf9f
Puts into place the DeploymentConfiguration yaml that
provides the options that should be configured by the site
design to the deployment (and update) workflows.
This change additionally refactors reused parts to common
modules as related to info passing (xcom)
Change-Id: Ib6470899b204dbc18d2a9a2e4f95540b3b0032b0
As the size of the YAMLs increases, the amount of time needed
to process the request increased as well. Hence there is a need
to make 'timeout' configurable for the deckhand client.
Change-Id: Iab91091cd8b9a900ad0daeac22e435d4e5c9c97d
This patch set updates the required dags and operators
for the redeploy server workflow. It also introduces the
Promenade Operator.
Note that many of the required functionalities in DryDock
and Promenade are being worked on and are not ready at the
moment. As such, this patch set is mainly providing the
skeleton framework for the redeploy server workflow. The
dags and relevant Operators will be updated at a later date
when the features and functionalities are ready for usage.
Change-Id: I4baae76ea9d8cde9c2b0bab3feac896d01400868
The following errors [0] were encountered during our end-to-end
testing. This is a result of extended execution of the workflow
that led to expiration of the keystone token.
It is also possible for the 'prepare_site' task to take more than
120 seconds to complete. Hence we are increasing the time out for
the 'prepare_site_task_timeout' variable to 300 seconds.
This P.S. addresses the above 2 observations
[0] Logs from DryDock
Authorization failed for token
Identity response: {"error": {"message": "Failed to validate token", "code": 404, "title": "Not Found"}}
Authorization failed for token
Change-Id: I4760e390822e6e8c9540216035e263d054fde400
There has been significant changes to the Shipyard code base
since the last major update to the UCP Health Check Operator.
This patch set is meant to align its implementation with the
rest of the Operators.
It removes the usage of 'urlopen' which can be a security
risk and make use of the python 'requests' module instead.
We are also adding 'timeout' parameters to the other Operators
that are using 'requests.get' as failure to do so can cause
the Operator(s) to hang indefinitely. The default time out
has been set to 30 seconds. It is noted that nearly all production
code should use this parameter in nearly all requests.
Change-Id: I1205aab38ff120cd239c236dc9bdffd1660c9afb
The current logic checks for nodes that started the join process
(based on the snapshot of the environment that was taken by the
operator at that point in time). It will not check the state of
nodes that it is not aware of, i.e. those that it did not capture
initially will not be checked. Hence there is a need to introduce
backoff time as it takes a while before all the nodes start to join
the Cluster.
This is a short term stop gap approach until the Promenade API is ready
for consumption
Change-Id: I2bdf9c970ecb509fe833fd353e6648a97118d79b
test_actions_api.py contains the unit tests for the ActionsResource
config.py was modified to remove the '/' at the end of the default
url for 'web_server' to follow the format of the live url.
Change-Id: Id164e57b1f1e4a921465dac7309ef1cccdf90d8d
The purpose of the document staging api is to provide a client of
shipyard to ingest and manipulate documents within the UCP platform
In support of the need to reach out to other services, this change
also introduces the use of keystone for service discovery and connection
to other UCP services.
Change-Id: I6fa113f8786cad2884c0b791788a4ef40cd1a6b6
This change introduces a large section of the API for the next major
version of Shipyard - the action api. By interfacing with Airflow,
Shipyard will invoke workflows and allow for controlling and querying
status of those workflows. Foundationally, this patchset introduces
a lot of framework code for other apis, including error handling
to a common output format, database interaction for persistence of
action information, and use of oslo_config for configuration support.
Add GET all actions primary code - db connection not yet impl
Update base classes to have more structure
Add POST actions framework
Add GET action by id
Add GET of validations and steps
Add control api
Add unit tests of action api methods
Re-Removed duplicate deps from test reqs
Add routes for API
Removed a lot of code better handled by falcon directly
Cleaned up error flows- handlers and defaults
Refactored existing airflow tests to match standard output format
Updated json validation to be more specific
Added basic start for alembic
Added alembic upgrade at startup
Added table creation definitions
Added base revision for alembic upgrade
Bug fixes - DB queries, airflow comm, logic issues, logging issues
Bug fixes - date formats and alignment of keys between systems
Exclusions to bandit / tox.ini
Resolved merge conflicts with integration of auth
Update to use oslo config and PBR
Update the context middleware to check uuid in a less contentious way
Removed routes and resources for regions endpoint - not used
Add auth policies for action api
Restructure execptions to be consistent class hierarchy and common handler
Add generation of config and policy examples
Update tests to init configs
Update database configs to not use env. vars
Removed examples directory, it was no longer accurate
Addressed/removed several TODOs - left some behind as well
Aligned input to DAGs with action: header
Retrieved all sub-steps for dags
Expanded step information
Refactored auth handling for better logging
rename create_actions policy to create_action
removed some templated file comments in env.py generated by alembic
updated inconsistent exception parameters
updated to use ulid instead of uuid for action ids
added action control audit code per review suggestion
Fixed correlation date betwen dags/actions by more string parsing
Change-Id: I2f9ea5250923f45456aa86826e344fc055bba762
Bryan Strassner