This PS bumps up Airflow version to the latest
2.8.2 and also bumps up openstack dependences to
Antelope 2023.1
Change-Id: I61b5b39079f9c3360f5dfaafb1f961b629795c1c
This PS updates python modules and code to match Airflow 2.6.2:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was perfrmed based on
airflow-2.6.2 constraints
- postgresql image updated to 14.8
Change-Id: Ibdcc75e600166c20b842508aa5539587cca466f0
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.
Change-Id: I695fb42efff29aeda737f2b9c1eaeb84b95aa57c
This PS delivers the following updates:
- fixed sample config and policy files generation
- rolled back chart version incremention
Change-Id: I0a7145afd8c81e2bbf36d9437d4eff3c0354667a
This PS implements the following changes:
- switches freeze approach to requirements-direct.txt and
requirements-frozen.txt files
- adjusts code tabulation style according to yapf recommendations
- replaces deprecated usage of responce.body attribute with
responce.text
- fixes integration tests in controlled by Makefile + tox
- uplifts Helm to v3.9.4
Change-Id: I751db72eb8f670825382f11a36657112faeb169a
- uplifted some python modules
- fixed tox4 requirements
- added focal build node as a default one
- added bindep.txt and bindep role to playbooks and docker image build process
- changes Makefile to reflect GoLang and dependency management changes
- upgraded Helm to v3 for chart build process
- uplifted postgresql version to 14.6
- fixed deprecated falcon.API - replaced with falcon.APP
- fixed upstream docker image publishing process
Change-Id: I307d72bb7680f6f5c71e42ad30666cf786420460
Readthedocs failed to render Drydock exceptions with error:
> WARNING: autodoc: failed to import exception xxx from module
> 'drydock_provisioner'; the following exception was raised: No module
> named 'drydock_provisioner'
Trying to add Drydock requirements to the installed requirements list,
so that Readthedocs has all modules, including those needed for the
Drydock itself.
Unify docs building by utilizing Zuul docs-on-readthedocs template job.
Cosmetic readability changes:
1. combined all Makefile .PHONY targets into one
2. merged multiple LABEL instructions in Dockerfile into one
Change-Id: I6a9b47cffc66d739968fa886c51e25b1e09ef124
The version of pyghmi being used depends on a vulnerable pycrypto
version. Updating to the latest, which depends on cryptography.
This also updates the `freeze` tox job to recreate to ensure proper
updating of the lockfile.
Change-Id: I4a7d82dc8bfad1609a5d5b4e663cf74266b7f6f0
Adding the baclient code to Drydock requires a refactor
of the build automation to support multiple languages
and multiple artifacts included in a single Docker image
NOTE: the go source here is a placeholder 'hello world' sample
Change-Id: I1b4883f018b33b3d4fcd7cbcb6cba660fcdc93de
- Support pkg_list bootactions in the MAAS driver by using
cloud-init user_data on deployment
- Add site definition caching to ease load on Deckhand
Change-Id: I2c8c7dfdd23992fae42fa32edab308f801d05867
We want to default to running all tox environments under python 3, so
set the basepython value in each environment.
We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.
We do not want to set the override once in testenv, because that
breaks the more specific versions used in default environments like
py35 and py36.
Change-Id: I0f865164b2e02577d5e57fc1fef836584dc627ea
- Support one or more repo specifications for a site
- Add object model for repository
- Add testing for repository parsing
Update freeze job with make target
- Update the requirements freeze job
to have a mark target that rebuilds the tox
virtualenv each run
- Update Dockerfile to create a valid /etc/protocols file
Change-Id: I9d09b7dd7226827995e23756ff968b36eaa4d16c
- New diagrams and documents for developer overview
- Update conf.py for docs to work w/ readthedocs.io
- Add policy and config gen to `make docs`
- Update zuul-linter to support checked in images
- Last fix to document publishing
Change-Id: I4faa1b87032ae5b0e786aa0fd998f809124b7987
This patchset renames the tox.ini jobs:
* coverage => cover
* lint => pep8
* unit => py35
to comply with OpenStack standards [0]. This is needed so that [1]
works seamlessly (it will enable py35 and cover jobs for Drydock).
[0] e.g. 04469a5181/tox.ini (L119)
[1] I174d5df008f6e000da1a3878afe75919312ea7aa
Change-Id: I9ead2156cde9b1bbbec8374845653825158dec19
* bson is incompatible w/ pip 10, migrate to pymongo bson
* Update tox to run tests w/ locked requirements file as this
is what production images are built with
Change-Id: Iacf3301ddc224f3525102b0204f349b733608138
- Update all validator rules to publish messages using
new format
- Update unit tests to follow new validation framework
- Add validation for no tagged VLANs on non-trunked links
- Add validation for node interfaces attaching to networks
not allowed on the interface link
- Unit test change to work around an issue where coverage testing
and unit testing show different results
- Update tox to skip E126 to match YAPF formatting
Change-Id: Ifef21112896b88c2bd2b361630e041806ebb6663
For Bootactions that use the template pipeline
segment, add the `action.design_ref` field to the
context for Jinja2.
- Add design_ref to the render_assets method signature
- Update calls to include the contextual design_ref
- Add unit test for design_ref rendering
Change-Id: I81017e050b6f1c0a3e66ee824ecb8ffd154e45dd
NOTE: This has become a monolithic commit to get gate
settings/scripts in place for CI
- Add Makefile with UCP standard entrypoints
- Move Dockerfile into images/drydock per UCP standards
- Add values.yaml entries for uWSGI threads and workers
- Add environment variables to chart Deployment manifest
for uWSGI thread and workers
- Add threads and workers specification to uWSGI commandline
in entrypoint
- Test that the Drydock API is responding
- Test that the Drydock API rejects noauth requests
- Fix Makefile utility script to work behind a proxy
Correct task success voting
Some tasks were incorrectly considered partial_success even when
no failure occurred.
- Network configuration erroneously marked messages as errors
- Update result propagation logic to only use the latest retry
The deploy_nodes task ended as incomplete due to a missing
subtask assignment
Also added a node check step to prepare_nodes so that nodes that
are already under provisioner control (MaaS) are not IPMI-rebooted.
Tangential changes:
- added config item to for leadership claim interval
- added some debug logging to bootaction_report task
- fix tasks list API endpoint to generate valid JSON
Improve task concurrency
When tasks are started with a scope of multiple nodes,
split the main task so each node is managed independently
to de-link the progression of nodes.
- Split the prepare_nodes task
- Begin reducing cyclomatic complexity to allow for
better unit testing
- Improved tox testing to include coverage by default
- Include postgresql integration tests in coverage
Closes #73
Change-Id: I600c2a4db74dd42e809bc3ee499fb945ebdf31f6
Route domains will allow multiple L3 networks
to self organize static routes. This allows additions
of L3 networks with manually updating previously deployed
networks.
- Fix a YamlIngester error with labels on NetworkLinks
- Update the CLI --block option
- Add routedomains attribue to Networks in schema and object model
- Add routedomain documentation
- Add unit test for routedomain route generation
- Add unit test coverage reporting
Change-Id: I059d2eae6da84c4f9ad909f0287432e6cf0970d0
- Implement boot action rendering and API
- Reorganize DB integration tests and add a tox -e postgres entrypoint
- Add boot action unit tests
- Add node filter unit test
- Add boot action context creation to deployment workflow
- Fix regression bug in MaaS Machines model
- Downgrade to Python 3.5 due to CICD limitations
Change-Id: I6c8f100cbe209f9b1c6c6ff1285365d89343ae2a
Refactor orchestrator to break large
monolithic functions into small functions
per action.
- Update orchestrator to match new statemgmt API
- Pull most code out of __init__.py files
- Create action classes for Orchestrator actions
- Create action classes for Driver actions
- Orchestrator consumes tasks from database queue
- Additional encapsulation of task functionality into Task class
- Create shared integration test fixtures
- Fix Sphinx entrypoint so package install works
- Disable bootdata API until BootAction implementation
- Bring codebase into PEP8 compliance
- Update documentation reflect code changes
- Mark SQL #nosec for bandit
Change-Id: Id9a7bdedcdd5bbf07aeabbdb52db0f0b71f1e4a4
Implement logic to use SQLalchemy to persist and query
task information as well as manage multiple instances
with a king-of-the-hill algorithm via Postgres
- Implement leadership claim
- Implement task insert/update/query
- Implement status message inserts
- Implement integration tests w/ Postgres
Change-Id: Ic657afef3c7e1eb678629fb8b30969fa9f52585e
Add drydock client access to the enquiry API
endpoint and CLI commands to access this endpoit.
Use PrettyTable to output the data.
Add a tox job for creating the frozen dependency list
Change-Id: Ie1724052eb9ae9500e6b0df8f0c78e25ae0617f4
This PS adds tooling and automation to automatically generate
Drydock's documentation into feature-rich HTML pages that can
be hosted.
To run the documentation job, simply execute:
tox -e docs
A future PS should add warning_is_error to 'build_sphinx' in
setup.py once the import warnings are addressed.
Change-Id: I91a3c585b2c27096e7fde12d180638d1ae4bdb81
- Add bandit target in tox.ini
- Fix source issue preventing bandit from completing
- Currently no found issues
Change-Id: Iaf264b3af77ff78843462137e44ee029efc03434
- Run codebase through YAPF for formatting
- Add tox configuration for yapf and pep8
- Fix some non-YAPF pep8 failures
- Enhance verify_site for better MaaS-integration testing
- Create initial basic functional test
Change-Id: Ie5b5275d7795693a6551764362aee916b99b3e56
Move sample config to etc/drydock
Update docs to generate a config with tox
Update configuration for Keystone
- Add config generation to tox.ini
- Fix default in bootdata config
- Add keystone dependencies
- Add config generator config
- Move sample config to a skeleton etc/drydock tree
Use PasteDeploy for WSGI integration
Using keystonemiddleware outside of a PasteDeploy
pipeline is deprecated. Move Drydock to use PasteDeploy
and integrate with keystonemiddleware
Update Falcon context object
Add keystone identity fields to context object
Clean up context marker field
Fix AuthMiddleware for keystone
Update falcon middleware to harvest headers injected
by keystonemiddleware
Fix context middleware
Update context middleware to enforce
a UUID-formatted external context marker
Lock keystonemiddleware version
Lock keystonemiddleware version to the Newton release
Sample drydock.conf with keystone
This drydock.conf file is known to integrate successfully
with Keystone via keystonemiddleware and the password plugin
Add .dockerignore
Stop adding .tox environment to docker images
Integrate with oslo.policy
Add oslo.policy 1.9.0 to requirements (Newton release)
Add tox job to generate sample policy.yaml
Create DrydockPolicy as facade for RBAC
Inject policy engine into API init
Create a DrydockPolicy instance and inject it into
the Drydock API resources.
Remove per-resource authorization
Update Drydock context and auth middleware
Update Drydock context to use keystone IDs instead of names as required
by oslo.policy
Update AuthMiddleware to capture headers when request provides
a service token
Add RBAC for /designs API
Add RBAC enforcement for GET and POST of
/api/v1.0/designs endpoint
Refactor check_policy
Refactor check_policy into the base class
Enforce RBAC for /designs/id endpoint
Enforce RBAC on /designs/id/parts endpoint
Enforce RBAC on /designs/id/parts/kind
Enforce RBAC on /designs/id/parts/kinds/
Enforce RBAC on /tasks/ endpoints
Create unit tests
- New unit tests for DrydockPolicy
- New unit tests for AuthMiddleware w/ Keystone integration
Address impacting keystonemiddleware bug
Use v4.9.1 to address https://bugs.launchpad.net/keystonemiddleware/+bug/1653646
Add oslo_config fixtures for unit testing
API base class fixes
Fix an import error in API resource base class
More graceful error handling in drydock_client
Create shared function for checking API response status codes
Create client errors for auth
Create specific Exceptions for Unauthorized
and Forbidden responses
Ignore generated sample configs
Lock iso8601 version
oslo.versionedobjects appears to be impcompatible with
iso8601 0.1.12 on Python 3.2+
Update docs for Keystone
Note Keystone as a external depdendency and
add notes on correctly configuring Drydock for
Keystone integration
Add keystoneauth1 to list_opts
Explicitly pull keystoneauth password plugin
options when generating a config template
Update reference config for keystone
Update the reference config template
for Keystone integration
Add keystoneauth1 to requirements
Need to directly include keystoneauth1 so that
oslo_config options can be pulled from it
Update config doc for keystoneauth1
Use the keystoneauth1 generated configuration options
for the configuration docs
Remove auth options
Force dependence on Keystone as the only authentication
backend
Clean up imports
Fix how falcon modules are imported
Default to empty role list
Move param extraction
Enforce RBAC before starting to parse parameters
Implement DocumentedRuleDefault
Use DocumentedRuleDefault for policy defaults at request
of @tlam. Requires v 1.21.1 of oslo_policy, which is tied
to the Pike openstack release.
Change sample output filenames
Update filenames to follow Openstack convention
Fix tests to use hex formatted IDs
Openstack resource IDs are not hyphenated, so update
unit tests to reflect this
Fix formating and whitespace
Refactor a few small items for code review
Update keystone integration to be more
robust with Newton codebase
Centralize policy_engine reference to
support a decorator-based model
RBAC enforcement decorator
Add units tests for decorator-based
RBAC and the tasks API
Minor refactoring and format changes
Change-Id: I35f90b0c88ec577fda1077814f5eac5c0ffb41e9