Commit Graph

38 Commits

Author SHA1 Message Date
Sergiy Markin 6b317dda05 Airflow stable 2.8.2
This PS bumps up Airflow version to the latest
2.8.2 and also bumps up openstack dependences to
Antelope 2023.1

Change-Id: I61b5b39079f9c3360f5dfaafb1f961b629795c1c
2024-02-26 20:59:52 +00:00
Sergiy Markin 1e180a3f51 Airflow stable 2.8.1
Change-Id: I7667fc5cfc5e02c360d0e964de75cf5d5e6fe825
2024-02-21 22:23:55 +00:00
Sergiy Markin f99abfa433 Airflow stable 2.6.2
This PS updates python modules and code to match Airflow 2.6.2:

- bionic py36 gates  were removed
- python code corrected to match new modules versions
- selection of python modules versions was perfrmed based on
  airflow-2.6.2 constraints
- postgresql image updated to 14.8

Change-Id: Ibdcc75e600166c20b842508aa5539587cca466f0
2023-08-29 00:50:54 +00:00
Sergiy Markin 73d22a488f Restored ubuntu_bionic image build
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.

Change-Id: I695fb42efff29aeda737f2b9c1eaeb84b95aa57c
2023-05-24 18:37:21 +00:00
Sergiy Markin 375abedb8a Drydock updates
This PS delivers the following updates:
- fixed sample config and policy files generation
- rolled back chart version incremention

Change-Id: I0a7145afd8c81e2bbf36d9437d4eff3c0354667a
2023-04-28 18:39:25 +00:00
Sergiy Markin d00eaf0303 Drydock focal related upgrades
This PS implements the following changes:
- switches freeze approach to requirements-direct.txt and
  requirements-frozen.txt files
- adjusts code tabulation style according to  yapf recommendations
- replaces deprecated usage of responce.body attribute with
  responce.text
- fixes integration tests in controlled by Makefile + tox
- uplifts Helm to v3.9.4

Change-Id: I751db72eb8f670825382f11a36657112faeb169a
2023-04-26 22:32:49 +00:00
Sergiy Markin 415a8b52c5 [focal] Python modules sync with Airship project
- uplifted some python modules
- fixed tox4 requirements
- added focal build node as a default one
- added bindep.txt and bindep role to playbooks and docker image build process
- changes Makefile to reflect GoLang and dependency management changes
- upgraded Helm to v3 for chart build process
- uplifted postgresql version to 14.6
- fixed deprecated falcon.API - replaced with falcon.APP
- fixed upstream docker image publishing process

Change-Id: I307d72bb7680f6f5c71e42ad30666cf786420460
2023-04-08 08:20:22 +00:00
Ruslan Aliev 389db640e4 Change MAAS api endpoints according to v3.0.0
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I9daa8b5b8d76f2da939ce0dffdb3471ba7cd8947
2023-03-10 01:44:09 -06:00
Roman Gorshunov 161326fe06 Fix: Drydock Exceptions docs rendering on RTD
Readthedocs failed to render Drydock exceptions with error:
> WARNING: autodoc: failed to import exception xxx from module
> 'drydock_provisioner'; the following exception was raised: No module
> named 'drydock_provisioner'

Trying to add Drydock requirements to the installed requirements list,
so that Readthedocs has all modules, including those needed for the
Drydock itself.

Unify docs building by utilizing Zuul docs-on-readthedocs template job.

Cosmetic readability changes:
1. combined all Makefile .PHONY targets into one
2. merged multiple LABEL instructions in Dockerfile into one

Change-Id: I6a9b47cffc66d739968fa886c51e25b1e09ef124
2019-08-27 22:56:32 +02:00
Mark Burnett fe78b9ab53 Security update: Update pyghmi to avoid pycrypto
The version of pyghmi being used depends on a vulnerable pycrypto
version.  Updating to the latest, which depends on cryptography.

This also updates the `freeze` tox job to recreate to ensure proper
updating of the lockfile.

Change-Id: I4a7d82dc8bfad1609a5d5b4e663cf74266b7f6f0
2018-09-11 20:37:08 -05:00
Scott Hussey 0f39a55942 Refactor build to include Go
Adding the baclient code to Drydock requires a refactor
of the build automation to support multiple languages
and multiple artifacts included in a single Docker image

NOTE: the go source here is a placeholder 'hello world' sample

Change-Id: I1b4883f018b33b3d4fcd7cbcb6cba660fcdc93de
2018-09-04 13:13:21 -05:00
Andreas Jaeger 1817211732 Add venv tox environment
Some OpenStack CI jobs need the venv tox environment to run, add it.

Change-Id: I285a1e941c540d3449cf81faf6c3ee6098ec6640
2018-08-24 21:29:39 +02:00
Roman Gorshunov 7613f1c624 Unifying proxy variables for tox and docker build
This patch adds proxy-related variables to tox and 'docker build'
commands.

Change-Id: I29fee630bd4c98406797ffec93f8f80bf1a3a043
2018-06-22 14:45:51 +02:00
Scott Hussey fda50be35e [411429] maasdriver support of packages
- Support pkg_list bootactions in the MAAS driver by using
  cloud-init user_data on deployment
- Add site definition caching to ease load on Deckhand

Change-Id: I2c8c7dfdd23992fae42fa32edab308f801d05867
2018-06-12 09:58:02 -05:00
huang.zhiping cbd96b13fe fix tox python3 overrides
We want to default to running all tox environments under python 3, so
set the basepython value in each environment.

We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.

We do not want to set the override once in testenv, because that
breaks the more specific versions used in default environments like
py35 and py36.

Change-Id: I0f865164b2e02577d5e57fc1fef836584dc627ea
2018-06-09 02:12:08 +00:00
Scott Hussey cc77125953 [411387] Schema update to support repo
- Support one or more repo specifications for a site
- Add object model for repository
- Add testing for repository parsing

Update freeze job with make target

- Update the requirements freeze job
  to have a mark target that rebuilds the tox
  virtualenv each run
- Update Dockerfile to create a valid /etc/protocols file

Change-Id: I9d09b7dd7226827995e23756ff968b36eaa4d16c
2018-06-07 23:08:20 +00:00
Zuul def51688e4 Merge "Update docs for developer overview" 2018-06-01 20:47:09 +00:00
Scott Hussey 20873ad4f9 Update docs for developer overview
- New diagrams and documents for developer overview
- Update conf.py for docs to work w/ readthedocs.io
- Add policy and config gen to `make docs`
- Update zuul-linter to support checked in images
- Last fix to document publishing

Change-Id: I4faa1b87032ae5b0e786aa0fd998f809124b7987
2018-06-01 14:50:05 -05:00
Felipe Monteiro e1463c6b79 [trivial] Rename tox jobs for zuul
This patchset renames the tox.ini jobs:

* coverage => cover
* lint => pep8
* unit => py35

to comply with OpenStack standards [0]. This is needed so that [1]
works seamlessly (it will enable py35 and cover jobs for Drydock).

[0] e.g. 04469a5181/tox.ini (L119)
[1] I174d5df008f6e000da1a3878afe75919312ea7aa

Change-Id: I9ead2156cde9b1bbbec8374845653825158dec19
2018-05-31 17:00:32 +01:00
Aaron Sheffield ae6a990940 Updating requirements-lock.txt
- Ran tox -e freeze to update requirements-lock.txt.  Removes pyinotify.

Change-Id: I87c62bc6ec4b0b746e60eef70e69762cb37b5c9e
2018-05-17 15:37:28 -05:00
Scott Hussey 94d351b364 [Fix] Migrate to pymongo BSON library
* bson is incompatible w/ pip 10, migrate to pymongo bson
* Update tox to run tests w/ locked requirements file as this
  is what production images are built with

Change-Id: Iacf3301ddc224f3525102b0204f349b733608138
2018-04-19 09:30:18 -05:00
Scott Hussey c27c8e6c9b Use new validator model for validation
- Update all validator rules to publish messages using
  new format
- Update unit tests to follow new validation framework
- Add validation for no tagged VLANs on non-trunked links
- Add validation for node interfaces attaching to networks
  not allowed on the interface link
- Unit test change to work around an issue where coverage testing
  and unit testing show different results

- Update tox to skip E126 to match YAPF formatting

Change-Id: Ifef21112896b88c2bd2b361630e041806ebb6663
2018-03-14 14:12:51 -05:00
Stacey Fletcher e7f630651f Update Makefile
This ps updates the build commands to allow the same
format to be used in att-comdev projects

Change-Id: Ibc5be98636b2d0f29654a66261df27367ded49a6
2018-02-25 20:08:04 -05:00
Scott Hussey 01c62f563b Add design_ref to template context
For Bootactions that use the template pipeline
segment, add the `action.design_ref` field to the
context for Jinja2.

- Add design_ref to the render_assets method signature
- Update calls to include the contextual design_ref
- Add unit test for design_ref rendering

Change-Id: I81017e050b6f1c0a3e66ee824ecb8ffd154e45dd
2018-01-04 21:16:12 -06:00
Scott Hussey ae87cd1714 Update image and chart mgmt
NOTE: This has become a monolithic commit to get gate
      settings/scripts in place for CI

- Add Makefile with UCP standard entrypoints
- Move Dockerfile into images/drydock per UCP standards
- Add values.yaml entries for uWSGI threads and workers
- Add environment variables to chart Deployment manifest
  for uWSGI thread and workers
- Add threads and workers specification to uWSGI commandline
  in entrypoint
- Test that the Drydock API is responding
- Test that the Drydock API rejects noauth requests
- Fix Makefile utility script to work behind a proxy

Correct task success voting

Some tasks were incorrectly considered partial_success even when
no failure occurred.

- Network configuration erroneously marked messages as errors
- Update result propagation logic to only use the latest retry

The deploy_nodes task ended as incomplete due to a missing
subtask assignment

Also added a node check step to prepare_nodes so that nodes that
are already under provisioner control (MaaS) are not IPMI-rebooted.

Tangential changes:
- added config item to for leadership claim interval
- added some debug logging to bootaction_report task
- fix tasks list API endpoint to generate valid JSON

Improve task concurrency

When tasks are started with a scope of multiple nodes,
split the main task so each node is managed independently
to de-link the progression of nodes.

- Split the prepare_nodes task
- Begin reducing cyclomatic complexity to allow for
  better unit testing
- Improved tox testing to include coverage by default
- Include postgresql integration tests in coverage

Closes #73

Change-Id: I600c2a4db74dd42e809bc3ee499fb945ebdf31f6
2017-12-15 15:33:14 -06:00
Scott Hussey 1964d7f9b0 Implement routedomain support
Route domains will allow multiple L3 networks
to self organize static routes. This allows additions
of L3 networks with manually updating previously deployed
networks.

- Fix a YamlIngester error with labels on NetworkLinks
- Update the CLI --block option
- Add routedomains attribue to Networks in schema and object model
- Add routedomain documentation
- Add unit test for routedomain route generation
- Add unit test coverage reporting

Change-Id: I059d2eae6da84c4f9ad909f0287432e6cf0970d0
2017-12-14 13:45:38 -06:00
Scott Hussey f4dba218ac Implement bootaction API
- Implement boot action rendering and API
- Reorganize DB integration tests and add a tox -e postgres entrypoint
- Add boot action unit tests
- Add node filter unit test
- Add boot action context creation to deployment workflow
- Fix regression bug in MaaS Machines model
- Downgrade to Python 3.5 due to CICD limitations

Change-Id: I6c8f100cbe209f9b1c6c6ff1285365d89343ae2a
2017-11-20 16:29:54 -06:00
Scott Hussey d12ef71f9f Refactor orchestrator
Refactor orchestrator to break large
monolithic functions into small functions
per action.

- Update orchestrator to match new statemgmt API
- Pull most code out of __init__.py files
- Create action classes for Orchestrator actions
- Create action classes for Driver actions
- Orchestrator consumes tasks from database queue
- Additional encapsulation of task functionality into Task class
- Create shared integration test fixtures
- Fix Sphinx entrypoint so package install works
- Disable bootdata API until BootAction implementation
- Bring codebase into PEP8 compliance
- Update documentation reflect code changes
- Mark SQL #nosec for bandit

Change-Id: Id9a7bdedcdd5bbf07aeabbdb52db0f0b71f1e4a4
2017-10-26 15:00:39 -05:00
Scott Hussey 72adce3d38 Add Task persistence and KOTH
Implement logic to use SQLalchemy to persist and query
task information as well as manage multiple instances
with a king-of-the-hill algorithm via Postgres

- Implement leadership claim
- Implement task insert/update/query
- Implement status message inserts
- Implement integration tests w/ Postgres

Change-Id: Ic657afef3c7e1eb678629fb8b30969fa9f52585e
2017-10-26 14:56:13 -05:00
Scott Hussey 27d54b3c46 Add client access to enquiry API
Add drydock client access to the enquiry API
endpoint and CLI commands to access this endpoit.
Use PrettyTable to output the data.

Add a tox job for creating the frozen dependency list

Change-Id: Ie1724052eb9ae9500e6b0df8f0c78e25ae0617f4
2017-10-18 16:18:07 -05:00
Felipe Monteiro 8afdedab30 Drydock documentation via build_sphinx.
This PS adds tooling and automation to automatically generate
Drydock's documentation into feature-rich HTML pages that can
be hosted.

To run the documentation job, simply execute:

    tox -e docs

A future PS should add warning_is_error to 'build_sphinx' in
setup.py once the import warnings are addressed.

Change-Id: I91a3c585b2c27096e7fde12d180638d1ae4bdb81
2017-10-06 15:05:41 -04:00
Scott Hussey 864f0d35a5 Add bandit scanning
- Add bandit target in tox.ini
- Fix source issue preventing bandit from completing
- Currently no found issues

Change-Id: Iaf264b3af77ff78843462137e44ee029efc03434
2017-09-21 10:04:18 -05:00
Scott Hussey 689445280e Node storage configuration support
- Refactor YAML schema for storage specification
- Add Drydock models for HostVolume/List, HostVolumeGroup/List
  HostStorageDevice/List, HostPartition/List
- Add MAAS API models for block device, partition, volume group
- Add implementation of ApplyNodeStorage driver task
- Add documentation for authoring storage configuration
- Add unit tests for YAML parsing
- Add unit tests for size calculation

Change-Id: I94fa00b2f2bcaff1607b645a421f7e54e6d1f11e
2017-09-21 10:04:18 -05:00
Scott Hussey 32be590a53 DRYD21 - YAPF formatting for 375626
- Run YAPF for the changes in Gerrit 375626

Change-Id: I0237af779610acb8b565d419b8382a63749e4194
2017-08-25 11:21:22 -05:00
Scott Hussey e892df58dc Fix issues failing CI pipeline
- Run codebase through YAPF for formatting
- Add tox configuration for yapf and pep8
- Fix some non-YAPF pep8 failures
- Enhance verify_site for better MaaS-integration testing
- Create initial basic functional test

Change-Id: Ie5b5275d7795693a6551764362aee916b99b3e56
2017-08-24 10:18:11 -05:00
Scott Hussey 4ae627be44 Add config generation to tox.ini
Move sample config to etc/drydock
Update docs to generate a config with tox

Update configuration for Keystone

- Add config generation to tox.ini
- Fix default in bootdata config
- Add keystone dependencies
- Add config generator config
- Move sample config to a skeleton etc/drydock tree

Use PasteDeploy for WSGI integration

Using keystonemiddleware outside of a PasteDeploy
pipeline is deprecated. Move Drydock to use PasteDeploy
and integrate with keystonemiddleware

Update Falcon context object

Add keystone identity fields to context object
Clean up context marker field

Fix AuthMiddleware for keystone

Update falcon middleware to harvest headers injected
by keystonemiddleware

Fix context middleware

Update context middleware to enforce
a UUID-formatted external context marker

Lock keystonemiddleware version

Lock keystonemiddleware version to the Newton release

Sample drydock.conf with keystone

This drydock.conf file is known to integrate successfully
with Keystone via keystonemiddleware and the password plugin

Add .dockerignore

Stop adding .tox environment to docker images

Integrate with oslo.policy

Add oslo.policy 1.9.0 to requirements (Newton release)
Add tox job to generate sample policy.yaml
Create DrydockPolicy as facade for RBAC

Inject policy engine into API init

Create a DrydockPolicy instance and inject it into
the Drydock API resources.

Remove per-resource authorization

Update Drydock context and auth middleware

Update Drydock context to use keystone IDs instead of names as required
by oslo.policy
Update AuthMiddleware to capture headers when request provides
a service token

Add RBAC for /designs API

Add RBAC enforcement for GET and POST of
/api/v1.0/designs endpoint

Refactor check_policy

Refactor check_policy into the base class

Enforce RBAC for /designs/id endpoint

Enforce RBAC on /designs/id/parts endpoint

Enforce RBAC on /designs/id/parts/kind

Enforce RBAC on /designs/id/parts/kinds/

Enforce RBAC on /tasks/ endpoints

Create unit tests

- New unit tests for DrydockPolicy
- New unit tests for AuthMiddleware w/ Keystone integration

Address impacting keystonemiddleware bug

Use v4.9.1 to address https://bugs.launchpad.net/keystonemiddleware/+bug/1653646

Add oslo_config fixtures for unit testing

API base class fixes

Fix an import error in API resource base class

More graceful error handling in drydock_client

Create shared function for checking API response status codes

Create client errors for auth

Create specific Exceptions for Unauthorized
and Forbidden responses

Ignore generated sample configs

Lock iso8601 version

oslo.versionedobjects appears to be impcompatible with
iso8601 0.1.12 on Python 3.2+

Update docs for Keystone

Note Keystone as a external depdendency and
add notes on correctly configuring Drydock for
Keystone integration

Add keystoneauth1 to list_opts

Explicitly pull keystoneauth password plugin
options when generating a config template

Update reference config for keystone

Update the reference config template
for Keystone integration

Add keystoneauth1 to requirements

Need to directly include keystoneauth1 so that
oslo_config options can be pulled from it

Update config doc for keystoneauth1

Use the keystoneauth1 generated configuration options
for the configuration  docs

Remove auth options

Force dependence on Keystone as the only authentication
backend

Clean up imports

Fix how falcon modules are imported

Default to empty role list

Move param extraction

Enforce RBAC before starting to parse parameters

Implement DocumentedRuleDefault

Use DocumentedRuleDefault for policy defaults at request
of @tlam. Requires v 1.21.1 of oslo_policy, which is tied
to the Pike openstack release.

Change sample output filenames

Update filenames to follow Openstack convention

Fix tests to use hex formatted IDs

Openstack resource IDs are not hyphenated, so update
unit tests to reflect this

Fix formating and whitespace

Refactor a few small items for code review

Update keystone integration to be more
robust with Newton codebase

Centralize policy_engine reference to
support a decorator-based model

RBAC enforcement decorator

Add units tests for decorator-based
RBAC and the tasks API

Minor refactoring and format changes

Change-Id: I35f90b0c88ec577fda1077814f5eac5c0ffb41e9
2017-08-21 14:35:56 -05:00
Scott Hussey 459ddf1023 Update tox with new requirements
Change requirements file naming to be
similar to the requirements-direct.txt name
and update tox.ini with the new name
2017-07-19 13:20:26 -05:00
Scott Hussey 07cb34e82d Initial Python skeleton and the model
classes to match the YAML schema
2017-02-22 17:32:55 -06:00