Commit Graph

48 Commits

Author SHA1 Message Date
Roman Gorshunov 161326fe06 Fix: Drydock Exceptions docs rendering on RTD
Readthedocs failed to render Drydock exceptions with error:
> WARNING: autodoc: failed to import exception xxx from module
> 'drydock_provisioner'; the following exception was raised: No module
> named 'drydock_provisioner'

Trying to add Drydock requirements to the installed requirements list,
so that Readthedocs has all modules, including those needed for the
Drydock itself.

Unify docs building by utilizing Zuul docs-on-readthedocs template job.

Cosmetic readability changes:
1. combined all Makefile .PHONY targets into one
2. merged multiple LABEL instructions in Dockerfile into one

Change-Id: I6a9b47cffc66d739968fa886c51e25b1e09ef124
2019-08-27 22:56:32 +02:00
Carter, Matt (mc981n) 2cf960f4fc Fix small inconsistency in API documentation
Change-Id: Iaaabf4f5f706020379c8369017575b99a5d5f923
2019-08-19 14:59:38 -05:00
Zuul 756a063c30 Merge "Support multiple rack controllers" 2018-10-23 18:52:25 +00:00
Scott Hussey 357cf7e455 Support multiple rack controllers
- Update the maasdriver to support two concurrent
  rack controllers when configuring networking.
- Identify a baremetal node from both the MAAS node and
  rack controller collections
- Relax validation to only require at least one healthy
  rack controller

Change-Id: I04beb56a8212b65061840021b13b412fbb37ae81
2018-10-18 17:04:00 -05:00
Zuul 03cf1517d6 Merge "Fix typo in doc" 2018-10-13 03:24:16 +00:00
skovaleff 7c5f328866 Fix typo in Site Topology doc
Change-Id: If62e95908e9141a23244bf11a606cdca078c9e10
2018-10-12 16:00:31 -07:00
skovaleff 6841aad00c Fix typo in doc
Change-Id: Icd9f860405de26d2199efca52b7f7cb67beee3e2
2018-10-12 14:34:08 -07:00
Zuul ebb712ff50 Merge "fix: Drydock exceptions documentation incorrectly rendering" 2018-10-05 21:06:51 +00:00
Zuul 43efbb0c63 Merge "Catchup YAPF formatting" 2018-10-02 20:52:13 +00:00
Zuul b52282db38 Merge "Fix: various documentation and URL fixes" 2018-10-01 19:18:16 +00:00
Scott Hussey 6ca7aa4bff Catchup YAPF formatting
Change-Id: Ic54f77b4b0bdd9199fbc10dfdfc43d3af8f0bfd1
2018-09-26 08:57:51 -05:00
Roman Gorshunov ec17405606 Fix: various documentation and URL fixes
1) UCP -> Airship
2) readthedocs.org -> readthedocs.io (there is redirect)
3) http -> https
4) attcomdev -> airshipit (repo on quay.io)
5) att-comdev -> openstack/airship-* (repo on github/openstack git)
6) many URLs have been verified and adjusted to be current
7) no need for 'en/latest/' path in URL of the RTD
8) added more info to some setup.cfg and setup.py files
9) ucp-integration docs are now in airship-in-a-bottle
10) various other minor fixes

Change-Id: I64fb508a3ff72258d82c795193e918f2945deb29
2018-09-26 08:21:14 +00:00
Soumitra Khuntia f879e3a88d Update node-labels through Kubernetes Provisioner
Blueprint: https://airshipit.readthedocs.io/projects/specs/en/latest/specs/approved/k8s_update_node_labels_workflow.html

This commit adds:

1. A new task action ''relabel_nodes'' added to update
nodes labels
2.A new Kubernetes driver added for Kubernetes cluster
interaction through Promenade.

Change-Id: I37c2d7bfda4966d907556036bc2b343df451994c
2018-09-25 17:54:59 +05:30
Felipe Monteiro 3fb095f0eb fix: Drydock exceptions documentation incorrectly rendering
This patch set changes Drydock's exceptions documentation
(contained underneath operators guide) because it isn't rendering
correctly as a list table on RTD (the autoexception information
is missing) [0].

The easy fix is to change the tabularized view (list table)
into basically a series of autoexception classes which sufficiently
captures the level of detail required, anyway.

Note that running `tox -e docs` locally and opening the resulting
index.html page appears to work -- but not when hosted on RTD.

[0] https://airship-drydock.readthedocs.io/en/latest/exceptions/exceptions.html

Change-Id: Ie4bc01f1fe6aee9b9a58fd956f5b350df44bef51
2018-09-23 12:27:54 -04:00
Scott Hussey 9a52dca199 Bootaction cleanup
- Clean up the docs and code around bootactions to support the
  baclient work

- Update the Jinja2 environment in the bootaction rendering
- Refactor authentication of bootaction signal API calls

Change-Id: Ic64f0c7ee09a487be750188953013f1ed3cd99cb
2018-09-06 16:33:11 -05:00
Scott Hussey 5344ed1957 Add destroy_node config items to sample
- Geneate a new sample configuration file including
  the items included in the destory_node action

Change-Id: Iaae8c95a7215b28b0fd54675582c86ba1e52a54a
2018-08-30 10:26:32 -05:00
Scott Hussey e2b3e8ee8e Database tunability
Add additional tunables to how SQLalchemy creates connection pools

Also start including static doc assets to they can be published

Change-Id: I268dc265a6b6cf1a200b235a5f99e65e89a95637
2018-08-21 14:09:36 -05:00
Rick Bartra 044aa2e2af Update Drydock sampleconf.rst with correct name
Update the Drydock samepleconf.rst which incorrectly references
Patrole instead of Drydock

Change-Id: I47208a652d2cc326f82c281c4eeb451b302bfb48
2018-08-14 14:00:36 -04:00
Scott Hussey e35712a573 [411430] Validate bootaction pkg_list
- Add a validator for bootactions to warn if a node doesn't
  have at least one
- Add a validator for bootactions to error if a package
  version specifier is invalid
- Unit tests for the validation

Change-Id: I61d8aa3831791af0484498e6fe9f7c1c83dbf540
2018-06-12 09:58:02 -05:00
Scott Hussey 1b0797440b [411428] Bootaction pkg_list support
- Support a list of debian packages as a bootaction asset
- Add unit testing for parsing the additional bootaction information
- Add __eq__ and __hash__ for DocumentReference to allow checking
  equality and list presence

Change-Id: I0ca42baf7aae6dc2e52efd5b311d0632e069dd79
2018-06-09 19:53:43 +00:00
Scott Hussey 20873ad4f9 Update docs for developer overview
- New diagrams and documents for developer overview
- Update conf.py for docs to work w/ readthedocs.io
- Add policy and config gen to `make docs`
- Update zuul-linter to support checked in images
- Last fix to document publishing

Change-Id: I4faa1b87032ae5b0e786aa0fd998f809124b7987
2018-06-01 14:50:05 -05:00
Scott Hussey d107e65a98 Libvirt support in maasdriver
- Add validations that OOB configs for nodes are valid for the
  oob type defined
- Add documentation for using Drydock/MAAS to deploy libvirt VMs
- Add logic to update the MAAS node power parameters to allow power
  control of libvirt VMs

Change-Id: Ia7d5fbd1659636d46cf1790fe3fc66ca6c6fee89
2018-05-25 11:13:53 -05:00
Aaron Sheffield 8af92eaf29 Change POST nodefilter Input
- Changed the expected input from site_design to design_ref.
- Converts the design_ref json/dict into a SiteDesign object.

Change-Id: Ie1f6738fec64749cde9d2843d9373175006794c0
2018-05-08 10:32:26 -05:00
Aaron Sheffield d052664f74 Add Drydock API Client Options
- Added builddata, subtaskerrors, and layers as options to get_task.
- Added get_nodes_for_filter to call POST /nodefilter.
- Updated endpoint for POST /nodes to POST /nodefilter.
- Made node_filter optional in POST /nodefilter.

Change-Id: I456a6e9991d03af3d375c448f5cbf07a21e91f1d
2018-05-08 14:11:15 +00:00
Scott Hussey 3b41868802 [373577] Fix hostnames with underscores
Hostnames with underscores caused a deployment failure.

Update to use a double underscore as a delimiter and provide
a design validator to check that hostnames do not contain it.

Closes #78

Change-Id: Ib148aed5cffe7fd8bc08441eaef8a45af6601bdd
2018-05-03 15:42:33 +00:00
Aaron Sheffield b138b3c179 New Drydock APIs for nodes and tasks
- Added a new query param 'layers=x' to /tasks/{task_id}
  where x is a number.
  This param allows the API to drill down to all the subtasks.
  If x is -1 it will reeturn all subtasks, otherwise it will return
  x layers of subtasks.
- Added a new query param 'subtaskerrors=true' to /tasks/{task_id}.
  If true then any errors from subtasks will be included in the
  subtask_errors element in the response.
- Added a POST to /nodes that requires node_filter and site_design
  in the body.  Then uses those parameters to return a list of nodes.

Change-Id: I9b4c06dd1c9de2b0500092aa4b4bfacac08eac54
2018-05-02 14:42:41 +00:00
Scott Hussey b628a1bfce [390773] Support SRIOV fields in definition profiles
- Add hugepages and cpu_sets stanzas to HardwareProfile as the
  size and count of hugepages and the exact CPUs to pin for SRIOV
  are dependent on hardware.
- Add sriov stanza to a node interface to specify vf_count and
  trustedmode. These will be passthrough values as Drydock doesn't
  configure SRIOV.
- Add sriov information to the bootaction context so it can be written
  to disk on a deployed node if needed
- Allow an interface configuration to be skipped when an interface has
  no defined network_link for things like SR-IOV interfaces.
- Add kernel parameter reference support to access hardware profile
  information
- Add unit tests
- Update topology documentation for usage of HardwareProfile
  and kernel parameter references

Change-Id: Iefd326f5c6fad19dbd21300ee249019a3dfd4848
2018-03-14 14:12:51 -05:00
Aaron Sheffield 8c2540c3f2 Lower cased dock in Drydock
Change-Id: Ie7f0d748744aaea820078d2cf3696bc5bd1075bf
2018-03-13 15:53:01 -05:00
Krysta 3fdebedf95 Documentation for Exceptions
Adds documentation for Drydock exceptions.

Change-Id: I99d5396a6deb46e8fb135f037292a295e855abd5
Note: Does not include unused exceptions.
2018-02-13 11:25:45 -05:00
Scott Hussey 74ce4aaef0 Add build data support to the API
- Add list of build data to detailed task response when builddata=true
  specifed in query string
- Add new endpoint for /nodes/nodename/builddata to retrieve
  build data for a particular node
- Update docs for new API capabilities
- Testing all around

Change-Id: If0fcd2962d4389789af45ad1fbe61d226ac6a403
2018-01-08 16:21:16 -06:00
Bryan Strassner fb8cbfc67d Merge "Documentation for Drydock Validate Design API" 2018-01-08 10:51:14 -05:00
Scott Hussey 22b574c9b8 Build data blueprint
Create a blueprint document describing the solution
design for collecing build data in Drydock

Change-Id: Ie2b83a6c86cb9ce8006a64e09a31c0b3ee7d2337
2018-01-04 21:16:12 -06:00
Scott Hussey 01c62f563b Add design_ref to template context
For Bootactions that use the template pipeline
segment, add the `action.design_ref` field to the
context for Jinja2.

- Add design_ref to the render_assets method signature
- Update calls to include the contextual design_ref
- Add unit test for design_ref rendering

Change-Id: I81017e050b6f1c0a3e66ee824ecb8ffd154e45dd
2018-01-04 21:16:12 -06:00
Krysta d9b3a678ec Documentation for Drydock Validate Design API
Change-Id: I10aa8ee6fca98168ac0a652e763d38322cd3220c
2018-01-04 09:05:22 -05:00
Scott Hussey 1964d7f9b0 Implement routedomain support
Route domains will allow multiple L3 networks
to self organize static routes. This allows additions
of L3 networks with manually updating previously deployed
networks.

- Fix a YamlIngester error with labels on NetworkLinks
- Update the CLI --block option
- Add routedomains attribue to Networks in schema and object model
- Add routedomain documentation
- Add unit test for routedomain route generation
- Add unit test coverage reporting

Change-Id: I059d2eae6da84c4f9ad909f0287432e6cf0970d0
2017-12-14 13:45:38 -06:00
Scott Hussey f4dba218ac Implement bootaction API
- Implement boot action rendering and API
- Reorganize DB integration tests and add a tox -e postgres entrypoint
- Add boot action unit tests
- Add node filter unit test
- Add boot action context creation to deployment workflow
- Fix regression bug in MaaS Machines model
- Downgrade to Python 3.5 due to CICD limitations

Change-Id: I6c8f100cbe209f9b1c6c6ff1285365d89343ae2a
2017-11-20 16:29:54 -06:00
Scott Hussey 575e7acbb8 Blueprint for Boot Actions
This design is for the framework of deploying actions and assets to
nodes being deployed by Drydock. Included is an API for nodes to
report back the result status of any running boot actions.

Change-Id: I2becd6af33d0202d0da73bab01f21b010fc3ea60
2017-10-27 13:42:49 -04:00
Scott Hussey d12ef71f9f Refactor orchestrator
Refactor orchestrator to break large
monolithic functions into small functions
per action.

- Update orchestrator to match new statemgmt API
- Pull most code out of __init__.py files
- Create action classes for Orchestrator actions
- Create action classes for Driver actions
- Orchestrator consumes tasks from database queue
- Additional encapsulation of task functionality into Task class
- Create shared integration test fixtures
- Fix Sphinx entrypoint so package install works
- Disable bootdata API until BootAction implementation
- Bring codebase into PEP8 compliance
- Update documentation reflect code changes
- Mark SQL #nosec for bandit

Change-Id: Id9a7bdedcdd5bbf07aeabbdb52db0f0b71f1e4a4
2017-10-26 15:00:39 -05:00
Scott Hussey e042811c76 DRYD47 - Task persistance and refactor
Refactor the task model and add database persistence for tasks.

- Document task schema
- Use Alembic for database creation
- Use SQLalchemy for database abstraction
- Update Task status and result enumerations

Change-Id: I247b88f293144a0bdf891958e19711d975c729ba
2017-10-26 14:54:25 -05:00
intlabs 15e4c96f4e Make docs conf.py Pep8 compliant
This PS makes the docs setup pep8 compliant.

Change-Id: Ibbd29dee6166de8e2afb6aef0b7be3c0b3d76e94
2017-10-17 09:17:09 -05:00
Felipe Monteiro 8afdedab30 Drydock documentation via build_sphinx.
This PS adds tooling and automation to automatically generate
Drydock's documentation into feature-rich HTML pages that can
be hosted.

To run the documentation job, simply execute:

    tox -e docs

A future PS should add warning_is_error to 'build_sphinx' in
setup.py once the import warnings are addressed.

Change-Id: I91a3c585b2c27096e7fde12d180638d1ae4bdb81
2017-10-06 15:05:41 -04:00
Pete Birley 46ee2a1683 Docs: Fix formatting and typos
This PS fixes the .rst formatting of the docs and corrects some
typos.

Change-Id: I2494098a5fbb126be3332a8e2ac490c695c8754a
2017-10-04 21:29:35 -05:00
Scott Hussey 48df97e133 DRYD-50 Drydock support of NIC bonding
Implement interface bonding via the MaaS
API

- Documentation on writing topology definition of networking and host
  network attachment
- Adjust topology YAML schema for interface definition
- Add MaaS API support for create_bond
- Fix some bugs from Gerrit #377818
- Update MaaS API client to support multi-select options

Change-Id: I1c42300ede3f67595ebc8029b0f375622b459254
2017-09-21 10:29:39 -05:00
Scott Hussey 689445280e Node storage configuration support
- Refactor YAML schema for storage specification
- Add Drydock models for HostVolume/List, HostVolumeGroup/List
  HostStorageDevice/List, HostPartition/List
- Add MAAS API models for block device, partition, volume group
- Add implementation of ApplyNodeStorage driver task
- Add documentation for authoring storage configuration
- Add unit tests for YAML parsing
- Add unit tests for size calculation

Change-Id: I94fa00b2f2bcaff1607b645a421f7e54e6d1f11e
2017-09-21 10:04:18 -05:00
Scott Hussey e892df58dc Fix issues failing CI pipeline
- Run codebase through YAPF for formatting
- Add tox configuration for yapf and pep8
- Fix some non-YAPF pep8 failures
- Enhance verify_site for better MaaS-integration testing
- Create initial basic functional test

Change-Id: Ie5b5275d7795693a6551764362aee916b99b3e56
2017-08-24 10:18:11 -05:00
Scott Hussey 4ae627be44 Add config generation to tox.ini
Move sample config to etc/drydock
Update docs to generate a config with tox

Update configuration for Keystone

- Add config generation to tox.ini
- Fix default in bootdata config
- Add keystone dependencies
- Add config generator config
- Move sample config to a skeleton etc/drydock tree

Use PasteDeploy for WSGI integration

Using keystonemiddleware outside of a PasteDeploy
pipeline is deprecated. Move Drydock to use PasteDeploy
and integrate with keystonemiddleware

Update Falcon context object

Add keystone identity fields to context object
Clean up context marker field

Fix AuthMiddleware for keystone

Update falcon middleware to harvest headers injected
by keystonemiddleware

Fix context middleware

Update context middleware to enforce
a UUID-formatted external context marker

Lock keystonemiddleware version

Lock keystonemiddleware version to the Newton release

Sample drydock.conf with keystone

This drydock.conf file is known to integrate successfully
with Keystone via keystonemiddleware and the password plugin

Add .dockerignore

Stop adding .tox environment to docker images

Integrate with oslo.policy

Add oslo.policy 1.9.0 to requirements (Newton release)
Add tox job to generate sample policy.yaml
Create DrydockPolicy as facade for RBAC

Inject policy engine into API init

Create a DrydockPolicy instance and inject it into
the Drydock API resources.

Remove per-resource authorization

Update Drydock context and auth middleware

Update Drydock context to use keystone IDs instead of names as required
by oslo.policy
Update AuthMiddleware to capture headers when request provides
a service token

Add RBAC for /designs API

Add RBAC enforcement for GET and POST of
/api/v1.0/designs endpoint

Refactor check_policy

Refactor check_policy into the base class

Enforce RBAC for /designs/id endpoint

Enforce RBAC on /designs/id/parts endpoint

Enforce RBAC on /designs/id/parts/kind

Enforce RBAC on /designs/id/parts/kinds/

Enforce RBAC on /tasks/ endpoints

Create unit tests

- New unit tests for DrydockPolicy
- New unit tests for AuthMiddleware w/ Keystone integration

Address impacting keystonemiddleware bug

Use v4.9.1 to address https://bugs.launchpad.net/keystonemiddleware/+bug/1653646

Add oslo_config fixtures for unit testing

API base class fixes

Fix an import error in API resource base class

More graceful error handling in drydock_client

Create shared function for checking API response status codes

Create client errors for auth

Create specific Exceptions for Unauthorized
and Forbidden responses

Ignore generated sample configs

Lock iso8601 version

oslo.versionedobjects appears to be impcompatible with
iso8601 0.1.12 on Python 3.2+

Update docs for Keystone

Note Keystone as a external depdendency and
add notes on correctly configuring Drydock for
Keystone integration

Add keystoneauth1 to list_opts

Explicitly pull keystoneauth password plugin
options when generating a config template

Update reference config for keystone

Update the reference config template
for Keystone integration

Add keystoneauth1 to requirements

Need to directly include keystoneauth1 so that
oslo_config options can be pulled from it

Update config doc for keystoneauth1

Use the keystoneauth1 generated configuration options
for the configuration  docs

Remove auth options

Force dependence on Keystone as the only authentication
backend

Clean up imports

Fix how falcon modules are imported

Default to empty role list

Move param extraction

Enforce RBAC before starting to parse parameters

Implement DocumentedRuleDefault

Use DocumentedRuleDefault for policy defaults at request
of @tlam. Requires v 1.21.1 of oslo_policy, which is tied
to the Pike openstack release.

Change sample output filenames

Update filenames to follow Openstack convention

Fix tests to use hex formatted IDs

Openstack resource IDs are not hyphenated, so update
unit tests to reflect this

Fix formating and whitespace

Refactor a few small items for code review

Update keystone integration to be more
robust with Newton codebase

Centralize policy_engine reference to
support a decorator-based model

RBAC enforcement decorator

Add units tests for decorator-based
RBAC and the tasks API

Minor refactoring and format changes

Change-Id: I35f90b0c88ec577fda1077814f5eac5c0ffb41e9
2017-08-21 14:35:56 -05:00
Scott Hussey 48a1f54966 Create a base getting started guide
Update README

Add dependencies
Add link to Getting Started

Document format fixes

Change-Id: I768bee88bc4776c06b1c1dcc1435c41e899cbed2
2017-08-07 12:55:31 -05:00
Scott Hussey 179459adc0 Fix some issues found during CLI testing
Found some bugs during REST client / CLI testing
in the API and downstream methods
2017-07-19 13:21:44 -05:00