This PS updates python modules and code to match Airflow 2.6.2:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was perfrmed based on
airflow-2.6.2 constraints
- postgresql image updated to 14.8
Change-Id: Ibdcc75e600166c20b842508aa5539587cca466f0
This PS implements the following changes:
- switches freeze approach to requirements-direct.txt and
requirements-frozen.txt files
- adjusts code tabulation style according to yapf recommendations
- replaces deprecated usage of responce.body attribute with
responce.text
- fixes integration tests in controlled by Makefile + tox
- uplifts Helm to v3.9.4
Change-Id: I751db72eb8f670825382f11a36657112faeb169a
- uplifted some python modules
- fixed tox4 requirements
- added focal build node as a default one
- added bindep.txt and bindep role to playbooks and docker image build process
- changes Makefile to reflect GoLang and dependency management changes
- upgraded Helm to v3 for chart build process
- uplifted postgresql version to 14.6
- fixed deprecated falcon.API - replaced with falcon.APP
- fixed upstream docker image publishing process
Change-Id: I307d72bb7680f6f5c71e42ad30666cf786420460
This patch adds an ability to match several NIC addresses by one
hardware profile by using regex as the address. The regex expression has
to have regex: prefix in order to be recognized.
Change-Id: I0bb067fb1783725e4ac485683eb898d5fc2d7bf2
Adding cronjob to purge the drydock DB based on retention day value. Additionally adding drydock API endpoint for purging the tasks and result_message tables and running vacuum full on drydock DB.
Change-Id: Ibcce61ecdafa637ca3ffec654152060aae26d4b8
During MAAS enlistment (and commissioning), an IPMI account (named
"maas" by default) is created on each node, which MAAS then uses for
power management.
This change allows MAAS to use the same credentials as the ones used by
the OOB driver, by overwriting the power parameters for the discovered
nodes. This includes the power type, so if the node is configured to use
Redfish, then Drydock will update a MAAS node discovered as IPMI to use
Redfish instead.
It also provides an option to instruct MAAS not to recreate IPMI
credentials during commissioning, which is passed through to the MAAS
API. Setting this to true is only supported in MAAS 2.7 or later [0].
The two maasdriver configuration options are introduced in drydock.conf,
along with their default values:
[maasdriver]
use_node_oob_params = false
skip_bmc_config = false
These options do not prevent MAAS from creating the IPMI account during
enlistment - this would require addition MAAS customization.
0: 8842d0bfd3
Change-Id: I24d3bc3b1cc94907d73bc247de3fc06dd4750ab1
The existing max_workers setting for the ThreadPoolExecutor caps the
number of nodes that can be deployed concurrently to 16. This change
allows more threads to be run in parallel, allowing the operator to
deploy more nodes at once.
Change-Id: I6d1dfa7ad8f5e03a8328311e6b59ee116a250462
The MAAS API call "GET /MAAS/api/2.0/machines/" retrieves information
about every machine known to MAAS, which is very slow. The API supports
filtering based on hostname and mac_address (among others), and querying
for power parameters for all nodes at once.
This change modifies identify_baremetal_node to avoid calling refresh on
the full machine list.
Also, the refresh method of ResourceCollectionBase is updated to allow
passing of params, which can be used to take advantage of the filtering.
Note that a filtered call to refresh overwrites the resources collection
to only contain the returned values.
Most calls to Machines.refresh() aren't really needed at all - they are
replaced with a call to Machines.empty_refresh(), which will still make
sure that the API endpoint is accessible but return an empty collection.
(This may get removed entirely in the future.)
Change-Id: Ie58c45e1790c5c827d9d47f5582214ca519946de
This patchset tunes the Drydock MAAS request factory to:
a) Implement retries for requests toward MAAS_URL/api/2.0/
b) Bumps the request timeout slightly
In addition, restricts threaded actions towards nodes to
using the same MAAS client, effectively rate limiting calls
to the MAAS api.
Change-Id: I2e66105ae332adaed62c9c3bc8cddc63e1f7bf23
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.
https://pypi.org/project/pylibyaml/
Change-Id: Iaddc0f30ed99b1f9a999f5365e9e8bf43349b82f
During drydock node deployments, sometimes MaaS node deployment for
some nodes fails when the node tries to pull the node bootaction
files, using drydock api.
Drydock api call fails with `500 Internal Server Error`, when
drydock tries to create the booaction files for the node. The logs,
however do no provide any additional clues on what caused drydock to
fail. This issue does not happen always, and subsequent site updates
will most of the deploy the failed nodes.
The additional checks and logs are added to help pinpoint the root cause
of the 500 return code, if/when this issue heppens again.
This ps also, uplifted `MarkupSafe` pip library from 1.0 to 1.1.1 to
address the issue with MarkupSafe and latest version of setuptools
described here: https://github.com/pallets/markupsafe/issues/116
Change-Id: I08a088d9690d8d9dd1f771dc5e84d1eb02fbd39f
This change allows node storage sizes to be specified using binary
prefixes (MiB, GiB, TiB) in addition to the existing supported formats
(MB, GB, TB).
Change-Id: Idef88b648a75bad87625acf1d73af011480cc0b9
A recent change[0] to address PEP8 issues resulted in an unintended
behavior modification, in some cases resulting in MAAS allocation of
multiple IP addresses to the same NIC.
This reverts to the original code logic.
[0] 1755930331
Change-Id: I6dccd1b60c414e3aa966085e81dc0b61244e9814
Flake8 version recently updated to include new PEP8 rules. Some of
the codebase is not compliant with the new rules.
Change-Id: I0f5b3d41ee54ff0d9ffa05f733f98c7e34f0f258
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
- Currently several failure paths won't log any messages
when doing a site validation. Add these messages
- Also, for validation steps that are dependent on external resources
make the resource inaccessibility a warning rather than a failure.
Change-Id: I431ed188e2f6cd3fc3fa41ae2729f3a099fdfbf5
MAAS only accepts CIDR IPs that do not have host bits set otherwise
MAAS sees the CIDR as a second network. This commit adds a Drydock
validation that checks if the CIDR has host bits and also suggests
which CIDR to use if the provided one is not acceptable to MAAS.
Change-Id: Ib6d4d8277d0e1634524426a08e138e39fb37f14b
tgt service is exposed in maas-rack-controller as a root-owned process
We are not utilizing tgt so let's just disable it.
Needed for this: https://review.openstack.org/#/c/639432/
Change-Id: I89da1c033baa72ac9c2755af9eeff6b8cce47517
- Drydock should support defined networks that MAAS cannot
see. This fixes an issue that caused this use-case to fail
by ensuring that the MAAS client models with no internal
resource IDs work.
Change-Id: I1a20d4730e94eee7268ff0cc3451e4b459a1e62b
- If two threads concurrently update the power parameters
of VMs on the same libvirt host, it seems to leave MAAS
with broken state. Add locking in Drydock so that Drydock
does not do this concurrently.
Change-Id: I85575f4ba48152b4dc79c646871f33b69f845ab9
This patch implements Refish as new OOB driver for Drydock.
All the existing Drydock Orchestrator actions are implemented.
Change-Id: I31d653fb41189a18c34cfafb0f490ca4f4d661b5
- Instead of forcing a user to provide the full URL for the MAAS API,
instead use the endpoints pattern and render the URI via HTK templates.
- Add secret name to chart to support HTK ingress
- Install libyaml to take advantage of faster parsing by pyyaml
- Add exception logging when node compiling fails.
- Add caching of parsed design to gain efficiency
- Add TLS certificate secret for use by the ingress document
Change-Id: I5a2dbc415483c336d38d67edcebdfc5812f7bb0c
- Update the maasdriver to support two concurrent
rack controllers when configuring networking.
- Identify a baremetal node from both the MAAS node and
rack controller collections
- Relax validation to only require at least one healthy
rack controller
Change-Id: I04beb56a8212b65061840021b13b412fbb37ae81
This patch fixes the issue that a reverted patch was causing [0].
In this patch we add a check for partitions which are not having
mountpoints and propably was causing the problem in airship-seaworthy.
[0] https://review.openstack.org/#/c/610594/
Story: #2002951
Change-Id: Ic3055ebdcf37461f52391cf564c29f8283637684
Signed-off-by: Dimitrios Markou <mardim@intracom-telecom.com>
- Add unit tests for node filtering based on rack name,
node tag and node label.
- Code updates to get all unit tests passing.
Change-Id: I79e7a8b35c5635364c37e22153b26687a684bfed
Prevent the partitions and logical volumes to
use the same mount point on the same baremetal
node
Story: #2002951
Change-Id: Ie936b5b20859d511e4cb858202c4ed50378bb72d
Signed-off-by: Dimitrios Markou <mardim@intracom-telecom.com>
Ruslan Aliev