Update kubernetes client to v26.1.0
Updating armada to focal base image
Remove xenial and opensuse dockerfiles
Update tox python from py35 to py38
Add apparmor for docker build
Uplift HTK chart version 0.2.52
Bumping up some python dependencies to get in sync with shipyard
Added clear-firewall role for airskiff-deploy playbook
Change-Id: If06a3f60466702d05a21c24a7cb8041bed41507a
For now we leave the tiller status enpdpoint, until
Shipyard has had a release to stop depending on it [0].
[0]: https://review.opendev.org/c/airship/shipyard/+/802718
Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: If8a02d7118f6840fdbbe088b4086aee9a18ababb
Helm 3 breaking changes (likely non-exhaustive):
- crd-install hook removed and replaced with crds directory in
chart where all CRDs defined in it will be installed before
any rendering of the chart
- test-failure hook annotation value removed, and test-success
deprecated. Use test instead
- `--force` no longer handles recreating resources which
cannot be updated due to e.g. immutability [0]
- `--recreate-pods` removed, use declarative approach instead [1]
[0]: https://github.com/helm/helm/issues/7082
[1]: https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: I20ff40ba55197de3d37e5fd647e7d2524a53248f
This removes release rollback/delete functionality. This functionality
was likely not being used and thus was likely not working.
This primary driver for this change is to ease introduction of Helm 3
support. Particularly to avoid having to make API changes related to
the namespacing of helm releases in Helm 3.
This also removes the swagger api documentation as it was not
maintained.
Change-Id: I7edb1c449d43690c87e5bb24726a9fcaf428c00b
This is a pre-requisite for Helm 3 integration, so that these
actions run regardless of whether we are going through the
tiller handler.
Change-Id: I97d7bcc823d11b527fcdaa7967fcab62af1c8161
Moving to falcon 3.0.0+ brings in some changes to the response
object. One of those is the deprecation of the "body" response
field. This PS attempts to get ahead and make the necessary
changes to no longer use deprecated fields.
Change-Id: Iac5d8cd2c658c337dfe7937db8124f3107d77e91
Flake8 version recently updated to include new PEP8 rules. Some of
the codebase is not compliant with the new rules.
Change-Id: Ieb9de200535b6e3fd743e1c0f2e851e81cae79f3
From recently merged document updates in [0] there is a desire to
standardize the Airship project python codebase. This is the effort
to do so for the Armada project.
[0] https://review.opendev.org/#/c/671291/
Change-Id: I4fe916d6e330618ea3a1fccfa4bdfdfabb9ffcb2
Armada remediates releases stuck in FAILED status, if not protected,
by purging and re-installing them. This implements the same for other
non-DEPLOYED statuses. For these statuses it guards this with a best
effort determination of whether a previous deployment of the release,
either through armada or the helm CLI, is likely still pending based
on whether it was last deployed within the chart's wait timeout. If
it is deemed likely pending an error is raised, however this
condition will eventually expire on future runs allowing for
eventual remediation.
Reasons why a release may get stuck in statuses other than DEPLOYED
or FAILED include:
1. tiller crashed mid-deployment
2. tiller could not reach kubernetes to update the release state
3. running `helm delete <rel>` (without --purge) (DELETED status)
Change-Id: Ia89cd59f056103dde47980a149c07a2984c4bbb4
We have seen issues with dangling threads in Armada. This is likely due to
a bug [0] in the version of gRPC that we were pinned to.
This patchset:
- moves us to the latest versions of the gRPC python libraries which add
a new `channel.close()` method to cleanup channels.
- implements the python context manager api in the tiller handler
- uses the context manager api to explicitly scope tiller channel creation
and cleanup to each Armada API and CLI call.
This also fixes a couples issues with error handling introduced in [1].
[0]: https://github.com/grpc/grpc/issues/14338
[1]: https://review.openstack.org/#/c/610384
Change-Id: I2577a20fc76c397aa33157dc12a0e1d36f49733e
This fixes the following issues with listing releases from tiller,
which could cause Armada to be confused about the state of the
latest release, and do the wrong thing.
- Was not filtering out old releases, so we could find both a
FAILED and DEPLOYED release for the same chart. When this is the
case it likely means the FAILED release is the latest, since
otherwise armada would have purged the release (and all its
history) upon seeing the FAILED release in a previous run.
The issue is that after the purge it would try to upgrade
rather than re-install, since it also sees the old DEPLOYED
release. Also if a release gets manually fixed (DEPLOYED)
outside of armada, armada still sees the old FAILED release,
and will purge the fixed release.
- Was only fetching DEPLOYED and FAILED releases from tiller, so if
the latest release has another status Armada won't see it at all.
This changes to:
- Fetch releases with all statuses.
- Filter out old releases.
- Raise an error if latest release has status other than DEPLOYED
or FAILED, since it's not clear what other action to take in
this scenario.
Change-Id: I84712c1486c19d2bba302bf3420df916265ba70c
This patch set consolidates two distinct/disparate base exception
classes together. Currently, Armada has base exceptions in
armada/errors.py and armada/exceptions/base_exceptions.py --
the former of which is a port from Shipyard and the latter of
which is "old-school" Armada code.
The problem is that the two implementations are completely
separate and different. Not only that, but Armada doesn't even
appear to be using armada.errors which it should be as it implements
the canonical UCP standard around error handling.
So this patch set merges the two implementations together. Note,
however, that Armada still doesn't use the same base exception
for all its exceptions which is not ideal because -- again --
Armada should be making use of the UCP error handling logic. This
is seen in the inheritance differentiation between `ArmadaBaseException`
which inherits from Exception and `ArmadaAPIException` which inherits
from `falcon.HTTPError`.
More work is needed to ensure that each exception actually has
an appropriate status code and that said status code is getting
bubbled up to the end user/client/other server as appropriate.
Change-Id: I7cf22fcbba4164f11fb01d9445ac575a14a5c3ab
Tiller has a non-configurable gRPC max response message size. If the
list releases response reaches this size it silently truncates the
results to be below this size. Thus for armada to be able to reliably
get back all the releases it requests, this patchset implements paging
with what should be a small enough page size to avoid the truncation.
Change-Id: Ic2de85f6eabcea8655b18b411b79a863160b0c81
This adds a `wait.resources` key to chart documents which allows
waiting on a list of k8s type+labels configurations to wait on.
Initially supported types are pods, jobs, deployments, daemonsets, and
statefulsets. The behavior for controller types is similar to that of
`kubectl rollout status`.
If `wait.resources` is omitted, it waits on pods and jobs (if any exist)
as before.
The existing `wait.labels` key still have the same behavior, but if
`wait.resources` is also included, the labels are added to each resource
wait in that array. Thus they serve to specify base labels that apply
to all resources in the release, so as to not have to duplicate them.
This may also be useful later for example to use them as labels to wait
for when deleting a chart.
Controller types additionaly have a `min_ready` field which
represents the minimum amount of pods of the controller which must
be ready in order for the controller to be considered ready. The value
can either be an integer or a percent string e.g. "80%", similar to e.g.
`maxUnavailable` in k8s. Default is "100%".
This also wraps up moving the rest of the wait code into its own module.
Change-Id: If72881af0c74e8f765bbb57ac5ffc8d709cd3c16
This changes unsequenced chart group deployments, such that each chart
in the group is deployed in parallel, including the install/upgrade,
wait, and tests.
Previously, whether and when to wait was entangled with whether or not
the chart group was sequenced, since running helm install/upgrade's
native wait (which cannot be run later) and armada's labels based wait,
delayed (or even prevented in the case of failure) the next chart from
being deployed, which is the intention for sequenced, but not for
unsequenced. With this patchset, sequencing and waiting are now
orthogonal. Hence we can now allow the user to explictly specify whether
to wait, which this patchset does for the case of helm's native wait
via a new `wait.native.enabled` flag, which defaults to true.
Previously, armada's labels-based wait sometimes occurred both between
charts and at the end of the chart group. It now occurs once directly
after chart deployment.
Previously, passing armada's --wait was documented to be equivalent to
forcing sequencing of chart groups, however helm tests did not run in
sequence as they normally would with sequenced chart groups, they now
do.
Since chart deploys can now occur in parallel, log messages for each
become interleaved, and thus when armada is deploying a chart, log
messages are updated to contain identifying information about which
chart deployment they are for.
Change-Id: I9d13245c40887712333aaccfb044dcdc4b83988e
This patchset changes the wait logic as follows:
- Move wait logic to own module
- Add framework for waiting on arbitrary resource types
- Unify pod and job wait logic using above framework
- Pass resource_version to k8s watch API for cleaner event tracking
- Only sleep for `k8s_wait_attempt_sleep` when successes not met
- Update to use k8s apps_v1 API where applicable
- Allow passing kwargs to k8s APIs
- Logging cleanups
This is in preparation for adding wait logic for other types of resources
and new wait configurations.
Change-Id: I92e12fe5e0dc8e79c5dd5379799623cf3f471082
In some use cases, some site level docs are only included in specific
manifests. This is so sites can call out what they want deployed, however
currently Armada is checking for all documents to exist and leads
to an invalid manifest exception.
This PS removes the '.build_charts_deps()' and 'build_chart_groups()' calls
in 'get_manifest()' so that only chart documents, and chart group documents
are built after finding them within 'build_armada_manfiest()' and
'build_chart_group()'. 'build_armada_manifest()' will now throw the
related 'Could not find chart group... exception' for related chart
and chart group issues. Additional subclass exceptions were added along
with adding traceback to capture the chained exceptions.
Change-Id: Idc8a75b290ac0afb1e177203535b012d589b708f
We were seeing false positives when diffing charts to determine
whether an upgrade was necessary. Previously we were serializing the
charts and values and diffing those, but these serializations often
output things in different and non-deterministic order, hence the
false positives. This removes the ordering concerns by puttings things
in maps instead of lists, and comparing those semantically rather than
via serialization. This also improves the diff output to be easier to
read. It also stops caring about diffs in Chart.yaml.
Change-Id: I4c92c2e7c814178c374623ea52d717bdb9f72b11
- Adding yapf diff to pep8 target
- Adding yapf tox target to do actual format
** The rest of this PS contains formatted code only, no other changes
Change-Id: Idfef60f53565add2d0cf65bb8e5b91072cf0aded
The helm test integration was severely broken, this fixes it by:
* correctly handle tiller test call response
* removes unnecessary call to tiller to get release content
* removes unnecessary call to k8s to check for test pod completion
* moves common logic into a test handler
* adds test coverage for the above
* adds logging for test results streamed from tiller
Change-Id: I09062387a1abc2fc3f6960f987c97248d9e1cb69
The `protected` parameter will be used to signify that we should
never purge a release in FAILED status. You can specify the
`continue_processing` param to either skip the failed release and
continue on, or to halt armada execution immediately.
- Add protected param to Chart schema and documentation.
- Implement protection logic.
- Moved purging of FAILED releases out of pre-flight and into sync
for finer control over protected params. This means failed
releases are now purged one at a time instead of all up front.
- Added purge and protected charts to final client `msg` return.
- Fix: Added missing dry-run protection in tiller delete resources.
Change-Id: Ia893a486d22cc1022b542ab7c22f58af12025523
This patchset simply removes the unused KnownReleasesException
from the code as well as its reference in documentation. This
exception was previously removed because it wasn't needed.
Change-Id: Ie98227e14dd493f083a10a211ec2913af6625df3
- fixing wait handling in multiple areas
-- wait for deleted pods before continuing Apply update
-- cleaning up and delineating wait for charts vs chartgroups
-- timeout exceptions to stop execution
- api/cli 'timeout' param now applies to all Charts
- api/cli 'wait' param now applies to all Charts
- update some docs
- several TODOs to be addressed in future PS
Closes #199
Change-Id: I5a697508ce6027e9182f3f1f61757319a3ed3593
This adds better exception handling ang logging to
_append_file_to_result helper in get_files. When reading
arbitrary file data and attempting to encode to utf-8
this can cause UnicodeDecodeError to be raised.
However, Armada will not skip over such files; it will
raise an exception with appropriate details instead.
Closes #195
Closes #196
Change-Id: Id7c32c17e351d1ffe042e3755c116c36b6380223
Enhance request logging (and scrub sensitive headers)
Enhance Tiller logging
Update grpcio, unpin from 1.6.0rc1
Plus a couple typo fixes
Plus a couple unused vars
Change-Id: I8afd679f6716c6e1af234a59ac44ba1fdc73cdc8
This PS adds validation logic recently implemented in
armada.utils.validate [0] for validating documents and
Armada-generated Manifests to the Test and Tests controller classes.
Also refactors some exception handling for both controller classes to
better bubble up the appropriate exception.
Finally unit tests have been added for the Armada Test controller
to verify above changes work.
[0] https://review.gerrithub.io/#/c/378700/
Change-Id: I01f73c1778bf7c2e38032d5fddabd327c013edbb
BREAKING CHANGE: Armada will no longer support
recursive monolithic documents such that a Manifest
fully defines ChartGroups inline and ChartGroups
fully define Charts inline. Only name-based references
to other documents is supported.
- Author document schemas in standalone
JSON schema files
- Update validation to return all failures available
- Removed unit tests for support of recursive monolithic
documents
Change-Id: Idb91fa552d3d7a3d7d525609d505fe7380443238
ARMADA-239: Documentation for validation error codes for Armada
3 of 3 commits (Inital Setup and Remaining Exceptions are in seperate commits)
-guide-exceptions.rst contains the files to include in the documentation.
-docs/.../exceptions files contains the format and content of the documenation.
-armada/exceptions files were modified in their comments in order to utilize the
sphinx-directed method.
Please Note: If the exception is not raised anywhere, it is not included in the
documentation.
Change-Id: Ie093cb3252b199bfc7fcd62284501e573d89ed8a
- Add support for SSH key auth using existing config file value
- Add authentication exceptions
- Remove redundant git error handling from Armada handler
Closes #169
Change-Id: Ia0f61e0b74893289bb90560a743a243393d89c56
ARMADA-239: Documentation for validation error codes for Armada
2 of 3 commits (Inital Setup and Remaining Exceptions are in seperate commits)
-guide-exceptions.rst contains the files to include in the documentation.
-docs/.../exceptions files contains the format and content of the documenation.
-armada/exceptions files were modified in their comments in order to utilize the
sphinx-directed method.
-conf.py was modified to indicate the full path for the autoexpection to work
when merged.
Please Note: If the exception is not raised anywhere, it is not included in the
documentation.
Change-Id: I6ddd598bfbb26cdd6ff33682844b0739c440d662
ARMADA-239: Documentation for validation error codes for Armada
1 of 3 Commits (Remaining Exception Documentation are in separate commits)
-guide-exceptions.rst contains the files to include in the documentation.
-docs/.../exceptions files contains the format and content of the documenation.
-armada/exceptions files were modified in their comments in order to utilize the
sphinx-directed method.
-Some files were modified to fix typos.
Please Note: If the exception is not raised anywhere, it is not included in the
documentation.
Change-Id: Icfe45f7b8fe116a0adab14e7f9919b46b022f169
- adding .editorconfig file
- minor cleanup in various files related to .editorconfig
- typos, whitespace, etc.
- other general housekeeping items on the codebase
Change-Id: I104f8dcb06aafb180da12f7ee4c0ded41fc07b9d
Support proxy for installing armada/Chart/v1, as data.source.proxy_server when data.source.type = 'git'.
Closes #191
Change-Id: I9bfd7dbd63c86f65b24b27bd5eec547f862e2311
This PS allows users to specify the manifest file to use
by the Armada handler by introducing a new flag called
`target_manifest`. This flag was added to the API and
CLI.
A foundation of unit tests for the manifest handler
is included in this PS. Most of the coverage is aimed
at checking the various success and failure cases
surrounding the new target_manifest feature.
Also updates documentation to convey information about
the new flag and clean up some documentation formatting
inconsistencies and typos.
Change-Id: I1d5a3ecc1e99b6479438d0ee5490610178be34fe
- Add labels to Tiller pod searching
- Add Tiller pod labels to default config options
- Add exception for case when Tiller pod cannot be found using labels
- Add exception for case when Tiller pod is not in running state
- Update exception documentation
Closes #172
Change-Id: I7e54c4b4a60638bca1073457c256030344832ef9
- using click framework
- added api client
- allow interactions between code and service endpoints
- documention on the command line
- updated gitignore
Change-Id: Ibe359025f5b35606d876c29fa88e04048f276cc8
- Add --set flag to override manifest values from CLI
- Add --values flag to override manifest values from values file
- Add support to override manifests values with API values option
Closes #146
Change-Id: Iefa14e4d3005aab3ee803ffb65dfe1a867507c0e
This patch set makes Armada pep8 compliant. Note the hapi/** is
autogenerated and therefore should be excluded from linting.
Change-Id: I123eefb543f9bd9cf0bc6bd98ed95646d8d72cc3
* Ensure that configurations are done via the global `cfg` object
* Ensure that the logger is configure through the global object
* Upload a configuration sample file with DEFAULT section having
the armada.conf and oslo_log namespace
-Add functionality to download a tarball from a URL, decompress/extract
the archive, and use as a chart source
-Compartmentalized functionality to later support extracting, but not
downloading, local tarballs
-Refactor specific git utils to general source utils
-Small exception handling bug fix
SPEARS, DUSTIN (ds443n)