Helm 3 breaking changes (likely non-exhaustive):
- crd-install hook removed and replaced with crds directory in
chart where all CRDs defined in it will be installed before
any rendering of the chart
- test-failure hook annotation value removed, and test-success
deprecated. Use test instead
- `--force` no longer handles recreating resources which
cannot be updated due to e.g. immutability [0]
- `--recreate-pods` removed, use declarative approach instead [1]
[0]: https://github.com/helm/helm/issues/7082
[1]: https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: I20ff40ba55197de3d37e5fd647e7d2524a53248f
This is a pre-requisite for Helm 3 integration, so that these
actions run regardless of whether we are going through the
tiller handler.
Change-Id: I97d7bcc823d11b527fcdaa7967fcab62af1c8161
This reverts commit c75898cd6a.
Airship 2 ended up using the Flux helm-controller instead:
https://github.com/fluxcd/helm-controller
So this is no longer needed. Removing it to get rid of tech
debt to ease introduction of Helm 3 support.
This retains the part of the commit which extracts the
chart download logic to its own handler as this is still useful.
Change-Id: Icb468be2d4916620fd78df250fd038ab58840182
This adds the release name to the beginning of the diff debug
log so that it's easier to correlate which diffs belong to which
components when reading the Armada logs.
Change-Id: If096f049a8057a46951ea70f2223a7a987636f5c
This commit will fix below warning message.
"Post upgrade actions are ignored by Armadaand will not affect deployment.:
Change-Id: I1a8397aa892815bdc96f94a8e49280a4b73a484c
In cases when chart is in a pending state and last deployment age
for chart is less than the chart wait timeout, armada can consume
all the retries without attempting to purge and re-install the
chart.
This PS addresses the above issue by waiting for the chart until
the wait timeout is met and then proceeding ahead with usual
armada process of purging and re-installing the chart if the status
is not DEPLOYED.
Other code changes in this PS is just re-arranging the existing logic to
fix some edge cases and solve pep8 issues.
Change-Id: Id11d0fb06201264015b3064a43e83903f99a30b1
Airship 2 is using Argo for workflow management, rather
than the builtin Armada workflow functionality. Hence, this
adds an apply_chart CLI command to apply a single chart at
a time, so that Argo can manage the higher level orchestration.
Airship 2 is also using kubernetes as opposed to Deckhand as the
document store. Hence this adds an ArmadaChart kubernetes CRD,
which can be consumed by the apply_chart CLI command. The chart
`dependencies` feature is intentionally not supported by the CRD,
as there are additional complexities to make that work, and ideally
this feature should be deprecated as charts should be building in
there dependencies before consumption by Armada.
Functional tests are included to excercise these features
against a minikube cluster.
Change-Id: I2bbed83d6d80091322a7e60b918a534188467239
Armada's dry-run option is incomplete, no longer maintained, and offers
little value for the complexity required to maintain it.
This commit is the final in a series of changes to remove the dry-run
feature. Specifically, this change removes the functionality associated
with the dry-run feature.
Story: 2005121
Change-Id: I7dfe5ab27511debe2b8ac01f8e0a696c6126a9f7
Signed-off-by: Drew Walters <andrew.walters@att.com>
This implements Prometheus metric integration, including metric
definition, collection, and exportation.
End user documentation for supported metric data and exportation
interface is included.
Change-Id: Ia0837f28073d6cd8e0220ac84cdd261b32704ae4
From recently merged document updates in [0] there is a desire to
standardize the Airship project python codebase. This is the effort
to do so for the Armada project.
[0] https://review.opendev.org/#/c/671291/
Change-Id: I4fe916d6e330618ea3a1fccfa4bdfdfabb9ffcb2
This is just a basic Helm CLI option, so this moves it to be alongside
the rest of the upgrade options.
Change-Id: I4cbb4f3bfe60240d793a30f7a7d58705024f633c
This introduces v2 docs in order to allow users to opt in to
breaking changes, while still supporting v1 docs for a time
so folks can migrate. At some point v1 doc support will be
removed.
This initial version of v2 docs is experimental. Further
breaking changes will be made before v2 docs are finalized.
A v1-v2 migration guide is included in the documentation.
This also refactors the internal data model to include the full
document structure, such as `metadata` and `schema`, so that
different behavior can be acheived for v1, v2, etc.
Change-Id: Ia0d44ff4276ef4c27f78706ab02c88aa421a307f
Currently, tests executed during chart deployment use the wait timeout
value, `wait.timeout`. This value can be too large of a timeout value
for Helm tests. This change introduces a timeout for tests,
`test.timeout` that is only used as a timeout for running Helm tests for
a release.
Story: 2003899
Depends-On: https://review.openstack.org/618355
Change-Id: Iee746444d5aede0b84b1805eb19f59f0f03c8f9e
Previously the timeout for deleting chart releases was 300s and
not configurable, this patchset makes it so via a new
`delete.timeout` property in the `armada/Chart/v1` schema.
Helm releases deleted which do not correspond to documents in this
schema still do not use a configurable timeout. Those will be
considered separately.
This also includes a minor logging fix.
Change-Id: Ia588faaafd18a3ac00eed3cda2f0556ffcec82c9
When running helm tests for a chart release multiple times in a site,
if the previous test pod is not deleted, then the test pod creation
can fail due to a name conflict. Armada/helm support immediate test pod
cleanup, but using this means that upon test failure, the test pod logs will
not be available for debugging purposes. Due to this, the recommended approach
for deleting test pods in Armada has been using `upgrade.pre.delete` actions.
So chart authors can accomplish test pod deletion using this
feature, however, it often takes awhile, usually not until they test upgrading
the chart for chart authors to realize that this is necessary and to get it
implemented.
This patchset automates deletion of test pods directly before running tests by
using the `wait.labels` field in the chart doc when they exist to find all pods
in the release and then using their annotations to determine if they are test
pods and deleting them if so.
A later patchset is planned to implement defaulting of the wait labels when
they are not defined.
Change-Id: I2092f448acb88b5ade3b31b397f9c874c0061668
Armada remediates releases stuck in FAILED status, if not protected,
by purging and re-installing them. This implements the same for other
non-DEPLOYED statuses. For these statuses it guards this with a best
effort determination of whether a previous deployment of the release,
either through armada or the helm CLI, is likely still pending based
on whether it was last deployed within the chart's wait timeout. If
it is deemed likely pending an error is raised, however this
condition will eventually expire on future runs allowing for
eventual remediation.
Reasons why a release may get stuck in statuses other than DEPLOYED
or FAILED include:
1. tiller crashed mid-deployment
2. tiller could not reach kubernetes to update the release state
3. running `helm delete <rel>` (without --purge) (DELETED status)
Change-Id: Ia89cd59f056103dde47980a149c07a2984c4bbb4
Fixes a bug where Armada Was looking for upgrade options
(force, recreate_pods currently) underneath `upgrade` directly
rather than `upgrade.options` where they are defined in the schema.
Change-Id: Ia95129a19c87f5d59eaefccd04a7ac9e2acb0b3b
While authoring [0], it was discovered that Armada has duplicate logic
for deciding if Helm test cleanup should be enabled as well as the tests
themselves. Because of this, changes to test logic (e.g. adding pre-test
actions), requires changing all traces of the repeated logic, which can
lead to inconsistent behavior if not properly addressed. This change
moves all test decision logic to a singular Test handler, implemented by
the `Test` class. This change does NOT change the expected behavior of
testing during upgrades; however, tests initiated from the API and CLI
will not execute when testing a manifest if they are disabled in a
chart, unless using the `--enable-all` flag.
[0] https://review.openstack.org/617834
Change-Id: I1530d7637b0eb6a83f048895053a5db80d033046
Armada does not perform post upgrade actions. This change adds a warning
to the documentation, comments, and output.
Change-Id: I4d37406e13a44759861ea179d06b26831efe2ac8
Previously if a chart is not updated, it would simply be skipped over.
Now, the wait/tests are run in this case to ensure the chart success
criteria is/was actually satisfied. It does still skip tests if there
is a last test result recorded as successful already, as an
optimization.
Change-Id: I5dc95fe0f16fe0989761e771c77d2c4fa8f6e7ea
When existing release found log that, and only log about upgrading
if actually doing that (due to changes found in the relase).
Change-Id: I54d224ce3abe4fabc74b3aaa32b1326eb1fd0d4f
This fixes the following issues with listing releases from tiller,
which could cause Armada to be confused about the state of the
latest release, and do the wrong thing.
- Was not filtering out old releases, so we could find both a
FAILED and DEPLOYED release for the same chart. When this is the
case it likely means the FAILED release is the latest, since
otherwise armada would have purged the release (and all its
history) upon seeing the FAILED release in a previous run.
The issue is that after the purge it would try to upgrade
rather than re-install, since it also sees the old DEPLOYED
release. Also if a release gets manually fixed (DEPLOYED)
outside of armada, armada still sees the old FAILED release,
and will purge the fixed release.
- Was only fetching DEPLOYED and FAILED releases from tiller, so if
the latest release has another status Armada won't see it at all.
This changes to:
- Fetch releases with all statuses.
- Filter out old releases.
- Raise an error if latest release has status other than DEPLOYED
or FAILED, since it's not clear what other action to take in
this scenario.
Change-Id: I84712c1486c19d2bba302bf3420df916265ba70c
This adds a `wait.resources` key to chart documents which allows
waiting on a list of k8s type+labels configurations to wait on.
Initially supported types are pods, jobs, deployments, daemonsets, and
statefulsets. The behavior for controller types is similar to that of
`kubectl rollout status`.
If `wait.resources` is omitted, it waits on pods and jobs (if any exist)
as before.
The existing `wait.labels` key still have the same behavior, but if
`wait.resources` is also included, the labels are added to each resource
wait in that array. Thus they serve to specify base labels that apply
to all resources in the release, so as to not have to duplicate them.
This may also be useful later for example to use them as labels to wait
for when deleting a chart.
Controller types additionaly have a `min_ready` field which
represents the minimum amount of pods of the controller which must
be ready in order for the controller to be considered ready. The value
can either be an integer or a percent string e.g. "80%", similar to e.g.
`maxUnavailable` in k8s. Default is "100%".
This also wraps up moving the rest of the wait code into its own module.
Change-Id: If72881af0c74e8f765bbb57ac5ffc8d709cd3c16
This changes unsequenced chart group deployments, such that each chart
in the group is deployed in parallel, including the install/upgrade,
wait, and tests.
Previously, whether and when to wait was entangled with whether or not
the chart group was sequenced, since running helm install/upgrade's
native wait (which cannot be run later) and armada's labels based wait,
delayed (or even prevented in the case of failure) the next chart from
being deployed, which is the intention for sequenced, but not for
unsequenced. With this patchset, sequencing and waiting are now
orthogonal. Hence we can now allow the user to explictly specify whether
to wait, which this patchset does for the case of helm's native wait
via a new `wait.native.enabled` flag, which defaults to true.
Previously, armada's labels-based wait sometimes occurred both between
charts and at the end of the chart group. It now occurs once directly
after chart deployment.
Previously, passing armada's --wait was documented to be equivalent to
forcing sequencing of chart groups, however helm tests did not run in
sequence as they normally would with sequenced chart groups, they now
do.
Since chart deploys can now occur in parallel, log messages for each
become interleaved, and thus when armada is deploying a chart, log
messages are updated to contain identifying information about which
chart deployment they are for.
Change-Id: I9d13245c40887712333aaccfb044dcdc4b83988e
francisy