Helm 3 breaking changes (likely non-exhaustive):
- crd-install hook removed and replaced with crds directory in
chart where all CRDs defined in it will be installed before
any rendering of the chart
- test-failure hook annotation value removed, and test-success
deprecated. Use test instead
- `--force` no longer handles recreating resources which
cannot be updated due to e.g. immutability [0]
- `--recreate-pods` removed, use declarative approach instead [1]
[0]: https://github.com/helm/helm/issues/7082
[1]: https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: I20ff40ba55197de3d37e5fd647e7d2524a53248f
Airship 2 is using Argo for workflow management, rather
than the builtin Armada workflow functionality. Hence, this
adds an apply_chart CLI command to apply a single chart at
a time, so that Argo can manage the higher level orchestration.
Airship 2 is also using kubernetes as opposed to Deckhand as the
document store. Hence this adds an ArmadaChart kubernetes CRD,
which can be consumed by the apply_chart CLI command. The chart
`dependencies` feature is intentionally not supported by the CRD,
as there are additional complexities to make that work, and ideally
this feature should be deprecated as charts should be building in
there dependencies before consumption by Armada.
Functional tests are included to excercise these features
against a minikube cluster.
Change-Id: I2bbed83d6d80091322a7e60b918a534188467239
Currently the chart `source` schema allows for a proxy server to be
specified, but it is only used for git repos. This patchset allows
the `proxy_server` to also be used for tarball url sources.
Change-Id: I6f90d056fa46f596b1fb248b6c596c58b6513d64
Armada's dry-run option is incomplete, no longer maintained, and offers
little value for the complexity required to maintain it.
This commit is the final in a series of changes to remove the dry-run
feature. Specifically, this change removes the functionality associated
with the dry-run feature.
Story: 2005121
Change-Id: I7dfe5ab27511debe2b8ac01f8e0a696c6126a9f7
Signed-off-by: Drew Walters <andrew.walters@att.com>
This implements Prometheus metric integration, including metric
definition, collection, and exportation.
End user documentation for supported metric data and exportation
interface is included.
Change-Id: Ia0837f28073d6cd8e0220ac84cdd261b32704ae4
The current logic to find valid releases is incorrect resulting in zero
valid releases being found and all currently installed chart releases
with the release_prefix being removed.
The data passed to this method is a list of chart groups with the
following format:
[{'data':
{'chart_group':
[{'data':
{'chart_name': 'ingress',
'release': 'kube-system-ingress'
...
This update adapts the code to the expected format of the data provided.
Change-Id: Ib285cdf21283b8ec73d583fb107ba49514a9dda7
Signed-off-by: Robert Church <robert.church@windriver.com>
From recently merged document updates in [0] there is a desire to
standardize the Airship project python codebase. This is the effort
to do so for the Armada project.
[0] https://review.opendev.org/#/c/671291/
Change-Id: I4fe916d6e330618ea3a1fccfa4bdfdfabb9ffcb2
This introduces v2 docs in order to allow users to opt in to
breaking changes, while still supporting v1 docs for a time
so folks can migrate. At some point v1 doc support will be
removed.
This initial version of v2 docs is experimental. Further
breaking changes will be made before v2 docs are finalized.
A v1-v2 migration guide is included in the documentation.
This also refactors the internal data model to include the full
document structure, such as `metadata` and `schema`, so that
different behavior can be acheived for v1, v2, etc.
Change-Id: Ia0d44ff4276ef4c27f78706ab02c88aa421a307f
While authoring [0], it was discovered that Armada has duplicate logic
for deciding if Helm test cleanup should be enabled as well as the tests
themselves. Because of this, changes to test logic (e.g. adding pre-test
actions), requires changing all traces of the repeated logic, which can
lead to inconsistent behavior if not properly addressed. This change
moves all test decision logic to a singular Test handler, implemented by
the `Test` class. This change does NOT change the expected behavior of
testing during upgrades; however, tests initiated from the API and CLI
will not execute when testing a manifest if they are disabled in a
chart, unless using the `--enable-all` flag.
[0] https://review.openstack.org/617834
Change-Id: I1530d7637b0eb6a83f048895053a5db80d033046
We have seen issues with dangling threads in Armada. This is likely due to
a bug [0] in the version of gRPC that we were pinned to.
This patchset:
- moves us to the latest versions of the gRPC python libraries which add
a new `channel.close()` method to cleanup channels.
- implements the python context manager api in the tiller handler
- uses the context manager api to explicitly scope tiller channel creation
and cleanup to each Armada API and CLI call.
This also fixes a couples issues with error handling introduced in [1].
[0]: https://github.com/grpc/grpc/issues/14338
[1]: https://review.openstack.org/#/c/610384
Change-Id: I2577a20fc76c397aa33157dc12a0e1d36f49733e
Caching and cleanup of git repository chart sources was previously
implemented. This adds these features for tarball sources as well.
This also implements transitive chart dependency sourcing. Previously
only a single level of dependencies were being downloaded, which
would lead to an error when multiple dependency levels exist.
Change-Id: I988e473a6ea29331e036d26c3ec7269374e0188f
This fixes the following issues with listing releases from tiller,
which could cause Armada to be confused about the state of the
latest release, and do the wrong thing.
- Was not filtering out old releases, so we could find both a
FAILED and DEPLOYED release for the same chart. When this is the
case it likely means the FAILED release is the latest, since
otherwise armada would have purged the release (and all its
history) upon seeing the FAILED release in a previous run.
The issue is that after the purge it would try to upgrade
rather than re-install, since it also sees the old DEPLOYED
release. Also if a release gets manually fixed (DEPLOYED)
outside of armada, armada still sees the old FAILED release,
and will purge the fixed release.
- Was only fetching DEPLOYED and FAILED releases from tiller, so if
the latest release has another status Armada won't see it at all.
This changes to:
- Fetch releases with all statuses.
- Filter out old releases.
- Raise an error if latest release has status other than DEPLOYED
or FAILED, since it's not clear what other action to take in
this scenario.
Change-Id: I84712c1486c19d2bba302bf3420df916265ba70c
This adds a `wait.resources` key to chart documents which allows
waiting on a list of k8s type+labels configurations to wait on.
Initially supported types are pods, jobs, deployments, daemonsets, and
statefulsets. The behavior for controller types is similar to that of
`kubectl rollout status`.
If `wait.resources` is omitted, it waits on pods and jobs (if any exist)
as before.
The existing `wait.labels` key still have the same behavior, but if
`wait.resources` is also included, the labels are added to each resource
wait in that array. Thus they serve to specify base labels that apply
to all resources in the release, so as to not have to duplicate them.
This may also be useful later for example to use them as labels to wait
for when deleting a chart.
Controller types additionaly have a `min_ready` field which
represents the minimum amount of pods of the controller which must
be ready in order for the controller to be considered ready. The value
can either be an integer or a percent string e.g. "80%", similar to e.g.
`maxUnavailable` in k8s. Default is "100%".
This also wraps up moving the rest of the wait code into its own module.
Change-Id: If72881af0c74e8f765bbb57ac5ffc8d709cd3c16
This changes unsequenced chart group deployments, such that each chart
in the group is deployed in parallel, including the install/upgrade,
wait, and tests.
Previously, whether and when to wait was entangled with whether or not
the chart group was sequenced, since running helm install/upgrade's
native wait (which cannot be run later) and armada's labels based wait,
delayed (or even prevented in the case of failure) the next chart from
being deployed, which is the intention for sequenced, but not for
unsequenced. With this patchset, sequencing and waiting are now
orthogonal. Hence we can now allow the user to explictly specify whether
to wait, which this patchset does for the case of helm's native wait
via a new `wait.native.enabled` flag, which defaults to true.
Previously, armada's labels-based wait sometimes occurred both between
charts and at the end of the chart group. It now occurs once directly
after chart deployment.
Previously, passing armada's --wait was documented to be equivalent to
forcing sequencing of chart groups, however helm tests did not run in
sequence as they normally would with sequenced chart groups, they now
do.
Since chart deploys can now occur in parallel, log messages for each
become interleaved, and thus when armada is deploying a chart, log
messages are updated to contain identifying information about which
chart deployment they are for.
Change-Id: I9d13245c40887712333aaccfb044dcdc4b83988e
This patchset changes the wait logic as follows:
- Move wait logic to own module
- Add framework for waiting on arbitrary resource types
- Unify pod and job wait logic using above framework
- Pass resource_version to k8s watch API for cleaner event tracking
- Only sleep for `k8s_wait_attempt_sleep` when successes not met
- Update to use k8s apps_v1 API where applicable
- Allow passing kwargs to k8s APIs
- Logging cleanups
This is in preparation for adding wait logic for other types of resources
and new wait configurations.
Change-Id: I92e12fe5e0dc8e79c5dd5379799623cf3f471082
This makes the following changes to release diffing:
* Move into own handler.
* Add unit tests.
* Include `chart.metadata.name` (from Chart.yaml / dependency dir name)
since that can affect the values passed to dependency charts, and
it's sometimes used by templates via `{{ .Chart.name }}` in ways
which may necessitate an upgrade.
Change-Id: I241384971ed85e5658d71348fee2d6320bbb3c45
We were seeing false positives when diffing charts to determine
whether an upgrade was necessary. Previously we were serializing the
charts and values and diffing those, but these serializations often
output things in different and non-deterministic order, hence the
false positives. This removes the ordering concerns by puttings things
in maps instead of lists, and comparing those semantically rather than
via serialization. This also improves the diff output to be easier to
read. It also stops caring about diffs in Chart.yaml.
Change-Id: I4c92c2e7c814178c374623ea52d717bdb9f72b11
This PS refines the logic in override.update_manifests when validating
documents to properly deduce the correct exception
that needs to be thrown. Added appropriate logic in armada.py to
handle the exceptions thrwon by override.update_manifests. Also
validate_armada_documents now logs appropriate error/debug messages.
Introduced ArmadaNegativeHandlerTestCase class in test_armada.py,
along with updating/adding unit tests in test_override.py.
Change-Id: I84051ae4901011093f987479861df5f89561bb2c
This PS fixes this_chart_should_wait value do not be set True when
user change timeout value.
In SKT CI pipeline, all charts are deployed parallely
(NOT sequentially) so needs more time to deploy than default 900.
But when use --timeout param to change timeout, charts waits
sequentially so can not deploy parallely.
Change-Id: Ibf67c0f98bf2a05b0369ebca68780efb94c4d709
- Armada will now run helm tests by default, and the charts must
disable tests if they choose. A helm test that does not exist
is still a happy-path resolution.
- Documentation and schema updates to signify new deault behavior.
- Preparing to deprecate `test_charts` in ChartGroup processing.
Change-Id: I8e51e33a5d9559b11c2b75e122ecfc97af084ca4
- This change will allow specification of wait labels, which still
apply during ChartGroup processing, without a timeout. The
action will be to take Armada's default timeout value for
waiting on the ChartGroup level, but will now apply labels to the
selector(s) instead of namespace only, if labels are provided.
- A couple logging tweaks.
Change-Id: I2492972abe15f5cd88fad318218a39c1b7ae3a53
Previously the chart `test` key was a boolean. This changes it to an
object which initially supports an `enabled` flag (covering the
previous use case) and adds support for helm's test cleanup option
(underneath an `options` key which mirrors what we have for `upgrade`).
Existing charts will continue to function the same, with cleanup always
turned on, and ability to use the old boolean `test` key for now. When
using the new `test` object however, cleanup defaults to false to match
helm's interface and allow for test pod debugging. Test pods can be
deleted on the next armada apply as well, to allow for debugging in the
meantime, by adding `pre`-`upgrade`-`delete` actions for the test pod.
The `test` commands in the API and CLI now support `cleanup` options as
well.
Change-Id: I92f8822aeaedb0767cb07515d42d8e4f3e088150
- Adding yapf diff to pep8 target
- Adding yapf tox target to do actual format
** The rest of this PS contains formatted code only, no other changes
Change-Id: Idfef60f53565add2d0cf65bb8e5b91072cf0aded
The helm test integration was severely broken, this fixes it by:
* correctly handle tiller test call response
* removes unnecessary call to tiller to get release content
* removes unnecessary call to k8s to check for test pod completion
* moves common logic into a test handler
* adds test coverage for the above
* adds logging for test results streamed from tiller
Change-Id: I09062387a1abc2fc3f6960f987c97248d9e1cb69
The `protected` parameter will be used to signify that we should
never purge a release in FAILED status. You can specify the
`continue_processing` param to either skip the failed release and
continue on, or to halt armada execution immediately.
- Add protected param to Chart schema and documentation.
- Implement protection logic.
- Moved purging of FAILED releases out of pre-flight and into sync
for finer control over protected params. This means failed
releases are now purged one at a time instead of all up front.
- Added purge and protected charts to final client `msg` return.
- Fix: Added missing dry-run protection in tiller delete resources.
Change-Id: Ia893a486d22cc1022b542ab7c22f58af12025523
The release name was being treated as multiple different values to
mean the same thing, when paired with the 'release_prefix'. This
commit addresses the bug, changing all instances to use the
'release' value instead of 'chart_name' or others.
Note: This is an impacting change, in the sense that it will
cause more reliable behavior in Armada's Apply processing which
could have actual impact while upgrading components installed with
a previous version of Armada. Previuosly undeleted FAILED releases
may now be deleted, and armada test and delete actions may now
run as expected where they didn't run before.
Change-Id: I9893e506274e974cdc8826b1812becf9b89a0ab6
This exposes helm's force and recreate pods flags for upgrade and
rollback. It exposes in the chart manifest an options field underneath
the upgrade field to hold options to pass through to helm, and
initializes it with these two flags. Since rollback is currently a
standalone operation which does not consume manifests, these flags are
directly exposed as api and cli arguments there.
Change-Id: If65c1e97d437d9cf9d5838111fd485c80c76aa1d
- revise wait on namespace+label, only wait on ns+label for
charts we've touched in the current apply loop
- skipping any actions that would change system during dry-run
- skip 'test' and 'wait' during dry-run
- tweaking some logs for insight and readability
Change-Id: I1223f01690832c26ce2faa96e7e64620cf413ac9
This patch set imports the constant module rather than the individual
constants. This allows adding future constant into the module easier with
needing to update import, also it is easier for developer to know the
constants are coming from a different module rather than local. Lastly,
this patch set also fixes the alphabetical order of imports.
Change-Id: I886b5368230ab2e11b9792ac7e830ded0fc91d06
Signed-off-by: Tin Lam <tin@irrational.io>
- fixing wait handling in multiple areas
-- wait for deleted pods before continuing Apply update
-- cleaning up and delineating wait for charts vs chartgroups
-- timeout exceptions to stop execution
- api/cli 'timeout' param now applies to all Charts
- api/cli 'wait' param now applies to all Charts
- update some docs
- several TODOs to be addressed in future PS
Closes #199
Change-Id: I5a697508ce6027e9182f3f1f61757319a3ed3593
This patch set fixes indexing improperly through the manifest
handler. This is a straightforward fix by using .get() in conjunction
with sensible defaults (rather than None for everything). However,
the better longterm solution is to use a series of helper functions
to retrieve common properties from Armada objects.
Change-Id: Ic3edf954e2a7b9041a01a254cb5971c02adeca3f
We have been deploying in the low 60's of charts in many environments,
but the current treasuremap environment is running 68.
This led to a situation where tiller would find all 68 charts, but then
return only the newest 64 (RELEASE_LIMIT) in its response. That meant
that armada did not see those charts when deciding whether to install or
upgrade.
Change-Id: I5d2a06f806006947ad48cc0e24b3f91baa3f37fd
- Additional logging to try to expose bug around deleted jobs
during an upgrade.
- Cleaner chart diff logging.
Change-Id: I5edfa1857aec417203e73565a39082328e3b677b
This PS fix to override values by --set option.
Now values come from --set are overriden on self.documents object.
But self.manifest object don't know what happens.
So need to update self.manifest object after override.
Change-Id: I714224788b4b6bd3da3508982821c59b5cff4a42
Armada was able to throw exceptions all the way up to invocation. To address:
- remove 'supermutes dotify', which was throwing exceptions
- refactor chartbuilder after removing dotify
- rework some helm wait/timeout logic, exposed during bug squash
- rename some variables to make their function more clear
Note: This has potentially breaking changes to existing charts, in the sense
that documents previously validated (improperly) may now give errors.
Change-Id: I9a6c99aa8ba3d666405e0ccf49a847fd01807b69
BREAKING CHANGE: Armada will no longer support
recursive monolithic documents such that a Manifest
fully defines ChartGroups inline and ChartGroups
fully define Charts inline. Only name-based references
to other documents is supported.
- Author document schemas in standalone
JSON schema files
- Update validation to return all failures available
- Removed unit tests for support of recursive monolithic
documents
Change-Id: Idb91fa552d3d7a3d7d525609d505fe7380443238
- Add support for SSH key auth using existing config file value
- Add authentication exceptions
- Remove redundant git error handling from Armada handler
Closes #169
Change-Id: Ia0f61e0b74893289bb90560a743a243393d89c56
- adding .editorconfig file
- minor cleanup in various files related to .editorconfig
- typos, whitespace, etc.
- other general housekeeping items on the codebase
Change-Id: I104f8dcb06aafb180da12f7ee4c0ded41fc07b9d
Ruslan Aliev