Helm 3 breaking changes (likely non-exhaustive):
- crd-install hook removed and replaced with crds directory in
chart where all CRDs defined in it will be installed before
any rendering of the chart
- test-failure hook annotation value removed, and test-success
deprecated. Use test instead
- `--force` no longer handles recreating resources which
cannot be updated due to e.g. immutability [0]
- `--recreate-pods` removed, use declarative approach instead [1]
[0]: https://github.com/helm/helm/issues/7082
[1]: https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: I20ff40ba55197de3d37e5fd647e7d2524a53248f
This is a pre-requisite for Helm 3 integration, so that these
actions run regardless of whether we are going through the
tiller handler.
Change-Id: I97d7bcc823d11b527fcdaa7967fcab62af1c8161
Airship 2 is using Argo for workflow management, rather
than the builtin Armada workflow functionality. Hence, this
adds an apply_chart CLI command to apply a single chart at
a time, so that Argo can manage the higher level orchestration.
Airship 2 is also using kubernetes as opposed to Deckhand as the
document store. Hence this adds an ArmadaChart kubernetes CRD,
which can be consumed by the apply_chart CLI command. The chart
`dependencies` feature is intentionally not supported by the CRD,
as there are additional complexities to make that work, and ideally
this feature should be deprecated as charts should be building in
there dependencies before consumption by Armada.
Functional tests are included to excercise these features
against a minikube cluster.
Change-Id: I2bbed83d6d80091322a7e60b918a534188467239
It doesn't appear to be compatible with newer versions of python
and the mock library, and wasn't working correctly anyways.
Change-Id: I117d01bed40849587b2d0337aad56fccdf77e192
This implements Prometheus metric integration, including metric
definition, collection, and exportation.
End user documentation for supported metric data and exportation
interface is included.
Change-Id: Ia0837f28073d6cd8e0220ac84cdd261b32704ae4
From recently merged document updates in [0] there is a desire to
standardize the Airship project python codebase. This is the effort
to do so for the Armada project.
[0] https://review.opendev.org/#/c/671291/
Change-Id: I4fe916d6e330618ea3a1fccfa4bdfdfabb9ffcb2
This introduces v2 docs in order to allow users to opt in to
breaking changes, while still supporting v1 docs for a time
so folks can migrate. At some point v1 doc support will be
removed.
This initial version of v2 docs is experimental. Further
breaking changes will be made before v2 docs are finalized.
A v1-v2 migration guide is included in the documentation.
This also refactors the internal data model to include the full
document structure, such as `metadata` and `schema`, so that
different behavior can be acheived for v1, v2, etc.
Change-Id: Ia0d44ff4276ef4c27f78706ab02c88aa421a307f
- Zuul updated ansible to 2.7, no longer uses missing variables.
- Using an if to try and address.
- Fixes a few formatting problems that are causing the gates to fail
Docker fix based on Aaron Sheffield's PS for Pegleg:
https://review.openstack.org/#/c/645631/
Change-Id: I14e8f3aac0af7a3abc4e2b6c4ece292a24bc4c6a
Previously the timeout for deleting chart releases was 300s and
not configurable, this patchset makes it so via a new
`delete.timeout` property in the `armada/Chart/v1` schema.
Helm releases deleted which do not correspond to documents in this
schema still do not use a configurable timeout. Those will be
considered separately.
This also includes a minor logging fix.
Change-Id: Ia588faaafd18a3ac00eed3cda2f0556ffcec82c9
When running helm tests for a chart release multiple times in a site,
if the previous test pod is not deleted, then the test pod creation
can fail due to a name conflict. Armada/helm support immediate test pod
cleanup, but using this means that upon test failure, the test pod logs will
not be available for debugging purposes. Due to this, the recommended approach
for deleting test pods in Armada has been using `upgrade.pre.delete` actions.
So chart authors can accomplish test pod deletion using this
feature, however, it often takes awhile, usually not until they test upgrading
the chart for chart authors to realize that this is necessary and to get it
implemented.
This patchset automates deletion of test pods directly before running tests by
using the `wait.labels` field in the chart doc when they exist to find all pods
in the release and then using their annotations to determine if they are test
pods and deleting them if so.
A later patchset is planned to implement defaulting of the wait labels when
they are not defined.
Change-Id: I2092f448acb88b5ade3b31b397f9c874c0061668
Fixes a bug where Armada Was looking for upgrade options
(force, recreate_pods currently) underneath `upgrade` directly
rather than `upgrade.options` where they are defined in the schema.
Change-Id: Ia95129a19c87f5d59eaefccd04a7ac9e2acb0b3b
While authoring [0], it was discovered that Armada has duplicate logic
for deciding if Helm test cleanup should be enabled as well as the tests
themselves. Because of this, changes to test logic (e.g. adding pre-test
actions), requires changing all traces of the repeated logic, which can
lead to inconsistent behavior if not properly addressed. This change
moves all test decision logic to a singular Test handler, implemented by
the `Test` class. This change does NOT change the expected behavior of
testing during upgrades; however, tests initiated from the API and CLI
will not execute when testing a manifest if they are disabled in a
chart, unless using the `--enable-all` flag.
[0] https://review.openstack.org/617834
Change-Id: I1530d7637b0eb6a83f048895053a5db80d033046
We have seen issues with dangling threads in Armada. This is likely due to
a bug [0] in the version of gRPC that we were pinned to.
This patchset:
- moves us to the latest versions of the gRPC python libraries which add
a new `channel.close()` method to cleanup channels.
- implements the python context manager api in the tiller handler
- uses the context manager api to explicitly scope tiller channel creation
and cleanup to each Armada API and CLI call.
This also fixes a couples issues with error handling introduced in [1].
[0]: https://github.com/grpc/grpc/issues/14338
[1]: https://review.openstack.org/#/c/610384
Change-Id: I2577a20fc76c397aa33157dc12a0e1d36f49733e
Previously if a chart is not updated, it would simply be skipped over.
Now, the wait/tests are run in this case to ensure the chart success
criteria is/was actually satisfied. It does still skip tests if there
is a last test result recorded as successful already, as an
optimization.
Change-Id: I5dc95fe0f16fe0989761e771c77d2c4fa8f6e7ea
Caching and cleanup of git repository chart sources was previously
implemented. This adds these features for tarball sources as well.
This also implements transitive chart dependency sourcing. Previously
only a single level of dependencies were being downloaded, which
would lead to an error when multiple dependency levels exist.
Change-Id: I988e473a6ea29331e036d26c3ec7269374e0188f
This fixes the following issues with listing releases from tiller,
which could cause Armada to be confused about the state of the
latest release, and do the wrong thing.
- Was not filtering out old releases, so we could find both a
FAILED and DEPLOYED release for the same chart. When this is the
case it likely means the FAILED release is the latest, since
otherwise armada would have purged the release (and all its
history) upon seeing the FAILED release in a previous run.
The issue is that after the purge it would try to upgrade
rather than re-install, since it also sees the old DEPLOYED
release. Also if a release gets manually fixed (DEPLOYED)
outside of armada, armada still sees the old FAILED release,
and will purge the fixed release.
- Was only fetching DEPLOYED and FAILED releases from tiller, so if
the latest release has another status Armada won't see it at all.
This changes to:
- Fetch releases with all statuses.
- Filter out old releases.
- Raise an error if latest release has status other than DEPLOYED
or FAILED, since it's not clear what other action to take in
this scenario.
Change-Id: I84712c1486c19d2bba302bf3420df916265ba70c
This changes unsequenced chart group deployments, such that each chart
in the group is deployed in parallel, including the install/upgrade,
wait, and tests.
Previously, whether and when to wait was entangled with whether or not
the chart group was sequenced, since running helm install/upgrade's
native wait (which cannot be run later) and armada's labels based wait,
delayed (or even prevented in the case of failure) the next chart from
being deployed, which is the intention for sequenced, but not for
unsequenced. With this patchset, sequencing and waiting are now
orthogonal. Hence we can now allow the user to explictly specify whether
to wait, which this patchset does for the case of helm's native wait
via a new `wait.native.enabled` flag, which defaults to true.
Previously, armada's labels-based wait sometimes occurred both between
charts and at the end of the chart group. It now occurs once directly
after chart deployment.
Previously, passing armada's --wait was documented to be equivalent to
forcing sequencing of chart groups, however helm tests did not run in
sequence as they normally would with sequenced chart groups, they now
do.
Since chart deploys can now occur in parallel, log messages for each
become interleaved, and thus when armada is deploying a chart, log
messages are updated to contain identifying information about which
chart deployment they are for.
Change-Id: I9d13245c40887712333aaccfb044dcdc4b83988e
We were seeing false positives when diffing charts to determine
whether an upgrade was necessary. Previously we were serializing the
charts and values and diffing those, but these serializations often
output things in different and non-deterministic order, hence the
false positives. This removes the ordering concerns by puttings things
in maps instead of lists, and comparing those semantically rather than
via serialization. This also improves the diff output to be easier to
read. It also stops caring about diffs in Chart.yaml.
Change-Id: I4c92c2e7c814178c374623ea52d717bdb9f72b11
This PS refines the logic in override.update_manifests when validating
documents to properly deduce the correct exception
that needs to be thrown. Added appropriate logic in armada.py to
handle the exceptions thrwon by override.update_manifests. Also
validate_armada_documents now logs appropriate error/debug messages.
Introduced ArmadaNegativeHandlerTestCase class in test_armada.py,
along with updating/adding unit tests in test_override.py.
Change-Id: I84051ae4901011093f987479861df5f89561bb2c
- Armada will now run helm tests by default, and the charts must
disable tests if they choose. A helm test that does not exist
is still a happy-path resolution.
- Documentation and schema updates to signify new deault behavior.
- Preparing to deprecate `test_charts` in ChartGroup processing.
Change-Id: I8e51e33a5d9559b11c2b75e122ecfc97af084ca4
Previously the chart `test` key was a boolean. This changes it to an
object which initially supports an `enabled` flag (covering the
previous use case) and adds support for helm's test cleanup option
(underneath an `options` key which mirrors what we have for `upgrade`).
Existing charts will continue to function the same, with cleanup always
turned on, and ability to use the old boolean `test` key for now. When
using the new `test` object however, cleanup defaults to false to match
helm's interface and allow for test pod debugging. Test pods can be
deleted on the next armada apply as well, to allow for debugging in the
meantime, by adding `pre`-`upgrade`-`delete` actions for the test pod.
The `test` commands in the API and CLI now support `cleanup` options as
well.
Change-Id: I92f8822aeaedb0767cb07515d42d8e4f3e088150
- Adding yapf diff to pep8 target
- Adding yapf tox target to do actual format
** The rest of this PS contains formatted code only, no other changes
Change-Id: Idfef60f53565add2d0cf65bb8e5b91072cf0aded
The helm test integration was severely broken, this fixes it by:
* correctly handle tiller test call response
* removes unnecessary call to tiller to get release content
* removes unnecessary call to k8s to check for test pod completion
* moves common logic into a test handler
* adds test coverage for the above
* adds logging for test results streamed from tiller
Change-Id: I09062387a1abc2fc3f6960f987c97248d9e1cb69
The `protected` parameter will be used to signify that we should
never purge a release in FAILED status. You can specify the
`continue_processing` param to either skip the failed release and
continue on, or to halt armada execution immediately.
- Add protected param to Chart schema and documentation.
- Implement protection logic.
- Moved purging of FAILED releases out of pre-flight and into sync
for finer control over protected params. This means failed
releases are now purged one at a time instead of all up front.
- Added purge and protected charts to final client `msg` return.
- Fix: Added missing dry-run protection in tiller delete resources.
Change-Id: Ia893a486d22cc1022b542ab7c22f58af12025523
The release name was being treated as multiple different values to
mean the same thing, when paired with the 'release_prefix'. This
commit addresses the bug, changing all instances to use the
'release' value instead of 'chart_name' or others.
Note: This is an impacting change, in the sense that it will
cause more reliable behavior in Armada's Apply processing which
could have actual impact while upgrading components installed with
a previous version of Armada. Previuosly undeleted FAILED releases
may now be deleted, and armada test and delete actions may now
run as expected where they didn't run before.
Change-Id: I9893e506274e974cdc8826b1812becf9b89a0ab6
This exposes helm's force and recreate pods flags for upgrade and
rollback. It exposes in the chart manifest an options field underneath
the upgrade field to hold options to pass through to helm, and
initializes it with these two flags. Since rollback is currently a
standalone operation which does not consume manifests, these flags are
directly exposed as api and cli arguments there.
Change-Id: If65c1e97d437d9cf9d5838111fd485c80c76aa1d
This is a multi-part PS because these patches may include small
fix-ups to the code base itself, so the intention is to keep
the patches small and easily reversible. This patchset introduces
the following:
* html coverage report (execute tox -e cover then open index.html
under htmlcov folder which is created by py.test)
* adds additional unit tests for pre_flight_ops
* adds more robust assertions for those tests
Change-Id: Ib29d7d8d0c3b686a36c5a87fc46d4594bb1838a6
- revise wait on namespace+label, only wait on ns+label for
charts we've touched in the current apply loop
- skipping any actions that would change system during dry-run
- skip 'test' and 'wait' during dry-run
- tweaking some logs for insight and readability
Change-Id: I1223f01690832c26ce2faa96e7e64620cf413ac9
- fixing wait handling in multiple areas
-- wait for deleted pods before continuing Apply update
-- cleaning up and delineating wait for charts vs chartgroups
-- timeout exceptions to stop execution
- api/cli 'timeout' param now applies to all Charts
- api/cli 'wait' param now applies to all Charts
- update some docs
- several TODOs to be addressed in future PS
Closes #199
Change-Id: I5a697508ce6027e9182f3f1f61757319a3ed3593
Armada was able to throw exceptions all the way up to invocation. To address:
- remove 'supermutes dotify', which was throwing exceptions
- refactor chartbuilder after removing dotify
- rework some helm wait/timeout logic, exposed during bug squash
- rename some variables to make their function more clear
Note: This has potentially breaking changes to existing charts, in the sense
that documents previously validated (improperly) may now give errors.
Change-Id: I9a6c99aa8ba3d666405e0ccf49a847fd01807b69
- Validation messaging to match UCP convention
- Adding some missing fields to Chart validation schema
- Minor update: Adding debug logging to each CLI call
- Fixing some typos and exception messages
Change-Id: I7dc1165432c8b3d138cabe6fd5f3a6e1878810ae
BREAKING CHANGE: Armada will no longer support
recursive monolithic documents such that a Manifest
fully defines ChartGroups inline and ChartGroups
fully define Charts inline. Only name-based references
to other documents is supported.
- Author document schemas in standalone
JSON schema files
- Update validation to return all failures available
- Removed unit tests for support of recursive monolithic
documents
Change-Id: Idb91fa552d3d7a3d7d525609d505fe7380443238
- Add support for SSH key auth using existing config file value
- Add authentication exceptions
- Remove redundant git error handling from Armada handler
Closes #169
Change-Id: Ia0f61e0b74893289bb90560a743a243393d89c56
This PS adds unit tests for the armada handler in
armada.handlers.armada. These unit tests are among
the most important as the Armada handler itself
interacts with most every other part of the application.
Note that the current unit tests are not only incorrect
(they don't pass) but they skip unconditionally as well.
So this PS is needed to correctly implement the intent
behind the original unit tests herein.
Change-Id: Iecb3e540e1d52eb5a25d9f6825b3d0f5339ede2a
* Ensure that configurations are done via the global `cfg` object
* Ensure that the logger is configure through the global object
* Upload a configuration sample file with DEFAULT section having
the armada.conf and oslo_log namespace
-No longer attempts to ping repo
-Move all source locating (git or local) to preflight
-Only unique repos are cloned now
-Catches any source/url errors in preflight and raises exception
- Register Oslo logging configurations in server.py, test_armada.py, test_chartbuilder.py, and test_tiller.py
- Add mock to test-requirements.txt
- Update .travis.yml
- Add mock case for tiller IP
-Update tiller.py and armada.py to support native tiller timeout
-Update documentation with the new yaml timeout keyword
-Update tiller version to 2.4.2
-Create tests for timeout ability, as well as structure for further test development
-Fix gRPC message size bug
Sean Eagan