This PS updates python modules and code to match Airflow 2.6.2:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was perfoemed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
Change-Id: If6f57325339995216d2553c7a5ff56e7673b5acc
- armada-airskiff-deploy is voting gate again
- fixed falcon.API deprecation - -> falcon.App
- fixed collections.abc.defaultdict not found error
- fixed tox4 requirements
- implemented requirements-frozen.txt approach to make allike as other
Airship projects
- uplifted docker version in the image building and publishing gate
Change-Id: I337ec07cd6d082acabd9ad65dd9eefb728a43b12
Update kubernetes client to v26.1.0
Updating armada to focal base image
Remove xenial and opensuse dockerfiles
Update tox python from py35 to py38
Add apparmor for docker build
Uplift HTK chart version 0.2.52
Bumping up some python dependencies to get in sync with shipyard
Added clear-firewall role for airskiff-deploy playbook
Change-Id: If06a3f60466702d05a21c24a7cb8041bed41507a
Bumping k8s client to v25.3.0
Cronjob batch v1beta1 no longer available in k8s 1.25
Update tox.ini file to be compatible with v4
Change-Id: Iac79c52c97c9ef1223ae8d502da1572ef8d068fa
Kubernetes v1.22 stopped serving the apiextensions.k8s.io/v1beta1 API
version of CustomResourceDefinition.
This change ensures that the locks.armada.process CRD is created using
the apiextensions.k8s.io/v1 API.
The kubernetes client package is also updated to take advantage of the
dynamic client.
Change-Id: Icd518ab5cbb78e8b15f63d19c51b5f5b9a67e995
Previously, existing dependency symlinks were causing
FileExistsErrors. This simplifies the logic to ensure
they do get deleted before being recreated. It also
moves to erroring if a chart's `charts` directory is
not a directory, since this is something a chart author
ought to fix, since it needs to be a directory in order
to add dependency sub-directories to it.
Change-Id: I7cc1d50f2bf09602866ce2cf13a4d6c8f6e118ce
For now we leave the tiller status enpdpoint, until
Shipyard has had a release to stop depending on it [0].
[0]: https://review.opendev.org/c/airship/shipyard/+/802718
Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: If8a02d7118f6840fdbbe088b4086aee9a18ababb
Helm 3 breaking changes (likely non-exhaustive):
- crd-install hook removed and replaced with crds directory in
chart where all CRDs defined in it will be installed before
any rendering of the chart
- test-failure hook annotation value removed, and test-success
deprecated. Use test instead
- `--force` no longer handles recreating resources which
cannot be updated due to e.g. immutability [0]
- `--recreate-pods` removed, use declarative approach instead [1]
[0]: https://github.com/helm/helm/issues/7082
[1]: https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: I20ff40ba55197de3d37e5fd647e7d2524a53248f
This removes release rollback/delete functionality. This functionality
was likely not being used and thus was likely not working.
This primary driver for this change is to ease introduction of Helm 3
support. Particularly to avoid having to make API changes related to
the namespacing of helm releases in Helm 3.
This also removes the swagger api documentation as it was not
maintained.
Change-Id: I7edb1c449d43690c87e5bb24726a9fcaf428c00b
This is a pre-requisite for Helm 3 integration, so that these
actions run regardless of whether we are going through the
tiller handler.
Change-Id: I97d7bcc823d11b527fcdaa7967fcab62af1c8161
This reverts commit c75898cd6a.
Airship 2 ended up using the Flux helm-controller instead:
https://github.com/fluxcd/helm-controller
So this is no longer needed. Removing it to get rid of tech
debt to ease introduction of Helm 3 support.
This retains the part of the commit which extracts the
chart download logic to its own handler as this is still useful.
Change-Id: Icb468be2d4916620fd78df250fd038ab58840182
Moving to falcon 3.0.0+ brings in some changes to the response
object. One of those is the deprecation of the "body" response
field. This PS attempts to get ahead and make the necessary
changes to no longer use deprecated fields.
Change-Id: Iac5d8cd2c658c337dfe7937db8124f3107d77e91
This PS:
1) Looks to improve specific logging in Armada, so that
it's easier to debug deployment related issues
2) Uplifts the k8s Python dependency to 12.0.0
3) Enforces 'watch' timeouts more strictly, as the call to
the Kubernetes Python watch function seemed unreliable.
4) Adds a field selector to the 'watch' stream to look for
the DELETE action to have been completed on the specific
pod/job/cronjob, rather than looking across the whole
namespace or via labels. This will narrow what the watch
is looking at, making the logs less busy.
Change-Id: I1952b0db32fb0b56ffffcddeae0532beb5a27b67
This adds the release name to the beginning of the diff debug
log so that it's easier to correlate which diffs belong to which
components when reading the Armada logs.
Change-Id: If096f049a8057a46951ea70f2223a7a987636f5c
In the occasion of a pod being evicted due to low resource availability,
armada keeps waiting for the Evicted pod to be ready. This commit
removes that behavior since Kubernetes will spun up a new pod.
Story: 2008645
Task: 41906
Signed-off-by: Thiago Brito <thiago.brito@windriver.com>
Change-Id: I7263eebe357b0952375d538555536dc9f7cceff4
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.
https://pypi.org/project/pylibyaml/
Change-Id: I4a3045e15df909fb4760800ce1fb87e298dd1e67
Update Helm chart for Armada to use Tiller version 2.16.9.
depends on: https://review.opendev.org/#/c/749497/
Change-Id: I16f7a5e8e571f067154e79a5f2ceb18be7d8db2d
This commit will fix below warning message.
"Post upgrade actions are ignored by Armadaand will not affect deployment.:
Change-Id: I1a8397aa892815bdc96f94a8e49280a4b73a484c
This change adds publishing to docs.airshipit.org and updates the theme
to match the other Airship projects on the site. This change also
updates orphaned links and removes the Read the Docs jobs.
The documentation can be found at docs.airshipit.org/armada when this
change merges.
Change-Id: I9641753f6084f911e3286c623d0c2de7b3f6040a
Signed-off-by: Drew Walters <andrew.walters@att.com>
In cases when chart is in a pending state and last deployment age
for chart is less than the chart wait timeout, armada can consume
all the retries without attempting to purge and re-install the
chart.
This PS addresses the above issue by waiting for the chart until
the wait timeout is met and then proceeding ahead with usual
armada process of purging and re-installing the chart if the status
is not DEPLOYED.
Other code changes in this PS is just re-arranging the existing logic to
fix some edge cases and solve pep8 issues.
Change-Id: Id11d0fb06201264015b3064a43e83903f99a30b1
Flake8 version recently updated to include new PEP8 rules. Some of
the codebase is not compliant with the new rules.
Change-Id: Ieb9de200535b6e3fd743e1c0f2e851e81cae79f3
Armada uses a Kubernetes watch to implement its chart wait logic. This
can be a fairly long-lived connection to the Kubernetes API server, and
is vulnerable to disruption (if, for example, the kubernetes apiserver
chart is being upgraded).
This change allows Armada to retry the wait for some specific errors,
including the establishment of a new watch, until the overall chart
timeout is reached.
https://github.com/kubernetes-client/python/issues/972
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
Change-Id: I3e68a54becadd5b2a2343960a120bdc3de8e8515
The mechanism to disable waits is to explictly set
wait.resources: []
After a recent update [https://review.opendev.org/#/c/697728/], this
configuration results in the default waits (pods and jobs).
This change restores the original functionality.
Change-Id: If5d09f649ca037855c88f928aee6b4dc35ca8b48
Airship 2 is using Argo for workflow management, rather
than the builtin Armada workflow functionality. Hence, this
adds an apply_chart CLI command to apply a single chart at
a time, so that Argo can manage the higher level orchestration.
Airship 2 is also using kubernetes as opposed to Deckhand as the
document store. Hence this adds an ArmadaChart kubernetes CRD,
which can be consumed by the apply_chart CLI command. The chart
`dependencies` feature is intentionally not supported by the CRD,
as there are additional complexities to make that work, and ideally
this feature should be deprecated as charts should be building in
there dependencies before consumption by Armada.
Functional tests are included to excercise these features
against a minikube cluster.
Change-Id: I2bbed83d6d80091322a7e60b918a534188467239
It doesn't appear to be compatible with newer versions of python
and the mock library, and wasn't working correctly anyways.
Change-Id: I117d01bed40849587b2d0337aad56fccdf77e192
This patch set adds in a missing space in a log message so it can be more
easily parsed, as the previous fields were separated by space.
Change-Id: I96cceb644c8193909a91fb42e13c43db0f83ba8d
Signed-off-by: Tin Lam <tin@irrational.io>
Armada intends to use `propagationPolicy: Foreground` when deleting
resources. However, the empty V1DeleteOptions object in the body of the
delete API call takes priority over the propogation_policy specified as
a query param, resulting in the per-resouce default. For Job and CronJob
resources, the default is Orphan, and for others it is Background.
This change includes the desired propogation_policy in V1DeleteOptions.
Reference: https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/#setting-the-cascading-deletion-policy
Change-Id: Iffee12b426ba1e7741eb5bd687ca1b2c11cb071d
Armada has previously named template files relative to the
`templates` dir, whereas the Helm CLI names them relative
to the chart root. This causes `include`s of these templates
to fail.
This change fixes this, for armada/Chart/v2 docs only, since it
is a breaking change, as some charts may have already aligned
with the existing Armada behavior. When updating a release
previously deployed with armada/Chart/v1, the fixed template
names alone will not cause the release to be updated, as the
diff logic accounts for this.
Change-Id: I243073ca4c2e1edbcb0d8f649475f568fc7c818f
Currently the chart `source` schema allows for a proxy server to be
specified, but it is only used for git repos. This patchset allows
the `proxy_server` to also be used for tarball url sources.
Change-Id: I6f90d056fa46f596b1fb248b6c596c58b6513d64
Armada's dry-run option is incomplete, no longer maintained, and offers
little value for the complexity required to maintain it.
This commit is the final in a series of changes to remove the dry-run
feature. Specifically, this change removes the functionality associated
with the dry-run feature.
Story: 2005121
Change-Id: I7dfe5ab27511debe2b8ac01f8e0a696c6126a9f7
Signed-off-by: Drew Walters <andrew.walters@att.com>
Armada's dry-run option is incomplete, no longer maintained, and offers
little value for the complexity required to maintain it.
This commit is the second in a series of changes to remove the dry-run
feature. Specifically, this change removes the flag as an option for the
CLI.
Story: 2005121
Change-Id: If0b512dcf5520e8a53521fbb5e4e83e06984c889
Signed-off-by: Drew Walters <andrew.walters@att.com>
Armada's dry-run option is incomplete, no longer maintained, and offers
little value for the complexity required to maintain it.
This commit is the first in a series of changes to remove the dry-run
feature. Specifically, this change removes the parameter as an option
for the API.
Story: 2005121
Change-Id: If5bd2639fe3e9af3f4cc669cd627b47c1d8fec16
Signed-off-by: Drew Walters <andrew.walters@att.com>
The metrics endpoint can be scraped frequently and produces a large
response body, hence this turns off logging of this data to prevent
the drowning out of other more salient logging.
Change-Id: I399480726ae39963b010bf5b054ce26d73fb28c5
This implements Prometheus metric integration, including metric
definition, collection, and exportation.
End user documentation for supported metric data and exportation
interface is included.
Change-Id: Ia0837f28073d6cd8e0220ac84cdd261b32704ae4
The current logic to find valid releases is incorrect resulting in zero
valid releases being found and all currently installed chart releases
with the release_prefix being removed.
The data passed to this method is a list of chart groups with the
following format:
[{'data':
{'chart_group':
[{'data':
{'chart_name': 'ingress',
'release': 'kube-system-ingress'
...
This update adapts the code to the expected format of the data provided.
Change-Id: Ib285cdf21283b8ec73d583fb107ba49514a9dda7
Signed-off-by: Robert Church <robert.church@windriver.com>
From recently merged document updates in [0] there is a desire to
standardize the Airship project python codebase. This is the effort
to do so for the Armada project.
[0] https://review.opendev.org/#/c/671291/
Change-Id: I4fe916d6e330618ea3a1fccfa4bdfdfabb9ffcb2
Also change enable to enabled.
Before if a user passed in --api and --url TEXT the error message would
return a confusing message that the user must pass in url when api is
enabled.
Now the error message says the user must provide url and token when api
is enabled.
Change-Id: I3284a7e1f920217dc476abcdd1fc17701c85655e
Sergiy Markin