Commit Graph

43 Commits

Author SHA1 Message Date
Sergiy Markin 386a686e69 [focal] Python modules sync with Airship project
- armada-airskiff-deploy is voting gate again
- fixed falcon.API deprecation - -> falcon.App
- fixed collections.abc.defaultdict not found error
- fixed tox4 requirements
- implemented requirements-frozen.txt approach to make allike as other
  Airship projects
- uplifted docker version in the image building and publishing gate

Change-Id: I337ec07cd6d082acabd9ad65dd9eefb728a43b12
2023-04-21 23:49:14 +00:00
SPEARS, DUSTIN (ds443n) 0e9d828fe6 Update armada to focal
Update kubernetes client to v26.1.0
Updating armada to focal base image
Remove xenial and opensuse dockerfiles
Update tox python from py35 to py38
Add apparmor for docker build
Uplift HTK chart version 0.2.52
Bumping up some python dependencies to get in sync with shipyard
Added clear-firewall role for airskiff-deploy playbook

Change-Id: If06a3f60466702d05a21c24a7cb8041bed41507a
2023-03-20 11:30:37 -04:00
SPEARS, DUSTIN (ds443n) 099de8aaf4 Bump k8s client to v25.3.0
Bumping k8s client to v25.3.0
Cronjob batch v1beta1 no longer available in k8s 1.25
Update tox.ini file to be compatible with v4

Change-Id: Iac79c52c97c9ef1223ae8d502da1572ef8d068fa
2023-01-18 11:25:05 -05:00
Ruslan Aliev 379c88d619 Bump helm to v3.9.4 and k8s client to 24.2.0
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I55d7b34e7525fe6359157d0fa042332a91ac9c95
2022-09-19 11:21:50 -05:00
Ruslan Aliev ef0ac2f252 Bump helm version from 3.6.3 to 3.8.2
Also bump k8s dependency to v1.23.

Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I188788a41419184cc3dd9172f54ab3a290cabe83
2022-06-30 22:19:56 -05:00
Phil Sphicas c5d39f27ca Create lock CRD as apiextensions.k8s.io/v1 object
Kubernetes v1.22 stopped serving the apiextensions.k8s.io/v1beta1 API
version of CustomResourceDefinition.

This change ensures that the locks.armada.process CRD is created using
the apiextensions.k8s.io/v1 API.

The kubernetes client package is also updated to take advantage of the
dynamic client.

Change-Id: Icd518ab5cbb78e8b15f63d19c51b5f5b9a67e995
2022-03-09 16:36:40 -08:00
Sean Eagan a5730f8db8 Remove Tiller
For now we leave the tiller status enpdpoint, until
Shipyard has had a release to stop depending on it [0].

[0]: https://review.opendev.org/c/airship/shipyard/+/802718

Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: If8a02d7118f6840fdbbe088b4086aee9a18ababb
2021-10-05 02:41:32 +00:00
Sean Eagan b37417253f Gate fixes
- Fix helm CLI download URL, the previous URL is no longer available,
  and is superceded by get.helm.sh
- Fix jsonschema dependency spec, 4.x alpha versions were causing the
  unit tests to fail for python 3.5
- Remove oslo.db dependency as it was not used and one of its recursive
  dependencies (tempita) was failing during setup

Change-Id: I03e8a738bb8aad620903119638210077d211770c
2021-09-30 17:21:33 -05:00
DeJaeger, Darren (dd118r) 9aadc14777 Armada improved logging, uplift dependency
This PS:

1) Looks to improve specific logging in Armada, so that
it's easier to debug deployment related issues
2) Uplifts the k8s Python dependency to 12.0.0
3) Enforces 'watch' timeouts more strictly, as the call to
the Kubernetes Python watch function seemed unreliable.
4) Adds a field selector to the 'watch' stream to look for
the DELETE action to have been completed on the specific
pod/job/cronjob, rather than looking across the whole
namespace or via labels. This will narrow what the watch
is looking at, making the logs less busy.

Change-Id: I1952b0db32fb0b56ffffcddeae0532beb5a27b67
2021-06-24 10:53:06 -04:00
Phil Sphicas ce5964a594 Accelerate YAML operations with LibYAML
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.

https://pypi.org/project/pylibyaml/

Change-Id: I4a3045e15df909fb4760800ce1fb87e298dd1e67
2020-09-25 14:29:41 +00:00
Rick Bartra 0c3aff1ba1 Update pip package versions in preparation of pip 20.3
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].

These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.

[0] https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020

Change-Id: Icd10a452a3b9f67b3431d75ba00e942fe42d7c22
2020-09-23 17:10:03 -04:00
Rick Bartra a1b693b84d fix: Gate fix for Armada cover, pep8, and docs gates
Now that the Armada gates are running on Focal(20.04), grpcio
and pyflakes need to use a version compatible with python3.8 and Ubuntu
20.04

Change-Id: I1071d211e5a6c4a5fcdad9818f735fcb3c97d962
2020-09-15 15:01:29 -04:00
Phil Sphicas 6b2c7245de Reestablish watch and retry wait for some errors
Armada uses a Kubernetes watch to implement its chart wait logic. This
can be a fairly long-lived connection to the Kubernetes API server, and
is vulnerable to disruption (if, for example, the kubernetes apiserver
chart is being upgraded).

This change allows Armada to retry the wait for some specific errors,
including the establishment of a new watch, until the overall chart
timeout is reached.

https://github.com/kubernetes-client/python/issues/972
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

Change-Id: I3e68a54becadd5b2a2343960a120bdc3de8e8515
2020-04-19 18:54:03 +00:00
Sean Eagan 268d7a3958 Move kubernetes client to >=11.0.0
This version had a breaking api change [0], which this
aligns with. This version also adds support for kubernetes
1.14 and 1.15 apis.

[0]: https://github.com/kubernetes-client/python/blob/master/CHANGELOG.md#v1100

Change-Id: I01866bd5739e4eebb3166cb583d07efb046360aa
2020-03-20 08:49:45 -05:00
Sean Eagan 0721ed43aa Implement Prometheus metric integration
This implements Prometheus metric integration, including metric
definition, collection, and exportation.

End user documentation for supported metric data and exportation
interface is included.

Change-Id: Ia0837f28073d6cd8e0220ac84cdd261b32704ae4
2019-08-15 16:12:17 +00:00
Sean Eagan 0a87c5a16a Move to kubernetes python client 9.0.0
This version [0] makes the adal package optional [1], which also
removes its dependency on the cryptography package.

It also fixes a thread pool issue [2] allowing us to remove the
workaround we had in place.

[0]: https://github.com/kubernetes-client/python/blob/master/CHANGELOG.md#v900
[1]: https://github.com/kubernetes-client/python-base/pull/108
[2]: https://github.com/kubernetes-client/gen/pull/91

Change-Id: I55aa8b97483b118fbde7e11df817ad8330da9bf1
2019-04-05 15:01:57 -05:00
Sean Eagan 8d54bd4c3a Move grpc tools to test requirements
Grpc tools is only needed for `make protoc` to generate the tiller grpc code.

This will result in it being removed from Armada's docker image as well as
Shipyard's once the Armada git reference is uplifted there to include this change.

Change-Id: Iede466db4fd2d157c2f86235dc6b6e35c706d8d8
2019-01-29 13:12:56 -06:00
Sean Eagan 7af22df7dc Implement tiller gRPC channel clean up
We have seen issues with dangling threads in Armada. This is likely due to
a bug [0] in the version of gRPC that we were pinned to.

This patchset:

- moves us to the latest versions of the gRPC python libraries which add
  a new `channel.close()` method to cleanup channels.
- implements the python context manager api in the tiller handler
- uses the context manager api to explicitly scope tiller channel creation
  and cleanup to each Armada API and CLI call.

This also fixes a couples issues with error handling introduced in [1].

[0]: https://github.com/grpc/grpc/issues/14338
[1]: https://review.openstack.org/#/c/610384

Change-Id: I2577a20fc76c397aa33157dc12a0e1d36f49733e
2018-11-12 13:32:52 -06:00
Sean Eagan 5c5ddf8e8c Move to semantic diffing of charts
We were seeing false positives when diffing charts to determine
whether an upgrade was necessary. Previously we were serializing the
charts and values and diffing those, but these serializations often
output things in different and non-deterministic order, hence the
false positives. This removes the ordering concerns by puttings things
in maps instead of lists, and comparing those semantically rather than
via serialization. This also improves the diff output to be easier to
read. It also stops caring about diffs in Chart.yaml.

Change-Id: I4c92c2e7c814178c374623ea52d717bdb9f72b11
2018-08-20 16:04:11 -05:00
Marshall Margenau 68a507e81b Update Helm version
- Update Helm to v2.9.1
- Update hapi protoc gen files
- Update kubernetes client to >=6

Change-Id: I53480e26683cbaa2b148aaa0f574ee7fb6147ce5
2018-07-20 16:08:28 +00:00
Marshall Margenau (mm8789) efd42dfab2 bug(chartbuilder): uncaught exceptions on bad manifests
Armada was able to throw exceptions all the way up to invocation. To address:
- remove 'supermutes dotify', which was throwing exceptions
- refactor chartbuilder after removing dotify
- rework some helm wait/timeout logic, exposed during bug squash
- rename some variables to make their function more clear

Note: This has potentially breaking changes to existing charts, in the sense
    that documents previously validated (improperly) may now give errors.

Change-Id: I9a6c99aa8ba3d666405e0ccf49a847fd01807b69
2018-03-29 15:15:00 -04:00
Marshall Margenau 13c4e3372a feat(hapi) updating hapi for new grpcio
Change-Id: I8283f5c1cda7d0042d371b382a6d7c49c1705d48
2018-03-09 22:33:39 -05:00
Marshall Margenau 3430283865 feat(logging): Enhance logging and update grpcio
Enhance request logging (and scrub sensitive headers)
Enhance Tiller logging
Update grpcio, unpin from 1.6.0rc1

Plus a couple typo fixes
Plus a couple unused vars

Change-Id: I8afd679f6716c6e1af234a59ac44ba1fdc73cdc8
2018-03-09 11:36:57 -05:00
gardlt 3b879fc846 Improved document validation
BREAKING CHANGE: Armada will no longer support
recursive monolithic documents such that a Manifest
fully defines ChartGroups inline and ChartGroups
fully define Charts inline. Only name-based references
to other documents is supported.

- Author document schemas in standalone
  JSON schema files
- Update validation to return all failures available
- Removed unit tests for support of recursive monolithic
  documents

Change-Id: Idb91fa552d3d7a3d7d525609d505fe7380443238
2018-02-23 11:11:09 -05:00
gardlt 7b26e59422 feat(cli): using-click-framework
- using click framework
- added api client
- allow interactions between code and service endpoints
- documention on the command line
- updated gitignore

Change-Id: Ibe359025f5b35606d876c29fa88e04048f276cc8
2017-11-02 20:59:57 +00:00
gardlt 0663a308d9 feat(tiller): updating-helm-version-2.6.0
- updated hapi lib
- implemented wait-resource-is-ready

Change-Id: Ia547bec0c83e5dca19c87a99dd2cdbe413d78c06
2017-10-27 16:08:47 +00:00
gardlt cba78d1d03 style(armada): converting py2 to py3
- Format code to python 3.5
- using absoulute paths

Change-Id: I7414b5de915429c2c7f85b99f2ab91f395c62121
2017-10-10 19:55:20 +00:00
gardlt bb26131ce2 feat(api): policy enforcement and api standard
- enhanced logging
- created base structure
- updated docs
- PasteDeploy auth
- Oslo Policy

Closes #107

Change-Id: I805863c57f17fcfb26dac5d03efb165e4be49a4e
2017-09-20 21:54:39 +00:00
Kochetov, Mikhail (mk899x) e9790db48d Fix tox/bandit tests
Change-Id: I9dc199ef0361aa4ba08b438f4eca2769a79c2a44
2017-09-14 14:19:49 -04:00
Mark Burnett 24ead6c303 Update grpc dependency to 1.6.0rc1
Version 1.4.0 was released after the `no_proxy` fix, but the fix does
not work in this version.  1.6.0rc1 does respect `no_proxy` as expected.

Change-Id: Ib0e81334765df6d5ffd025b48294fe52724c4506
2017-08-25 12:46:06 -05:00
Mark Burnett 0a47e2f472 Update grpc version
The previous version did not respect no_proxy settings, causing Armada
to attempt to use the proxy to talk to tiller even when not appropriate,
e.g. on localhost.

Change-Id: Ib4e7bd322c7f5781642800d01f3f984f4e6c4133
2017-08-24 11:16:13 -05:00
Tim Heyer ea6ad23672 Migrate from pygit2 to GitPython
Change-Id: Ib477b945e922fd0a1767bd6643c0a45751599fad
2017-08-09 21:20:25 +00:00
drewwalters96 05818f6d00 [Feat] Add API support for core features
- Refactor API structure
- Add API support for existing CLI flags
- Add Keystone token and RBAC authentication
- Add API documentation
- Add API unit tests
2017-07-22 17:06:32 -05:00
Tim Heyer 74a324e298 Update theme to sphinx_rtd 2017-06-28 23:33:55 -05:00
gardlt c1fa5c3e52 testing-utils-lib
* added unittest for the utils module
2017-06-28 12:47:26 -05:00
drewwalters96 928c5f81da [Feat] Add support for Oslo logging configuration
- Adds Oslo logging libraries
- Enables logging configuration with a config file
- Enables debug logging with --trace flag
- Supports Docker logs
- Adds logging for tiller
2017-06-27 16:04:30 -05:00
Omar Rivera 781f2cd3ea [feature] Adds setup for Armada configuration file
* Includes oslo.config library
* Configures initial settings object structure
* Uses command ``tox -egenconfig`` to generate armada.conf template
2017-06-22 10:59:39 -05:00
drewwalters96 368c89664b [WIP][Feat] Add support for Oslo logging capabilities
- Add 'oslo.log' package to requirements.txt
- Replace Python logging library with Oslo logging library in server.py
and log.py
- Update iso8601 silencincing to default option
2017-06-15 16:42:57 -05:00
drewwalters96 d8a1a32b0d fix missing requests dependency
- added requests module to requirements.txt
2017-06-13 12:42:27 -05:00
gardlt 68d95bdcc5 [feature] restructure-clean-up-project
* updating file-structure
* update docker file
* update develop docs
* update api and cmd
2017-06-12 09:06:17 -05:00
gardlt 629a118f49 [feature] implement-api-service
* created tiller status
* created apply api
* fixing prefix-issues
2017-05-01 17:16:11 -05:00
Alexis Rivera DeLa Torre 72c3310707 [script] installing-libgit2-deps (#22)
* removed pip from requirements.txt
* added bash for libgit install
* updated tox
2017-03-12 07:22:29 -07:00
Alan Meadows 501d3d5ca2 Initial Armada Commit (#1)
* Initial commit of Armada framework

* python based .gitignore

* incorporate pull request feedback
2017-02-10 15:27:04 -08:00