This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.
Change-Id: Id31d97ced8732d823937fb1f218e7ad8760d735c
This PS delivers focal version of Pegleg image and has the following updates:
- removed release-notes-jobs-python3 gate job because of incompartibility with Sphinx from current requirements
- added focal gate node and switched gates to use it
- added bindep.txt file into project root
- added bindep role into gate jobs
- added ubuntu_focal dockerfile for building focal pegleg image
- switched tox profiles to py38
- uplifted references to shipyard_client, promenade and deckhand projects
- resolved required dependencies conflicts by weakening constraints in Pipfile
- updated tox profile update-requirements for generate requirements.txt and test-requirements.txt
- generated new Pipfile.lock, requirements.txt and test-requirements.txt from Pipfile
- switched tox profiles to use requirements.txt and test-requirements.txt instead of pipenv because of upstream zuul nodes Pypi mirrorring issue
- updated reference to seaworthy site certificates in treasuremap repo
- fixed unit tests issues caused by pytest/mock updates and new openssl version
- fixed focal docker image publishing issue
- added multiprocessing into coverage tests running process
- made unit and coveraget tests more verbosive
Change-Id: I5c4c519dc725cfb8c7b4e14756347c9336028aff
Uplift promenade to include:
https://review.opendev.org/c/airship/promenade/+/855432
Small typo. The extra trailing newline disappeared when I added
the comment.
Also removing dependency on gitdb, as it conflicts with gitdb2
which seems to be required.
Change-Id: I8fb9413bf3bf46a68b88635c76b9192e1f9f8b21
Uplift promenade to include the following patchset:
Ensure haproxy.cfg ends with newline to support Haproxy
versions >=2.3
https://review.opendev.org/c/airship/promenade/+/854466
Change-Id: Ifecfd093220226ece45fdb62ec6cb48c6b8732d0
* disable python 35, leave only 36
* switch to bionic nodeset only
* update requirements.txt, Pipfile and lock
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: Ia3c4df9d1f39f0f2cebf8ba0d89aebc5eec6f674
Uplift the embedded version of Promenade to produce a genesis bundle
with updated validation script and pod templates.
Relevant changes:
* Remove log-test pod if validation succeeds [0]
* Update tolerations and priority classes [1]
Full list of changes in Promenade since the last uplift:
* e4d9d99 Update charts to use stable Kubernetes APIs
* e14854b Update HTK stable commit (Ingress)
* 0890626 Update tolerations and priority classes
* e43b6f0 Remove log-test pod if validation succeeds
* 2f823c6 Helm 3: Fix Job labels
[0] https://review.opendev.org/c/airship/promenade/+/814471
[1] https://review.opendev.org/c/airship/promenade/+/814486
Change-Id: I19d790aca9d3f8f23c07e88d5bdb314686fe5528
Updates Pegleg to include the latest Promenade updates.
List of Pegleg changes since the previous uplift:
* 7692b36 Kubelet warning fix
* 183b977 Fix deprecated warning in Promenade controller-manager chart
* 1401664 Fix deprecated warning in Promenade apiserver chart
* 9da1262 Add configMap to proxy chart
Change-Id: I3f36c1575de4c748edc1c640ba9b66b59ca3de26
Updates Pegleg to include the latest Promenade updates, including
changes to support Kubernetes v1.20.
Complete list of Pegleg changes since the previous uplift:
* 06254b3 CoreDNS: Uplift to 1.7.0
* ae6782b Kubernetes: Uplift to v1.20.5
* 5cf854e CoreDNS: Migrate Corefile to version 1.6.4
* 9533be3 Add required apiserver serviceaccount flags
* f3febea Gate stability improvements
* 300a399 apiserver(-webhook): Allow fileless kube-apiserver command_options
Change-Id: I61fb95e0d35bb10b6f22f4dd1cff79a6d5f92df4
Uplift the Promenade commit to pick up a change that disables the
kube-apiserver insecure-port in the Armada bootstrap pod. [0]
List of Promenade changes since the last uplift:
* fd9f3d6 Stop using kube-apiserver insecure-port
* ef66d10 Remove TaintBasedEvictions feature gate references
* c6b62ff apiserver(-webhook): Allow volume overrides
* 27f181a Add configmap-etc-hash to apiserver anchor
* a57158d Disable kubernetes-etcd anchor cleanup in gates
0: https://review.opendev.org/c/airship/promenade/+/784016
Change-Id: Idfb28343b7ab3a69b420f3a63ef0d42a5259a84f
Uplifts promenade to the latest verson. Notably, this removes support
for kubelet extraction from the hyperkube image (which has not been
possible since Kubernetes v1.16).
Complete list of promenade changes since the last uplift:
* 5bb5886 Uplift Kubernetes to v1.19.7
* 023e7d4 Uplift etcd to v3.4.13
* e2324e7 Remove remaining hyperkube references
* 5323ca2 Deploy with standalone kubernetes images
* c7e7294 Remove hyperkube extraction functionality
* 0307391 Update cleanup.sh developer script
Change-Id: I51766a1b9fb8bb2e86f60370625a2bb81fd9e8f6
Uplifts promenade to a version that supports specifying a direct url as
the source for a file in the HostSystem and Genesis schema.
Complete list of promenade changes since the last uplift:
* c9862e5 Allow url as source of file to be deployed to host
* d161528 Avoid calico-etcd crashloop
* 77c7624 Fix ubuntu_xenial build (use pip <21.0)
* 630e504 Update to container image repo k8s.gcr.io
* 5e70957 Merge "Makefile; clean should include .tox"
* 946a28d Use HostToContainer mountPropagation
* f29d6df Ignore upstream chart repos when installing Helm
* 2f2a872 Makefile; clean should include .tox
* 922e3b2 Uplift HTK for etcd backup/restore delete archive capability
* de9f841 kube-apiserver: disable http2
Change-Id: Ia054136956d0a6c3ac24ae1658085a62157427ea
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].
These changes account for the fact that Pegleg imports Shipyard, Promenade,
and Deckhand. Having said that, with pip 20.3, the pip packages amongst
those projects cannot conflict. A follow-up change may be needed if more
conflicts are found.
[0] https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020
Change-Id: Iedf7227c468d678430a5525a2d71d22ce210c557
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.
https://pypi.org/project/pylibyaml/
Change-Id: Ic48d2234ca3107404d9f883ca6038a12ca06a408
Gates are currently failing due to some python requirement conflicts
in Deckhand and Promenade. Promenade was updated in [0] to address
these. Uplifting Pegleg's version of Promenade to resolve gate
errors.
Additional minor fixes:
- Some formatting addressed in Pipfile to remove duplicated entries
ex: oslo.utils and oslo-utils
- Update the commits of Deckhand Promenade and Shipyard used in
building the Docker image
[0] https://review.opendev.org/#/c/734122/
Change-Id: I370e89f9bdd23eaf2ecddec25ace5cc82f2046d7
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
This patchset updates pegleg airship clients for shipyard and
deckhand to use the new clients, which support bionic base image.
Change-Id: I266747b84c39984b941afd6454647fe0d5510ca3
Periodic uplift of Pegleg dependencies, notably:
1. deckhand, shipyard, promenade now on latest
2. uplift of other packages in attempt to resolve security scans
identified on quay.io [0]
[0] https://quay.io/repository/airshipit/pegleg?tab=tags
Change-Id: Id4a2a61fe1748b865e6eca55ea7ce13686855497
This patch uplifts promenade and deckhand versions Pegleg consumes.
Also included are minor package updates.
Change-Id: I1921ae8e215031e36b024ed3badafa67a1f43beb
Upgrades Deckhand to revision supporting six 1.12.0
https://review.opendev.org/#/c/677272/
Installs python3 and overrides python3-six version in OpenSUSE image
Reenables OpenSUSE image build gate
Change-Id: Id72dad8e3668d77b06aa8af4278fcdff0cb678eb
A recent change to implement Pipenv caused VCS dependencies (Promenade,
Deckhand, and Shipyard) to not be fully installed in Docker images. This
change removes the "editable" tags from the VCS dependencies to ensure a
full install as having "editable" enabled will only install dependencies
in development mode.
Unfortunately, the "editable" tag is required to install the
requirements.txt for a VCS dependency. To get the lower-level
dependencies installed from VCS dependencies, I implemented a few
commands in the Dockerfiles to retrieve the appropriate requirements.txt
and install them before fully installing Pegleg. An upcoming release of
Pipenv will fix the existing problems with VCS dependency resolution at
which point this temporary solution may be removed.
Adds manual installation of VCS dependency requirements.txt in Docker
Removes "editable" tags from VCS dependencies
Moves docker package to deployment packages from dev packages
Adds .env file to track VCS refs used by Docker for requirements.txt
Change-Id: Ifdb1fe960b32280dcb3c5308e56b2d608f848975
Pipenv is a tool that brings better package dependency management to
python. It can automatically create and manage virtualenv as well as
managing package dependencies using Pipfile and Pipfile.lock. Adding
this dependency manager into Airship projects will decrease package
version conflicts between projects and help increase security through
hash validation of packages and vulnerability scans.
Changes:
- Imports requirements.txt type files into Pipfile
- Pipenv dependency management in tox
- Switches Safety package for "pipenv check", an implementation of
Safety
- Adds `-e` flag on VCS dependencies to resolve good versions on all
packages
- Unpins or loosens pins on "dev"-type packages
Pipenv Docs: https://docs.pipenv.org/en/latest/
Helpful Pipenv Guide: https://realpython.com/pipenv-guide/
Change-Id: I47e7e7b440d76103b4984499e6ffce4482a59353
I recently received a request to add additional features to Pegleg's
generate passphrases command. The desire was to support multiple
types of secrets:
1. passphrases (24+ characters, including characters from upper,
lower, number, symbol).
2. base64 encoded passphrases.
3. UUID4.
As well as adding an additional flag to prevent Pegleg from
regenerating specific passphrases that are sensitive to rotation.
Finally, responding to an enhancement request interactive
passphrase generation can now be specified via the command line for
all passphrases, or by specifying 'prompt': True for specific
passphrases in passphrase-catalog.yaml
These objectives were completed by:
1. Updating passphrase_catalog.py to support a type field. If a
type is not specified, default to existing passphrase generation.
If an invalid value is specified, raise an exception.
2. Updating passphrase_catalog.py to support a regenerable field. If
the regenerable field is not specified, default to True. If an
invalid value is specified, raise an exception. When regenerable
is determined, secrets of 'uuid' type always use regenerable=False
as they should be one time values created at time of deployment
but not rotated.
3. Updating passphrase_catalog.py to support a prompt field. If the
prompt field is not specified, default to False. If an invalid
value is specified, raise an exception.
4. Adding appropriate exceptions.
5. Updating passphrase_generator.py to handle the new type checks,
UUID will use UUID4, base64 uses the existing logic of generating
a random passphrase and base64 encoding it, and existing logic
remains for generating a random passphrase.
6. Updating passphrase_generator.py to handle the regenerable field.
It checks if a file is present at the expected save path, and if
regenerable=False. If both are true, the passphrase is skipped so
the passphrase is not overwritten.
7. Updating unit tests to validate the new type checks.
NOTE: # nosec is used in passphrase_generator.py on the
'if passphrase_type == <special type>' statements. These are not a
security concern, but do cause Bandit error B105. See documentation
for B105 in [0]
Local testing of the generate passphrase command with the following
passphrase types:
passphrase_b64 : base64
passphrase_uuid : uuid
passphrase_specified : passphrase (specified)
passphrase_defaulted : passphrase (defaulted)
Resulted in the following data for each:
passphrase_b64.yaml:data: !!binary |
UDI1SGFFZHFlbWhITjBrdGJHZGFWRkp6UlZWdFdVNUQ=
passphrase_uuid.yaml:data: 5ce7c6bc-00d2-4b2c-9222-54891f075656
passphrase_specified.yaml:data: cYTenMYXFHUKn6ppYjx#+Hdx
passphrase_defaulted.yaml:data: 13ryjaM?I@sP#3&YQXuQEik4
[0] https://bandit.readthedocs.io/en/latest/plugins/b105_hardcoded_password_string.html
Change-Id: I389316c5194ffa06f3df5114f7ac5f4f2887b319
Uplift deckhand dependency to include support for v2 schemas [0].
[0]: https://review.opendev.org/#/c/666659/
Change-Id: I2970fef857cacf64aa99141d489c3d024456c637
Pegleg is a consumer of deckhand shipyard and promenade. Ensure we have
the latest versions of each project on a periodic basis.
Change-Id: I75b2ebc978dca2f0d57aff41b4b6bc3805af5493
Previously the pins resolved tox tests, but were added to the wrong
requirements file. The dependency error is still present in the
pegleg images. Moving pins to correct file to resolve.
Change-Id: I1796491cbf0e548fac50e01fbe03a379a5c82f9e
Promenade needs to be upgraded to the most recent version for Pegleg to
help transition from using Promenade directly to using Promenade through
Pegleg instead.
Change-Id: I7c62aaf0e257483f4a0aeb06a9798d65c157120f
This patch handles the case where CA certs or authorities are loaded as
byte strings. It also disables parsing YAML documents with python/object
types directly into (non-dict) Python objects (which is PyYaml's
default behavior), as it creates issues with the PeglegManagedDocument
module.
The patch also fixes a bug where attempting to re-encrypt an already
encrypted file would result in a serialized python object being written
rather than the expected output YAML.
Change-Id: I4b84ee8f9922ae042411e70242ffda4622647e86
The dockerfile and some unit tests were still pointing to review.openstack.org
update those references to review.opendev.org
Change-Id: I161158ac0d66533a1775957864d1bd69dfa9530b
The rstr library relies on the normal random.SystemRandom() and does not
perform additional functionalites that would require a whole dependency
to be added to pegleg. This patch set places in the logic into pegleg's
crypto generation rather than using the whole rstr library.
Change-Id: I20c0ceae8ac7d11468f325cd6a4cc035fc176b14
Signed-off-by: Tin Lam <tin@irrational.io>
Added a pegleg cli command to build genesis.sh bundle for
a site deployment.
Pegleg imports promenade engine, and uses promenade to build
and encrypt the genesis.sh deployment bundle.
Change-Id: I1a489459b2c56b7b53018c32aab5e6550c69e1d2
Sergiy Markin