This PS bumps up Airflow version to the latest
2.8.2 and also bumps up openstack dependences to
Antelope 2023.1
Change-Id: I7bbcbc8f9e6334100f47702f4546f0963d11b2ca
This PS updates python modules and code to match Airflow 2.6.2 as well
as deploys new Airflow:
- bionic py36 gates were removed
- python code corrected to match new modules versions
- selection of python modules versions was performed based on
airflow-2.6.2 constraints
- airskiff deploy pipeline was aligned with latest in treasuremap v1.9
- shipyard chart was corrected to match new airflow cli, configuration
items and their default values
- added new celery configuration items and their values
- updated airflow runtime logging config
- disabled deprecation and future python warnings in airflow images
- added celery to the list of airflow providers
- adjusted airflow runtime scripts to match new cli
- shipyard SQL queries to airflow DB were adjusted to match new SQL
schema of the db
- shipyard_airflow and shipyard_client unit tests were updated to match
new DB structure and new cli
- airflow db sync job is using db upgrade command
- helm version uplifted to v3.12.2
Change-Id: Ife88e53ce0dd8dc77bf267de1f5e6b8361ca76fd
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.
Change-Id: I4c4c7186e67c771f8249b2017b2507aac17760b7
This PS adds default values for chart values and resolves some issues
in python code that utilizes these values:
validation_connect_timeout: 20
validation_read_timeout: 300
deckhand_client_connect_timeout: 20
deckhand_client_read_timeout: 300
drydock_client_connect_timeout: 20
drydock_client_read_timeout: 300
Change-Id: Ic5b1920257859239613a3ce77134e6b05bd7e9dd
This PS is created to :
- roll back apache-airflow back to 1.10.5
- change default chart values from bionic to ubuntu_focal
- save freshly generated config and policy files samples
- in focal Dockerfile postgres client version is pubped up to v15
- change airflow docker image requirements from limited to fully frozen for shipyard-airflow project
- adjusted requirements-direct.txt for shipyard_airflow
- adjusted requirements-direct.txt for shipyard_client
- regenerated requirements-frozen for both projects
- fixed unit tests after upgrading click module
- gen_all tox profile processing has been moved over to py8 gate because it requires focal zuul node to run
- upgraded airskiff gate playbook to include latest treasuremap patchset with updated airskiff site admada manifests
Change-Id: I47e44f5cfa19b2649697e7cc5a31557a6f4fcfea
- upgraded Airflow to 1.10.15 -
https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html
- disabled xenial, bionic and opensuse images build gates
- added focal image build gate
- added focal zuul build node
- adjusted Makefile for focal
- added bindep.txt to utilize bindep zuul base role for zuul build node
pre-setup
- added focal Dockerfile
- implemented freeze requirements.txt approach like in other Airship
projects
- removed specific requirements.txt for airflow in favor of using
requirements-frozen.txt from shipyard_airflow project when building
airflow docker image
- fixed docker image publishing to Quay
- replaces deprecated LOG.warn with new LOG.warning call
- replaced deprecated body attribute in responce wiht responce.text
attribute
- update of falcon module deprecated .API call - replaced wiht
falcon.App call
- deprecated routing.create_http_method_map method replaced with
routing.map_http_methods
- re-formatted code tabulations based on yapf recommendations
- replaced deprecated protocol attribute in Pytest create_environ() with
http_version attribute
- replaced deprecated app attribute in Pytest create_environ() with
root_path attribute
- fixed airflow CLI commands to match 1.10.15 version
- updated zuul gates to work on focal nodes and added focal specific
node setup items by adding appriate ansible tasks and roles
- uplifted Helm to 3.9.4
- uplifted stable HTK commit id
- updated tox.in to with with tox v4
- uplifted dependences references to other Airship projects
- common python dependences were syncronized with other Airship
projects(Promenade, Deckhand, Armada, Drydock)
- fixed airskiff deployment gate
- fixed genconfig* profiles in shipyard-airflow tox.ini responsible for
maintanance of policy.yaml.sample and shipyard.conf.sample
Change-Id: I0c85187dc9bacf0849382563dd5ff7e9b2814c59
Update Zuul config and tox settings to not run python3.5 jobs, and to
make the xenial image jobs non-voting.
Change-Id: Icc9f71e60e6b48254917ea1d41ce9bc823cce512
* Install older version of pip<21.0 for ubuntu_xenial images
* Install setuptools via pip for ubuntu_xenial images
* Pin typing-extensions to 3.7.2 and apache-airflow to 1.10.5
* Move promenade Dependencies under UCP components in requirements.txt
* With apache-airflow=1.10.5, strip ANSI escape sequences
in test_deployment_group_manager.py
* Update tox.ini to support apache-airflow=1.10.5
* airskiff gate fixes
- Pin treasuremap to v1.9 branch
- Pin openstack-helm-infra to master
- Remove openstack client setup as it's not used
Change-Id: Iee4ce59fdceacb165120a69d11c44e6e47feaea8
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].
These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.
Change-Id: Idd9ea0d57b5be063b133036cfc9ebaa69956f4fc
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.
https://pypi.org/project/pylibyaml/
Change-Id: Ic3f7ba1ce6c4db1a4bc18fe1aef8e0675f7cd69e
Now that the Shipyard gates are running on Focal(20.04), flake8, grpcio,
and pyflakes need to use a version compatible with python3.8 and Ubuntu
20.04 which is the default python version in Ubuntu focal. Also unpinning
setuptools so that it is compatible with python3.8.
Additionally, address pep8 violations that arise from using a newer
version of flake8
Change-Id: Idc3c5d66b48fc9e4497a71d1b640bcd2872c22eb
Uplifted celery to 4.4.2 to address memory leak issue raised
in: https://github.com/celery/celery/issues/4843.
This issue is addressed in celery 4.4.2
Change-Id: I26c403513ba48308044d69130d33561b314fd209
- Adds the information related to the test_site action.
- Reformats, slightly, the output from 'shipyard help actions'
- Adds tests that use an externalized list of actions to keep the help
documentation in alignment with the actions supported in the API.
Change-Id: I2efd473da0dbf6c8cbadfc9fae575c303996c43b
Setuptools is currently listed as a requirement for shipyard-client, but
it is not used. Pegging setuptools forces downgrading when installing
shipyard-client, which can cause issues for packages that attempt to
utilize features found in newer versions of setuptools. Removing the
setuptools requirements will allow installing users to choose their own
version of setuptools to use.
Change-Id: I88570b69346ec7019e753bdd8681fd63d55dc8c8
This change is going to be done in conjunction with Pegleg's shipyard
helper code [0] such that an auth token can be set as an environment
variable. If present it will be used rather than authenticating
against Keystone. This is a requested change by Darren Dejaeger so
that when using automation such as Jenkins pipelines the user is not
forced to provide their credentials to have shipyard automatically
obtain the token - instead they can obtain their token separately and
provide that token for shipyard to use.
[0]: https://opendev.org/airship/pegleg/src/branch/master/pegleg/engine/util/shipyard_helper.py
Change-Id: I859bff978069e8cc3502b952273237e84de83456
Shipyard Client's version of PyYAML (3.13) conflicts with the version
used by Pegleg (5.1). This conflict can cause issues when trying to
install Pegleg as it uses Shipyard Client as a dependency. This update
to Shipyard Client's PyYAML version will fix the dependency conflict
with Pegleg and will also fix a possible security vulnerability
associated with older versions of PyYAML.
Upgrades PyYAML to version 5.1, which makes several changes:
- Deprecates yaml.load in favor of FullLoader and UnsafeLoader due to
yaml.load's vulnerability with code injection [0]
- Drops support for py33 and py26
- Changes to default_flow_style=False
- Several other bug fixes and additional changes
[0] https://nvd.nist.gov/vuln/detail/CVE-2017-18342
Change-Id: I8eb4bb02dcc99eab3d983640d6c769cadb3b97e8
Deckhand's requirement for keystoneauth1 (3.11.1) does not match with
the version used by Shipyard Client (3.11.0). In addition, Shipyard
Client uses requests-2.19.1 which is an insecure version of requests.
Both of these requirements conflict with other Airship project
dependencies. These conflicts become apparent in Pegleg, which uses both
Shipyard Client and Deckhand. The updates made in this change make
Shipyard Client compatible with Pegleg's own dependencies and other
projects it imports from.
Upticks keystoneauth1 to 3.11.1, a bugfix which should not impact
Shipyard Client behavior.
Upticks requests to 2.20.0, which contains several bug fixes and
deprecates support for py26.
Change-Id: I8716c56351e90bca8e7dbdf672638a45f15c655a
This PS adds funtionality to Shipyard to validate the existence of
the Pegleg-generated "deployment-version" document (Pegleg change id:
I7919b02d70c9797f689cdad85066d3953b978901) when a user runs create
configdocs. This validation only checks the presence of the document
(by name and schema) and does not care about the document's other
contents.
The severity of a failed validation is configurable through the
"validations" config section in shipyard.conf. The default severity
is "Skip", meaning the validation is not ran at all.
Note that with the default configuration of new validation, Shipyard
functionality should be unchanged.
Change-Id: I754617de81f628a24232e890b12b157ba6731c25
Currently shipyard raises same error for 2 scenarios-
- Collection empty or resulted in no revision (Error code - 400)
- Buffer is either not empty or the collection already
exists in buffer (Error code - 409)
This PS enables shipyard to raise specific errors for the above
given scenarios which would equip consumers of Shipyard API client
to handle the exception gracefully.
Change-Id: I10860ca60f4fde4088cbb146283a2db305418320
Adds an option to create configdocs as an empty colleciton. This is done
as an explicit flag (--empty-collection) on the command as opposed to
using empty files to prevent accidental emtpy file loads leading to
frustration.
Since this introduced a new flag value for the CLI, the CLIs using flag
values were updated to use the standard is_flag=True instead of the
flag_value=True or some other value when a boolean flag is expected.
Minor updates to CLI tests due to moving to responses 0.10.2
Depends-On: https://review.openstack.org/#/c/614421/
Change-Id: I489b0e1183335cbfbaa2014c1458a84dadf6bb0b
This PS enables shipyard to use PBR as rest
of the other OpenStack project.
Currently if shipyard is used as a requirement in another
project, the dependencies for shipyard doesn't get installed.
This PS fixes this bug.
Change-Id: I34d452649a886af9d865d4ad51eeab70399fe395
- Adds a query parameter 'cleartext-secrets' to get full raw documents.
- Adds CLI flag to get full raw documents.
Change-Id: If38974c8433c8360cc47ae1273720ad76e87a6fd
While iterating on the next steps of using notes, it became clear that
several changes to the output and access methods for notes needed
enhancements. This change introduces a new way to access a note's URL
information via a new API/CLI, while removing the resolution of URLs
from the existing note output. This supports the concept of "builddata"
coming back with sizes of 800kb or more - which really can never work
out inline in other data, especially in cases where there is
multiplicity of the information across many items.
New API: GET /notedetails/{id}
CLI: shipyard get notedetails/{id} and/or shipyard get notedetails {id}
Returns the resolution of the URL for a note, outputting the raw info as
the response (not structured in a JSON response).
The CLI will attempt to minimally format the response if it has inline
\n characters by replacing them will real newlines in the output (if the
output-format is set to either cli or format. Raw format will be
returned as-is.
The existing notes responses are changed to not include the resolution
of the URL information inline, but rather provide the text:
Details at notedetails/{id}
The CLI will interpret this and present:
- Info available with 'describe notedetails/09876543210987654321098765'
This is an attempt to inform the user to access the note details that
way - luckily the API and CLI align on the term notedetails, as the word
details works well enough in the singular form presented by the CLI and
the plural form used by the API.
The ID returned is the unique id of the note (ULID format).
Notes that have no URL will return a 404 response from the API (and
an appropriately formatted value from the CLI).
This approach solves an issue beyond the large inline values from URLs;
providing a means to NOT resolve the URLs except in a one-at-a-time way.
Long lists of notes will no longer have the risk of long waits nor
needing of parallelization of retrieval of URLs for notes.
This change introduces an API-side sorting of notes by timestamp,
providing a chronological presentation of the information that may or
may not match the ULID or insertion ordering of the notes.
Additional feedback from peers about the output of noted indicated that
the CLI formatting of notes in general was in need of visual tuning. As
such, this change introduces changes to the formatting of the output
of notes from the CLI:
- Notes for describing an item will be presented with a more specific
header, e.g.: Action Notes: or Step Notes: instead of simply Notes.
- Tables with notes will change the header from "Notes" to "Footnotes"
give the user a better marker that the notes follow the current
table.
- Table footnotes will be presented in a table format similar to
the following, with headings matching the kind of note being
produced.
Step Footnotes Note
(1) > blah blah blah
> yakkity yakkity
(2) > stuff stuff stuff stuff stuff stuff stuff
stuff stuff stuff
- Info available with 'describe notedetails/...
> things things things
Change-Id: I1680505d5c555b2293419179ade995b0e8484e6d
Enhance the Shipyard API and CLI to retrieve notes that have been
specified against actions and steps. Includes a new reusable parameter
for verbosity.
Change-Id: I1c7f47c0346ce783dacd62b8bbc1fd35a0bf285b
This commit introduces an action, `test_site`, that invokes Helm
tests for all deployed releases using the
`ArmadaTestReleasesOperator` introduced in [1]. This action supports
the ability to invoke Helm tests for a specific release using the
`release` parameter and cleanup resources if the `cleanup` parameter
is set to `true`.
[1] https://review.openstack.org/#/c/603236/
Depends-On: https://review.openstack.org/#/c/603236/
Change-Id: Ib5f38fe4b8a6516ee2afae62774ec84f1d2eb1ad
Updates the image building to Airflow 1.10, including necessary
configuration changes and a general update of dependencies.
Airflow 1.10 includes many enhancements and bugfixes since 1.9 [0]
This change introduces many "unused" configuration parameters to satsify
Airflow's expectations[1]. An ugly, but likely harmless change to the
log output with interleaved newline characters from Airflow steps[2].
Changes to the chart and other dependendencies have also been introduced
to match this update.
[0] https://github.com/apache/incubator-airflow/blob/master/CHANGELOG.txt
[1] https://issues.apache.org/jira/browse/AIRFLOW-3099
[2] https://issues.apache.org/jira/browse/AIRFLOW-1917
Change-Id: I179dcf1f0369650b8c4519f704abb7fb495f4248
1) UCP -> Airship
2) readthedocs.org -> readthedocs.io (there is redirect)
3) http -> https
4) attcomdev -> airshipit (repo on quay.io)
5) att-comdev -> openstack/airship-* (repo on github/openstack git)
6) many URLs have been verified and adjusted to be current
7) no need for 'en/latest/' path in URL of the RTD
8) added more info to some setup.cfg and setup.py files
9) ucp-integration docs are now in airship-in-a-bottle
10) various other minor fixes
Change-Id: I4b8cc6ddf491e35d600a83f5f82d7717108e31dd
The Shipyard API docs and shipyard_api_client incorrectly reference
the wrong API endpoints for some APIs. This commit updates the API
docs to reference the correct API endpoints for:
- getting action validation details
- triggering a control action against an activity
The following is updated in the shipyard_api_client.py:
- API endpoint for getting action validation details
Change-Id: I04b770acfd64f331efce6f83f51cb41d0818a6b2
This change adds the global zuul pep8 tox job, which runs both
bandit and pep8 using tox. This also removes the two other airship
specific lint-pep8 and bandit zuul jobs since they are both covered
by the default openstack global one.
Change-Id: Iebea5b872f78762d6f401b574d53965b2e1c090b
Adds the functionality to redeploy a server in an unguarded fashion,
meaning that the server will not be pre-validated to be in a state that
workloads have been removed.
This is the first targeted action for Shipyard, so a refactoring of the
validators to support more flexibility has been done.
Also adds RBAC controls for specific actions being created, rather than
a binary create action privilege.
Change-Id: I39e3dab6595c5187affb9f2b6edadd0f5f925b0c
Adds a #nosec exclusion to a known Bandit error that has been evaluated
as low severity, and included comments about how the severity is further
reduced.
Adds a target Python version for Bandit job so that it does not fail to
scan several more files that are not working with a Python 2 run of
Bandit.
Change-Id: I251abd092b3049a663b8758bbec0926f4b4836f7
A new Shipyard site statuses API and CLI supporting nodes
provisoning status and node power state. This API
can be further developed to support new status
requirements by expanding the filters option.
Change-Id: I620aefd82d4a17b616f3f253265605e519506257