Commit Graph

41 Commits

Author SHA1 Message Date
Sergiy Markin 9c28c832dd Shipyard timeout issue
This PS  adds default values for chart values and resolves some issues
in python code that utilizes these values:

      validation_connect_timeout: 20
      validation_read_timeout: 300
      deckhand_client_connect_timeout: 20
      deckhand_client_read_timeout: 300
      drydock_client_connect_timeout: 20
      drydock_client_read_timeout: 300

Change-Id: Ic5b1920257859239613a3ce77134e6b05bd7e9dd
2023-05-16 20:37:29 +00:00
Sergiy Markin b7e7d25308 Rollback apache-airflow back to 1.10.5
This PS is created to :
- roll back apache-airflow back to 1.10.5
- change default chart values from bionic to ubuntu_focal
- save freshly generated config and policy files samples
- in focal Dockerfile postgres client version is pubped up to v15
- change airflow docker image requirements from limited to fully frozen for shipyard-airflow project
- adjusted requirements-direct.txt for shipyard_airflow
- adjusted requirements-direct.txt for shipyard_client
- regenerated requirements-frozen for both projects
- fixed unit tests after upgrading click module
- gen_all tox profile processing has been moved over to py8 gate because it requires focal zuul node to run
- upgraded airskiff gate playbook to include latest treasuremap patchset with updated airskiff site admada manifests

Change-Id: I47e44f5cfa19b2649697e7cc5a31557a6f4fcfea
2023-05-08 21:58:34 +00:00
Sergiy Markin 154a099b28 Shipyard upgrade for focal
- upgraded Airflow to 1.10.15  -
  https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html
- disabled xenial, bionic and opensuse images build  gates
- added focal image build  gate
- added focal zuul build node
- adjusted Makefile for focal
- added bindep.txt to utilize bindep zuul base role for zuul build node
  pre-setup
- added focal Dockerfile
- implemented freeze requirements.txt approach like in other Airship
  projects
- removed specific requirements.txt for airflow in favor of using
  requirements-frozen.txt from shipyard_airflow project when building
  airflow docker image
- fixed docker image publishing to Quay
- replaces deprecated LOG.warn with new LOG.warning call
- replaced deprecated body attribute in responce wiht responce.text
  attribute
- update of falcon module deprecated .API call - replaced wiht
  falcon.App call
- deprecated routing.create_http_method_map method replaced with
  routing.map_http_methods
- re-formatted code tabulations based on yapf recommendations
- replaced deprecated protocol attribute in Pytest create_environ() with
  http_version attribute
- replaced deprecated app attribute in Pytest create_environ() with
  root_path attribute
- fixed airflow CLI commands to match 1.10.15 version
- updated zuul gates to work on focal nodes and added focal specific
  node setup items by adding appriate ansible tasks and roles
- uplifted Helm to 3.9.4
- uplifted stable HTK commit id
- updated tox.in to with with tox v4
- uplifted dependences references to other Airship projects
- common python dependences were syncronized with other Airship
  projects(Promenade, Deckhand, Armada, Drydock)
- fixed airskiff deployment gate
- fixed genconfig* profiles in shipyard-airflow tox.ini responsible for
  maintanance of policy.yaml.sample and shipyard.conf.sample

Change-Id: I0c85187dc9bacf0849382563dd5ff7e9b2814c59
2023-04-28 20:40:50 +00:00
Ruslan Aliev 9a8218aa25 Bump airship dependencies
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I61d65874864f233372df72787cdb25a54b2edbc3
2022-07-07 14:14:51 -05:00
Sean Eagan e64f17b91b Remove Helm 2 based Armada integrations
- Tiller status check
- Test cleanup arg removed

Depends-On: https://review.opendev.org/c/airship/armada/+/812047
Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: I77ef3fb8e952ad28132e3476138d34bbb5a6fd3d
2021-10-04 15:03:04 -05:00
Ahmad Mahmoudi 24f6a01e0b Add support for Ubuntu bionic base image
Added support to buid shipyard and airflow images using either a xenial
or Ubuntu bionic base image.
The default base image is set to bionic.

Change-Id: I6ad4d42dede081586b3ccea87a42e250979ac106
2020-02-04 13:38:39 -06:00
Zuul 9c5f68cf97 Merge "Add support domain-scoped token for CLI" 2019-08-07 22:34:14 +00:00
Svetlana Shturm d8332bcebb Add support domain-scoped token for CLI
Change-Id: I4c0b4a6de9598e234e0b49b9cfc0b4dc5daaf7be
2019-08-02 10:15:20 -05:00
Andreas Jaeger 9b2c000e90 Update OpenStack api-ref location
The api documentation is now published on docs.openstack.org instead
of developer.openstack.org. Update all links that are changed to the
new location.

Note that redirects will be set up as well but let's point now to the
new location.

For details, see:
http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007828.html

Change-Id: I09b5d45fb5ca505b1f97d0a78cc082c1e0254edd
2019-07-30 19:44:33 +02:00
Rajeshwari Dharwadkar 6a54c62828 Update base image from leap15.0 to leap15.1
Changes made in opensuse_15 dockerfile to support opensuse leap15.1

Change-Id: I3b010c15745b505896066fc144cbc57487dd6688
2019-07-11 17:15:06 -07:00
Michael Beaver 02533cb085 Add unit tests and fix comments/samples
Follow up change for deployment-status ConfigMap [0] adding a few
things that were either missed or need to be added, especially unit
tests for the new DeploymentStatusOperator class

[0] https://review.opendev.org/#/c/660197/

Change-Id: Ieaf55dc7725ea92062fde7997a676726b2396253
2019-05-29 15:02:59 -05:00
Arun Kant cd6f154da5 Adding opensuse support in image building of airflow and shipyard
Adding DISTRO parameter for makefile to invoke distribution specific
Dockerfile and build image accordingly.

Also qualifying image job name to distinguish ubuntu vs opensuse
check jobs and passing distro specific variable.
Similar logic for pushing distro specific images.

Added new document section for multiple distro support.

Change-Id: I215f8a107cab8770181c0472e0cc29053b2b978e
2019-05-14 09:52:24 -07:00
Carter, Matthew (mc981n) 0761099337 Validate existence of "deployment-version" during create configdocs
This PS adds funtionality to Shipyard to validate the existence of
the Pegleg-generated "deployment-version" document (Pegleg change id:
I7919b02d70c9797f689cdad85066d3953b978901) when a user runs create
configdocs. This validation only checks the presence of the document
(by name and schema) and does not care about the document's other
contents.
The severity of a failed validation is configurable through the
"validations" config section in shipyard.conf. The default severity
is "Skip", meaning the validation is not ran at all.

Note that with the default configuration of new validation, Shipyard
functionality should be unchanged.

Change-Id: I754617de81f628a24232e890b12b157ba6731c25
2019-05-09 08:33:52 -05:00
Carter, Matthew (mc981n) 64171aacf4 Validate existence of "deployment-version" doc on configdocs commit
This PS adds funtionality to Shipyard to validate the existence of
the Pegleg-generated "deployment-version" document (Pegleg change id:
I7919b02d70c9797f689cdad85066d3953b978901). As implied, this new
validation only checks for the presence of the document (by name and
schema) and currently does not care about any of the document's
contents under "data".
The severity of a failed validation is configurable through the new
"validations" configuration section in shipyard.conf, and is
defaulted to skip the validation altogether. This means that by
default, this patch set does not alter the functionality of Shipyard

Note that with the default configuration of this new validation,
Shipyard functionality should be unchanged.

Change-Id: I5e7269066f769804710a0fd1f2c8d0aece0d3314
2019-05-09 08:25:37 -05:00
Carter, Matthew (mc981n) 222b074cb1 Be configuration driven when referencing document names/schemas
Currently, any document name or schema referenced in the Shipyard
code base is a hard-coded string. Often times, these strings are
repeated throughout the code. This patch set adds a new configuration
section to shipyard.conf to define document names and schemas so they
can then be referenced in the Shipyard code via the oslo
configuration object. This functionality will be important for
upcoming Shipyard features which will call for more documents to be
validated as well as some new Shipyard-created docs.

Change-Id: I34ae8cd578bab730d004c3d176e3817b5a45c89e
2019-05-02 10:47:38 -05:00
Bryan Strassner a11e962eef Move Airflow web container into Shipyard pod
Moves the airflow web server container from its own pod into the
Shipyard pod. This removes exposed network surface area from the
Shipyard suite of software. Shipyard, after this change accesses the
Airflow API using localhost in the same k8s pod.

Change-Id: Ied4bd415a8d78c393b7256ead27a6a2176f4a2d6
2019-01-29 09:41:16 -06:00
Bryan Strassner 6b75c7119a Move airflow scheduler to worker statefulset
Moves the airflow sceduler to a container in the airflow-worker
statefulset so that its version lifecycle matches that of the worker.
Leaves the stand-alone scheduler in place to support upgradability from
prior installations that included a standalone scheduler. New
installations are advised to turn off the scheduler template from
rendering using the values.yaml flag.

This is an attempt to make disruptive upgrades to airflow less impactful
to a "update_site" action from Shipyard.

Additionally this removes the template for airflow-flower, which is not in use.

Change-Id: I0608793ee6aba1eb3ce0f5e9567655287014a0ca
2019-01-29 09:41:16 -06:00
Zuul ce8e8264a3 Merge "fix typos in API.rst & output.py" 2019-01-22 20:01:11 +00:00
Zuul c1b12b9a9e Merge "Updates cleartext-secrets RBAC Permissions" 2018-12-21 14:41:02 +00:00
melissaml 7b40eb190f fix typos in API.rst & output.py
Change-Id: I5bb9f8e55580a926b2684aa2e62bb8cc139de3be
2018-12-12 01:15:48 +08:00
Bryan Strassner 667a538330 Support clearing collections of configdocs
Adds an option to create configdocs as an empty colleciton. This is done
as an explicit flag (--empty-collection) on the command as opposed to
using empty files to prevent accidental emtpy file loads leading to
frustration.

Since this introduced a new flag value for the CLI, the CLIs using flag
values were updated to use the standard is_flag=True instead of the
flag_value=True or some other value when a boolean flag is expected.

Minor updates to CLI tests due to moving to responses 0.10.2

Depends-On: https://review.openstack.org/#/c/614421/

Change-Id: I489b0e1183335cbfbaa2014c1458a84dadf6bb0b
2018-11-30 13:36:14 -06:00
Aaron Sheffield 0cac1cbe2f Updates cleartext-secrets RBAC Permissions
- Adds an RBAC check when returning raw configdocs.

Change-Id: Ia4967ba4e1dfc49d44a3914cfa151177a49c3799
2018-11-28 10:01:40 -06:00
Zuul 03d7269b6a Merge "Ensure SY gets redacted rendered documents" 2018-11-14 22:34:23 +00:00
anthony.bellino 407cdc76d0 Ensure SY gets redacted rendered documents
- Introduced query parameter 'cleartext-secrets' yields
  cleartext secrets in encrypted documents if true, else
  the values are redacted.
- Introduced CLI flag '--cleartext-secrets' yields cleartext
  secrets in encrypted documents.

Change-Id: I8a950c8bded3550f4aab8e6dc662ea330c2fb73f
2018-11-14 18:32:37 +00:00
Bryan Strassner 84d99967bc Add configurable timeout for Drydock client
Adds configs to allow a drydock client to use a non-default read
timeout.

Change-Id: Id4e4a235861165bfb5eb571684c8ce0be4181543
2018-11-13 13:10:07 -06:00
zhouxinyong 077bf7653a fix some errors for ill-syntax in CLI.rst
Change-Id: I11043e82da9de50b8444055e3602554d355540ee
2018-11-13 10:11:19 +08:00
Aaron Sheffield d62f15123d Expects Redacted Raw Documents
- Adds a query parameter 'cleartext-secrets' to get full raw documents.
- Adds CLI flag to get full raw documents.

Change-Id: If38974c8433c8360cc47ae1273720ad76e87a6fd
2018-10-23 11:12:56 -05:00
Bryan Strassner 5a9abc73dd Modify note access methods
While iterating on the next steps of using notes, it became clear that
several changes to the output and access methods for notes needed
enhancements. This change introduces a new way to access a note's URL
information via a new API/CLI, while removing the resolution of URLs
from the existing note output. This supports the concept of "builddata"
coming back with sizes of 800kb or more - which really can never work
out inline in other data, especially in cases where there is
multiplicity of the information across many items.

New API: GET /notedetails/{id}
CLI: shipyard get notedetails/{id} and/or shipyard get notedetails {id}
Returns the resolution of the URL for a note, outputting the raw info as
the response (not structured in a JSON response).
The CLI will attempt to minimally format the response if it has inline
\n characters by replacing them will real newlines in the output (if the
output-format is set to either cli or format.  Raw format will be
returned as-is.

The existing notes responses are changed to not include the resolution
of the URL information inline, but rather provide the text:

  Details at notedetails/{id}

The CLI will interpret this and present:

  - Info available with 'describe notedetails/09876543210987654321098765'

This is an attempt to inform the user to access the note details that
way - luckily the API and CLI align on the term notedetails, as the word
details works well enough in the singular form presented by the CLI and
the plural form used by the API.
The ID returned is the unique id of the note (ULID format).

Notes that have no URL will return a 404 response from the API (and
an appropriately formatted value from the CLI).

This approach solves an issue beyond the large inline values from URLs;
providing a means to NOT resolve the URLs except in a one-at-a-time way.
Long lists of notes will no longer have the risk of long waits nor
needing of parallelization of retrieval of URLs for notes.

This change introduces an API-side sorting of notes by timestamp,
providing a chronological presentation of the information that may or
may not match the ULID or insertion ordering of the notes.

Additional feedback from peers about the output of noted indicated that
the CLI formatting of notes in general was in need of visual tuning. As
such, this change introduces changes to the formatting of the output
of notes from the CLI:

-  Notes for describing an item will be presented with a more specific
   header, e.g.: Action Notes: or Step Notes: instead of simply Notes.

-  Tables with notes will change the header from "Notes" to "Footnotes"
   give the user a better marker that the notes follow the current
   table.

-  Table footnotes will be presented in a table format similar to
   the following, with headings matching the kind of note being
   produced.

   Step Footnotes    Note
   (1)               > blah blah blah
                     > yakkity yakkity
   (2)               > stuff stuff stuff stuff stuff stuff stuff
                     stuff stuff stuff
                       - Info available with 'describe notedetails/...
                     > things things things

Change-Id: I1680505d5c555b2293419179ade995b0e8484e6d
2018-10-16 07:45:02 -05:00
Bryan Strassner 06de84e0ab Add notes processing to the Shipyard API+CLI
Enhance the Shipyard API and CLI to retrieve notes that have been
specified against actions and steps. Includes a new reusable parameter
for verbosity.

Change-Id: I1c7f47c0346ce783dacd62b8bbc1fd35a0bf285b
2018-10-11 15:36:22 -05:00
Zuul 282c5699b9 Merge "Add notes common code for Shipyard" 2018-10-09 20:24:13 +00:00
Bryan Strassner 35d6089d32 Add notes common code for Shipyard
Adds the common/shared code to support notes in Shipyard such that this
component can be reused between the Shipyard API and the workflows.
Includes a single example of creating and retreiving notes as part of
action creation, that will likely be changed in a future change.

Note that this change doesn't include changes to the CLI to represent
notes outwardly, but the first example can be seen using the
--output-format=format option (shipyard get actions)

Change-Id: I2f87713eb74dae312912ff4c36e6ae30a569ea38
2018-10-05 15:40:48 -05:00
Zuul af1a3dd900 Merge "actions: Add Shipyard action to test site" 2018-10-05 15:46:40 +00:00
Andrey Volkov 0e6486be8b Return non-200 response if Airflow log request failed
This patch modifies API behavior for
GET /v1.0/actions/{action_id}/steps/{step_id}/logs
such way:
- it returns the same status code as Airflow HTTP request returned
  if Airflow responds with a status code of 400 or greater,
- it returns 500 error status code if an exception happens during
  Airflow HTTP request (200 was before).

Warning: this change breaks API backward compatibility, now a client
could get 4xx or 5xx codes proxied from Airflow.

Change-Id: Ic5dceb3abc34415d21b4d8d4e71b4e5661a7363d
2018-10-04 07:23:21 -07:00
Drew Walters 84921b31d2 actions: Add Shipyard action to test site
This commit introduces an action, `test_site`, that invokes Helm
tests for all deployed releases using the
`ArmadaTestReleasesOperator` introduced in [1]. This action supports
the ability to invoke Helm tests for a specific release using the
`release` parameter and cleanup resources if the `cleanup` parameter
is set to `true`.

[1] https://review.openstack.org/#/c/603236/

Depends-On: https://review.openstack.org/#/c/603236/
Change-Id: Ib5f38fe4b8a6516ee2afae62774ec84f1d2eb1ad
2018-09-27 16:38:54 -05:00
Bryan Strassner 5c2f921d6f [docs] Update docs to match site_statuses API
Also rebuilds sample yaml files.

Change-Id: I740bdd9aba2464d1656e57c9f8b30886f552cfd6
2018-09-26 15:43:39 -05:00
Zuul c879583939 Merge "Add relabel_nodes action" 2018-09-25 20:50:04 +00:00
Zuul e574aac0dc Merge "Minor: docs location fix" 2018-09-25 18:47:53 +00:00
Crank, Daniel (dc6350) 9b3cd66156 Add relabel_nodes action
This adds the Shipyard workflow for the new relabel_nodes
action. Related changes are https://review.openstack.org/593609
(Drydock) and https://review.openstack.org/584925 (Promenade).

Change-Id: I2881199c1622e873b5fc3e948231bacd10a1beba
2018-09-25 07:34:43 -05:00
Roman Gorshunov 7fa3136470 Fix: various documentation and URL fixes
1) UCP -> Airship
2) readthedocs.org -> readthedocs.io (there is redirect)
3) http -> https
4) attcomdev -> airshipit (repo on quay.io)
5) att-comdev -> openstack/airship-* (repo on github/openstack git)
6) many URLs have been verified and adjusted to be current
7) no need for 'en/latest/' path in URL of the RTD
8) added more info to some setup.cfg and setup.py files
9) ucp-integration docs are now in airship-in-a-bottle
10) various other minor fixes

Change-Id: I4b8cc6ddf491e35d600a83f5f82d7717108e31dd
2018-09-24 12:53:27 +02:00
Roman Gorshunov 392f75e9a4 Minor: docs location fix
Change-Id: Ifdb02451ee9feada4a18e8e67c1ac64a77efe973
2018-09-14 23:38:29 +02:00
Roman Gorshunov 785c4ca5f5 Set up publishing of docs
Set up publishing of docs to the readthedocs.

Change-Id: Idfafa228e9136de3cec72d9df82b537ebd8fc8d3
2018-09-14 21:32:41 +02:00