Commit Graph

38 Commits

Author SHA1 Message Date
Sergiy Markin 7ea83c25ef Restored ubuntu_bionic image build
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.

Change-Id: I4c4c7186e67c771f8249b2017b2507aac17760b7
2023-05-30 22:39:59 +00:00
Sergiy Markin 154a099b28 Shipyard upgrade for focal
- upgraded Airflow to 1.10.15  -
  https://airflow.apache.org/docs/apache-airflow/1.10.15/changelog.html
- disabled xenial, bionic and opensuse images build  gates
- added focal image build  gate
- added focal zuul build node
- adjusted Makefile for focal
- added bindep.txt to utilize bindep zuul base role for zuul build node
  pre-setup
- added focal Dockerfile
- implemented freeze requirements.txt approach like in other Airship
  projects
- removed specific requirements.txt for airflow in favor of using
  requirements-frozen.txt from shipyard_airflow project when building
  airflow docker image
- fixed docker image publishing to Quay
- replaces deprecated LOG.warn with new LOG.warning call
- replaced deprecated body attribute in responce wiht responce.text
  attribute
- update of falcon module deprecated .API call - replaced wiht
  falcon.App call
- deprecated routing.create_http_method_map method replaced with
  routing.map_http_methods
- re-formatted code tabulations based on yapf recommendations
- replaced deprecated protocol attribute in Pytest create_environ() with
  http_version attribute
- replaced deprecated app attribute in Pytest create_environ() with
  root_path attribute
- fixed airflow CLI commands to match 1.10.15 version
- updated zuul gates to work on focal nodes and added focal specific
  node setup items by adding appriate ansible tasks and roles
- uplifted Helm to 3.9.4
- uplifted stable HTK commit id
- updated tox.in to with with tox v4
- uplifted dependences references to other Airship projects
- common python dependences were syncronized with other Airship
  projects(Promenade, Deckhand, Armada, Drydock)
- fixed airskiff deployment gate
- fixed genconfig* profiles in shipyard-airflow tox.ini responsible for
  maintanance of policy.yaml.sample and shipyard.conf.sample

Change-Id: I0c85187dc9bacf0849382563dd5ff7e9b2814c59
2023-04-28 20:40:50 +00:00
Sean Eagan a0da68409a Use helm 3 in chart build
`helm serve` is removed in helm 3 so this moves
to using local `file://` dependencies [0] instead.

[0]: https://helm.sh/docs/chart_best_practices/dependencies/#repository-urls

Signed-off-by: Sean Eagan <seaneagan1@gmail.com>
Change-Id: Ibdea9f1ccda043259620add022643e5c9800259a
2021-10-01 11:48:37 -05:00
Ahmad Mahmoudi df751e302c Upgrade apache-airflow to 1.10.5
Upgraded apache-airflow to 1.10.5 to address the database migration issue
https://issues.apache.org/jira/browse/AIRFLOW-6046.

Change-Id: Ib8678cddb2deb4f5ef944d026e0d2633a326d9cc
2020-03-30 05:21:31 +00:00
Ahmad Mahmoudi 24f6a01e0b Add support for Ubuntu bionic base image
Added support to buid shipyard and airflow images using either a xenial
or Ubuntu bionic base image.
The default base image is set to bionic.

Change-Id: I6ad4d42dede081586b3ccea87a42e250979ac106
2020-02-04 13:38:39 -06:00
Ahmad Mahmoudi f0f876afef (airflow) - Apache airflow uplift to 1.10.4
Updated airflow config and Dockerfiles for apache airflow 1.10.4
For details see: https://github.com/apache/airflow/blob/master/UPDATING.md

Also updated the kubectl to 1.16.2, as part of kubernetes 1.16 uplift.

Change-Id: Ib24ff0304b6279ff0be749141854d6a604473597
2019-10-22 14:46:57 +00:00
Ahmad Mahmoudi f42e85d7ce (airflow) - Added no_proxy env variable
Added no_proxy environment variable to be able to reach internal
repos when http_proxy is used.

Change-Id: If23fb644bfc2a6486e9089ad5ca1eb21f81e7df6
2019-09-19 17:56:55 +00:00
Scott Hussey 185dc9823c (airflow) Configure source of Airflow pkg
- Allow a Docker build-arg to specify the source of the Airflow
  python package in any format supported by pip

Change-Id: Ifa2dd62d91570035cff91ff07868b0fcf659add8
2019-09-13 11:09:18 -05:00
Arun Kant cd6f154da5 Adding opensuse support in image building of airflow and shipyard
Adding DISTRO parameter for makefile to invoke distribution specific
Dockerfile and build image accordingly.

Also qualifying image job name to distinguish ubuntu vs opensuse
check jobs and passing distro specific variable.
Similar logic for pushing distro specific images.

Added new document section for multiple distro support.

Change-Id: I215f8a107cab8770181c0472e0cc29053b2b978e
2019-05-14 09:52:24 -07:00
Drew Walters 3af62864d6 images: Update default image tag to `latest`
This commit changes the default image tag for Shipyard and Airflow
images from `untagged` to `latest` to match other Airship projects.

Change-Id: Id1f9d56784e51f0631280188ba1b898fde8f56ea
2019-03-22 18:42:48 +00:00
Bryan Strassner 919868e16c Use UBUNTU_BASE_IMAGE instead of BASE_IMAGE
Offering better alignment with some downstream build jobs.

Change-Id: I90332926b4e5dd04b00b6008d06d36624086e313
2019-02-06 16:55:00 -06:00
Bryan Strassner 8550346b78 Switch to ubuntu base image
Change to use ubuntu base image instead of python
Refactor Shipyard Dockerfile to reduce image size significantly

BREAKING CHANGE: The `make images` PYTHON_BASE_IMAGE arg is now renamed to BASE_IMAGE.

Change-Id: I3338dfbbb91b5514fa4fd205bdfc4136d0abc2e5
2019-02-02 08:21:55 -06:00
Roman Gorshunov 8621e440c7 Minor: meaningful default label
Making default label meaningful and conformant to "key=value" requirement.

Change-Id: I19886ea8bc9a1a3a164ee1ab0d8adfc831af02c1
2018-11-08 19:32:19 +01:00
Roman Gorshunov 1863d36c62 Fix: adding back the possibility to add arbitrary labels
Arbitrary labels could be added as `make` parameter `LABEL=`, which is used in
att-comdev/cicd Jenkins pipelines.

Sample parameter:
'LABEL=org.label-schema.vcs-url=${GERRIT_CHANGE_URL} \
--label org.label-schema.base-image=${base_sha256}'

Sample usage:
See Jenkinsfile files under images/ directory in att-comdev/cicd repo.

In addition to that, if `COMMIT` variable is undefined when invoking `make`, we
use result of `git rev-parse HEAD` command, which should output latest git
commit ID.

[0] https://github.com/att-comdev/cicd

Change-Id: I7c6b63c865ac0ff63f4a85ec54863c20781fe98d
2018-10-24 18:28:52 +02:00
j330k c90e017bea Making Python version uniform across all airship components.
Change-Id: I53febd3da36812e0bce0be5a80e57dca0c7804c3
2018-10-19 15:29:41 -05:00
Roman Gorshunov a4211b9081 Minor fix: duplicate items in .title label of container images
Fixing duplicate items in "org.opencontainers.image.title" label of
the container images. Was: "airflow shipyard", now - separate and
correct for each if the images being built.

Change-Id: Iad180a100f8ce5ccc3ac718a447d10b0813aa665
2018-10-09 15:36:03 +02:00
Zuul 12d79ce423 Merge "Fix: git commit id labels on images" 2018-09-28 14:15:35 +00:00
Roman Gorshunov 655b1ae3c3 Fix: git commit id labels on images
1) Use OCI Image Specs for labels instead of custom 'commit-id=xxxxx'
   or legacy "Label Schema"
2) Fix missing git commit id labels on images (.revision)
3) Add human-readable title (.title) of the image, URL (.url), and
   a few other properties (annotations) according to the latest Specs

Change-Id: Iedd64fa70016880d5df241d27d8dad101e10130b
2018-09-21 03:31:13 +02:00
Roman Gorshunov 785c4ca5f5 Set up publishing of docs
Set up publishing of docs to the readthedocs.

Change-Id: Idfafa228e9136de3cec72d9df82b537ebd8fc8d3
2018-09-14 21:32:41 +02:00
Drew Walters bb1db91a31 Remove temp dir deletion statement from make clean
Invoking make charts initializes the clean recipe, resulting in the
deletion of the temp directory, $(BUILD_DIR), which is created when
make is invoked. Since this is the location the helm client is
installed by tools/helm_install.sh, the script fails. This commit
removes the statement that deletes the temp directory created when make
is invoked.

Change-Id: I87d41df979a5a4daeb539bcc1330f126a5a407c7
2018-09-04 17:19:57 -05:00
Bryan Strassner 5e542cef0b [FIX] Update dangerous makefile clean command
In the corner case that the tmp directory would not be created, the old
form could attempt to delete much more than intended. This changes to
more simply fail in the case of a missing tmp directory

Change-Id: I3a4cd800e329cd477904d00f6dcb77bbbc2ff90b
2018-08-29 08:00:51 -05:00
Bryan Strassner f3749ca3f9 Add redeploy_server processing
Adds the functionality to redeploy a server in an unguarded fashion,
meaning that the server will not be pre-validated to be in a state that
workloads have been removed.

This is the first targeted action for Shipyard, so a refactoring of the
validators to support more flexibility has been done.

Also adds RBAC controls for specific actions being created, rather than
a binary create action privilege.

Change-Id: I39e3dab6595c5187affb9f2b6edadd0f5f925b0c
2018-08-21 09:42:40 -05:00
Jerome Brette 4b8e0220b0 Update Dockerfile to allow override of FROM variable
l is to let user customize the base image of the component
by passing FROM=myimage during the build process. This would let any
project leveraging Airship ensure that the base image is matching the
security requirements for that project and still use the same Dockerfile.
This will also ease the control of the /etc/apt/source.list
and thereby the result of apt-get update/upgrade procedure.
2. The above goal is achievable by using docker-ce feature such as:
ARG FROM="defaultbaseimage:xx"
FROM ${FROM}
For this reason, the installation of docker.io in the Zuul gating is beeing
replaced by docker-ce.
3. Third Goal is to bring consistency with the other compoenents leveraging
Helm such as the openstack-helm and potentially use bindep the same way
the LOCI images are to ensure
4. The new syntax in the Dockerfile is still commented out until the associated
image builder have been updated to use docker-ce as they have been for the LOCI
images.

Change-Id: Ife7e1be53c7c139bdc42dee42f0798e83f4fd271
2018-07-17 16:35:42 -05:00
anthony.lin c7a9c65c88 Update Shipyard Chart - HTK OSH Infra
Updated configurations to point to openstack-helm-infra
for reference to helm-toolkit as helm-toolkit has been
removed from the openstack-helm repo [0]

Also aligned with changes to the keystone user set up in
OSH using Helm ToolKit so as to get pass Helm Lint.

Updated Makefile targets to install helm dynamically

[0] https://review.openstack.org/#/c/558065/

Change-Id: I0a0813516f9ad176ff005b4693e6b933013a99fd
2018-06-25 17:14:28 -05:00
Roman Gorshunov 1e9a21897a Unifying proxy variables for tox and docker build
This patch adds proxy-related variables to tox and 'docker build'
commands.

Change-Id: I019bb0ff5a5eb48b0a9bec6f202a5d3022b8095c
2018-06-22 19:37:20 +02:00
Bryan Strassner 04906cce68 Workflow to support deployment groups
Updates the Shipyard/Airflow workflow for deploy_site and
update_site to use the deployment group/deployment strategy
information from the design.

This allows for baremetal nodes to be deployed in a design-
specified order, with criticality and success criteria driving
the success and failure of deployment.

Includes refactoring of service endpoints to reduce the need
for so much data passing.

Change-Id: Ib5e9fca535ca74d1819fe46959695acfed5b65c2
2018-06-20 09:55:15 -05:00
Bryan Strassner 6a6e3ec46e Add proxy options to docker run portions of image build
Updates the Makefile and associated shell scripts that perform docker
run to utilize proxy info.

Change-Id: Ic37ae9bc25500357f980ed327148eacac7555915
2018-06-08 15:26:42 -05:00
anthony.lin 8a393dfc80 Add docker image publishing
1) Refactor shipyard image build
2) Add shipyard docker image publishing

Change-Id: I1a007ae21a88d6468bf42724e5def5cd04ebe48e
2018-06-07 14:39:38 -05:00
Scott Hussey 4d0bbb546b (zuul) Add a bandit scanning gate
- Add a gate for bandit code scanning

Change-Id: I27bf7e6fb5ddc5bf854796c4d15ef0e40eebfbff
2018-05-31 16:21:20 -05:00
Bryan Strassner 769d0ded47 Refactor shipyard to UCP target layout
Refactor Shipyard to be better able to leverage common
packages and conform with the target UCP standard layout.

This change supports the same tox entrypoints at
the root level, but the preferred approach is to use make
targets defined in the Makefile such as 'make tests' and
'make lint'

The previous tox.ini has moved and been
tailored to the specifics of each subproject at
src/bin/*/tox.ini

Autotmatic generation of the policy and configuration
files has been removed from the sphinx build for now
but these files will be automatically generated locally
into the docs source by using a 'make docs' command.
This may need to be revisited later to re-enable the
automatic generation of these files such that readthedocs
would still support the project layout.

Change-Id: Ifdc1cd4cf35fb3c5923414c677b781a60a9bae42
2018-04-24 16:47:13 -05:00
Stacey Fletcher 78b82c76a8 Removing Whitespace
Change-Id: I341261a439a8cc4952d7ca997a90fa69f6a09c3a
2018-02-25 12:40:27 -06:00
Stacey Fletcher a4bd79630d Update Makefile
This ps updates the build commands to allow the same
format to be used in att-comdev projects

Change-Id: Iafc95d6c6a5cfa5cfe1e3c562a86ddacd928a079
2018-02-25 13:23:33 -05:00
Stacey Fletcher 17c3fb0eec Add labels to Docker Build process
Adding the ability to add a label to the docker image at build
time

Change-Id: I96a4800256af164de140f4554ab9b09d8c2e6722
2018-02-19 16:38:52 -05:00
Bryan Strassner 93877bbcc3 Update makefile with run commands
Set up and do a rudimentary test of the images created
by the shipyard project to see if the images are okish.

Change-Id: I1f1c8fbcfeeafff66764ae99a176ff6a6766edce
2017-12-11 12:07:44 -06:00
Bryan Strassner 22e065bdd6 Add docs target to Makefile
Also adds the successful build of docs to the linting
target

Change-Id: I1771125b1208819806c47aabed62a8f994a771ab
2017-12-04 15:17:14 -06:00
Kochetov Mikhail 2a826d5a43 Fix: deps before charts
Change-Id: I2248d86ca7d6dac4659d997d3f6fd62c86780a7b
2017-11-06 11:10:41 -07:00
Bryan Strassner b6212a3f1b Update makefile with additional functionality
Adds linting and build of charts

Change-Id: I8ccb9978e419a84ed73081f8e27b31c9dbf9982c
2017-10-27 11:09:21 -05:00
Anthony Lin 772b3b74e7 Move Shipyard and Airflow Dockerfiles
This PS migrates the Shipyard and Airflow Dockerfiles
into this repo and adds a Makefile

We will run the following command from the root directory
to build the airflow and shipyard images

$ sudo make build_airflow
$ sudo make build_shipyard

Change-Id: I9a9fb761ce193b1c5b9c5d9589982366eb73e396
2017-10-20 19:31:06 +00:00