deckhand/doc/source/testing.rst

Testing

Unit testing

Unit testing currently uses an in-memory sqlite database. Since Deckhand's primary function is to serve as the back-end storage for UCP, the majority of unit tests perform actual database operations. Mocking is used sparingly because Deckhand is a fairly insular application that lives at the bottom of a very deep stack; Deckhand only communicates with Keystone and Barbican. As such, validating database operations is paramount to correctly testing Deckhand.

To run unit tests using sqlite, execute:

$ tox -epy27
$ tox -epy35

against a py27- or py35-backed environment, respectively. To run individual unit tests, run:

$ tox -e py27 -- deckhand.tests.unit.db.test_revisions

for example.

To run unit tests using postgresql, postgresql must be installed in your environment. Then execute:

$ tox -epy27-postgresql
$ tox -epy35-postgresql

against a py27- or py35-backed environment, respectively. Individual unit tests can be executed the same way as above.

pifpaf is used to spin up a temporary postgresql database. The URL is set up as an environment variable via PIFPAF_URL.

Warning

It is not recommended to run postgresql-backed unit tests concurrently. Only run them serially. This is because, to guarantee true test isolation, the DB tables are re-created each test run. Only one instance of postgresql is created across all threads, thus causing major conflicts if concurrency > 1.

Functional testing

Prerequisites

Deckhand requires Docker to run its functional tests. A basic installation guide for Docker for Ubuntu can be found here.

Overview

Deckhand uses gabbi as its functional testing framework. Functional tests can be executed via:

$ tox -e functional

You can also run a subset of tests via a regex:

$ tox -e functional -- gabbi.suitemaker.test_gabbi_document-crud-success-multi-bucket

The command executes tools/functional-tests.sh which:

1. Launches Postgresql inside a Docker container.
2. Sets up a basic Deckhand configuration file that uses Postgresql in its oslo_db connection string.
3. Sets up a custom policy file with very liberal permissions so that gabbi can talk to Deckhand without having to authenticate against Keystone and pass an admin token to Deckhand.
4. Instantiates Deckhand via uwisgi.
5. Calls gabbi which runs a battery of functional tests.
6. An HTML report that visualizes the result of the test run is output to results/index.html.

At this time, there are no functional tests for policy enforcement verification. Negative tests will be added at a later date to confirm that a 403 Forbidden is raised for each endpoint that does policy enforcement absent necessary permissions.

CICD

Since it is important to validate the Deckhand image itself, CICD:

• Generates the Deckhand image from the new patchset
• Runs functional tests against the just-produced Deckhand image

Deckhand uses the same script -- tools/functional-tests.sh -- for CICD testing. To test Deckhand against a containerized image, run, for example:

export DECKHAND_IMAGE=quay.io/attcomdev/deckhand:latest
tox -e functional

Which will result in the following script output:

Running Deckhand via Docker
+ sleep 5
+ sudo docker run --rm --net=host -p 9000:9000 -v /opt/stack/deckhand/tmp.oBJ6XScFgC:/etc/deckhand quay.io/attcomdev/deckhand:latest

Warning

For testing dev changes, it is not recommended to follow this approach, as the most up-to-date code is located in the repository itself. Running tests against a remote image will likely result in false positives.