Commit Graph

121 Commits

Author SHA1 Message Date
Alexander Hughes d460e0aaa0 Fix image build checks missing setuptools
Use apt to install python3-pip, and use pip3 in event system has
both pip2 and pip3 installed. Use apt to install setuptools for
Ansible's consumption.

Change-Id: I31f40936648ebd9cf6612af2564ae959f9d3232e
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-06-17 14:16:35 -04:00
Drew Walters 1405672087 Update Airship vulnerability link
The Airship vulnerability documentation has moved [0]. This change
updates SECURITY.md to point to the correct location.

[0] https://docs.airshipit.org/learn/vulnerabilities.html

Change-Id: Iea843a3399bc7836f5645c3ca81603e2e9ca7356
Signed-off-by: Drew Walters <andrew.walters@att.com>
2020-05-18 21:28:18 +00:00
Drew Walters 3e715e16eb Add SECURITY.md
All Airship projects are moving to GitHub issues. This change adds a
GitHub security policy that links to the official Airship vulnerability
management process [0]. When users on GitHub click "New Issue" on this
GitHub repository, they will see an option to report a security
vulnerability, which will direct them to our official policy.

[0] https://airship-docs.readthedocs.io/en/latest/security/vulnerabilities.html

Change-Id: Iaf060dd0085c21f0c4f18f100e3e053b5ceedbed
Signed-off-by: Drew Walters <andrew.walters@att.com>
2020-02-20 17:13:28 -06:00
Zuul c1be6eb7e3 Merge "Allow rules to be input by user" 2020-02-03 20:03:08 +00:00
Ryan Schroder cddf30092b Docker image guide
Added steps to setting up, creating, and running spyglass in Docker image

Change-Id: Ib168c12306608d19df66fc7961d1ceb28d46f5f1
2020-01-16 11:27:29 -06:00
Ryan Schroder 386d7c0e57 Allow rules to be input by user
New tag -r added so that a rules yaml file can be input.
Rules file outlines rules for data manipulation in the engine.
Preexisting rules left in as the default.

Change-Id: Ide8af31b018b4f888486ae6d48ffb441bf9634a7
2020-01-16 16:48:48 +00:00
Ryan Schroder 161528365d Spyglass Docs Update
Changed paths for all examples, removed references to Tugboat
and Formation

Change-Id: Ibeb80a1c239169bd4f4ecb124e86671891bef99a
2019-12-16 17:01:00 +00:00
Ryan Schroder 9d983e4874 [FIX] Bug fix for template path ValueError
Add cleaning step to remove ending slash in template path entered from CLI

Change-Id: I1ec8f48918b54b4733658ed7001b6e6462bef7a8
2019-12-11 14:44:51 +00:00
Ryan Schroder f53c51edc1 Update Spyglass to run as “airship” user in container
Spyglass previously ran as root in container

Change-Id: I341679fcb6173ca667f009dfc100237f32275ef2
2019-12-10 19:16:23 +00:00
Drew Walters ddd6bca645 docs: Update copyright footer
During the recent Airship Working Committee meeting, the committee
addressed feedback from the Airship confirmation review [0]. One such
item was concerned with copyright footers mistakenly claiming rights to
all Airship documentation.

This change updates the footer to attribute documentation to the
Spyglass authors.

[0] https://etherpad.openstack.org/p/airship-wc-meeting-2019-12-09

Change-Id: If9db77b1bc6a0abc710898f5f9c0d2ff7fc25bdf
Signed-off-by: Drew Walters <andrew.walters@att.com>
2019-12-09 22:08:12 +00:00
Ryan Schroder 261b064b65 Update requirements install in docs
Change pip3 install to pipenv install to match new dependency management

Change-Id: Idbfb47744ece56b5a4549687f39c8e3c36be60a5
2019-11-15 22:06:35 +00:00
Ian H Pittwood e65715d1a1 Update Spyglass documentation
Updates CLI commands to match current code

Fixes title formatting on README

Pins six back to 1.12.0 for OpenSUSE incompatibility

Updates package dependencies list

Change-Id: Iedf6a438fbe441315dc1124364eea38056090065
2019-11-14 20:43:01 +00:00
Ian H Pittwood 0bed580daa Implement intermediary file validation
Implements JSON schema validation for intermediary YAML files

Adds tests for intermediary validation

Change-Id: Iaa385d265b027426f8e5f2376462ffb4c0d1d3fa
2019-08-29 08:39:24 -05:00
Ian H. Pittwood 7f1ed8bcf9 Import GPG keys in OpenSUSE image
Change-Id: Ia2a0504c4d1d47b90de88b9d4c53928d22fbde1e
2019-08-21 11:01:14 -05:00
Ian H Pittwood cf673e7d80 Fix OpenSUSE image build
Change-Id: I17e72e8239a5f4c0875e6ff736ca74393e86284c
2019-08-16 20:02:57 +00:00
Zuul 5d756ece9a Merge "Address TODO notes in engine" 2019-08-15 12:52:50 +00:00
Zuul 9a0329f49f Merge "Use Pipfile.lock only in tox gates" 2019-08-14 20:20:39 +00:00
Zuul 77c00c5d01 Merge "Use py36 job for safety checks" 2019-08-14 20:20:38 +00:00
Ian H Pittwood dae192efc6 Upgrade yapf to 0.28
Upgrades yapf to newest version, 0.28.0, and runs the formatter to
update all existing code.

Adds entry to .gitignore to ignore pyenv installations.

Change-Id: I11512a8a522cc530165461cc8f52f7ff010dd092
2019-08-14 19:15:04 +00:00
Ian H Pittwood 8cc2a62255 Use Pipfile.lock only in tox gates
Pipenv validates that the Pipfile.lock that it uses for installing
packages is not out of date before following through on execution. If it
is not, it will throw an error. This change adds an env that tells
Pipenv to ignore the Pipfile and only use Pipfile.lock so gates do not
fail due to out of date packages.

Change-Id: I2fa541b31e14561c7cefc330e7ef2cc0b0e4abc6
2019-08-14 19:14:51 +00:00
Ian H. Pittwood 61ecee4624 Use py36 job for safety checks
When using pipenv, the targeted version must be used when attempting to
do pipenv check. This change updates the safety dependency check to use
openstack-tox-py36  so the correct python version will be used.

Change-Id: I4b9f71cfaaa74724ecee7b40ea6e9362e836d261
2019-08-14 19:14:25 +00:00
Ian H Pittwood 81092e6d7c Address TODO notes in engine
This change addresses some of the TODOs made in the Spyglass engine.
There will be additional follow-up patchsets that will address issues
with the rules engine and intermediary validation.

Change-Id: Iba70a51d291659bf827e46fc9070a898303082d1
2019-08-14 19:14:19 +00:00
Ian H Pittwood 166483d6ad Fix Spyglass gates
Zuul gates broke for Spyglass due to an update made to either Zuul or
Openstack's Zuul jobs (I wasn't able to pin it down). The update caused
Zuul to attempt to install Spyglass's requirements.txt file instead of
referring to the method given in the tox configuration that utilizes
Pipenv. To bypass this issue, the requirements.txt will be removed and
Spyglass will be fully moved over to using Pipenv.

Removes requirements.txt

Updates Pipfile and Pipfile.lock for new package releases

Updates Dockerfile to install requirements from Pipfile.lock

Change-Id: I347ef6db18a44e1b88c811aa071ba8c60bccbd53
2019-08-14 14:13:47 -05:00
Alexander Hughes 9689dae61f Standardize Spyglass code with YAPF
From recently merged document updates in [0] there is a desire to
standardize the Airship project python codebase.  This is the effort
to do so for the Spyglass project.

[0] https://review.opendev.org/#/c/671291/

Co-Authored-By: Ian Pittwood <pittwoodian@gmail.com>
Change-Id: I2b0f2491d83675d7742b168ecb9e1dfb9a11e719
2019-07-25 17:01:33 +00:00
Zuul 534fe50b9e Merge "Add tests for site processor init" 2019-07-24 13:42:43 +00:00
Zuul 3a4bc68a10 Merge "Move safety checks to new Zuul job" 2019-07-24 13:35:14 +00:00
Ian H. Pittwood 8304add63e Add tests for site processor init
This change adds a couple tests for init in Spyglass' site processor
class, bringing total test coverage up to the goal of 90%.

Change-Id: I81c1dfdf2a9d65589d961451401802621058a9f2
2019-07-23 09:01:47 -05:00
Ian H Pittwood 8349197be8 Move safety checks to new Zuul job
The pyup.io database that is used to check dependencies for
vulnerabilities in Spyglass is currently down. When first implemented,
this check was placed in the tox pep8 job. This change moves the
vulnerability check into its own Zuul job so it can be disabled in
instances such as this where the database is down.

Specifies basepython to python3 for safety and bandit jobs.

Change-Id: I4e13c5341a9e62095587ec1820be79a621380f09
2019-07-23 08:40:51 -05:00
Ian H Pittwood 4e2de1dae1 Adds tests for the plugin base
This change adds tests for the abstract plugin base for any non-abstract
methods.

Change-Id: I6be3af09cec97bf925b274622d64f3019eb389e1
2019-07-22 14:09:30 -05:00
Zuul 1b5d64fe32 Merge "Add tests for parsing engine" 2019-07-22 14:39:58 +00:00
Ian H Pittwood fd3d935941 Restructure plugin base
This change makes some mild restructuring and naming changes to the
data-extractor plugin base. Data extraction will now be separated into
two main workflows: loading raw data and source specific parsing into
model objects. Naming of methods was changed to help more accurately
reflect their function in these workflows.

Change-Id: Ia3bc892994ff96cce5c1672fadf35ef2d1c4164b
2019-07-18 12:26:46 -05:00
Ian H Pittwood b7c2bc7ccd Add tests for parsing engine
Increases test coverage of the Spyglass parser engine from 0% to 82%,
bringing overall test coverage to 86.4%. This change moves minimum
coverage to 84%.

Change-Id: I2de496b8d7f4c4252be22c713605fae6fd565b66
2019-07-18 14:27:45 +00:00
Zuul 4dd1cea32a Merge "Update OpenSuse image to 15.1 from 15.0" 2019-07-17 15:59:28 +00:00
Zuul 880f6d455a Merge "Use init to configure plugins" 2019-07-17 15:02:14 +00:00
Zuul 46ccaece21 Merge "Upgrade Sphinx package" 2019-07-17 15:00:14 +00:00
Ian H. Pittwood b3f1e59148 Use init to configure plugins
Plugins currently use a couple configuration methods to set up field
data for the class. This seems superfluous. This change moves these
configuration steps into the init method so plugin classes can ingest
data as kwargs.

Here is an example of how the change will be implemented in plugins:
https://review.opendev.org/#/c/670171/

Change-Id: Ib26636f1eb4146902ee801af5bcce53d137be2ad
2019-07-15 21:08:57 +00:00
Alexander Hughes 597bdba490 Update OpenSuse image to 15.1 from 15.0
This change took place in Pegleg, adding to Spyglass for project
consistency.

Pegleg change: https://review.opendev.org/#/c/670421/

Change-Id: I1c3610ce6041393c94252629194295cc28eb129e
2019-07-15 19:07:42 +00:00
Ian H. Pittwood e58167af68 Centralizes shared CLI options
This change moves all the shared CLI options between Spyglass and its
plugins back into the main Spyglass CLI file. The plugins will then
reference the main CLI file for these options instead of redefining
them in a future change.

Change-Id: I9a6fe7a7d84fed71c372beea9cf7d74a2f6430b1
2019-07-10 15:43:46 +00:00
Ian H. Pittwood ee18e3e94b Upgrade Sphinx package
Sphinx 2.1.0 has a bug [0] that causes whitespace to be excluded in
outputted docs. This change updates Sphinx to peg any version >2.1.0.

Safety dependency vulnerability checks now will also cover the doc
requirements.txt.

[0] https://github.com/sphinx-doc/sphinx/issues/6440

Change-Id: I35f1acf4385821969ffa7c9807cf209a59c1d73e
2019-07-10 10:30:05 -05:00
Zuul 601f281191 Merge "Combines all exceptions into a single file" 2019-07-09 20:29:59 +00:00
Ian H. Pittwood 43b6d78d00 Combines all exceptions into a single file
Spyglass has three different files that define exceptions. This change
merges all of those definitions into a single file for simplicity.
Before this change can be merged, spyglass-plugin-xls needs to move the
exceptions needed for its functions.

Related Change: https://review.opendev.org/#/c/667243/

Depends-On: I81c5ca2d9083aece3641bc8b5405dfd44baec810
Change-Id: Ibf34771653247850903e529beb17dbd60ba35fbf
2019-07-09 12:29:01 -05:00
HUGHES, ALEXANDER (ah8742) ec827aa4d5 Add voting to OpenSuse jobs
Voting was previously turned off due to instability with OpenSUSE
mirrors.  This appears to have been resolved over the last week so
this patch re-adds voting.

Change-Id: I293d298996468fee43db410ea54911aef7852a9a
2019-07-02 07:54:31 -05:00
Ian H. Pittwood 4909870115 Implement Pipenv dependency management
Pipenv is a tool that brings better package dependency management to
python. It can automatically create and manage virtualenv as well as
managing package dependencies using Pipfile and Pipfile.lock. Adding
this dependency manager into Airship projects will decrease package
version conflicts between projects and help increase security through
hash validation of packages and vulnerability scans.

Changes:
- Imports requirements.txt type files into Pipfile
- Pipenv dependency management in tox
- Switches Safety package for "pipenv check", an implementation of
Safety
- Unpins or loosens pins on all development packages
- Soft pins package dependencies to encourage adoption of bug fixes and
prevent small version mismatches

Pipenv Docs: https://docs.pipenv.org/en/latest/

Helpful Pipenv Guide: https://realpython.com/pipenv-guide/

Change-Id: I2c9cec8acf6b6c6157f807b010def873c349d3ae
2019-07-01 16:42:34 +00:00
Zuul ca9756de09 Merge "Use data objects for document generation" 2019-07-01 13:51:37 +00:00
Ian H Pittwood 746d7ca2ec Add site_config.yaml back into examples
Spyglass uses additional configuration files to add in any information
not included by a data source. This configuration file should be usable
independent of the data source used. This change adds the
site_config.yaml example file back into the examples folder after it was
initially removed in the project separation of spyglass-plugin-xls.

Change-Id: I9be76c89076327b8ff928aa3da6c91cca0397cc2
2019-06-26 14:58:54 -05:00
Ian H. Pittwood efe24d8a5f Use data objects for document generation
This is a follow-up change to [0] which further implements data objects
to be used in the generation of documents in Jinja2. The following
additions and changes are made:

- Adds helper functions to data objects to filter hosts and networks for
ease of use in Jinja2 templates
- Adds SiteDocumentData factory function to convert intermediary yaml
dictionaries into a SiteDocumentData object with all associated objects
- Updates Jinja2 templates to use data objects
- Cleans up overly complex looping in Jinja2
- Adds tests for new code in models.py

[0] https://review.opendev.org/#/c/662092/

Change-Id: I66ebfeaf5d6ca76b6dee5a2285a74bad8b06b720
2019-06-26 12:03:40 -05:00
Ian H. Pittwood 4747222641 Implements data object models
This change implements data object models from [0] in data extraction
and parsing. The change results in minor modifications to the outputted
intermediary, which can be seen between these two example intermeidary
files [1].

This fully implements the data objects from models.py in data extraction
and parsing. A follow-up change will implement use of the data objects
in Jinja2. Temporarily, all objects will be converted to dictionaries
for generating documents from templates.

[0] https://review.opendev.org/#/c/658917/
[1] https://www.diffchecker.com/NnjjJrb2

Change-Id: Ifd867787aab541be5dabecf9f6026faa2ec7049e
2019-06-21 15:00:46 -05:00
Ian H. Pittwood f9226d2f4a Disable voting on openSUSE
There's been issues in the past week with openSUSE image builds timing
out. This may be due to issues with openSUSE's infrastructure [0]. For
the time being, we should disable the openSUSE docker build until we are
able to consistently build without failures.

[0] https://status.opensuse.org/

Change-Id: I6f3cb9867898a3371cedc58835633ac6eb9ad99b
2019-06-20 13:44:29 +00:00
Ian H. Pittwood 313058b8fb Adds tests for Spyglass data objects
Adds unit tests for all data objects created in [0]. The changes in [0]
were merged in short succession with [1], causing the test coverage gate
to fail with all the newly introduced lines of code. This change adds
tests for all of the newly added code and increases test coverage
requirement to 60% (currently at 65.93%).

[0] https://review.opendev.org/#/c/658917/
[1] https://review.opendev.org/#/c/663729/

Change-Id: I96931e3e415af80ca5ab9202c2bda0344a9901f0
2019-06-18 15:15:28 -05:00
Zuul 6b8d6f2aae Merge "Data objects for Spyglass" 2019-06-17 17:47:58 +00:00