diff --git a/doc/source/cli.rst b/doc/source/cli.rst new file mode 100644 index 0000000..737838d --- /dev/null +++ b/doc/source/cli.rst @@ -0,0 +1,232 @@ +.. + Copyright 2019 AT&T Intellectual Property. + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +============ +Spyglass CLI +============ + +The Spyglass CLI is used in conjunction with the script ``tools/spyglass.sh``. + +.. note:: + + The default workspace for the ``spyglass.sh`` script is ``/workspace``. The + examples below require that this workspace be used. + +CLI Options +=========== + +**-v / --verbose** (Optional). False by default. + +Enable debug logging. + +Generate Intermediary +--------------------- + +Generates an intermediary file from passed excel data. + +.. code-block:: bash + + ./spyglass.sh i -p -x \ + -e -c -s + +Options +^^^^^^^ + +**-p / --plugin-type** "tugboat" by default. + +The plugin to use to open engineering data. Two plugins are available by +default: "tugboat" and "formation". Tugboat can be used for reading Excel data. +Formation can be used to read data from remote sources. + +**-f / --formation-target** (Required for "formation" plugin). + +Target remote for the formation plugin. Accepts a url and a username and +password to access the url. + +:: + + -f + +**-d / --intermediary-dir** (Optional). + +Path where the intermediary file will be created. Must be a writeable +directory. + +**-x / --excel-file** (Required for "tugboat" plugin). + +Path to the engineering excel file. Multiple files can be included, provided +they follow the same specification. Must be readable file(s) in a Microsoft +Excel supported format (.xls, .xslx, etc...). + +**-e / --excel-spec** (Required for "tugboat" plugin). + +Path to the specification YAML that defines the content of the provided +engineering excel files. Must be a readable file in YAML format. + +**-c / --site-configuration** (Optional). + +Path to site specific configuration YAML. Must be a readable file. + +**-s / --site-name** (Optional). + +Name of the site for which the intermediary is generated. + +Generate Manifests +------------------ + +Generates manifests from intermediary file created from passed excel data. +Intermediary data is always generated, but will not be saved unless specified. + +.. code-block:: bash + + ./spyglass.sh m -t -x \ + -e -c \ + -s -t + +Options +^^^^^^^ + +**-i / --save-intermediary** (Optional). False by default. + +Saves the intermediary file used to make the manifests created by the command. + +**-p / --plugin-type** "tugboat" by default. + +The plugin to use to open engineering data. Two plugins are available by +default: "tugboat" and "formation". Tugboat can be used for reading Excel data. +Formation can be used to read data from remote sources. + +**-f / --formation-target** (Required for "formation" plugin). + +Target remote for the formation plugin. Requires a url, a username, and a +password to access the url. + +:: + + -f + +**-d / --intermediary-dir** (Optional). + +Path where the intermediary file will be created. Must be a writeable +directory. + +**-x / --excel-file** (Required for "tugboat" plugin). + +Path to the engineering excel file. Multiple files can be included, provided +they follow the same specification. Must be readable file(s) in a Microsoft +Excel supported format (.xls, .xslx, etc...). + +**-e / --excel-spec** (Required for "tugboat" plugin). + +Path to the specification YAML that defines the content of the provided +engineering excel files. Must be a readable file in YAML format. + +**-c / --site-configuration** (Optional). + +Path to site specific configuration YAML. Must be a readable file. + +**-s / --site-name** (Optional). + +Name of the site for which the intermediary is generated. + +**-t / --template-dir** (Required). + +Path to the Jinja2 template files that will be used to generate manifest files. +Must be a readable directory with Jinja2 files using the .j2 extension. + +**-m / --manifest-dir** (Optional). + +Path where generated manifest files should be written. Must be a writeable +directory. + +Generate Manifests from Intermediary +------------------------------------ + +Generates manifests using an existing intermediary file. This is a shortcut to +skip intermediary generation if it has already been completed. + +.. code-block:: bash + + ./spyglass.sh mi -t + +Arguments +^^^^^^^^^ + +**INTERMEDIARY_FILE** (Required). + +Path to an existing intermediary YAML file that can be used to generate +manifests. + +Options +^^^^^^^ + +**-t / --template-dir** (Required). + +Path to the Jinja2 template files that will be used to generate manifest files. +Must be a readable directory with Jinja2 files using the .j2 extension. + +**-m / --manifest-dir** (Optional). + +Path where generated manifest files should be written. Must be a writeable directory. + +Examples +======== + +Running Spyglass with Excel Plugin +---------------------------------- + +.. code-block:: bash + + spyglass m -i -p tugboat -x -e -c \ + -s -t + +Generating intermediary and manifests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: bash + + spyglass m -i -p tugboat -x SiteDesignSpec_v0.1.xlsx \ + -e excel_spec_upstream.yaml -c site_config.yaml \ + -s airship-seaworthy -t + +Generating intermediary without manifests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: bash + + spyglass i -p tugboat -x SiteDesignSpec_v0.1.xlsx \ + -e excel_spec_upstream.yaml -c site_config.yaml \ + -s airship-seaworthy + +Generating manifests without intermediary +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: bash + + spyglass m -p tugboat -x SiteDesignSpec_v0.1.xlsx \ + -e excel_spec_upstream.yaml -c site_config.yaml \ + -s airship-seaworthy --template_dir= + +Generating manifests using intermediary +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: bash + + spyglass mi -t + +Where sample 'excel_spec_upstream.yaml', 'SiteDesignSpec_v0.1.xlsx' +'site_config.yaml' and J2 templates can be found under 'spyglass/examples' +folder. \ No newline at end of file diff --git a/doc/source/conf.py b/doc/source/conf.py index 2e68d14..05967c1 100755 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -1,3 +1,16 @@ +# Copyright 2019 AT&T Intellectual Property. All other rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. # -*- coding: utf-8 -*- # # shipyard documentation build configuration file, created by @@ -51,18 +64,18 @@ source_suffix = '.rst' master_doc = 'index' # General information about the project. -project = 'tugboat' +project = 'spyglass' copyright = '2018 AT&T Intellectual Property.' -author = 'Tugboat Authors' +author = 'Spyglass Authors' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. -version = '0.1.0' +version = '0.0.1' # The full version, including alpha/beta/rc tags. -release = '0.1.0' +release = '0.0.1' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. @@ -104,7 +117,7 @@ html_static_path = [] # -- Options for HTMLHelp output ------------------------------------------ # Output file base name for HTML help builder. -htmlhelp_basename = 'ucpintdoc' +htmlhelp_basename = 'spyglassdoc' # -- Options for LaTeX output --------------------------------------------- diff --git a/doc/source/developer_quickstart.rst b/doc/source/developer_quickstart.rst index b816494..1a0d90c 100644 --- a/doc/source/developer_quickstart.rst +++ b/doc/source/developer_quickstart.rst @@ -1,5 +1,5 @@ .. - Copyright 2018 AT&T Intellectual Property. + Copyright 2019 AT&T Intellectual Property. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may @@ -25,43 +25,42 @@ airship-spyglass directory. .. code-block:: console - sudo apt install -y python3-pip - sudo apt install -y tox + sudo apt install -y python3-pip + sudo apt install -y tox 2. Set up an environment with tox. .. code-block:: console - tox -e py36 --notest + tox -e py36 --notest 3. Enter the tox environment. .. code-block:: console - source .tox/py36/bin/activate + source .tox/py36/bin/activate 4. Install spyglass in the tox environment. .. code-block:: console - pip install -e . + pip install -e . 5. Run spyglass on the example files to generate an intermediate document. .. code-block:: console - mkdir intermediate - spyglass -g -s airship-seaworthy -t tugboat \ - -idir intermediate \ - --excel_spec spyglass/examples/excel_spec.yaml \ - --excel spyglass/examples/SiteDesignSpec_v0.1.xlsx \ - --additional_config spyglass/examples/site_config.yaml \ - --template_dir spyglass/examples/templates/ + mkdir intermediate + spyglass m -s airship-seaworthy -p tugboat -d intermediate \ + --excel-spec spyglass/examples/excel_spec.yaml \ + --excel-file spyglass/examples/SiteDesignSpec_v0.1.xlsx \ + --site-configuration spyglass/examples/site_config.yaml \ + --template-dir spyglass/examples/templates/ 6. Run spyglass on the intermediate document to generate manifests. .. code-block:: console - mkdir manifest_dir - spyglass -m -i intermediate/airship-seaworthy_intermediary.yaml \ - -mdir manifest_dir/ -tdir spyglass/examples/templates/ + mkdir manifest_dir + spyglass mi intermediate/airship-seaworthy_intermediary.yaml \ + -m manifest_dir/ -t spyglass/examples/templates/ \ No newline at end of file diff --git a/doc/source/getting_started.rst b/doc/source/getting_started.rst index dcb30a7..ca0645d 100644 --- a/doc/source/getting_started.rst +++ b/doc/source/getting_started.rst @@ -1,5 +1,5 @@ .. - Copyright 2018 AT&T Intellectual Property. + Copyright 2019 AT&T Intellectual Property. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may @@ -28,7 +28,7 @@ for a site deployment. These site manifest YAML files generated by spyglass will be saved in a Git repository, from where Pegleg can access and aggregate them. This aggregated file can then be fed to Shipyard for site deployment / updates. -Reference: https://review.opendev.org/#/c/605227 +Reference: https://airshipit.readthedocs.io/projects/specs/en/latest/specs/approved/data_config_generator.html Architecture ------------ @@ -75,9 +75,9 @@ Supported Features Future Work ----------- 1) Schema based manifest generation instead of Jinja2 templates. It shall -be possible to cleanly transition to this schema based generation keeping a unique -mapping between schema and generated manifests. Currently this is managed by -considering a mapping of j2 templates with schemas and site type. +be possible to cleanly transition to this schema based generation keeping a +unique mapping between schema and generated manifests. Currently this is +managed by considering a mapping of j2 templates with schemas and site type. List of Generated Site Manifests: --------------------------------- @@ -129,72 +129,10 @@ Before using Spyglass you must: .. code-block:: console - git clone https://opendev.org/airship/spyglass.git + git clone https://opendev.org/airship/spyglass.git 2. Install the required packages in spyglass: .. code-block:: console - pip3 install -r airship-spyglass/requirements.txt - - -CLI Options ------------ - -Usage: spyglass [OPTIONS] - -Options: - -s, --site TEXT Specify the site for which manifests to be - generated - -t, --type TEXT Specify the plugin type formation or tugboat - -f, --formation_url TEXT Specify the formation url - -u, --formation_user TEXT Specify the formation user id - -p, --formation_password TEXT Specify the formation user password - -i, --intermediary PATH Intermediary file path generate manifests, - use -m also with this option - -d, --additional_config PATH Site specific configuraton details - -g, --generate_intermediary Dump intermediary file from passed excel and - excel spec - -idir, --intermediary_dir PATH The path where intermediary file needs to be - generated - -m, --generate_manifests Generate manifests from the generated - intermediary file - -mdir, --manifest_dir PATH The path where manifest files needs to be - generated - -x, --excel PATH Path to engineering excel file, to be passed - with generate_intermediary - -e, --excel_spec PATH Path to excel spec, to be passed with - generate_intermediary - -l, --loglevel INTEGER Loglevel NOTSET:0 ,DEBUG:10, INFO:20, - WARNING:30, ERROR:40, CRITICAL:50 [default: - 20] - --help Show this message and exit. - --------- -Examples --------- - -1. Running Spyglass with Remote Data Source Plugin - -spyglass -mg --type formation -f -u -p -d -s --template_dir= - -2. Running Spyglass with Excel Plugin - -spyglass -mg --type tugboat -x -e -d -s --template_dir= - -for example: - 2.1 Generating intermediary and manifests - spyglass -mg -t tugboat -x SiteDesignSpec_v1.1.xlsx -e excel_spec_upstream.yaml -d site_config.yaml -s airship-seaworthy --template_dir= - - 2.2 Generating intermediary without manifests - spyglass -g -t tugboat -x SiteDesignSpec_v1.1.xlsx -e excel_spec_upstream.yaml -d site_config.yaml -s airship-seaworthy - - 2.3 Generating manifests without intermediary - spyglass -m -t tugboat -x SiteDesignSpec_v1.1.xlsx -e excel_spec_upstream.yaml -d site_config.yaml -s airship-seaworthy --template_dir= - - 2.4 Generating manifests using intermediary - spyglass -mi --template_dir= - -Where sample 'excel_spec_upstream.yaml', 'SiteDesignSpec_v0.1.xlsx' -'site_config.yaml' and J2 templates can be found under 'spyglass/examples' -folder + pip3 install -r airship-spyglass/requirements.txt diff --git a/doc/source/index.rst b/doc/source/index.rst index 777122c..5a528c1 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -1,5 +1,5 @@ .. - Copyright 2018 AT&T Intellectual Property. + Copyright 2019 AT&T Intellectual Property. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may @@ -33,4 +33,5 @@ fed to Shipyard for site deployment / updates. getting_started developer_quickstart + cli tugboat diff --git a/doc/source/tugboat.rst b/doc/source/tugboat.rst index c17508d..2adf9f1 100644 --- a/doc/source/tugboat.rst +++ b/doc/source/tugboat.rst @@ -1,5 +1,5 @@ .. - Copyright 2018 AT&T Intellectual Property. + Copyright 2019 AT&T Intellectual Property. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may @@ -37,51 +37,56 @@ need to be filled by the Deployment Engineer. Below is the definition for each key in the Excel spec -:: - - - ipmi_sheet_name - name of the sheet from where IPMI and host profile information is to be read - start_row - row number from where the IPMI and host profile information starts - end_row - row number from where the IPMI and host profile information ends - hostname_col - column number where the hostnames are to be read from - ipmi_address_col - column number from where the ipmi addresses are to be read - host_profile_col - column number from where the host profiles are to be read - ipmi_gateway_col - column number from where the ipmi gateways are to be read - private_ip_sheet - name of the sheet which has the private IP information - net_type_col - column number from where the network type is to be read - vlan_col - column number from where the network vlan is to be read - vlan_start_row - row number from where the vlan information starts - vlan_end_row - row number from where the vlan information ends - net_start_row - row number from where the network information starts - net_end_row - row number from where the network information ends - net_col - column number where the IP ranges for network is to be read - net_vlan_col - column number where the vlan information is present in the pod wise network section - public_ip_sheet - name of the sheet which has the public IP information - oam_vlan_col - column number from where the OAM vlan information is to be read from - oam_ip_row - row number from where the OAM network information is to be read from - oam_ip_col - column number from where the OAM network information is to be read from - oob_net_row - row number which has the OOB network subnet ranges - oob_net_start_col - column number from where the OOB network ranges start - oob_net_end_col - column number from where the OOB network ranges end - ingress_ip_row - row number from where the Ingress network information is to be read from - dns_ntp_ldap_sheet - name of the sheet which has the DNS, NTP and LDAP information - login_domain_row - row number which has the ldap login domain - ldap_col - column number which has the all ldap related information - global_group - row number which has the ldap group information - ldap_search_url_row - row number which has the ldap url - ntp_row - row number which has the ntp information - ntp_col - column number which has the ntp information - dns_row - row number which has the dns information - dns_col - column number which has the dns information - domain_row - row number which has the domain information - domain_col - column number which has the domain information - location_sheet - name of the sheet which has the location information - column - column number which has all the information - corridor_row - row number which has the corridor information - site_name_row - row number which has the site name - state_name_row - row number which has the state name - country_name_row - row number which has the country name - clli_name_row - row number which has CLLI information +* ipmi_sheet_name - name of the sheet from where IPMI and host profile + information is to be read +* start_row - row number from where the IPMI and host profile information + starts +* end_row - row number from where the IPMI and host profile information ends +* hostname_col - column number where the hostnames are to be read from +* ipmi_address_col - column number from where the ipmi addresses are to be read +* host_profile_col - column number from where the host profiles are to be read +* ipmi_gateway_col - column number from where the ipmi gateways are to be read +* private_ip_sheet - name of the sheet which has the private IP information +* net_type_col - column number from where the network type is to be read +* vlan_col - column number from where the network vlan is to be read +* vlan_start_row - row number from where the vlan information starts +* vlan_end_row - row number from where the vlan information ends +* net_start_row - row number from where the network information starts +* net_end_row - row number from where the network information ends +* net_col - column number where the IP ranges for network is to be read +* net_vlan_col - column number where the vlan information is present in the + pod wise network section +* public_ip_sheet - name of the sheet which has the public IP information +* oam_vlan_col - column number from where the OAM vlan information is to be + read from +* oam_ip_row - row number from where the OAM network information is to be read + from +* oam_ip_col - column number from where the OAM network information is to be + read from +* oob_net_row - row number which has the OOB network subnet ranges +* oob_net_start_col - column number from where the OOB network ranges start +* oob_net_end_col - column number from where the OOB network ranges end +* ingress_ip_row - row number from where the Ingress network information is to + be read from +* dns_ntp_ldap_sheet - name of the sheet which has the DNS, NTP and LDAP + information +* login_domain_row - row number which has the ldap login domain +* ldap_col - column number which has the all ldap related information +* global_group - row number which has the ldap group information +* ldap_search_url_row - row number which has the ldap url +* ntp_row - row number which has the ntp information +* ntp_col - column number which has the ntp information +* dns_row - row number which has the dns information +* dns_col - column number which has the dns information +* domain_row - row number which has the domain information +* domain_col - column number which has the domain information +* location_sheet - name of the sheet which has the location information +* column - column number which has all the information +* corridor_row - row number which has the corridor information +* site_name_row - row number which has the site name +* state_name_row - row number which has the state name +* country_name_row - row number which has the country name +* clli_name_row - row number which has CLLI information Example: Tugboat Plugin Usage ----------------------------- @@ -95,6 +100,9 @@ Example: Tugboat Plugin Usage e) Site name: airship-seaworthy 2. Spyglass CLI Command: -:: - spyglass -mg -t tugboat -x SiteDesignSpec_v0.1.xlsx -e excel_spec_upstream.yaml -d site_config.yaml -s airship-seaworthy --template_dir= \ No newline at end of file