Merge branch 'master' into the-great-makeover

10 files changed, 73 insertions, 48 deletions

diff --git a/docs/Configuration.md b/docs/Configuration.md
index 6bbbb61..661e65a 100644
--- a/docs/Configuration.md
+++ b/docs/Configuration.md
@@ -4,7 +4,7 @@
 
 In case a model plugin doesn't work correctly (ios, procurve, etc.), you can enable live debugging of SSH/Telnet sessions. Just add a `debug` option containing the value true to the `input` section. The log files will be created depending on the parent directory of the logfile option.
 
-The following example will log an active ssh/telnet session `/home/oxidized/.config/oxidized/log/<IP-Adress>-<PROTOCOL>`. The file will be truncated on each consecutive ssh/telnet session, so you need to put a `tailf` or `tail -f` on that file!
+The following example will log an active ssh/telnet session `/home/oxidized/.config/oxidized/log/<IP-Address>-<PROTOCOL>`. The file will be truncated on each consecutive ssh/telnet session, so you need to put a `tailf` or `tail -f` on that file!
 
 ```yaml
 log: /home/oxidized/.config/oxidized/log
@@ -29,18 +29,18 @@ vars:
 
 ## Removing secrets
 
-To strip out secrets from configurations before storing them, Oxidized needs the the remove_secrets flag. You can globally enable this by adding the following snippet to the global sections of the configuration file.
+To strip out secrets from configurations before storing them, Oxidized needs the `remove_secret` flag. You can globally enable this by adding the following snippet to the global section of the configuration file.
 
 ```yaml
 vars:
   remove_secret: true
 ```
 
-Device models can contain substitution filters to remove potentially sensitive data from configs.
+Device models that contain substitution filters to remove sensitive data will now be run on any fetched configuration.
 
 As a partial example from ios.rb:
 
-```yaml
+```ruby
   cmd :secret do |cfg|
     cfg.gsub! /^(snmp-server community).*/, '\\1 <configuration removed>'
     (...)
@@ -64,7 +64,11 @@ vars:
 
 ## SSH Proxy Command
 
-Oxidized can `ssh` through a proxy as well. To do so we just need to set `ssh_proxy` variable.
+Oxidized can `ssh` through a proxy as well. To do so we just need to set `ssh_proxy` variable with the proxy host information.
+
+This can be provided on a per-node basis by mapping the proper fields from your source.
+
+An example for a `csv` input source that maps the 4th field as the `ssh_proxy` value.
 
 ```yaml
 ...
@@ -79,7 +83,7 @@ vars_map:
 
 ## FTP Passive Mode
 
-Oxidized uses ftp passive mode by default. Some devices require passive mode to be disabled. To do so, we can set `input.ftp.passive` to false
+Oxidized uses ftp passive mode by default. Some devices require passive mode to be disabled. To do so, we can set `input.ftp.passive` to false - this will make use of FTP active mode.
 
 ```yaml
 input:
@@ -192,10 +196,12 @@ rest: 10.0.0.1:8000/oxidized
 
 ## Triggered backups
 
-A node can be moved to head-of-queue via the REST API `GET/POST /node/next/[NODE]`.
+A node can be moved to head-of-queue via the REST API `GET/POST /node/next/[NODE]`. This can be useful to immediately schedule a fetch of the configuration after some other event such as a syslog message indicating a configuration update on the device.
 
-In the default configuration this node will be processed when the next job worker becomes available, it could take some time if existing backups are in progress. To execute moved jobs immediately a new job can be added:
+In the default configuration this node will be processed when the next job worker becomes available, it could take some time if existing backups are in progress. To execute moved jobs immediately a new job can be added automatically:
 
 ```yaml
 next_adds_job: true
 ```
+
+This will allow for a more timely fetch of the device configuration.
diff --git a/docs/Creating-Models.md b/docs/Creating-Models.md
index d7a8155..3cb87c3 100644
--- a/docs/Creating-Models.md
+++ b/docs/Creating-Models.md
@@ -4,6 +4,8 @@ Oxidized supports a growing list of [operating system types](Supported-OS-Types.
 
 A user may wish to extend an existing model to collect the output of additional commands. Oxidized offers smart loading of models in order to facilitate this with ease, without the need to introduce changes to the upstream source code.
 
+This methodology allows local site changes to be preserved during Oxidized version updates / gem updates.
+
 ## Extending an existing model with a new command
 
 The example below can be used to extend the `JunOS` model to collect the output of `show interfaces diagnostics optics` and append the output to the configuration file as a comment. This command retrieves DOM information on pluggable optics present in a `JunOS`-powered chassis.
@@ -25,13 +27,16 @@ class JunOS
 end
 ```
 
-Due to smart loading, the user-supplied `~/.config/oxidized/model/junos.rb` file will take precedence over the model with the same name included in Oxidized. The code then uses `require` to load the included model, and extends the class defined therein with an additional command.
+Due to smart loading, the user-supplied `~/.config/oxidized/model/junos.rb` file will take precedence over the model with the same name included in the Oxidized distribution.
+
+The code then uses `require` to initially load the Oxidized-supplied model, and extends the class defined therein with an additional command.
 
 Intuitively, it is also possible to:
 
 * Completely re-define an existing model by creating a file in `~/.config/oxidized/model/` with the same name as an existing model, but not `require`-ing the upstream model file.
-* Create a named variation of an existing model, by creating a file with a new name (such as `~/.config/oxidized/model/junos-extra.rb`), Then `require` the original model and extend its base class as in the above example. The named variation can then be specified as an OS type for some devices but not others when defining sources.
+* Create a named variation of an existing model, by creating a file with a new name (such as `~/.config/oxidized/model/junos-extra.rb`), Then `require` the original model and extend its base class as in the above example. The named variation can then be specified as an OS type for specific devices that can benefit from the extra functionality. This allows for preservation of the base functionality for the default model types.
 * Create a completely new model, with a new name, for a new operating system type.
+* Testing/validation of an updated model from the [Oxidized GitHub repo models](https://github.com/ytti/oxidized/tree/master/lib/oxidized/model) by placing an updated model in the proper location without disrupting the gem-supplied model files.
 
 ## Advanced features
 
@@ -70,4 +75,4 @@ class JunOS
 end
 ```
 
-The output of the `show configuration | display set` command is marked with a new arbitrary alternative output type, `junos-set`.  The `git` output will use the output type to create a new subdirectory by the same name. In this subdirectory, the `git` output will create files with the name `<devicename>--set` that will contain the output of this command for each device.
\ No newline at end of file
+The output of the `show configuration | display set` command is marked with a new arbitrary alternative output type, `junos-set`.  The `git` output will use the output type to create a new subdirectory by the same name. In this subdirectory, the `git` output will create files with the name `<devicename>--set` that will contain the output of this command for each device.
diff --git a/docs/Hooks.md b/docs/Hooks.md
index fb80a19..ff430ca 100644
--- a/docs/Hooks.md
+++ b/docs/Hooks.md
@@ -1,6 +1,6 @@
 # Hooks
 
-You can define arbitrary number of hooks that subscribe different events. The hook system is modular and different kind of hook types can be enabled.
+You can define an arbitrary number of hooks that subscribe to different events. The hook system is modular and different kind of hook types can be enabled.
 
 ## Configuration
 
@@ -38,10 +38,10 @@ OX_REPO_COMMITREF
 OX_REPO_NAME
 ```
 
-Exec hook recognizes following configuration keys:
+Exec hook recognizes the following configuration keys:
 
-* `timeout`: hard timeout for the command execution. SIGTERM will be sent to the child process after the timeout has elapsed. Default: 60
-* `async`: influences whether main thread will wait for the command execution. Set this true for long running commands so node pull is not blocked. Default: false
+* `timeout`: hard timeout (in seconds) for the command execution. SIGTERM will be sent to the child process after the timeout has elapsed. Default: `60`
+* `async`: Execute the command in an asynchronous fashion. The main thread by default will wait for the hook command execution to complete. Set this to `true` for long running commands so node configuration pulls are not blocked. Default: `false`
 * `cmd`: command to run.
 
 ### exec hook configuration example
@@ -79,8 +79,8 @@ For ssh key-based authentication, it is possible to set the environment variable
 * `remote_repo`: the remote repository to be pushed to.
 * `username`: username for repository auth.
 * `password`: password for repository auth.
-* `publickey`: public key for repository auth.
-* `privatekey`: private key for repository auth.
+* `publickey`: public key file path for repository auth.
+* `privatekey`: private key file path for repository auth.
 
 When using groups, each group must have a unique entry in the `remote_repo` config.
 
@@ -95,7 +95,7 @@ hooks:
 
 ### githubrepo hook configuration example
 
-Authenticate with a username and a password:
+Authenticate with a username and a password without groups in use:
 
 ```yaml
 hooks:
@@ -130,6 +130,11 @@ Fields sent in the message:
 * `model`: Model name (e.g. `eos`)
 * `node`: Device hostname
 
+
+The AWS SNS hook requires the following configuration keys:
+
+* `region`: AWS Region name
+* `topic_arn`: ASN Topic reference
 ### awssns hook configuration example
 
 ```yaml
@@ -141,11 +146,6 @@ hooks:
     topic_arn: arn:aws:sns:us-east-1:1234567:oxidized-test-backup_events
 ```
 
-AWS SNS hook requires the following configuration keys:
-
-* `region`: AWS Region name
-* `topic_arn`: ASN Topic reference
-
 Your AWS credentials should be stored in `~/.aws/credentials`.
 
 ## Hook type: slackdiff
diff --git a/docs/Model-Notes/ArbOS.md b/docs/Model-Notes/ArbOS.md
index f68467f..ebac997 100644
--- a/docs/Model-Notes/ArbOS.md
+++ b/docs/Model-Notes/ArbOS.md
@@ -8,3 +8,5 @@ If you are running ArbOS version 7 or lower then you may need to update the mode
     pre_logout 'exit'
   end
 ```
+
+Back to [Model-Notes](README.md)
diff --git a/docs/Model-Notes/Comware.md b/docs/Model-Notes/Comware.md
index e7a2198..048f312 100644
--- a/docs/Model-Notes/Comware.md
+++ b/docs/Model-Notes/Comware.md
@@ -10,3 +10,5 @@ info-center source default channel 1 log state off debug state off
 ```
 
 [Reference](https://github.com/ytti/oxidized/issues/1171)
+
+Back to [Model-Notes](README.md)
diff --git a/docs/Model-Notes/Netgear.md b/docs/Model-Notes/Netgear.md
index 9bfd094..d82bdcc 100644
--- a/docs/Model-Notes/Netgear.md
+++ b/docs/Model-Notes/Netgear.md
@@ -6,7 +6,7 @@ There are several models available with CLI management via telnet (port 60000),
 ```
 Connected to 192.168.3.201.
 
-(GS748Tv4) 
+(GS748Tv4)
 Applying Interface configuration, please wait ...admin
 Password:********
 (GS748Tv4) >enable
@@ -42,7 +42,7 @@ Configuration for older/newer models: make sure you have defined variable 'enabl
 
 One possible configuration:
 - oxidized config
-```
+```yaml
 source:
   default: csv
   csv:
@@ -63,4 +63,6 @@ switchOldFW:netgear:admin:adminpw::60000
 switchNewFW:netgear:admin:adminpw:true:60000
 ```
 
-[Reference](https://github.com/ytti/oxidized/pull/1268)
\ No newline at end of file
+[Reference](https://github.com/ytti/oxidized/pull/1268)
+
+Back to [Model-Notes](README.md)
diff --git a/docs/Outputs.md b/docs/Outputs.md
index 92c672c..fab3bf8 100644
--- a/docs/Outputs.md
+++ b/docs/Outputs.md
@@ -14,7 +14,7 @@ output:
 
 This uses the rugged/libgit2 interface. So you should remember that normal Git hooks will not be executed.
 
-For a single repositories for all devices:
+For a single repository containing all devices:
 
 ```yaml
 output:
@@ -25,7 +25,7 @@ output:
     repo: "/var/lib/oxidized/devices.git"
 ```
 
-And for groups repositories:
+And for group-based repositories:
 
 ```yaml
 output:
@@ -68,7 +68,7 @@ output:
 This uses the gem git and system git-crypt interfaces. Have a look at [GIT-Crypt](https://www.agwa.name/projects/git-crypt/) documentation to know how to install it.
 Additionally to user and email informations, you have to provide the users ID that can be a key ID, a full fingerprint, an email address, or anything else that uniquely identifies a public key to GPG (see "HOW TO SPECIFY A USER ID" in the gpg man page).
 
-For a single repositories for all devices:
+For a single repository containing all devices:
 
 ```yaml
 output:
@@ -82,7 +82,7 @@ output:
       - "<user@example.com>"
 ```
 
-And for groups repositories:
+And for group-based repositories:
 
 ```yaml
 output:
@@ -130,8 +130,9 @@ Please note that user list is only updated once at creation.
 
 ## Output: Http
 
-POST a config to the specified URL
+The HTTP output will POST a config to the specified HTTP URL. Basic username/password authentication is supported.
 
+Example HTTP output configuration:
 ```yaml
 output:
   default: http
@@ -185,4 +186,4 @@ which will result in the following layout
 diff/$FQDN--show_running_config
 nodiff/$FQDN--show_version
 nodiff/$FQDN--show_inventory
-```
\ No newline at end of file
+```
diff --git a/docs/Ruby-API.md b/docs/Ruby-API.md
index 8621870..a9963b3 100644
--- a/docs/Ruby-API.md
+++ b/docs/Ruby-API.md
@@ -6,7 +6,7 @@ The following objects exist in Oxidized.
 
 * gets config from nodes
 * must implement 'connect', 'get', 'cmd'
-* 'ssh', 'telnet, ftp, and tftp' implemented
+* 'ssh', 'telnet', 'ftp', and 'tftp' implemented
 
 ## Output
 
@@ -18,12 +18,12 @@ The following objects exist in Oxidized.
 
 * gets list of nodes to poll
 * must implement 'load'
-* source can have 'name', 'model', 'group', 'username', 'password', 'input', 'output', 'prompt'
-  * name - name of the devices
-  * model - model to use ios/junos/xyz, model is loaded dynamically when needed (Also default in config file)
-  * input - method to acquire config, loaded dynamically as needed (Also default in config file)
-  * output - method to store config, loaded dynamically as needed (Also default in config file)
-  * prompt - prompt used for node (Also default in config file, can be specified in model too)
+* source can have 'name', 'model', 'group', 'username', 'password', 'input', 'output', 'prompt' for each device.
+  * `name` - name of the device
+  * `model` - model to use ('ios', 'junos', etc).The model is loaded dynamically by the first node of that model type. (Also default in config file)
+  * `input` - method to acquire config, loaded dynamically as needed (Also default in config file)
+  * `output` - method to store config, loaded dynamically as needed (Also default in config file)
+  * `prompt` - prompt used for node (Also default in config file, can be specified in model too)
 * 'sql', 'csv' and 'http' (supports any format with single entry per line, like router.db)
 
 ## Model
@@ -116,16 +116,20 @@ the password prompt. If not specified, the default of `/^Password/` is used.
 #### `post_login`
 
 Used inside `cfg` invocations to specify commands to run once Oxidized has
-logged in to the switch. Takes one argument that is either a block (taking zero
+logged in to the device. Takes one argument that is either a block (taking zero
 parameters) or a string containing a command to execute.
 
+This allows `post_login` to be used for any model-specific items prior to running the regular commands. This could include disabling the output pager or timestamp outputs that would cause constant differences.
+
 #### `pre_logout`
 
 Used to specify commands to run before Oxidized closes the connection to the
-switch. Takes one argument that is either a block (taking zero parameters) or a
+device. Takes one argument that is either a block (taking zero parameters) or a
 string containing a command to execute.
 
+This allows `pre_logout` to be used to 'undo' any changes that may have been needed via `post_login` (restore pager output, etc.)
+
 #### `send`
 
 Usually used inside `expect` or blocks passed to `post_login`/`pre_logout`.
-Takes a single parameter: a string to be sent to the switch.
+Takes a single parameter: a string to be sent to the device.
diff --git a/docs/Sources.md b/docs/Sources.md
index 6d8e19e..eb3cd0c 100644
--- a/docs/Sources.md
+++ b/docs/Sources.md
@@ -59,6 +59,9 @@ pinentry-mode loopback
 
  Oxidized uses the `sequel` ruby gem. You can use a variety of databases that aren't explicitly listed. For more information visit https://github.com/jeremyevans/sequel Make sure you have the correct adapter!
 
+**NOTE** - Many database engines have reserved keywords that may conflict with Oxidized configuration field names (such as 'name', 'group', etc). Pay attention to any names that are used and observed proper quoting methods to avoid errors or unpredictable results.
+
+
 ## Source: MYSQL
 
 `sudo apt-get install libmysqlclient-dev`
diff --git a/docs/Supported-OS-Types.md b/docs/Supported-OS-Types.md
index 4556946..8839cc3 100644
--- a/docs/Supported-OS-Types.md
+++ b/docs/Supported-OS-Types.md
@@ -32,12 +32,12 @@
   * [VOSS (VSP Operating System Software)](/lib/oxidized/model/voss.rb)
   * [BOSS (Baystack Operating System Software)](/lib/oxidized/model/boss.rb)
 * Brocade
-  * [FabricOS](lib/oxidized/model/fabricos.rb)
-  * [Ironware](lib/oxidized/model/ironware.rb)
-  * [NOS (Network Operating System)](lib/oxidized/model/nos.rb)
-  * [Vyatta](lib/oxidized/model/vyatta.rb)
-  * [6910](lib/oxidized/model/br6910.rb)
-  * [SLX-OS](lib/oxidized/model/slxos.rb)
+  * [FabricOS](/lib/oxidized/model/fabricos.rb)
+  * [Ironware](/lib/oxidized/model/ironware.rb)
+  * [NOS (Network Operating System)](/lib/oxidized/model/nos.rb)
+  * [Vyatta](/lib/oxidized/model/vyatta.rb)
+  * [6910](/lib/oxidized/model/br6910.rb)
+  * [SLX-OS](/lib/oxidized/model/slxos.rb)
 * Casa
   * [Casa](/lib/oxidized/model/casa.rb)
 * Check Point