diff options
33 files changed, 923 insertions, 67 deletions
diff --git a/CHANGELOG.md b/CHANGELOG.md index 0e0c0a1..bf3d8d5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,23 @@ +# 0.8.1 +- BUGFIX: restore ruby 1.9.3 compatibility + +# 0.8.0 +- FEATURE: hooks (by @aakso) +- FEATURE: MRV MasterOS support (by @kwibbly) +- FEATURE: EdgeOS support (by @laf) +- FEATURE: FTP input and Zyxel ZynOS support (by @ytti) +- FEATURE: version and diffs API For oxidized-web (by @FlorianDoublet) +- BUGFIX: aosw, ironware, routeros, xos models +- BUGFIX: crash with 0 nodes +- BUGFIX: ssh auth fail without keyboard-interactive +- Full changelog https://github.com/ytti/oxidized/compare/0.7.1...HEAD + +# 0.7.0 +- FEATURE: support http source (by @laf) +- FEATURE: support Palo Alto PANOS (by @rixxxx) +- BUGFIX: screenos fixes (by @rixxxx) +- BUGFIX: allow 'none' auth in ssh (spotted by @SaldoorMike, needed by ciscosmb+aireos) + # 0.6.0 - FEATURE: support cumulus linux (by @FlorianDoublet) - FEATURE: support HP Comware SMB siwtches (by @sid3windr) diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000..b2aecfb --- /dev/null +++ b/Dockerfile @@ -0,0 +1,11 @@ +FROM debian:latest +MAINTAINER Samer Abdel-Hafez <sam@arahant.net> + +RUN apt-get update && \ + apt-get install -y ruby ruby-dev libsqlite3-dev libssl-dev pkg-config make cmake + +RUN gem install oxidized oxidized-web --no-ri --no-rdoc + +RUN apt-get remove -y ruby-dev pkg-config make cmake + +RUN apt-get -y autoremove
\ No newline at end of file @@ -2,7 +2,7 @@ [](http://badge.fury.io/rb/oxidized) -Oxidized is a network device configuration backup tool. It's a RANCID replacment! +Oxidized is a network device configuration backup tool. It's a RANCID replacement! * automatically adds/removes threads to meet configured retrieval interval * restful API to move node immediately to head-of-queue (GET/POST /node/next/[NODE]) @@ -12,6 +12,7 @@ Oxidized is a network device configuration backup tool. It's a RANCID replacment * restful API to reload list of nodes (GET /reload) * restful API to fetch configurations (/node/fetch/[NODE] or /node/fetch/group/[NODE]) * restful API to show list of nodes (GET /nodes) +* restful API to show list of version for a node (/node/version[NODE]) and diffs [Youtube Video: Oxidized TREX 2014 presentation](http://youtu.be/kBQ_CTUuqeU#t=3h) @@ -22,7 +23,8 @@ Oxidized is a network device configuration backup tool. It's a RANCID replacment * [CentOS, Oracle Linux, Red Hat Linux version 6](#centos-oracle-linux-red-hat-linux-version 6) 3. [Initial Configuration](#configuration) 4. [Installing Ruby 2.1.2 using RVM](#installing-ruby-2.1.2-using-rvm) -5. [Cookbook](#cookbook) +5. [Running with Docker](#running-with-docker) +6. [Cookbook](#cookbook) * [Debugging](#debugging) * [Privileged mode](#privileged-mode) * [Source: CSV](#source-csv) @@ -30,8 +32,9 @@ Oxidized is a network device configuration backup tool. It's a RANCID replacment * [Source: HTTP](#source-http) * [Output: GIT](#output-git) * [Output: File](#output-file) + * [Output types](#output-types) * [Advanced Configuration](#advanced-configuration) -6. [Ruby API](#ruby-api) +7. [Ruby API](#ruby-api) * [Input](#input) * [Output](#output) * [Source](#source) @@ -60,6 +63,7 @@ Oxidized is a network device configuration backup tool. It's a RANCID replacment * DELL PowerConnect * Extreme Networks XOS * Force10 FTOS + * Force10 NDOS * FortiGate FortiOS * HP Comware (HP A-series, H3C, 3Com) * HP ProCurve @@ -67,8 +71,10 @@ Oxidized is a network device configuration backup tool. It's a RANCID replacment * Juniper JunOS * Juniper ScreenOS (Netscreen) * Mikrotik RouterOS + * MRV Master-OS * Ubiquiti AirOS * Palo Alto PAN-OS + * Zyxel ZyNOS # Installation @@ -159,6 +165,43 @@ rvm install 2.1.2 rvm use --default 2.1.2 ``` +# Running with Docker +1. clone git repo: + +``` + root@bla:~# git clone https://github.com/ytti/oxidized +``` +2. build container locally: +``` + root@bla:~# docker build -q -t oxidized/oxidized:latest oxidized/ +``` +3. create config directory in main system: +``` + root@bla~:# mkdir /etc/oxidized +``` +4. run container the first time: +``` + root@bla:~# docker run -v /etc/oxidized:/root/.config/oxidized -p 8888:8888/tcp -t oxidized/oxidized:latest oxidized +``` +5. add 'router.db' to /etc/oxidized: +``` + root@bla:~# vim /etc/oxidized/router.db + [ ... ] + root@bla:~# +``` +6. run container again: +``` + root@bla:~# docker run -v /etc/oxidized:/root/.config/oxidized -p 8888:8888/tcp -t oxidized/oxidized:latest oxidized + oxidized[1]: Oxidized starting, running as pid 1 + oxidized[1]: Loaded 1 nodes + Puma 2.13.4 starting... + * Min threads: 0, max threads: 16 + * Environment: development + * Listening on tcp://0.0.0.0:8888 + ^C + + root@bla:~# +``` ## Cookbook ### Debugging @@ -169,7 +212,7 @@ The following example will log an active ssh session to ```/home/fisakytt/.confi ``` input: default: ssh, telnet - debug: ~/.config/oxidized/log_input + debug: /tmp/oxidized_log_input ssh: secure: false ``` @@ -265,6 +308,66 @@ output: repo: "/var/lib/oxidized/devices.git" ``` +### Output types + +If you prefer to have different outputs in different files and/or directories, you can easily do this by modifying the corresponding model. To change the behaviour for IOS, you would edit `lib/oxidized/model/ios.rb`. + +For example, let's say you want to split out `show version` and `show inventory` into separate files in a directory called `nodiff` which your tools will not send automated diffstats for. You can apply a patch along the lines of + +``` +- cmd 'show version' do |cfg| +- comment cfg.lines.first ++ cmd 'show version' do |state| ++ state.type = 'nodiff' ++ state + +- cmd 'show inventory' do |cfg| +- comment cfg ++ cmd 'show inventory' do |state| ++ state.type = 'nodiff' ++ state ++ end + +- cmd 'show running-config' do |cfg| +- cfg = cfg.each_line.to_a[3..-1].join +- cfg.gsub! /^Current configuration : [^\n]*\n/, '' +- cfg.sub! /^(ntp clock-period).*/, '! \1' +- cfg.gsub! /^\ tunnel\ mpls\ traffic-eng\ bandwidth[^\n]*\n*( ++ cmd 'show running-config' do |state| ++ state = state.each_line.to_a[3..-1].join ++ state.gsub! /^Current configuration : [^\n]*\n/, '' ++ state.sub! /^(ntp clock-period).*/, '! \1' ++ state.gsub! /^\ tunnel\ mpls\ traffic-eng\ bandwidth[^\n]*\n*( + (?:\ [^\n]*\n*)* + tunnel\ mpls\ traffic-eng\ auto-bw)/mx, '\1' +- cfg ++ state = Oxidized::String.new state ++ state.type = 'nodiff' ++ state +``` + +which will result in the following layout + +``` +diff/$FQDN--show_running_config +nodiff/$FQDN--show_version +nodiff/$FQDN--show_inventory +``` + +### RESTful API and Web Interface + +The RESTful API and Web Interface is enabled by configuring the `rest:` parameter in the config file. This parameter can optionally contain a relative URI. + +``` +# Listen on http://127.0.0.1:8888/ +rest: 127.0.0.1:8888 +``` + +``` +# Listen on http://10.0.0.1:8000/oxidized/ +rest: 10.0.0.1:8000/oxidized +``` + ### Advanced Configuration Below is an advanced example configuration. You will be able to (optinally) override options per device. The router.db format used is ```hostname:model:username:password:enable_password```. Hostname and model will be the only required options, all others override the global configuration sections. @@ -313,6 +416,57 @@ model_map: juniper: junos ``` +# Hooks +You can define arbitrary number of hooks that subscribe different events. The hook system is modular and different kind of hook types can be enabled. + +## Configuration +Following configuration keys need to be defined for all hooks: + + * `events`: which events to subscribe. Needs to be an array. See below for the list of available events. + * `type`: what hook class to use. See below for the list of available hook types. + +### Events + * `node_success`: triggered when configuration is succesfully pulled from a node and right before storing the configuration. + * `node_fail`: triggered after `retries` amount of failed node pulls. + * `post_store`: triggered after node configuration is stored. + +## Hook type: exec +The `exec` hook type allows users to run an arbitrary shell command or a binary when triggered. + +The command is executed on a separate child process either in synchronous or asynchronous fashion. Non-zero exit values cause errors to be logged. STDOUT and STDERR are currently not collected. + +Command is executed with the following environment: +``` +OX_EVENT +OX_NODE_NAME +OX_NODE_FROM +OX_NODE_MSG +OX_NODE_GROUP +OX_JOB_STATUS +OX_JOB_TIME +``` + +Exec hook recognizes following configuration keys: + + * `timeout`: hard timeout for the command execution. SIGTERM will be sent to the child process after the timeout has elapsed. Default: 60 + * `async`: influences whether main thread will wait for the command execution. Set this true for long running commands so node pull is not blocked. Default: false + * `cmd`: command to run. + + +## Hook configuration example +``` +hooks: + name_for_example_hook1: + type: exec + events: [node_success] + cmd: 'echo "Node success $OX_NODE_NAME" >> /tmp/ox_node_success.log' + name_for_example_hook2: + type: exec + events: [post_store, node_fail] + cmd: 'echo "Doing long running stuff for $OX_NODE_NAME" >> /tmp/ox_node_stuff.log; sleep 60' + async: true + timeout: 120 +``` # Ruby API @@ -345,3 +499,22 @@ The following objects exist in Oxidized. * cfg is executed in input/output/source context * cmd is executed in instance of model * 'junos', 'ios', 'ironware' and 'powerconnect' implemented + + +# License and Copyright + +Copyright 2013-2015 Saku Ytti <saku@ytti.fi> + 2013-2015 Samer Abdel-Hafez <sam@arahant.net> + + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. diff --git a/extra/nagios_check_failing_nodes.rb b/extra/nagios_check_failing_nodes.rb index 1c81f66..27a5c66 100644 --- a/extra/nagios_check_failing_nodes.rb +++ b/extra/nagios_check_failing_nodes.rb @@ -6,20 +6,30 @@ require 'open-uri' require 'json' critical = false +pending = false critical_nodes = [] +pending_nodes = [] json = JSON.load(open("http://localhost:8888/nodes.json")) json.each do |node| - if node['last']['status'] != 'success' - critical_nodes << node['name'] - critical = true + if not node['last'].nil? + if node['last']['status'] != 'success' + critical_nodes << node['name'] + critical = true + end + else + pending_nodes << node['name'] + pending = true end end -if critical - puts 'Unable to backup: ' + critical_nodes.join(' ') +if pending + puts '[WARN] Pending backup: ' + pending_nodes.join(',') + exit 1 +elsif critical + puts '[CRIT] Unable to backup: ' + critical_nodes.join(',') exit 2 else - puts 'Backup of all nodes completed successfully.' + puts '[OK] Backup of all nodes completed successfully.' exit 0 end diff --git a/extra/oxidized.service b/extra/oxidized.service new file mode 100644 index 0000000..65063b7 --- /dev/null +++ b/extra/oxidized.service @@ -0,0 +1,12 @@ +#For debian 8 put it in /lib/systemd/system/ +#and call it with systemctl start oxidized.service + +[Unit] +Description=Oxidized - Network Device Configuration Backup Tool + +[Service] +ExecStart=/usr/local/bin/oxidized +User=root + +[Install] +WantedBy=multi-user.target diff --git a/extra/syslog.rb b/extra/syslog.rb index 2af83a0..e364cf9 100644 --- a/extra/syslog.rb +++ b/extra/syslog.rb @@ -8,10 +8,18 @@ # set system syslog host SERVER interactive-commands notice # set system syslog host SERVER match "^mgd\[[0-9]+\]: UI_COMMIT: .*" -# Ports < 1024 need extra privileges, use a port higher than this by passing the first argument a number -# To use the default port for syslog (514) you shouldnt pass an argument, but you will need to allow this with: +# Ports < 1024 need extra privileges, use a port higher than this by setting the port option in your oxidized config file. +# To use the default port for syslog (514) you shouldn't pass an argument, but you will need to allow this with: # sudo setcap 'cap_net_bind_service=+ep' /usr/bin/ruby +# Config options are: +# syslogd +# port (Default = 514) +# file (Default = messages) +# resolve (Default = true) + +# To stop the resolution of IP's to PTR you can set resolve to false + # exit if fork ## TODO: proper daemonize require 'socket' @@ -19,26 +27,44 @@ require 'resolv' require_relative 'rest_client' module Oxidized + + require 'asetus' + class Config + Root = File.join ENV['HOME'], '.config', 'oxidized' + end + + CFGS = Asetus.new :name=>'oxidized', :load=>false, :key_to_s=>true + CFGS.default.syslogd.port = 514 + CFGS.default.syslogd.file = 'messages' + CFGS.default.syslogd.resolve = true + + begin + CFGS.load + rescue => error + raise InvalidConfig, "Error loading config: #{error.message}" + ensure + CFG = CFGS.cfg # convenienence, instead of Config.cfg.password, CFG.password + end + class SyslogMonitor NAME_MAP = { /(.*)\.ip\.tdc\.net/ => '\1', /(.*)\.ip\.fi/ => '\1', } - PORT = 514 - FILE = 'messages' MSG = { :ios => /%SYS-(SW[0-9]+-)?5-CONFIG_I:/, :junos => 'UI_COMMIT:', + :eos => /%SYS-5-CONFIG_I:/, + :nxos => /%VSHD-5-VSHD_SYSLOG_CONFIG_I:/, } class << self - def udp port=PORT, listen=0 - port ||= PORT + def udp port=Oxidized::CFG.syslogd.port, listen=0 io = UDPSocket.new io.bind listen, port new io, :udp end - def file syslog_file=FILE + def file syslog_file=Oxidized::CFG.syslogd.file io = open syslog_file, 'r' io.seek 0, IO::SEEK_END new io, :file @@ -102,12 +128,16 @@ module Oxidized end def getname ip - name = (Resolv.getname ip.to_s rescue ip) - NAME_MAP.each { |re, sub| name.sub! re, sub } - name + if Oxidized::CFG.syslogd.resolve == false + ip + else + name = (Resolv.getname ip.to_s rescue ip) + NAME_MAP.each { |re, sub| name.sub! re, sub } + name + end end end end -Oxidized::SyslogMonitor.udp ARGV[0] +Oxidized::SyslogMonitor.udp #Oxidized::SyslogMonitor.file '/var/log/poop' diff --git a/lib/oxidized/config.rb b/lib/oxidized/config.rb index 96c8fbf..f45004a 100644 --- a/lib/oxidized/config.rb +++ b/lib/oxidized/config.rb @@ -9,10 +9,11 @@ module Oxidized OutputDir = File.join Directory, %w(lib oxidized output) ModelDir = File.join Directory, %w(lib oxidized model) SourceDir = File.join Directory, %w(lib oxidized source) + HookDir = File.join Directory, %w(lib oxidized hook) Sleep = 1 end class << self - attr_accessor :mgr + attr_accessor :mgr, :Hooks end CFGS = Asetus.new :name=>'oxidized', :load=>false, :key_to_s=>true CFGS.default.username = 'username' diff --git a/lib/oxidized/core.rb b/lib/oxidized/core.rb index 71267dd..6e7a352 100644 --- a/lib/oxidized/core.rb +++ b/lib/oxidized/core.rb @@ -6,6 +6,7 @@ module Oxidized require 'oxidized/worker' require 'oxidized/nodes' require 'oxidized/manager' + require 'oxidized/hook' class << self def new *args Core.new args @@ -13,9 +14,13 @@ module Oxidized end class Core + class NoNodesFound < OxidizedError; end + def initialize args Oxidized.mgr = Manager.new + Oxidized.Hooks = HookManager.from_config CFG nodes = Nodes.new + raise NoNodesFound, 'source returns no usable nodes' if nodes.size == 0 @worker = Worker.new nodes trap('HUP') { nodes.load } if CFG.rest? diff --git a/lib/oxidized/hook.rb b/lib/oxidized/hook.rb new file mode 100644 index 0000000..7f1942b --- /dev/null +++ b/lib/oxidized/hook.rb @@ -0,0 +1,88 @@ +module Oxidized +class HookManager + class << self + def from_config cfg + mgr = new + cfg.hooks.each do |name,h_cfg| + h_cfg.events.each do |event| + mgr.register event.to_sym, name, h_cfg.type, h_cfg + end + end + mgr + end + end + + # HookContext is passed to each hook. It can contain anything related to the + # event in question. At least it contains the event name + class HookContext < OpenStruct; end + + # RegisteredHook is a container for a Hook instance + class RegisteredHook < Struct.new(:name, :hook); end + + Events = [ + :node_success, + :node_fail, + :post_store, + ] + attr_reader :registered_hooks + + def initialize + @registered_hooks = Hash.new {|h,k| h[k] = []} + end + + def register event, name, hook_type, cfg + unless Events.include? event + raise ArgumentError, + "unknown event #{event}, available: #{Events.join ','}" + end + + Oxidized.mgr.add_hook hook_type + begin + hook = Oxidized.mgr.hook.fetch(hook_type).new + rescue KeyError + raise KeyError, "cannot find hook #{hook_type.inspect}" + end + + hook.cfg = cfg + + @registered_hooks[event] << RegisteredHook.new(name, hook) + Log.debug "Hook #{name.inspect} registered #{hook.class} for event #{event.inspect}" + end + + def handle event, ctx_params={} + ctx = HookContext.new ctx_params + ctx.event = event + + @registered_hooks[event].each do |r_hook| + begin + r_hook.hook.run_hook ctx + rescue => e + Log.error "Hook #{r_hook.name} (#{r_hook.hook}) failed " + + "(#{e.inspect}) for event #{event.inspect}" + end + end + end +end + +# Hook abstract base class +class Hook + attr_accessor :cfg + + def initialize + end + + def cfg=(cfg) + @cfg = cfg + validate_cfg! if self.respond_to? :validate_cfg! + end + + def run_hook ctx + raise NotImplementedError + end + + def log(msg, level=:info) + Log.send(level, "#{self.class.name}: #{msg}") + end + +end +end diff --git a/lib/oxidized/hook/exec.rb b/lib/oxidized/hook/exec.rb new file mode 100644 index 0000000..af2aeb1 --- /dev/null +++ b/lib/oxidized/hook/exec.rb @@ -0,0 +1,84 @@ +class Exec < Oxidized::Hook + include Process + + def initialize + super + @timeout = 60 + @async = false + end + + def validate_cfg! + # Syntax check + if cfg.has_key? "timeout" + @timeout = cfg.timeout + raise "invalid timeout value" unless @timeout.is_a?(Integer) && + @timeout > 0 + end + + if cfg.has_key? "async" + @async = !!cfg.async + end + + if cfg.has_key? "cmd" + @cmd = cfg.cmd + raise "invalid cmd value" unless @cmd.is_a?(String) || @cmd.is_a?(Array) + end + + rescue RuntimeError => e + raise ArgumentError, + "#{self.class.name}: configuration invalid: #{e.message}" + end + + def run_hook ctx + env = make_env ctx + log "Execute: #{@cmd.inspect}", :debug + th = Thread.new do + begin + run_cmd! env + rescue => e + raise e unless @async + end + end + th.join unless @async + end + + def run_cmd! env + pid, status = nil, nil + Timeout.timeout(@timeout) do + pid = spawn env, @cmd , :unsetenv_others => true + pid, status = wait2 pid + unless status.exitstatus.zero? + msg = "#{@cmd.inspect} failed with exit value #{status.exitstatus}" + log msg, :error + raise msg + end + end + rescue TimeoutError + kill "TERM", pid + msg = "#{@cmd} timed out" + log msg, :error + raise TimeoutError, msg + end + + def make_env ctx + env = { + "OX_EVENT" => ctx.event.to_s + } + if ctx.node + env.merge!( + "OX_NODE_NAME" => ctx.node.name.to_s, + "OX_NODE_FROM" => ctx.node.from.to_s, + "OX_NODE_MSG" => ctx.node.msg.to_s, + "OX_NODE_GROUP" => ctx.node.group.to_s, + "OX_EVENT" => ctx.event.to_s, + ) + end + if ctx.job + env.merge!( + "OX_JOB_STATUS" => ctx.job.status.to_s, + "OX_JOB_TIME" => ctx.job.time.to_s, + ) + end + env + end +end diff --git a/lib/oxidized/hook/noophook.rb b/lib/oxidized/hook/noophook.rb new file mode 100644 index 0000000..d4673ba --- /dev/null +++ b/lib/oxidized/hook/noophook.rb @@ -0,0 +1,9 @@ +class NoopHook < Oxidized::Hook + def validate_cfg! + log "Validate config" + end + + def run_hook ctx + log "Run hook with context: #{ctx}" + end +end diff --git a/lib/oxidized/input/ftp.rb b/lib/oxidized/input/ftp.rb new file mode 100644 index 0000000..70db60c --- /dev/null +++ b/lib/oxidized/input/ftp.rb @@ -0,0 +1,54 @@ +module Oxidized + require 'net/ftp' + require 'timeout' + require_relative 'cli' + + class FTP < Input + RescueFail = { + :debug => [ + #Net::SSH::Disconnect, + ], + :warn => [ + #RuntimeError, + #Net::SSH::AuthenticationFailed, + ], + } + include Input::CLI + + def connect node + @node = node + @node.model.cfg['ftp'].each { |cb| instance_exec(&cb) } + @log = File.open(Oxidized::Config::Crash + "-#{@node.ip}-ftp", 'w') if CFG.input.debug? + @ftp = Net::FTP.new @node.ip, @node.auth[:username], @node.auth[:password] + connected? + end + + def connected? + @ftp and not @ftp.closed? + end + + def cmd file + Log.debug "FTP: #{file} @ #{@node.name}" + @ftp.getbinaryfile file, nil + end + + # meh not sure if this is the best way, but perhaps better than not implementing send + def send my_proc + my_proc.call + end + + def output + "" + end + + private + + def disconnect + @ftp.close + #rescue Errno::ECONNRESET, IOError + ensure + @log.close if CFG.input.debug? + end + + end +end diff --git a/lib/oxidized/input/ssh.rb b/lib/oxidized/input/ssh.rb index 46f90f9..21fb02c 100644 --- a/lib/oxidized/input/ssh.rb +++ b/lib/oxidized/input/ssh.rb @@ -20,11 +20,12 @@ module Oxidized @output = '' @node.model.cfg['ssh'].each { |cb| instance_exec(&cb) } secure = CFG.input.ssh.secure - @log = File.open(CFG.input.debug?.to_s + '-ssh', 'w') if CFG.input.debug? - @ssh = Net::SSH.start @node.ip, @node.auth[:username], + @log = File.open(Oxidized::Config::Crash + "-#{@node.ip}-ssh", 'w') if CFG.input.debug? + port = vars(:ssh_port) || 22 + @ssh = Net::SSH.start @node.ip, @node.auth[:username], :port => port.to_i, :password => @node.auth[:password], :timeout => CFG.timeout, :paranoid => secure, - :auth_methods => %w(publickey password), + :auth_methods => %w(none publickey password keyboard-interactive), :number_of_password_prompts => 0 unless @exec shell_open @ssh diff --git a/lib/oxidized/input/telnet.rb b/lib/oxidized/input/telnet.rb index 13fccf7..bf0140c 100644 --- a/lib/oxidized/input/telnet.rb +++ b/lib/oxidized/input/telnet.rb @@ -10,14 +10,19 @@ module Oxidized @node = node @timeout = CFG.timeout @node.model.cfg['telnet'].each { |cb| instance_exec(&cb) } + port = vars(:telnet_port) || 23 - opt = { 'Host' => @node.ip, 'Timeout' => @timeout, - 'Model' => @node.model } - opt['Output_log'] = CFG.input.debug?.to_s + '-telnet' if CFG.input.debug? + opt = { 'Host' => @node.ip, + 'Port' => port.to_i, + 'Timeout' => @timeout, + 'Model' => @node.model } + opt['Output_log'] = Oxidized::Config::Crash + "-#{@node.ip}-telnet" if CFG.input.debug? @telnet = Net::Telnet.new opt - expect username - @telnet.puts @node.auth[:username] + if @node.auth[:username] and @node.auth[:username].length > 0 + expect username + @telnet.puts @node.auth[:username] + end expect password @telnet.puts @node.auth[:password] begin diff --git a/lib/oxidized/manager.rb b/lib/oxidized/manager.rb index b4eaecd..bf28ae7 100644 --- a/lib/oxidized/manager.rb +++ b/lib/oxidized/manager.rb @@ -23,12 +23,13 @@ module Oxidized end end end - attr_reader :input, :output, :model, :source + attr_reader :input, :output, :model, :source, :hook def initialize @input = {} @output = {} @model = {} @source = {} + @hook = {} end def add_input method method = Manager.load Config::InputDir, method @@ -53,5 +54,13 @@ module Oxidized return false if _source.empty? @source.merge! _source end + def add_hook _hook + return nil if @hook.key? _hook + name = _hook + _hook = Manager.load File.join(Config::Root, 'hook'), name + _hook = Manager.load Config::HookDir, name if _hook.empty? + return false if _hook.empty? + @hook.merge! _hook + end end end diff --git a/lib/oxidized/model/aosw.rb b/lib/oxidized/model/aosw.rb index 203c5e0..43e1cff 100644 --- a/lib/oxidized/model/aosw.rb +++ b/lib/oxidized/model/aosw.rb @@ -5,7 +5,7 @@ class AOSW < Oxidized::Model # Also Dell controllers comment '# ' - prompt /^\([^)]+\) #/ + prompt /^\([^)]+\) [#>]/ cmd :all do |cfg| cfg.each_line.to_a[1..-2].join @@ -36,7 +36,16 @@ class AOSW < Oxidized::Model end cfg :telnet, :ssh do + if vars :enable + post_login do + send 'enable\n' + send vars(:enable) + '\n' + end + end post_login 'no paging' + if vars :enable + pre_logout 'exit' + end pre_logout 'exit' end @@ -50,7 +59,7 @@ class AOSW < Oxidized::Model next if line.match /[0-9]+ (RPM|mV|C)$/ out << line.strip end - out = out.join "\n" + out = comment out.join "\n" out << "\n" end diff --git a/lib/oxidized/model/asa.rb b/lib/oxidized/model/asa.rb index d257e9e..547afd7 100644 --- a/lib/oxidized/model/asa.rb +++ b/lib/oxidized/model/asa.rb @@ -23,16 +23,16 @@ class ASA < Oxidized::Model comment cfg end + cmd 'show inventory' do |cfg| + comment cfg + end + cmd 'more system:running-config' do |cfg| cfg = cfg.each_line.to_a[3..-1].join cfg.gsub! /^: [^\n]*\n/, '' cfg end - cmd 'show inventory' do |cfg| - comment cfg - end - cfg :ssh do if vars :enable post_login do diff --git a/lib/oxidized/model/c4cmts.rb b/lib/oxidized/model/c4cmts.rb new file mode 100644 index 0000000..150029c --- /dev/null +++ b/lib/oxidized/model/c4cmts.rb @@ -0,0 +1,52 @@ +class C4CMTS < Oxidized::Model + + # Arris C4 CMTS + + prompt /^([\w.@:\/-]+[#>]\s?)$/ + comment '! ' + + cmd :all do |cfg| + cfg.each_line.to_a[1..-2].map{|line|line.delete("\r").rstrip}.join("\n") + "\n" + end + + cmd :secret do |cfg| + cfg.gsub! /(.+)\s+encrypted-password\s+\w+\s+(.*)/, '\\1 <secret hidden> \\2' + cfg.gsub! /(snmp-server community)\s+".*"\s+(.*)/, '\\1 <secret hidden> \\2' + cfg.gsub! /(tacacs.*\s+key)\s+".*"\s+(.*)/, '\\1 <secret hidden> \\2' + cfg.gsub! /(cable authstring)\s+\w+\s+(.*)/, '\\1 <secret hidden> \\2' + cfg + end + + cmd 'show environment' do |cfg| + cfg.gsub! /\s+[\-\d]+\s+C\s+[\(\s\d]+\s+\F\)/, '' # remove temperature readings + cfg.each_line.to_a[1..-2].join + comment cfg + end + + cmd 'show version' do |cfg| + # remove uptime readings at char 55 and beyond + cfg = cfg.each_line.map{|line|line.rstrip.slice(0..54)}.join("\n") + "\n" + comment cfg + end + + cmd 'show running-config' do |cfg| + cfg = cfg.each_line.to_a[1..-2].join + cfg + end + + cfg :telnet do + username /^Username:/ + password /^Password:/ + end + + cfg :telnet, :ssh do + if vars :enable + post_login do + send "enable\n" + send vars(:enable) + "\n" + end + end + pre_logout 'exit' + end + +end diff --git a/lib/oxidized/model/comware.rb b/lib/oxidized/model/comware.rb index bfc1524..9b36e8b 100644 --- a/lib/oxidized/model/comware.rb +++ b/lib/oxidized/model/comware.rb @@ -1,7 +1,8 @@ class Comware < Oxidized::Model # HP (A-series)/H3C/3Com Comware - prompt /^(<[\w.-]+>)$/ + # sometimes the prompt might have a leading nul + prompt /^\0*(<[\w.-]+>)$/ comment '# ' # example how to handle pager @@ -38,7 +39,7 @@ class Comware < Oxidized::Model end cmd 'display version' do |cfg| - cfg = cfg.each_line.select {|l| not l.match /uptime/ }.join + cfg = cfg.each_line.select {|l| not l.match /uptime/i }.join comment cfg end diff --git a/lib/oxidized/model/dnos.rb b/lib/oxidized/model/dnos.rb new file mode 100644 index 0000000..1c31aad --- /dev/null +++ b/lib/oxidized/model/dnos.rb @@ -0,0 +1,47 @@ +class DNOS < Oxidized::Model + + # Force10 DNOS model # + + comment '! ' + + cmd :all do |cfg| + cfg.each_line.to_a[2..-2].join + end + + cmd :secret do |cfg| + cfg.gsub! /^(snmp-server community).*/, '\\1 <configuration removed>' + cfg.gsub! /secret (\d+) (\S+).*/, '<secret hidden>' + cfg + end + + cmd 'show inventory' do |cfg| + comment cfg + end + + cmd 'show inventory media' do |cfg| + comment cfg + end + + cmd 'show running-config' do |cfg| + cfg = cfg.each_line.to_a[3..-1].join + cfg + end + + cfg :telnet do + username /^Login:/ + password /^Password:/ + end + + cfg :telnet, :ssh do + post_login 'terminal length 0' + post_login 'terminal width 0' + if vars :enable + post_login do + send "enable\n" + send vars(:enable) + "\n" + end + end + pre_logout 'exit' + end + +end diff --git a/lib/oxidized/model/edgeos.rb b/lib/oxidized/model/edgeos.rb new file mode 100644 index 0000000..2a8d663 --- /dev/null +++ b/lib/oxidized/model/edgeos.rb @@ -0,0 +1,27 @@ +class Edgeos < Oxidized::Model + + # EdgeOS # + + prompt /\@.*?\:~\$\s/ + + cmd :all do |cfg| + cfg = cfg.lines.to_a[1..-2].join + end + + cmd :secret do |cfg| + cfg.gsub! /community (\S+) {/, 'community <hidden> {' + cfg + end + + cmd 'show configuration | no-more' + + cfg :telnet do + username /login:\s/ + password /^Password:\s/ + end + + cfg :telnet, :ssh do + pre_logout 'exit' + end + +end diff --git a/lib/oxidized/model/ironware.rb b/lib/oxidized/model/ironware.rb index e18902b..87a51a6 100644 --- a/lib/oxidized/model/ironware.rb +++ b/lib/oxidized/model/ironware.rb @@ -1,6 +1,6 @@ class IronWare < Oxidized::Model - prompt /^.+[>#]\s?$/ + prompt /^.*(telnet|ssh)\@.+[>#]\s?$/i comment '! ' #to handle pager without enable @@ -16,22 +16,34 @@ class IronWare < Oxidized::Model #end cmd :all do |cfg| - cfg.each_line.to_a[1..-2].join - end - - cmd 'show running-config' do |cfg| - cfg = cfg.each_line.to_a[3..-1].join - cfg + # sometimes ironware inserts arbitrary whitespace after commands are + # issued on the CLI, from run to run. this normalises the output. + cfg.each_line.to_a[1..-2].drop_while { |e| e.match /^\s+$/ }.join end cmd 'show version' do |cfg| - cfg.gsub! /(^((.*)system uptime(.*))$)/, '' #remove unwanted line system uptime + cfg.gsub! /(^((.*)[Ss]ystem uptime(.*))$)/, '' #remove unwanted line system uptime + cfg.gsub! /[Uu]p\s?[Tt]ime is .*/,'' + comment cfg end cmd 'show chassis' do |cfg| - cfg.gsub! "\xFF", '' # ugly hack - avoids JSON.dump utf-8 breakage on 1.9.. + cfg.encode!("UTF-8", :invalid => :replace) #sometimes ironware returns broken encoding cfg.gsub! /(^((.*)Current temp(.*))$)/, '' #remove unwanted lines current temperature + cfg.gsub! /Speed = [A-Z]{3} \(\d{2}\%\)/, '' #remove unwanted lines Speed Fans + cfg.gsub! /current speed is [A-Z]{3} \(\d{2}\%\)/, '' + cfg.gsub! /\d{2}\.\d deg-C/, 'XX.X deg-C' + if cfg.include? "TEMPERATURE" + sc = StringScanner.new cfg + out = '' + temps = '' + out << sc.scan_until(/.*TEMPERATURE/) + temps << sc.scan_until(/.*Fans/) + out << sc.rest + cfg = out + end + comment cfg end @@ -40,22 +52,31 @@ class IronWare < Oxidized::Model end cmd 'show module' do |cfg| + cfg.gsub! /^((Invalid input)|(Type \?)).*$/, '' # some ironware devices are fixed config comment cfg end + cmd 'show running-config' do |cfg| + arr = cfg.each_line.to_a + arr[2..-1].join unless arr.length < 2 + end + cfg :telnet do - username /^Username:/ - password /^Password:/ + # match expected prompts on both older and newer + # versions of IronWare + username /^(Please Enter Login Name|Username):/ + password /^(Please Enter )Password:/ end #handle pager with enable cfg :telnet, :ssh do if vars :enable post_login do - send "enable\n" - send vars(:enable) + "\n" + send "enable\r\n" + send vars(:enable) + "\r\n" end end + post_login '' post_login 'skip-page-display' post_login 'terminal length 0' pre_logout 'logout' @@ -63,4 +84,4 @@ class IronWare < Oxidized::Model pre_logout 'exit' end -end
\ No newline at end of file +end diff --git a/lib/oxidized/model/junos.rb b/lib/oxidized/model/junos.rb index e43d71a..0e921d2 100644 --- a/lib/oxidized/model/junos.rb +++ b/lib/oxidized/model/junos.rb @@ -9,7 +9,7 @@ class JunOS < Oxidized::Model cmd :all do |cfg| # we don't need screen-scraping in ssh due to exec cfg = cfg.lines.to_a[1..-2].join if telnet - cfg.lines.map { |line| line.rstrip }.join "\n" + cfg.lines.map { |line| line.rstrip }.join("\n") + "\n" end cmd :secret do |cfg| diff --git a/lib/oxidized/model/masteros.rb b/lib/oxidized/model/masteros.rb new file mode 100644 index 0000000..3f5a2fc --- /dev/null +++ b/lib/oxidized/model/masteros.rb @@ -0,0 +1,46 @@ +class MasterOS < Oxidized::Model + + # MRV MasterOS model # + +comment '!' + + cmd :secret do |cfg| + cfg.gsub! /^(snmp-server community).*/, '\\1 <configuration removed>' + cfg.gsub! /username (\S+) password encrypted (\S+) class (\S+).*/, '<secret hidden>' + cfg + end + + cmd :all do |cfg| + cfg.each_line.to_a[1..-2].join + end + + cmd 'show inventory' do |cfg| + cfg = cfg.each_line.to_a[0..-2].join + comment cfg + end + + cmd 'show plugins' do |cfg| + comment cfg + end + + cmd 'show hw-config' do |cfg| + comment cfg + end + + cmd 'show running-config' do |cfg| + cfg = cfg.each_line.to_a[3..-1].join + cfg + end + + cfg :telnet, :ssh do + post_login 'no pager' + if vars :enable + post_login do + send "enable\n" + send vars(:enable) + "\n" + end + end + pre_logout 'exit' + end + +end
\ No newline at end of file diff --git a/lib/oxidized/model/powerconnect.rb b/lib/oxidized/model/powerconnect.rb index 0b28f37..ec15402 100644 --- a/lib/oxidized/model/powerconnect.rb +++ b/lib/oxidized/model/powerconnect.rb @@ -25,7 +25,7 @@ class PowerConnect < Oxidized::Model cmd 'show running-config' - cfg :telnet do + cfg :telnet, :ssh do username /^User( Name)?:/ password /^\r?Password:/ end diff --git a/lib/oxidized/model/routeros.rb b/lib/oxidized/model/routeros.rb index bd588b7..4822500 100644 --- a/lib/oxidized/model/routeros.rb +++ b/lib/oxidized/model/routeros.rb @@ -1,5 +1,5 @@ class RouterOS < Oxidized::Model - prompt /^\[\w+@\S+\]\s?>\s?$/ + prompt /\[\w+@\S+\]\s?>\s?$/ comment "# " cmd '/system routerboard print' do |cfg| @@ -7,10 +7,16 @@ class RouterOS < Oxidized::Model end cmd '/export' do |cfg| + cfg.gsub! /\x1B\[([0-9]{1,3}((;[0-9]{1,3})*)?)?[m|K]/, '' # strip ANSI colours cfg = cfg.split("\n").select { |line| not line[/^\#\s\w{3}\/\d{2}\/\d{4}.*$/] } cfg.join("\n") + "\n" end + cfg :telnet do + username /^Login:/ + password /^Password:/ + end + cfg :ssh do exec true end diff --git a/lib/oxidized/model/xos.rb b/lib/oxidized/model/xos.rb index 88c81ed..de8ec39 100644 --- a/lib/oxidized/model/xos.rb +++ b/lib/oxidized/model/xos.rb @@ -6,7 +6,9 @@ class XOS < Oxidized::Model comment '# ' cmd :all do |cfg| - cfg.each_line.to_a[1..-2].join.rstrip + # xos inserts leading \r characters and other trailing white space. + # this deletes extraneous \r and trailing white space. + cfg.each_line.to_a[1..-2].map{|line|line.delete("\r").rstrip}.join("\n") + "\n" end cmd 'show version' do |cfg| @@ -35,6 +37,7 @@ class XOS < Oxidized::Model cfg :telnet, :ssh do post_login 'disable clipaging' pre_logout 'exit' + pre_logout 'n' end end diff --git a/lib/oxidized/model/zynos.rb b/lib/oxidized/model/zynos.rb new file mode 100644 index 0000000..89be8af --- /dev/null +++ b/lib/oxidized/model/zynos.rb @@ -0,0 +1,12 @@ +class ZyNOS < Oxidized::Model + + # Used in Zyxel DSLAMs, such as SAM1316 + + comment '! ' + + cmd 'config-0' + + cfg :ftp do + end + +end diff --git a/lib/oxidized/node.rb b/lib/oxidized/node.rb index 253de53..d50317f 100644 --- a/lib/oxidized/node.rb +++ b/lib/oxidized/node.rb @@ -29,6 +29,9 @@ module Oxidized def run status, config = :fail, nil @input.each do |input| + # don't try input if model is missing config block, we may need strong config to class_name map + cfg_name = input.to_s.split('::').last.downcase + next unless @model.cfg[cfg_name] and not @model.cfg[cfg_name].empty? @model.input = input = input.new if config=run_input(input) status = :success diff --git a/lib/oxidized/nodes.rb b/lib/oxidized/nodes.rb index cb2fbc5..cb2ce7b 100644 --- a/lib/oxidized/nodes.rb +++ b/lib/oxidized/nodes.rb @@ -111,7 +111,7 @@ module Oxidized end def find_index node - index { |e| e.name == node } + index { |e| e.name == node or e.ip == node} end # @param node node which is removed from nodes list @@ -148,6 +148,35 @@ module Oxidized end end end + + public + + def version node, group + with_lock do + i = find_node_index node + output = self[i].output.new + raise Oxidized::NotSupported unless output.respond_to? :fetch + output.version node, group + end + end + + def get_version node, group, oid + with_lock do + i = find_node_index node + output = self[i].output.new + raise Oxidized::NotSupported unless output.respond_to? :fetch + output.get_version node, group, oid + end + end + + def get_diff node, group, oid1, oid2 + with_lock do + i = find_node_index node + output = self[i].output.new + raise Oxidized::NotSupported unless output.respond_to? :fetch + output.get_diff node, group, oid1, oid2 + end + end end end diff --git a/lib/oxidized/output/git.rb b/lib/oxidized/output/git.rb index 64497c9..3757cfc 100644 --- a/lib/oxidized/output/git.rb +++ b/lib/oxidized/output/git.rb @@ -2,7 +2,6 @@ module Oxidized class Git < Output class GitError < OxidizedError; end begin - gem 'rugged', '~> 0.21.0' require 'rugged' rescue LoadError raise OxidizedError, 'rugged not found: sudo gem install rugged' @@ -20,6 +19,7 @@ class Git < Output CFGS.save :user raise NoConfig, 'no output git config, edit ~/.config/oxidized/config' end + @cfg.repo = File.expand_path @cfg.repo end def store file, outputs, opt={} @@ -51,18 +51,95 @@ class Git < Output def fetch node, group begin repo = @cfg.repo - if group - repo = File.join File.dirname(repo), group + '.git' - end + repo = File.join File.dirname(repo), group + '.git' if group and not @cfg.single_repo? repo = Rugged::Repository.new repo index = repo.index index.read_tree repo.head.target.tree unless repo.empty? - repo.read(index.get(node)[:oid]).data + file = node + file = File.join(group, node) if group and @cfg.single_repo? + repo.read(index.get(file)[:oid]).data rescue 'node not found' end end + #give a hash of all oid revision for the givin node, and the date of the commit + def version node, group + begin + repo = @cfg.repo + if group + repo = File.join File.dirname(repo), group + '.git' + end + repo = Rugged::Repository.new repo + walker = Rugged::Walker.new(repo) + walker.sorting(Rugged::SORT_DATE) + walker.push(repo.head.target) + i = -1 + tab = [] + walker.each do |commit| + if commit.diff(paths: [node]).size > 0 + hash = {} + hash[:date] = commit.time.to_s + hash[:oid] = commit.oid + hash[:author] = commit.author + hash[:message] = commit.message + tab[i += 1] = hash + end + end + walker.reset + tab + rescue + 'node not found' + end + end + + #give the blob of a specific revision + def get_version node, group, oid + begin + repo = @cfg.repo + if group && group != '' + repo = File.join File.dirname(repo), group + '.git' + end + repo = Rugged::Repository.new repo + repo.blob_at(oid,node).content + rescue + 'version not found' + end + end + + #give a hash with the patch of a diff between 2 revision and the stats (added and deleted lines) + def get_diff node, group, oid1, oid2 + begin + repo = @cfg.repo + diff_commits = nil + if group && group != '' + repo = File.join File.dirname(repo), group + '.git' + end + repo = Rugged::Repository.new repo + commit = repo.lookup(oid1) + #if the second revision is precised + if oid2 + commit_old = repo.lookup(oid2) + diff = repo.diff(commit_old, commit) + diff.each do |patch| + if /#{node}\s+/.match(patch.to_s.lines.first) + diff_commits = {:patch => patch.to_s, :stat => patch.stat} + break + end + end + #else gives the diffs between the first oid and his first parrent + else + stat = commit.parents[0].diff(commit).stat + stat = [stat[1],stat[2]] + patch = commit.parents[0].diff(commit).patch + diff_commits = {:patch => patch, :stat => stat} + end + diff_commits + rescue + 'no diffs' + end + end + private def update repo, file, data @@ -105,7 +182,9 @@ class Git < Output :parents => repo.empty? ? [] : [repo.head.target].compact, :update_ref => 'HEAD', ) + index.write + true end end end diff --git a/lib/oxidized/worker.rb b/lib/oxidized/worker.rb index 7ed70ac..eea747e 100644 --- a/lib/oxidized/worker.rb +++ b/lib/oxidized/worker.rb @@ -34,11 +34,17 @@ module Oxidized @jobs.duration job.time node.running = false if job.status == :success + Oxidized.Hooks.handle :node_success, :node => node, + :job => job msg = "update #{node.name}" msg += " from #{node.from}" if node.from msg += " with message '#{node.msg}'" if node.msg - node.output.new.store node.name, job.config, + if node.output.new.store node.name, job.config, :msg => msg, :user => node.user, :group => node.group + Log.info "Configuration updated for #{node.group}/#{node.name}" + Oxidized.Hooks.handle :post_store, :node => node, + :job => job + end node.reset else msg = "#{node.name} status #{job.status}" @@ -49,6 +55,8 @@ module Oxidized else msg += ", retries exhausted, giving up" node.retry = 0 + Oxidized.Hooks.handle :node_fail, :node => node, + :job => job end Log.warn msg end diff --git a/oxidized.gemspec b/oxidized.gemspec index 90b77b0..d2956b1 100644 --- a/oxidized.gemspec +++ b/oxidized.gemspec @@ -1,10 +1,10 @@ Gem::Specification.new do |s| s.name = 'oxidized' - s.version = '0.6.0' + s.version = '0.8.1' s.licenses = %w( Apache-2.0 ) s.platform = Gem::Platform::RUBY - s.authors = [ 'Saku Ytti', 'Samer Abdel-Hafez' ] - s.email = %w( saku@ytti.fi sam@arahant.net ) + s.authors = [ 'Saku Ytti', 'Samer Abdel-Hafez', 'Anton Aksola' ] + s.email = %w( saku@ytti.fi sam@arahant.net aakso@iki.fi) s.homepage = 'http://github.com/ytti/oxidized' s.summary = 'feeble attempt at rancid' s.description = 'software to fetch configuration from network devices and store them' @@ -18,4 +18,5 @@ Gem::Specification.new do |s| s.add_runtime_dependency 'slop', '~> 3.5' s.add_runtime_dependency 'net-ssh', '~> 2.8' s.add_runtime_dependency 'rugged', '~> 0.21', '>= 0.21.4' + s.add_development_dependency 'pry', '~> 0' end |