Expect - Using expect to Automate Processes or Generate Reports

I have 51 Linux servers that I manage (soon to grow to well over 70).  Over the past year the company I am with has moved and we have redesigned the network (a few times).  During this redesign we changed which servers provide DNS and NTP services.  I like to think I am a pretty thorough person and believe I updated all 51 servers with the correct DNS and NTP IP Addresses, but I also want to validate my thoroughness as a sanity check and a c.y.a. BTW - I use IP Addresses for the DNS and NTP settings just in case DNS is unavailable

I really don't want to log in to 51 different servers and verify the contents of 3 different configuration files on each of these servers.  It would be nice if I could spend a few minutes writing a script that could poll each server and write out a report that I could review.  So that is just what I did.

A few things I needed to have in place before I got started.
1.  A linux account defined on all 51 servers that has remote SSH permissions and the ability to read the three configuration files I am interested in.  I don't allow root to remotely SSH to any server.
2.  On the computer I will be running the script from (my Linux laptop) a linux utility called expect.
3.  A list of all 51 servers in a text file.

Number 1 is easy as I have a service account (we'll call it saccount) that has access to every server but has very little permissions (but enough to read the files I am interested in).  For number 2 I had to install expect on my laptop which is running a flavor/type-of Redhat Linux.  Expect should be available in your repository for updates.  Number 3 was easy too.  I had a file containing all of my Linux servers.

I ended up with 3 files (not including the report file generated after running the script/s).
File 1:  serverlist.txt - this file contains a list of my servers.  One server name per line.  Example:
serverA
serverB
serverC
server1
server2
server3
   you get the idea...

File 2:  dnsntpreport.exp - you can call it anything you want.  Just make sure it is executable.  the contents of the files are as follows:

   #!/usr/bin/expect -f
   spawn ./dnsntpreport.ksh
   expect {
   "*re you sure you want to continue connecting (yes/no)?"
   {send -- "yes\r\n"
   exp_continue}
   "*assword:*"
   {send "#######\r\n"
   exp_continue}
   }
   exit

where you see #######, you would put the actual password for the user you are using.  This script will watch for certain prompts and answer them with the text you entered automatically.

File 3:  dnsntpreport.ksh - you can call it whatever you want but notice that the above script will call this script so if you change the file name you will need to edit the script above.  The contents of this script are as follows:

   #!/bin/ksh
   for line in $(cat ./serverlist.txt)
   do
   echo -e "\n###$line###" >> dnsntp_report.txt
   echo -e "/etc/resolv.conf file" >> dnsntp_report.txt
   ssh saccount@$line grep -e "10\." /etc/resolv.conf >> dnsntp_report.txt
   echo -e "\n/etc/ntp.conf file" >> dnsntp_report.txt
   ssh saccount@$line grep -e "10\." /etc/ntp.conf >> dnsntp_report.txt
   echo -e "\n/etc/ntp/step-tickers file" >> dnsntp_report.txt
   ssh saccount@$line grep -e "10\." /etc/ntp/step-tickers >> dnsntp_report.txt
   echo -e "###" >> dnsntp_report.txt
   done

So what is going on here?  File 3 will SSH to a server and look through three files for IP Addresses starting with a "10" and record its finds to a file called dnsntp_report.txt.  During our moves and reconfigures the first octet has remained "10" but the others have changed.  Of course when you SSH to a server (assuming you do not have Passwordless SSH setup) you are sometimes prompted whether you trust the key and then for a password.  This is where File 2 comes in and is actually the file you execute from the command-line since it will call File 3.  This file (File 2) will look for two specific prompts and answer them automatically so we don't have to respond 51 or more times.  Obviously, where you see "saccount" in the above script replace with the account you are using.  Remember, the password is stored in File2.

Assuming you have all three files in the same directory and File 2 and File 3 executable all you need to do is run File 2 from the command line.  After the script runs you should have a text file called dnsntp_report.txt that indicates the settings you were (or in this case I was) interested in.

I hope this helps someone else out there.