Expect - Using expect to Automate Processes or Generate Reports

I have 51 Linux servers that I manage (soon to grow to well over 70).  Over the past year the company I am with has moved and we have redesigned the network (a few times).  During this redesign we changed which servers provide DNS and NTP services.  I like to think I am a pretty thorough person and believe I updated all 51 servers with the correct DNS and NTP IP Addresses, but I also want to validate my thoroughness as a sanity check and a c.y.a. BTW - I use IP Addresses for the DNS and NTP settings just in case DNS is unavailable

I really don't want to log in to 51 different servers and verify the contents of 3 different configuration files on each of these servers.  It would be nice if I could spend a few minutes writing a script that could poll each server and write out a report that I could review.  So that is just what I did.

A few things I needed to have in place before I got started.
1.  A linux account defined on all 51 servers that has remote SSH permissions and the ability to read the three configuration files I am interested in.  I don't allow root to remotely SSH to any server.
2.  On the computer I will be running the script from (my Linux laptop) a linux utility called expect.
3.  A list of all 51 servers in a text file.

Number 1 is easy as I have a service account (we'll call it saccount) that has access to every server but has very little permissions (but enough to read the files I am interested in).  For number 2 I had to install expect on my laptop which is running a flavor/type-of Redhat Linux.  Expect should be available in your repository for updates.  Number 3 was easy too.  I had a file containing all of my Linux servers.

I ended up with 3 files (not including the report file generated after running the script/s).
File 1:  serverlist.txt - this file contains a list of my servers.  One server name per line.  Example:
serverA
serverB
serverC
server1
server2
server3
   you get the idea...

File 2:  dnsntpreport.exp - you can call it anything you want.  Just make sure it is executable.  the contents of the files are as follows:

   #!/usr/bin/expect -f
   spawn ./dnsntpreport.ksh
   expect {
   "*re you sure you want to continue connecting (yes/no)?"
   {send -- "yes\r\n"
   exp_continue}
   "*assword:*"
   {send "#######\r\n"
   exp_continue}
   }
   exit

where you see #######, you would put the actual password for the user you are using.  This script will watch for certain prompts and answer them with the text you entered automatically.

File 3:  dnsntpreport.ksh - you can call it whatever you want but notice that the above script will call this script so if you change the file name you will need to edit the script above.  The contents of this script are as follows:

   #!/bin/ksh
   for line in $(cat ./serverlist.txt)
   do
   echo -e "\n###$line###" >> dnsntp_report.txt
   echo -e "/etc/resolv.conf file" >> dnsntp_report.txt
   ssh saccount@$line grep -e "10\." /etc/resolv.conf >> dnsntp_report.txt
   echo -e "\n/etc/ntp.conf file" >> dnsntp_report.txt
   ssh saccount@$line grep -e "10\." /etc/ntp.conf >> dnsntp_report.txt
   echo -e "\n/etc/ntp/step-tickers file" >> dnsntp_report.txt
   ssh saccount@$line grep -e "10\." /etc/ntp/step-tickers >> dnsntp_report.txt
   echo -e "###" >> dnsntp_report.txt
   done

So what is going on here?  File 3 will SSH to a server and look through three files for IP Addresses starting with a "10" and record its finds to a file called dnsntp_report.txt.  During our moves and reconfigures the first octet has remained "10" but the others have changed.  Of course when you SSH to a server (assuming you do not have Passwordless SSH setup) you are sometimes prompted whether you trust the key and then for a password.  This is where File 2 comes in and is actually the file you execute from the command-line since it will call File 3.  This file (File 2) will look for two specific prompts and answer them automatically so we don't have to respond 51 or more times.  Obviously, where you see "saccount" in the above script replace with the account you are using.  Remember, the password is stored in File2.

Assuming you have all three files in the same directory and File 2 and File 3 executable all you need to do is run File 2 from the command line.  After the script runs you should have a text file called dnsntp_report.txt that indicates the settings you were (or in this case I was) interested in.

I hope this helps someone else out there.

A Quick and Dirty Virtual IP (VIP) Address for HA Purposes

A virtual IP address?  Why do I want one of those?  I have two servers (not clustered) set up to run an application.  High Availability is important, but I do not need automatic failover.  So I have the application run on a specific IP address on Server-A while the application is off on Server-B.  In the event Server-A needs to be brought down for maintenance or has an issue, I want to be able to start the application on Server-B with the same IP Address.


Here is the quick and dirty way to do it.
Assumption:  IP Network is 192.168.1.0/255.255.255.0
We will pick 192.168.1.100 for our Virtual IP (VIP) Address


On Server-A:
1.  Create a file called start_adm_vip.sh in /usr/local/sbin
Its contents should be as follows:
/sbin/ifconfig eth0:1 192.168.1.100 netmask 255.255.255.0
/sbin/arping -q -U -c 3 -I eth0 192.168.1.100

2.  Create a file called shutdown_adm_vip.sh
Its contents should be as follows:
/sbin/ifconfig eth0:1 down

3.  Modify permissions on these files so they are executable. 750 should suffice.
chmod 750 *adm_vip.sh

4.  Edit the /etc/hosts file to add a friendly name to the VIP.  Obviously use the name and the IP Address you choose here:
192.168.1.100     appadm.mydomain.net     appadm

5.  Copy the two scripts you just created to Server-B and place them in /usr/local/sbin as well.

On Server-B:
1.  Edit the /etc/hosts file to add a friendly name to the VIP.  Obviously use the name and the IP Address you choose here:

192.168.1.100     appadm.mydomain.net     appadm

Starting the VIP

Now run the script "start_adm_vip.sh" on Server-A.  You should be able to ping "appadm" from both Server-A and Server-B.  Do not go and run the start script on Server-B.  If you do you will have duplicate IP Addresses on the network.  If you want to move the VIP to Server-B, shut it down on Server-A first.

Shutting the VIP down
If you want to manually move the VIP to Server-B you need to shut it down on Server-A first.
On Server-A run the "shutdown_adm_vip.sh".  Now "appadm" should not be pingable from either server.

Go to Server-B and run the script "start_adm_vip.sh".  "appadm" should now be live on Server-B and pingable from both Server-A and Server-B.


Like I said, this is a quick and dirty way to have a VIP.  Hope this helps someone out there.