snmpd Information Filling up the /var/log/messages File

I am using net-snmp on my linux servers so cacti can poll for data and graph statistics.  I noticed that the /var/log/messages file was filling up with snmpd messages.  All of which were merely informational and benign.  I know snmpd works and is configured properly and those log messages in my /var/log/messages file makes it hard to find anything useful in it.

I found that (at least in the version of net-snmp that I am using) debug logging is turned on by default.  Well, I don't want debug level logging.  In fact I don't want any logging for snmpd to go to my /var/log/messages file.

I run Oracle Enterprise Linux and Ubuntu.

On any flavor of Red Hat Enterprise Linux (example:  RHEL, OEL, CentOS) modify the /etc/sysconfig/snmpd.options file.  If it doesn't exist, create it.

The contents of that file should be changed to this:

     # snmpd command line options
     OPTIONS="-Lf /dev/null -p /var/run/snmpd.pid -a"

This will turn off all logging for snmpd.  Remember to restart snmpd for the changes to take affect.
     service snmpd restart

On Ubuntu edit the /etc/init.d/snmpd file and change the line that looks like this:
     SNMPDOPTS='-Lsd -Lf /dev/null -p /var/run/snmpd.pid'
to this
     SNMPDOPTS='-Lf /dev/null -p /var/run/snmpd.pid'

That's it.  Remember to restart snmpd for the changes to take affect
     /etc/init.d/snmpd restart

Interactive Tape Backups using TAR and Linux

I sometimes want to run an on-demand backup of either a particular directory or file system.  I wrote an interactive script to do this and thought I would share it.
Assumptions:
1.  You have a tape device attached to your Computer or Server.
2.  You know what device your tape drive is.  Example: /dev/st0
3.  You have the mt-st package installed to manage the tape device.
4.  The tape you are using will be overwritten.
5.  You copy the contents of the script below in to a utility like notepad and check the contents.  Then copy from there in to a script called (whatever you want).
6.  Pay close attention to the command that starts like this:  TAPECHK=$(mt  It show up correctly in this post but if you cut and past the script in to notepad the lines do not match.  Edit it so it looks like it does here in the post.
7.  You make the script executable.
8.  I placed the script in /usr/local/sbin but you can put it where ever it make sense to you.

Here is the script:

#!/bin/bash
#
# SCRIPT: Interactive_2_tape.bash
# AUTHOR: Bob
# DATE: 02/24/2010
# REV:
#
# PURPOSE: This script is used to backup files
# from $SOURCE to $TAPEDEV
#
# set -x # Uncomment to debug this script
#
# set -n # Uncomment to check the script.s syntax
#        # without any execution. Do not forget to
#        # recomment this line!
#
####################
# Define Variables #
####################

# Capture the shell script file name
THIS_SCRIPT=$(basename $0)

# Define the start time of the script
STARTTIME=`date +%T`

# Define current directory to return to at end of script
CURRENTDIR=$PWD

# Ask for the source of the backup
echo "What directory or filesystem do you want to backup?"
echo "Type the directory in this format /dir1/dir2 followed by [ENTER]:"
read SOURCE
echo "Using $SOURCE as the source directory you want to backup."

# Ask for tape device
TAPEDEV="/dev/st0"
echo "I assume your tape device is $TAPEDEV"
read -p "Am I correct? yes/no: "
if [ "$REPLY" = "no" ]; then
     echo "What is your tape device? "
     read TAPEDEV
     echo "Using $TAPEDEV as your tape device."
else
     echo "Using $TAPEDEV as your tape device."
fi


################
# Main Section #
################

# Verify there is a tape in the
# drive and rewind the tape
TAPECHK=$(mt -f $TAPEDEV rewind 2>&1 1>/dev/null)
# If there is no tape tell me and exit out
#  If the mt command return any data then there is an error
#  Check the results of the mt command
if [ "$TAPECHK" != "" ]; then
     echo $TAPECHK
     echo "Check to see if there is a tape in the drive or if the device you entered is valid."
     exit
fi

# If we made it here, there is a tape in the drive and it has rewinded.
# Change to the directory to be backed up
cd $SOURCE
echo "Changing to the $SOURCE directory."

# Back up data
tar cvf $TAPEDEV .

# Rewind the tape again
mt -f $TAPEDEV rewind

# Change back to the directory from where you came
cd $CURRENTDIR
echo "Changing back to the directory you started from: $CURRENTDIR"
# Define the end time of this script
ENDTIME=`date +%T`

# Display the start and end time of this script
echo "$THIS_SCRIPT began at $STARTTIME and finished at $ENDTIME"

exit
#################
# End of Script #
#################

I hope this helps others out there trying to do the same thing.

Using tar to Copy a Large File into a Tight Space

I do a lot of work with virtual machine images.  I ran in to a situation where I wanted to copy a file called System.img from one server to another and even though I knew I had enough room to do it, I would get messages stating that there was not enough space.  What in the world was going on?  The file system I wanted to copy the file to was 33G in size.  The file was just under 33G in size.  I knew it should fit.  I knew this because the file system this file is coming from is also 33GB in size (same identical size).  When it was all said and done I should of had about 150M of free space according to the source.

I tried FTP, SCP, and various other mechanisms to copy the file from the one server to the other.  No joy.
So I copied the file to the destination server but to a different and larger file system.  That obviously was successful, but I still wanted it on my 33G file system.  So I tried copying the file locally from the larger file system to the 33G file system.  No joy again... I got a message after a few minutes stating there was not enough space, and the process errored out.

I found a solution!  Now, to be honest, I do not know why it works, but it does.

Assumptions: You are in the directory the System.img file is located.  The file resides on the same server you are copying to.  You have done the math and according to the calculator the file will fit on the destination file system.

Run this command with the appropriate path of your destination.

Note:  The following command is all on one line.
# tar cvf - System.img | ( cd /destination file system/destination folder/;tar xvf - )

Works like a charm.  I have used this little gem a dozen times in the past few months.

This should be obvious but...  Remember, you can not copy/place a file that is larger than the space available on the destination.  Hope this helps others out there.

Filesystem Replication Using rsync and SHH

There are many reasons you may need to replicate a file system.  My reason was for DR purposes.  In a previous post I set up Passwordless SSH sessions between two systems.  This is a requirement if you want to sync file systems on an automated schedule.  I looked on the web for a script that would do what I wanted and I could not find something that met my needs.  So I wrote the script below.  To give credit where credit is due, I borrowed from some ideas and code from Randal K. Michael, author of Mastering UNIX Shell Scripting.  I placed the script below in /usr/local/bin directory on the "source" node and called it fsrsync.bash.  This script will replicated a designated filesystem from a source node to two different nodes after confirming they are "alive".  I used rsync because after the initial sync, future replications are much faster since only update and/or changes are sent and not the entire filesystem.

Here is the script below.  Highlight the contents and select copy.
Note:  When you grab the test here and paste it, do a sanity check on the text to verify the formatting has not changed.
Edit/create /usr/local/bin/fsrsync.bash and paste in the contents.
# vi /usr/local/bin/fsrsync.bash

Script starts below.

#!/bin/bash
#
# SCRIPT: fsrsync.bash
# AUTHOR:
# DATE:
# REV:
#
# PURPOSE: This script is used to replicate the
# /somedir/test filesystem from Node A to Node B and C
#
# set -x # Uncomment to debug this script
#
# set -n # Uncomment to check the script.s syntax
#        # without any execution. Do not forget to
#        # recomment this line!
#
##############################################
# DEFINE FILES AND GLOBAL VARIABLES HERE
##############################################

# Define the target machines to copy data to.
# To specify more than one host enclose the
# hostnames in double quotes and put at least
# one space between each hostname
#
# EXAMPLE: MACHINE_LIST="fred yogi booboo"

MACHINE_LIST="nodeB nodeC"

# Capture the shell script file name

THIS_SCRIPT=$(basename $0)

# The FS_PATTERN variable defines the regular expression
# matching the filesystems we want to replicate with rsync.
# Example:  FS_PATTERN="/home"

FS_PATTERN="/somedir/test"

# Query the system for the hostname

THIS_HOST=$(hostname)

##############################################
# BEGINNING OF MAIN
##############################################

# Comfirm the nodes are alive and replicate
# the filesystems.
echo -e "\n####################################################\n"
echo -e "$THIS_SCRIPT started execution $(date)\n"
echo -e "Verifying the node is alive..."

for M in $MACHINE_LIST
do
    echo "Pinging $M..."
    ping -c1 $M >/dev/null 2>&1
    if (( $? != 0 ))
    then
        echo -e "ERROR: $M host is not pingable...cannot continue..."
        echo -e "...EXITING...\n"
        echo -e "####################################################"
        exit 2
    else
        echo -e "$M is alive... Starting rsync process!\n"
        echo -e "Replicating $FS_PATTERN/ from $THIS_HOST to $M\n"
        #The rsync command is all on one line although it doesn't appear so here.  This comment can be removed.
        rsync -aqz --delete -e ssh $FS_PATTERN/ root@$M:$FS_PATTERN
    fi
echo -e "$THIS_SCRIPT finished execution $(date)\n"
echo -e "####################################################"
done

###############################################
# END OF SCRIPT
###############################################

Make sure you make the file executable.
# chmod 754 /usr/local/bin/fsrsync.bash

You can manually run the file by simply running this command as a user with the appropriate rights.
# /usr/local/bin/fsrsync.bash

Do you want to schedule this script to replicate the file system every 5 minutes and log results?  Add the following entry to the crontab.
# crontab -e
0,5,10,15,20,25,30,35,40,45,50,55 * * * *     /usr/local/bin/fsrsync.bash 2>&1 >> /var/log/fsrsync.log


Do you want to rotate your log file (assuming you use logrotate)?  If so create a file in /etc/logrotate.d/ called fsrsync.
# vi /etc/logrotate.d/fsrsync

The contents of the file should look like this:
/var/log/fsrsync.log {
        weekly
        rotate 4
        nocompress
        missingok
}

I hope this helps someone out there.

PS:  If you want to test logrotate without having to wait a week you can do the following.
# /usr/sbin/logrotate -v /etc/logrotate.d/fsrsync
This will give you details on what it will do and rotate the log if needed.  If you want to force a log rotation, do the following.
# /usr/sbin/logrotate -f /etc/logrotate.d/fsrsync

Passwordless SSH Setup

Disclaimer:  I would not suggest doing this as root.  I am only using root as an example.

I needed to replicate a file system from a production server to a DR server.  I wanted to script and schedule this so there was no intervention needed from an end user.  The first step was to setup passwordless SSH between the source and destination.  I found a few tutorials out on the web but they were not as clear as would have liked.  So, I documented my process and thought I would share it with you.

We will assume we have two hosts, host1 and host2.  For my purposes, I want host2 to be able to run commands through ssh to host1 without being prompted for a password.  In this example, we’ll assume the user running these commands is “root”.

On host2 you will need to do the following.
 - Log in as root to host2
 - Verify the following directory exists
/root/.ssh
 - You can do this by issuing the following commands
# cd
# ls -al | grep .ssh
 - If in the output returned you see .ssh, then the directory exists.  If you are returned to a command prompt without seeing .ssh you will need to create the directory.
 - If you need to create the directory issue the following commands.
# mkdir -p /root/.ssh
# chmod 700 /root/.ssh
- Now run the following command
# ssh-keygen -t rsa
*** You will see output similar to below ***
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
 - You can hit enter at the above prompt and accept the defaults for the two prompts below
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.

On host1 you will need to do the following.
- Log in as root to host1
 - Verify the following directory exists
/root/.ssh
 - You can do this by issuing the following commands
# cd
# ls -al | grep .ssh
 - If in the output returned you see .ssh, then the directory exists.  If you are returned to a command prompt without seeing .ssh you will need to create the directory.
 - If you need to create the directory issue the following commands.
# mkdir -p /root/.ssh
# chmod 700 /root/.ssh
- Now copy host2’s id_rsa.pub key to host1 (assuming you are still on host1) renaming it to host2.pub
# scp host2:/root/.ssh/id_rsa.pub /root/.ssh/host2.pub
***  Note:  my version(s) require authorized_keys2, your file may need to be named authorized_keys
- Now copy /root/.ssh/host2.pub to /root/.ssh/authorized_keys2
# cp /root/.ssh/host2.pub /root/.ssh/authorized_keys2

Now from host2 you should be able to ssh to host1 without being prompted for a password.
- Run the following command from host2 as a test
# ssh host1 ls
You should be returned a directory listing of host1 on host2 without being prompted for a password.

The file system replication script (using rsync) and scheduling (using cron) will be posted in a future blog update.

Windows 7 - God Mode

I found this little gem of a trick.  By simply creating a folder you can access (what seems like) all of Windows 7's controls and settings.


I tried it and it works!


Give it a read/try.
http://news.cnet.com/8301-13860_3-10423985-56.html

Managing and Archiving Log Files Using Find and Tar

You could probably write a script and put this in the cron scheduler...  Here is the quick and dirty manual way to do it.

I wanted a way to archive everything in a particular directory and remove files (and only files) in that directory.  I was running out of space on one of my filesystems and I knew "log" files were eating up most of my space.  I still had plenty of space on my /tmp filesystem.  I wanted to get rid of files older than 15 days after I archived them so here is what I did.

I created a place for my archive files.
# mkdir -p /tmp/logs

I then changed directories to the place the log files lived.
# cd /(path to log files)/

I then, as a sanity check, ran this command to make sure it included the files I wanted and excluded those I didn't.
# find . -mtime +15 | xargs ls -l

After I validated this list was the files I wanted to archive, I ran the following command to tar and compress the files.
# find . -mtime +15 | xargs tar cvfz /tmp/logs/111609logs.tar.gz

This will tar up and compress everything older than 15 days.  I named my log file using a date stamp 15 days in the past. Now I want to remove files older than 15 days, but only files, not directories.  I used this command.
# find . -type f -mtime +15 | xargs rm -f

The "find" command will find everything so using the "-type f" option it will only find files.

I was able to free up about 3GB.  I hope this helps others out there.