Tuesday, January 12, 2010

Filesystem Replication Using rsync and SHH

There are many reasons you may need to replicate a file system.  My reason was for DR purposes.  In a previous post I set up Passwordless SSH sessions between two systems.  This is a requirement if you want to sync file systems on an automated schedule.  I looked on the web for a script that would do what I wanted and I could not find something that met my needs.  So I wrote the script below.  To give credit where credit is due, I borrowed from some ideas and code from Randal K. Michael, author of Mastering UNIX Shell Scripting.  I placed the script below in /usr/local/bin directory on the "source" node and called it fsrsync.bash.  This script will replicated a designated filesystem from a source node to two different nodes after confirming they are "alive".  I used rsync because after the initial sync, future replications are much faster since only update and/or changes are sent and not the entire filesystem.

Here is the script below.  Highlight the contents and select copy.
Note:  When you grab the test here and paste it, do a sanity check on the text to verify the formatting has not changed.
Edit/create /usr/local/bin/fsrsync.bash and paste in the contents.
# vi /usr/local/bin/fsrsync.bash

Script starts below.

#!/bin/bash
#
# SCRIPT: fsrsync.bash
# AUTHOR:
# DATE:
# REV:
#
# PURPOSE: This script is used to replicate the
# /somedir/test filesystem from Node A to Node B and C
#
# set -x # Uncomment to debug this script
#
# set -n # Uncomment to check the script.s syntax
#        # without any execution. Do not forget to
#        # recomment this line!
#
##############################################
# DEFINE FILES AND GLOBAL VARIABLES HERE
##############################################

# Define the target machines to copy data to.
# To specify more than one host enclose the
# hostnames in double quotes and put at least
# one space between each hostname
#
# EXAMPLE: MACHINE_LIST="fred yogi booboo"

MACHINE_LIST="nodeB nodeC"

# Capture the shell script file name

THIS_SCRIPT=$(basename $0)

# The FS_PATTERN variable defines the regular expression
# matching the filesystems we want to replicate with rsync.
# Example:  FS_PATTERN="/home"

FS_PATTERN="/somedir/test"

# Query the system for the hostname

THIS_HOST=$(hostname)

##############################################
# BEGINNING OF MAIN
##############################################

# Comfirm the nodes are alive and replicate
# the filesystems.
echo -e "\n####################################################\n"
echo -e "$THIS_SCRIPT started execution $(date)\n"
echo -e "Verifying the node is alive..."

for M in $MACHINE_LIST
do
    echo "Pinging $M..."
    ping -c1 $M >/dev/null 2>&1
    if (( $? != 0 ))
    then
        echo -e "ERROR: $M host is not pingable...cannot continue..."
        echo -e "...EXITING...\n"
        echo -e "####################################################"
        exit 2
    else
        echo -e "$M is alive... Starting rsync process!\n"
        echo -e "Replicating $FS_PATTERN/ from $THIS_HOST to $M\n"
        #The rsync command is all on one line although it doesn't appear so here.  This comment can be removed.
        rsync -aqz --delete -e ssh $FS_PATTERN/ root@$M:$FS_PATTERN
    fi
echo -e "$THIS_SCRIPT finished execution $(date)\n"
echo -e "####################################################"
done

###############################################
# END OF SCRIPT
###############################################

Make sure you make the file executable.
# chmod 754 /usr/local/bin/fsrsync.bash

You can manually run the file by simply running this command as a user with the appropriate rights.
# /usr/local/bin/fsrsync.bash

Do you want to schedule this script to replicate the file system every 5 minutes and log results?  Add the following entry to the crontab.
# crontab -e
0,5,10,15,20,25,30,35,40,45,50,55 * * * *     /usr/local/bin/fsrsync.bash 2>&1 >> /var/log/fsrsync.log


Do you want to rotate your log file (assuming you use logrotate)?  If so create a file in /etc/logrotate.d/ called fsrsync.
# vi /etc/logrotate.d/fsrsync

The contents of the file should look like this:
/var/log/fsrsync.log {
        weekly
        rotate 4
        nocompress
        missingok
}

I hope this helps someone out there.

PS:  If you want to test logrotate without having to wait a week you can do the following.
# /usr/sbin/logrotate -v /etc/logrotate.d/fsrsync
This will give you details on what it will do and rotate the log if needed.  If you want to force a log rotation, do the following.
# /usr/sbin/logrotate -f /etc/logrotate.d/fsrsync

No comments:

Post a Comment