Filesystem Replication Using rsync and SHH

There are many reasons you may need to replicate a file system.  My reason was for DR purposes.  In a previous post I set up Passwordless SSH sessions between two systems.  This is a requirement if you want to sync file systems on an automated schedule.  I looked on the web for a script that would do what I wanted and I could not find something that met my needs.  So I wrote the script below.  To give credit where credit is due, I borrowed from some ideas and code from Randal K. Michael, author of Mastering UNIX Shell Scripting.  I placed the script below in /usr/local/bin directory on the "source" node and called it fsrsync.bash.  This script will replicated a designated filesystem from a source node to two different nodes after confirming they are "alive".  I used rsync because after the initial sync, future replications are much faster since only update and/or changes are sent and not the entire filesystem.

Here is the script below.  Highlight the contents and select copy.
Note:  When you grab the test here and paste it, do a sanity check on the text to verify the formatting has not changed.
Edit/create /usr/local/bin/fsrsync.bash and paste in the contents.
# vi /usr/local/bin/fsrsync.bash

Script starts below.

#!/bin/bash
#
# SCRIPT: fsrsync.bash
# AUTHOR:
# DATE:
# REV:
#
# PURPOSE: This script is used to replicate the
# /somedir/test filesystem from Node A to Node B and C
#
# set -x # Uncomment to debug this script
#
# set -n # Uncomment to check the script.s syntax
#        # without any execution. Do not forget to
#        # recomment this line!
#
##############################################
# DEFINE FILES AND GLOBAL VARIABLES HERE
##############################################

# Define the target machines to copy data to.
# To specify more than one host enclose the
# hostnames in double quotes and put at least
# one space between each hostname
#
# EXAMPLE: MACHINE_LIST="fred yogi booboo"

MACHINE_LIST="nodeB nodeC"

# Capture the shell script file name

THIS_SCRIPT=$(basename $0)

# The FS_PATTERN variable defines the regular expression
# matching the filesystems we want to replicate with rsync.
# Example:  FS_PATTERN="/home"

FS_PATTERN="/somedir/test"

# Query the system for the hostname

THIS_HOST=$(hostname)

##############################################
# BEGINNING OF MAIN
##############################################

# Comfirm the nodes are alive and replicate
# the filesystems.
echo -e "\n####################################################\n"
echo -e "$THIS_SCRIPT started execution $(date)\n"
echo -e "Verifying the node is alive..."

for M in $MACHINE_LIST
do
    echo "Pinging $M..."
    ping -c1 $M >/dev/null 2>&1
    if (( $? != 0 ))
    then
        echo -e "ERROR: $M host is not pingable...cannot continue..."
        echo -e "...EXITING...\n"
        echo -e "####################################################"
        exit 2
    else
        echo -e "$M is alive... Starting rsync process!\n"
        echo -e "Replicating $FS_PATTERN/ from $THIS_HOST to $M\n"
        #The rsync command is all on one line although it doesn't appear so here.  This comment can be removed.
        rsync -aqz --delete -e ssh $FS_PATTERN/ root@$M:$FS_PATTERN
    fi
echo -e "$THIS_SCRIPT finished execution $(date)\n"
echo -e "####################################################"
done

###############################################
# END OF SCRIPT
###############################################

Make sure you make the file executable.
# chmod 754 /usr/local/bin/fsrsync.bash

You can manually run the file by simply running this command as a user with the appropriate rights.
# /usr/local/bin/fsrsync.bash

Do you want to schedule this script to replicate the file system every 5 minutes and log results?  Add the following entry to the crontab.
# crontab -e
0,5,10,15,20,25,30,35,40,45,50,55 * * * *     /usr/local/bin/fsrsync.bash 2>&1 >> /var/log/fsrsync.log


Do you want to rotate your log file (assuming you use logrotate)?  If so create a file in /etc/logrotate.d/ called fsrsync.
# vi /etc/logrotate.d/fsrsync

The contents of the file should look like this:
/var/log/fsrsync.log {
        weekly
        rotate 4
        nocompress
        missingok
}

I hope this helps someone out there.

PS:  If you want to test logrotate without having to wait a week you can do the following.
# /usr/sbin/logrotate -v /etc/logrotate.d/fsrsync
This will give you details on what it will do and rotate the log if needed.  If you want to force a log rotation, do the following.
# /usr/sbin/logrotate -f /etc/logrotate.d/fsrsync

Passwordless SSH Setup

Disclaimer:  I would not suggest doing this as root.  I am only using root as an example.

I needed to replicate a file system from a production server to a DR server.  I wanted to script and schedule this so there was no intervention needed from an end user.  The first step was to setup passwordless SSH between the source and destination.  I found a few tutorials out on the web but they were not as clear as would have liked.  So, I documented my process and thought I would share it with you.

We will assume we have two hosts, host1 and host2.  For my purposes, I want host2 to be able to run commands through ssh to host1 without being prompted for a password.  In this example, we’ll assume the user running these commands is “root”.

On host2 you will need to do the following.
 - Log in as root to host2
 - Verify the following directory exists
/root/.ssh
 - You can do this by issuing the following commands
# cd
# ls -al | grep .ssh
 - If in the output returned you see .ssh, then the directory exists.  If you are returned to a command prompt without seeing .ssh you will need to create the directory.
 - If you need to create the directory issue the following commands.
# mkdir -p /root/.ssh
# chmod 700 /root/.ssh
- Now run the following command
# ssh-keygen -t rsa
*** You will see output similar to below ***
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
 - You can hit enter at the above prompt and accept the defaults for the two prompts below
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.

On host1 you will need to do the following.
- Log in as root to host1
 - Verify the following directory exists
/root/.ssh
 - You can do this by issuing the following commands
# cd
# ls -al | grep .ssh
 - If in the output returned you see .ssh, then the directory exists.  If you are returned to a command prompt without seeing .ssh you will need to create the directory.
 - If you need to create the directory issue the following commands.
# mkdir -p /root/.ssh
# chmod 700 /root/.ssh
- Now copy host2’s id_rsa.pub key to host1 (assuming you are still on host1) renaming it to host2.pub
# scp host2:/root/.ssh/id_rsa.pub /root/.ssh/host2.pub
***  Note:  my version(s) require authorized_keys2, your file may need to be named authorized_keys
- Now copy /root/.ssh/host2.pub to /root/.ssh/authorized_keys2
# cp /root/.ssh/host2.pub /root/.ssh/authorized_keys2

Now from host2 you should be able to ssh to host1 without being prompted for a password.
- Run the following command from host2 as a test
# ssh host1 ls
You should be returned a directory listing of host1 on host2 without being prompted for a password.

The file system replication script (using rsync) and scheduling (using cron) will be posted in a future blog update.