Using "tar" to backup and exclude files in a script.

I needed to create a manual backup script that I could easily modify to backup specific directories and files while excluding others.  I was having a difficult time excluding files and directories without having to type out each exclusion using --exclude=somefile --exclude=some_other_file --exclude=somedirectory.  My "tar" command would have grown quite large and, in my opinion, is not easy to read. So, I decided to use an array to accomplish my goal.

#!/bin/bash # # SCRIPT: manual_backup.bash # AUTHOR: Bob # DATE: 3-1-2011 # R_DATE: 11-14-2012 # REV: 1.3.T (Valid are A, B, D, T, Q, and P) # (For Alpha, Beta, Dev, Test, QA, and Production) # # PLATFORM: Not platform dependent # # PURPOSE: Backup modified configuration files # # # set -n # Uncomment to check script syntax, without execution. # # NOTE: Do not forget to put the # comment back in or # # the shell script will never execute! # set -x # Uncomment to debug this shell script # ########################################################## # DEFINE FILES AND VARIABLES HERE ########################################################## SCRIPTNAME=$(basename $0 .bash) # List files or directories you want backed up FILELIST=" /etc/hosts /etc/fstab /var/log/*" # List files or directories you want excluded from the backup. # This is an array. EXCLUDELIST=(--exclude=/var/log/messages) EXCLUDELIST+=(--exclude=/var/log/lastlog) EXCLUDELIST+=(--exclude=/var/log/mail) ########################################################## # DEFINE FUNCTIONS HERE ########################################################## function get_date_stamp { DATENOW=$(date +'%m-%d-%y') } function get_host_name { HOST=$(hostname) } ########################################################## # BEGINNING OF MAIN ########################################################## get_date_stamp get_host_name tar -czvf /tmp/$HOST.$SCRIPTNAME.$DATENOW.tar.gz "${EXCLUDELIST[@]}" $FILELIST # End of script

Backing Up a django Web Server

I have a webserver I want to backup weekly and keep 4 weeks of backup history.
The webserver runs CentOS 5.4 and Django.
In the event of a server crash I'd like to be able to recover my website rather easily.  I figure it would be relatively easy to reinstall the OS and install the applications.  Then all I would need to do is restore the configuration files and the database.

Assumptions:
You have django installed and working
$DBDUMPDIR = where ever you want the Database Dump files to go
$BKUPDIR = where ever you want the Backup files to go.  Note:  This backup directory could be a remote server such as an NFS mount or SMB mount.  This is suggested so if this server crashes, your backups would be on another server.
Edit the scripts below to for "your" appropriate directories and file names.



Here are the four files I will need to backup:
*  /etc/httpd/conf/httpd.conf
*  /etc/httpd/conf/mysite.conf
*  /etc/www/django
*  $DBDUMPDIR/data.json

I created a script called dumpfiles.bash and put it in root's bin directory "/root/bin/dumpfiles.bash".
Make sure that permissions are right to be able to execute the file.
chmod 770 /root/bin/dumpfiles.bash

Here is the script:
----- start of script -----
#!/bin/bash
#
# SCRIPT: dumpfiles.bash
# AUTHOR: Bob
# DATE: 07/01/2010
# REV: 1.blah
#
# PURPOSE: This script is used to backup webserver specific data
#
# set -x # Uncomment to debug this script
#
# set -n # Uncomment to check the script.s syntax
#        # without any execution. Do not forget to
#        # recomment this line!
#
####################
# Define Variables #
####################
# Capture the shell script file name
THIS_SCRIPT=$(basename $0)
#
# Define the start time of the script
STARTTIME=`date +%T`
#
#Set Backup and DB Dump Directories
#Change the directories below to match your environment
BKUPDIR=/mnt/backup/webserver
DBDUMPDIR=/root
#
##################
# Increment Backups #
##################
rm -f $BKUPDIR/bkup4.tar.gz
mv -f $BKUPDIR/bkup3.tar.gz $BKUPDIR/bkup4.tar.gz
mv -f $BKUPDIR/bkup2.tar.gz $BKUPDIR/bkup3.tar.gz
mv -f $BKUPDIR/bkup1.tar.gz $BKUPDIR/bkup2.tar.gz
#
###################
# Dump Database #
###################
# export PYTHONPATH for dumpdata script
export PYTHONPATH='/var/www/django':'/var/www/django/apps'
#
#Change Directories to /var/www/django/mysite
cd /var/www/django/mysite
#
#Backup the DJango database and files to a flat file
python manage.py dumpdata > $DBDUMPDIR/data.json
#
#Get out of the /var/www/django/mysite directory
#Let us go home
cd /root
#
###############################
# Backup and compress important files #
###############################
tar cvfz $BKUPDIR/bkup1.tar.gz /etc/httpd/conf/httpd.conf /etc/httpd/conf.d/mysite.conf /var/www/django $DBDUMPDIR/data.json
----- end of script -----

I want to schedule a weekly backup so I'll use root's cron to do this.
Edit root's cron (assuming you are logged in as root).

Enter the following line to root's cron.
0 4 * * 1 /root/bin/dumpfiles.bash

This will run the script every Monday morning at 4:00 AM.
Now each time the script runs bkup1.tar.gz will be created.  If it already exists the old files will be incremented up to the number 4 giving you four weeks of backup files.

In the event of a disaster and you rebuilt a new server, you'd untar the backup files and copy the files back to their appropriate place and you the manage.py script to restore the django database.

Hope this helps someone out there.

Interactive Tape Backups using TAR and Linux

I sometimes want to run an on-demand backup of either a particular directory or file system.  I wrote an interactive script to do this and thought I would share it.
Assumptions:
1.  You have a tape device attached to your Computer or Server.
2.  You know what device your tape drive is.  Example: /dev/st0
3.  You have the mt-st package installed to manage the tape device.
4.  The tape you are using will be overwritten.
5.  You copy the contents of the script below in to a utility like notepad and check the contents.  Then copy from there in to a script called (whatever you want).
6.  Pay close attention to the command that starts like this:  TAPECHK=$(mt  It show up correctly in this post but if you cut and past the script in to notepad the lines do not match.  Edit it so it looks like it does here in the post.
7.  You make the script executable.
8.  I placed the script in /usr/local/sbin but you can put it where ever it make sense to you.

Here is the script:

#!/bin/bash
#
# SCRIPT: Interactive_2_tape.bash
# AUTHOR: Bob
# DATE: 02/24/2010
# REV:
#
# PURPOSE: This script is used to backup files
# from $SOURCE to $TAPEDEV
#
# set -x # Uncomment to debug this script
#
# set -n # Uncomment to check the script.s syntax
#        # without any execution. Do not forget to
#        # recomment this line!
#
####################
# Define Variables #
####################

# Capture the shell script file name
THIS_SCRIPT=$(basename $0)

# Define the start time of the script
STARTTIME=`date +%T`

# Define current directory to return to at end of script
CURRENTDIR=$PWD

# Ask for the source of the backup
echo "What directory or filesystem do you want to backup?"
echo "Type the directory in this format /dir1/dir2 followed by [ENTER]:"
read SOURCE
echo "Using $SOURCE as the source directory you want to backup."

# Ask for tape device
TAPEDEV="/dev/st0"
echo "I assume your tape device is $TAPEDEV"
read -p "Am I correct? yes/no: "
if [ "$REPLY" = "no" ]; then
     echo "What is your tape device? "
     read TAPEDEV
     echo "Using $TAPEDEV as your tape device."
else
     echo "Using $TAPEDEV as your tape device."
fi


################
# Main Section #
################

# Verify there is a tape in the
# drive and rewind the tape
TAPECHK=$(mt -f $TAPEDEV rewind 2>&1 1>/dev/null)
# If there is no tape tell me and exit out
#  If the mt command return any data then there is an error
#  Check the results of the mt command
if [ "$TAPECHK" != "" ]; then
     echo $TAPECHK
     echo "Check to see if there is a tape in the drive or if the device you entered is valid."
     exit
fi

# If we made it here, there is a tape in the drive and it has rewinded.
# Change to the directory to be backed up
cd $SOURCE
echo "Changing to the $SOURCE directory."

# Back up data
tar cvf $TAPEDEV .

# Rewind the tape again
mt -f $TAPEDEV rewind

# Change back to the directory from where you came
cd $CURRENTDIR
echo "Changing back to the directory you started from: $CURRENTDIR"
# Define the end time of this script
ENDTIME=`date +%T`

# Display the start and end time of this script
echo "$THIS_SCRIPT began at $STARTTIME and finished at $ENDTIME"

exit
#################
# End of Script #
#################

I hope this helps others out there trying to do the same thing.

Using tar to Copy a Large File into a Tight Space

I do a lot of work with virtual machine images.  I ran in to a situation where I wanted to copy a file called System.img from one server to another and even though I knew I had enough room to do it, I would get messages stating that there was not enough space.  What in the world was going on?  The file system I wanted to copy the file to was 33G in size.  The file was just under 33G in size.  I knew it should fit.  I knew this because the file system this file is coming from is also 33GB in size (same identical size).  When it was all said and done I should of had about 150M of free space according to the source.

I tried FTP, SCP, and various other mechanisms to copy the file from the one server to the other.  No joy.
So I copied the file to the destination server but to a different and larger file system.  That obviously was successful, but I still wanted it on my 33G file system.  So I tried copying the file locally from the larger file system to the 33G file system.  No joy again... I got a message after a few minutes stating there was not enough space, and the process errored out.

I found a solution!  Now, to be honest, I do not know why it works, but it does.

Assumptions: You are in the directory the System.img file is located.  The file resides on the same server you are copying to.  You have done the math and according to the calculator the file will fit on the destination file system.

Run this command with the appropriate path of your destination.

Note:  The following command is all on one line.
# tar cvf - System.img | ( cd /destination file system/destination folder/;tar xvf - )

Works like a charm.  I have used this little gem a dozen times in the past few months.

This should be obvious but...  Remember, you can not copy/place a file that is larger than the space available on the destination.  Hope this helps others out there.

Managing and Archiving Log Files Using Find and Tar

You could probably write a script and put this in the cron scheduler...  Here is the quick and dirty manual way to do it.

I wanted a way to archive everything in a particular directory and remove files (and only files) in that directory.  I was running out of space on one of my filesystems and I knew "log" files were eating up most of my space.  I still had plenty of space on my /tmp filesystem.  I wanted to get rid of files older than 15 days after I archived them so here is what I did.

I created a place for my archive files.
# mkdir -p /tmp/logs

I then changed directories to the place the log files lived.
# cd /(path to log files)/

I then, as a sanity check, ran this command to make sure it included the files I wanted and excluded those I didn't.
# find . -mtime +15 | xargs ls -l

After I validated this list was the files I wanted to archive, I ran the following command to tar and compress the files.
# find . -mtime +15 | xargs tar cvfz /tmp/logs/111609logs.tar.gz

This will tar up and compress everything older than 15 days.  I named my log file using a date stamp 15 days in the past. Now I want to remove files older than 15 days, but only files, not directories.  I used this command.
# find . -type f -mtime +15 | xargs rm -f

The "find" command will find everything so using the "-type f" option it will only find files.

I was able to free up about 3GB.  I hope this helps others out there.

Copying Directory Structures Between Linux or UNIX Systems

As I build or rebuild servers that need a particular directory structure, I wanted a fast way to replicate or reproduce the empty directory structure with the appropriate owners and permissions.  Historically, I would have used the tar command for this and only selected directories.  There is another, and in my opinion more efficient, way to do this using the cpio command.

I wanted to recreate the following two directory structures (including all sub-directories, which there are many of):
/u02/prn/app
/u02/prn/oradata

On the server you want to model the directory structure(s) after you would run the following commands.
# find /u02/prn/app -type d | cpio -ov >/tmp/appdirs.cpio
# find /u02/prn/oradata -type d | cpio -ov >/tmp/oradatadirs.cpio

Now using "scp" I can copy these files to the server I want to recreate the directory structure on.
# scp /tmp/*.cpio destination_server_here:/tmp/.

Now I will extract the archive which will create the directory structures I want (with the appropriate owners and permissions).  Note:  run this command from the "/" filesystem.  If you run it from /tmp, it will look for u02 in tmp.  It obviously doesn't live there.  So, do it from /
# cpio -iv </tmp/appdirs.cpio
# cpio -iv </tmp/oradatadirs.cpio

Done!  Not quite magic, but almost.

Assumptions:  You have the same users and groups created on the destination server as you do on the source server.

Hope this helps someone out there.