Tuesday, December 1, 2009

Managing and Archiving Log Files Using Find and Tar

You could probably write a script and put this in the cron scheduler...  Here is the quick and dirty manual way to do it.

I wanted a way to archive everything in a particular directory and remove files (and only files) in that directory.  I was running out of space on one of my filesystems and I knew "log" files were eating up most of my space.  I still had plenty of space on my /tmp filesystem.  I wanted to get rid of files older than 15 days after I archived them so here is what I did.

I created a place for my archive files.
# mkdir -p /tmp/logs

I then changed directories to the place the log files lived.
# cd /(path to log files)/

I then, as a sanity check, ran this command to make sure it included the files I wanted and excluded those I didn't.
# find . -mtime +15 | xargs ls -l

After I validated this list was the files I wanted to archive, I ran the following command to tar and compress the files.
# find . -mtime +15 | xargs tar cvfz /tmp/logs/111609logs.tar.gz

This will tar up and compress everything older than 15 days.  I named my log file using a date stamp 15 days in the past. Now I want to remove files older than 15 days, but only files, not directories.  I used this command.
# find . -type f -mtime +15 | xargs rm -f

The "find" command will find everything so using the "-type f" option it will only find files.

I was able to free up about 3GB.  I hope this helps others out there.