Managing Self Signed Certs and When They Expire

Have you ever walked in to work one morning to have people jump on you stating nothing is working?  You never got any pages from your monitoring system.  Everything appears to still be up.  Come to find out one of your self sign SSL certificates expired and needs updating.  Well, it has happened to some of the Windows Admins I have worked with.  It is difficult to track and manage self signed certificates.  Even if you have a good handle on it, and have automated reminders about certificates that will expire, it can slip by.  I came across a utility (script) called "ssl-cert-check.bash" found here:  http://prefetch.net/articles/checkcertificate.html  And, this does almost exactly what I want.  This will show you and even e-mail you if you have any servers with expired certificates.  I wanted to take it one simple step further...  I want a report e-mailed to my department regarding the status of all certificated whether they expired or not.  So here is what I did.  On a linux server I put the "ssl_cert-check.bash" file in my utility user's home "bin" directory.  I then created a file called ssldomains.txt (following the format described in the link above) containing the list of servers that have ssl certificates.  I then created a file called "cert_report.bash" in the same bin directory.  It's contents looks like this:
-----------------------------------------
#!/bin/bash
#
# SCRIPT: cert_report.bash
# AUTHOR: Bob
# DATE: Sep 24, 2012
# REV: 1.1.P (Valid are A, B, D, T, Q, and P)
# (For Alpha, Beta, Dev, Test, QA, and Production)
#
# PLATFORM: Not platform dependent
#
# PURPOSE: This script will call another script to report SSL Cert
# expiration days, save to a text file, and e-mail that text
# file.
#
# set -n # Uncomment to check script syntax, without execution.
# # NOTE: Do not forget to put the # comment back in or
# # the shell script will never execute!
# set -x # Uncomment to debug this shell script
#
##########################################################
# DEFINE FILES AND VARIABLES HERE
##########################################################
THIS_SCRIPT=$(basename )

##########################################################
# DEFINE FUNCTIONS HERE
##########################################################
　

##########################################################
# BEGINNING OF MAIN
##########################################################
~/bin/ssl_cert_check.bash -f ~/bin/ssldomains.txt > /tmp/SSL_Cert_Report.txt

echo "Attached is the bi-monthly SSL Certificate Expiration Report." | mutt -s "SSL Cert Check Report" -a /tmp/SSL_Cert_Report.txt mydepartment@mydomain.com

# End of script
----------------------------------------------

I then added an entry in cron for this to run twice a month.
0 8 1,15 * * ~/bin/cert_report.bash

This delivers a nicely formatted report of my servers and their certificate status.  Here is an example of the e-mailed report.

Host                                                    Status        Expires        Days
-------------------------------------------  ------------ ------------      ----
appsrv.mydomain.com:443                Valid       Jul 7 2014      643
lync1001.mydomain.com:443            Valid       Feb 13 2013   134
monitor01.mydomain.com:443          Valid       Feb 13 2013   134
idm.mydomain.com:443                     Valid       Oct 14 2021   3299
mail.mydomain.com:443                    Valid       Jul 10 2014   646
ntsrv1.mydomain.net:3471                 Valid       Aug 30 2013 332
ops1003.mydomain.net:3471              Valid       Feb 13 2013 134
web1003.mydomain.net:4443             Valid       Oct 14 2021 3299
web1004.mydomain.net:4443             Valid       Oct 14 2021 3299
report.mydomain.com:443                  Valid       Jul 7 2014    643
reportdev.mydomain.com:443            Valid       Jul 7 2014    643
srvavautil.mydomain.net:443              Valid      Mar 15 2019 2355
srvmail1002.mydomain.net:443          Valid      Jul 10 2014   646
srvmom01.mydomain.net:443             Valid      Mar 3 2028   5631
srvmoss001.mydomain.net:443           Valid      Aug 1 2013   303
srvocs2001.mydomain.net:443            Valid     Dec 29 2012   88
srvrecord2001.mydomain.net:443       Valid      Feb 13 2013   134
srvslshd1001.mydomain.net:443          Valid      Dec 14 2014   803
srvunibackup1001.mydomain.net:443  Valid      Sep 6 2014    704
srvvault2001.mydomain.net:443          Valid       Sep 6 2014   704
srvwas2001.mydomain.net:443            Valid       Jan 5 2020   2651
srvwcs2001.mydomain.net:443            Expired   Dec 16 2011 -291
app.mydomain.com:443                       Valid       Jul 9 2014     645
sftp.mydomain.com:443                       Valid       Feb 10 2015  861
solarwinds.mydomain.com:443            Valid       Feb 13 2013  134
support.mydomain.com:443                 Valid       Jul 9 2014     645
vcenter01.mydomain.com:443             Valid       Sep 27 2021   3282

Easy way to clean up old (unneeded) files and directories on Windows

As a System Administrator, Care and Feeding of your servers is a crucial task.  Imagine you have a directory containing daily transaction (substitute log, backup, or just about anything else) files.  You need to keep these files for a period of time for auditing purposes, but space is limited and you don't want or need to keep them indefinitely.  What is an Admin to do?  Set a quarterly task to do general clean up?  Yuck.

In Linux it is pretty easy to accomplish this task with the "find" command and its various switches.  How do you do it on a Windows server?  What I am about to show you works on Windows 2008 Servers.  It may not work on Windows 2003 as I believe the syntax for our magic command "forfiles" was a little different.

Below is are the commands you need in a batch file to delete files and/or folders older than "x" days.
Open up notepad or your favorite Windows plain text editor (I like Notepad++ btw).
Enter the following text between the "-----" but not including the "-----"

--------------------------------------------------------
@echo off
rem - The following line is a cya testing line.  It will display the files or
rem - folder that are older than "x" days but does not delete them.
rem - Where you see /d -x
rem - replace x with your criteria.  I will use 30 do anything older than 30 days.
rem - also change your path.  In this example, I want to clean up files in the
rem - E:\SharePointBackup directory older than 30 days.
rem - Uncomment the following "forfiles" line for testing,
rem - but make sure all other lines are commented out.
rem FORFILES /p "E:\SharePointBackup" /s /m *.* /d -30 /c "CMD /C ECHO @FILE"

rem - The following line will delete files older than "30" in given directory
rem - This will just delete files and not remove directories or anything else.
rem - Uncomment the following "forfiles" line to delete files older than 30 days, but make sure
rem - all other lines are commented out.
rem FORFILES /p "E:\SharePointBackup" /s /m *.* /d -30 /c "CMD /C del /Q @FILE"

rem - This will look for directories in a given path older than 30 days
rem - and delete it and all of its contents.  All files and subdirectories in directories
rem - that are older than 30 days.
rem - Uncomment the following "forfiles" line to delete directories older than 30 days, but make sure
rem - all other lines are commented out.
rem FORFILES /p "E:\SharePointBackup" /d -30 /c "CMD /C if @isdir==TRUE RMDIR /S /Q @path"
----------------------------------------------------------------

Save this file on your server as whatever you want with a ".bat" extension.  Example:  delete_older_than_30.bat

Make sure you "test" this on data you wouldn't mind losing before putting it in production.  Once you have tested and have verified it will do what you want it to do, you can schedule a daily (or whatever you'd like) task using the scheduler calling this batch file.

I hope this helps some of you out there.

Thermaltake - Dual Bay Docking Station for SATA drives

I bought one of these little gems at Best Buy recently.  I love it.

I use it on my Linux and Windows Servers/PCs to copy, image, and access SATA drives.  It works with 3.5 inch and 2.5 inch SATA drives.  I bought mine on-sale for $59.00.  It can access two drives at a time and can be connected to your computer with the supplied USB or eSATA cable.  This comes in very useful in my line of work.  It sure beats an external inclosure.
You can read more about it here:  http://www.thermaltakeusa.com/Product.aspx?C=1346&ID=1895

Backing Up your Laptop or PC - Easy!

I was using a program called G4L to image my laptop or pc but I have found a much easier way.  It works with Linux or Windows, no installation is needed, allows you to resize partitions, backup systems, access files, and recover lost data.  It is a live CD called "Redo".  Check it out here http://redobackup.org/.  It rocks!

Taken from the site:

Redo Backup and Recovery is so simple that anyone can use it. It is the easiest, most complete disaster recovery solution available. It allows bare-metal restore. Bare metal restore means that even if your hard drive melts or gets completely erased by a virus, you can have a completely-functional system back up and running in as little as 10 minutes.


All your documents and settings will be restored to the exact same state they were in when the last snapshot was taken. Redo Backup and Recovery is a live CD, so it does not matter if you use Windows or Linux. You can use the same tool to backup and restore every machine. And because it is open source released under the GPL, it is completely free for personal and commercial use.

Active Directory Account Lockout Notification

Scenario: You want to be notified by e-mail if one of your Active Directory users locks their account by entering their password incorrectly too many times. This allows you to be proactive in identifying locked accounts and contacting said user to see if there is an issue they need assistance with.

Assumptions: You host your mail server and it will relay mail internally for a domain user. You have a group policy enabled so that an Active Directory account will “lock” after “x” number of incorrect login attempts.

What is needed:

* Blat - a Win32 command line utility that sends eMail using SMTP - http://www.blat.net/
* unlock.exe - Command line Active Directory unlock tool. Will show you currently locked accounts and unlock accounts. One simple command to unlock all locked accounts in a domain. - http://www.joeware.net/freetools/tools/unlock/index.htm

Details:

On one of your Active Directory servers you will want to create a directory structure like this:

c:\utils\script
c:\utils\temp
c:\utils\blat
c:\utils\unlock

Place unlock.exe in the c:\utils\unlock folder.
Place blat.exe, blat.dll, blat.lib in the c:\utils\blat folder.

In the script folder, create a file named lockout_check.bat.
The contents of this batch file should look something like this:

@echo off
:: Run the unlock utility to determine if there are locked accounts
:: Use the -view option to only view locked accounts
:: and do not unlock them
:: Send the output to a temp file
c:\utils\unlock\unlock . * -view > c:\utils\temp\lockstatus.txt

:: Read contents of lockstatus.txt
:: If contents of file contains No objects found. then goto END
:: Otherwise send someone an e-mail that there are locked accounts
type "c:\utils\temp\lockstatus.txt" | find "No objects found." > nul
if errorlevel 1 goto LOCKS
goto END

:LOCKS
:: Edit the to and from addresses and put in your mail server
:: The next next line is one continuous line.
c:\utils\blat\blat.exe c:\utils\temp\lockstatus.txt -to helpdesk@mydomain.com -subject "Locked Active Directory Accounts!" -f helpdesk@mydomain.com -server mail.mydomain.com

:END
exit

You can now (on another computer) log in to an account incorrectly “x” number of times to lock the account.

You can now run this command to verify the account is locked:
c:\utils\unlock\unlock . * -view

Assuming you see a “locked” account you can run the script you created to see if you get an e-mail stating there was a locked account.

Assuming that works correctly, you can schedule a task to run this script as often as you deem necessary. I have it set for every 10 minutes.

Hot Copy File System Snapshot - Installation, Setup and Use

What is HotCopy

(Description taken from the R1Soft website)

R1Soft Hot Copy (hcp) is the answer to taking online point-in-time disk and volume snapshots in Linux.  Use the hcp command line utility to take an instant snapshot of any mounted file system on almost any block device.

·        Add point-in-time open file backups to your existing backup scripts for free e.g. tar and rsync

·        Check your disk for errors with fsck without rebooting and without unmounting your file system!

·        Test scripts and programs in an instant snapshot of your live environment before you use them on real data

·        Keep instantly recoverable snapshots available by taking periodic snapshots via cron

Installation

This document will assume the Operating System this tool will be installed to is a 64-bit OS.  At the time this document was created HotCopy was at version 3.6.1.

There are two components needed for install.

1.      r1soft-hotcopy-3.6.1.x86_64.rpm

2.      r1soft-setup-3.6.1.x86_64.rpm

The first package is the HotCopy utility.  The second package is the setup utility to get and install the kernel driver.

r1soft-hotcopy-3.6.1.x86_64.rpm

To install this package you need to be root.  Copy the packages to the server’s /tmp directory.

# cd /tmp

# rpm -ivh r1soft-hotcopy-3.6.1.x86_64.rpm

r1soft-setup-3.6.1.x86_64.rpm

To install this package you need to be root.  Copy the packages to the server’s /tmp directory.

# cd /tmp

# rpm -ivh r1soft-setup-3.6.1.x86_64.rpm

Setup

Now that the packages are installed you need to download and install the Kernel driver.  Your server will need access to the Internet to perform the following command.

As a root user run the following command and you should see the following output.

# /usr/sbin/hcp-setup --get-module

Checking for binary module

Waiting

Complete.

Saving kernel module to ‘/lib/modules/r1soft/hcpdriver-cki-2.6.18-194.17.4.el5.ko’

#

Usage

Creating a Snapshot

Verify the filesystem you want to snapshot.

# df -h

Filesystem                                                        Size        Used      Avail      Use%     Mounted on

/dev/mapper/VolGroup00-LogVol00         7.8G       3.4G       4.0G       47%       /

/dev/sda1                                                         99M       19M       75M       21%       /boot

Tmpfs                                                                1006M  0             1006M  0%          /dev/shm

Create the snapshot

Verify the Snapshot Exists

Example of  “foo”

In this example we’ll “accidentally” delete a file in the /tmp directory but be able to retrieve it from the snapshot.

Good thing I took a snapshot before I began messing around.  I will now retrieve the file from the snapshot.

We’re back!  File has been retrieved.  Disaster averted.

Removing a Snapshot

Additional Cool Things

You can have the Snapshot created and mounted to a completely different filesystem.

Why would you want to do that?

Backups of course.  Let’s assume we have a system that can have very little downtime and cannot be backed up while its special application is running.  Backups take 2 or 3 hours and that kind of downtime in not acceptable.  Here is what we would do (assuming there is a separate disk to snap to)

1.      Bring down the application

2.      Create a Snap of the filesystem to another disk.  This process takes a second or two.

3.      Bring the application up

4.      Take a backup of the Snap instead of the live filesystem.

Your application would be down only for a matter of seconds and backing up the Snap would not affect disk I/O on production filesystem.

Example:

Verify the New Disk is Seen

[root@centos55-test /]# fdisk -lu /dev/sdb

Disk /dev/sdb: 12.8 GB, 12884901888 bytes

255 heads, 63 sectors/track, 1566 cylinders, total 25165824 sectors

Units = sectors of 1 * 512 = 512 bytes

   Device Boot      Start         End      Blocks   Id  System

Partition the Disk

[root@centos55-test /]# fdisk /dev/sdb

The number of cylinders for this disk is set to 1566.

There is nothing wrong with that, but this is larger than 1024,

and could in certain setups cause problems with:

1) software that runs at boot time (e.g., old versions of LILO)

2) booting and partitioning software from other OSs

   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): u

Changing display/entry units to sectors

Command (m for help): n

Command action

   e   extended

   p   primary partition (1-4)

p

Partition number (1-4): 1

First sector (63-25165823, default 63):

Using default value 63

Last sector or +size or +sizeM or +sizeK (63-25165823, default 25165823):

Using default value 25165823

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

Create a Linux FileSystem on the New Partition

[root@centos55-test /]# mkfs.ext3 /dev/sdb1

mke2fs 1.39 (29-May-2006)

Filesystem label=

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

1572864 inodes, 3145720 blocks

157286 blocks (5.00%) reserved for the super user

First data block=0

Maximum filesystem blocks=3221225472

96 block groups

32768 blocks per group, 32768 fragments per group

16384 inodes per group

Superblock backups stored on blocks:

        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Writing inode tables: done                           

Creating journal (32768 blocks): done

Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 26 mounts or

180 days, whichever comes first.  Use tune2fs -c or -i to override.

Create a Directory to Mount the New Filesystem to

[root@centos55-test /]# mkdir -p /mnt/bkup

Mount it

[root@centos55-test /]# mount /dev/sdb1 /mnt/bkup

Verify it is Mounted

[root@centos55-test /]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/VolGroup00-LogVol00

                      7.8G  3.4G  4.1G  46% /

/dev/sda1              99M   19M   75M  21% /boot

tmpfs                1006M     0 1006M   0% /dev/shm

/dev/sdb1              12G  159M   12G   2% /mnt/bkup

Stop Application or Database

Do what is needed to stop the application or database

Create a Snapshot of the FileSystem we want to backup

-m   and  -c

This takes all “additional” I/O off of the production filesystem

[root@centos55-test /]# hcp -m /mnt/bkup -c /dev/sdb1 /dev/mapper/VolGroup00-LogVol00

R1Soft Hot Copy    3.6.1 build 10439 (http://www.r1soft.com)

Documentation      http://wiki.r1soft.com

Forums             http://forum.r1soft.com

Thank you for using Hot Copy!

R1Soft makes the only Continuous Data Protection software for Linux.

Starting Hot Copy: /dev/mapper/VolGroup00-LogVol00.

Changed blocks stored: /dev/sdb1

Snapshot completed: 0.001 seconds

File system frozen: 0.128 seconds

Hot Copy created: Fri Oct 15:12:05 CDT 2010

Creating Hot Copy snapshot device: /dev/hcp1, Please Wait...

Hot Copy created at: /dev/hcp1

Mounting /dev/hcp1 read-write

Hot Copy mounted at: /mnt/bkup

[root@centos55-test /]#

Start Application or Database again

Do what is need to start the DB or APP again

Verify the Snap has been made and mounted.

[root@centos55-test /]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/VolGroup00-LogVol00

                      7.8G  3.4G  4.1G  46% /

/dev/sda1              99M   19M   75M  21% /boot

tmpfs                1006M     0 1006M   0% /dev/shm

/dev/sdb1             7.8G  3.4G  4.1G  46% /mnt/bkup

/dev/hcp1             7.8G  3.4G  4.1G  46% /mnt/bkup

[root@centos55-test /]#

Run the backup on the Snap

Run Backup Against /mnt/bkup

Once the backup is complete you can remove the Snap

[root@centos55-test /]# hcp -r /dev/hcp1

R1Soft Hot Copy    3.6.1 build 10439 (http://www.r1soft.com)

Documentation      http://wiki.r1soft.com

Forums             http://forum.r1soft.com

Thank you for using Hot Copy!

R1Soft makes the only Continuous Data Protection software for Linux.

Hot Copy Session has successfully been stopped.

All active Hot Copy sessions have been stopped. It is now safe to restart the R1Soft Backup Agent.

[root@centos55-test /]#

Backing Up a django Web Server

I have a webserver I want to backup weekly and keep 4 weeks of backup history.
The webserver runs CentOS 5.4 and Django.
In the event of a server crash I'd like to be able to recover my website rather easily.  I figure it would be relatively easy to reinstall the OS and install the applications.  Then all I would need to do is restore the configuration files and the database.

Assumptions:
You have django installed and working
$DBDUMPDIR = where ever you want the Database Dump files to go
$BKUPDIR = where ever you want the Backup files to go.  Note:  This backup directory could be a remote server such as an NFS mount or SMB mount.  This is suggested so if this server crashes, your backups would be on another server.
Edit the scripts below to for "your" appropriate directories and file names.



Here are the four files I will need to backup:
*  /etc/httpd/conf/httpd.conf
*  /etc/httpd/conf/mysite.conf
*  /etc/www/django
*  $DBDUMPDIR/data.json

I created a script called dumpfiles.bash and put it in root's bin directory "/root/bin/dumpfiles.bash".
Make sure that permissions are right to be able to execute the file.
chmod 770 /root/bin/dumpfiles.bash

Here is the script:
----- start of script -----
#!/bin/bash
#
# SCRIPT: dumpfiles.bash
# AUTHOR: Bob
# DATE: 07/01/2010
# REV: 1.blah
#
# PURPOSE: This script is used to backup webserver specific data
#
# set -x # Uncomment to debug this script
#
# set -n # Uncomment to check the script.s syntax
#        # without any execution. Do not forget to
#        # recomment this line!
#
####################
# Define Variables #
####################
# Capture the shell script file name
THIS_SCRIPT=$(basename $0)
#
# Define the start time of the script
STARTTIME=`date +%T`
#
#Set Backup and DB Dump Directories
#Change the directories below to match your environment
BKUPDIR=/mnt/backup/webserver
DBDUMPDIR=/root
#
##################
# Increment Backups #
##################
rm -f $BKUPDIR/bkup4.tar.gz
mv -f $BKUPDIR/bkup3.tar.gz $BKUPDIR/bkup4.tar.gz
mv -f $BKUPDIR/bkup2.tar.gz $BKUPDIR/bkup3.tar.gz
mv -f $BKUPDIR/bkup1.tar.gz $BKUPDIR/bkup2.tar.gz
#
###################
# Dump Database #
###################
# export PYTHONPATH for dumpdata script
export PYTHONPATH='/var/www/django':'/var/www/django/apps'
#
#Change Directories to /var/www/django/mysite
cd /var/www/django/mysite
#
#Backup the DJango database and files to a flat file
python manage.py dumpdata > $DBDUMPDIR/data.json
#
#Get out of the /var/www/django/mysite directory
#Let us go home
cd /root
#
###############################
# Backup and compress important files #
###############################
tar cvfz $BKUPDIR/bkup1.tar.gz /etc/httpd/conf/httpd.conf /etc/httpd/conf.d/mysite.conf /var/www/django $DBDUMPDIR/data.json
----- end of script -----

I want to schedule a weekly backup so I'll use root's cron to do this.
Edit root's cron (assuming you are logged in as root).

Enter the following line to root's cron.
0 4 * * 1 /root/bin/dumpfiles.bash

This will run the script every Monday morning at 4:00 AM.
Now each time the script runs bkup1.tar.gz will be created.  If it already exists the old files will be incremented up to the number 4 giving you four weeks of backup files.

In the event of a disaster and you rebuilt a new server, you'd untar the backup files and copy the files back to their appropriate place and you the manage.py script to restore the django database.

Hope this helps someone out there.