So I am working on my first attempt at python... so far I like it but not that much. The white space thing is actually kinda annoying but whatever. Anyway the idea here was to write a script that would take in a parameter (process name) and track its cpu usage. My main reason for this is mono has been chewing up cpu on a regular basis and I don't feel like logging in to kill the process all the time. Anyway this is just a first pass so I'll probably edit it as I make modifications.

 

'''
Created on Mar 30, 2011

@author: Keith
'''
import os
import psutil
import pickle
import signal
import smtplib

from_addr = 'root'
to_addrs = 'root'
watchFileFormat = 'watchProcess'

def getProcessIds(name):
    processes = psutil.get_pid_list()
    pids = []
    for pid in processes:
        p = psutil.Process(pid)
        if p.name == name:
            pids.append(pid)
    return pids


def checkThreshold(percentage, threshold):
    for utilization in percentage:
        if utilization < threshold:
            return False
    return True

def cleanWatch(pids):
    for file in os.listdir('.'):
        pid = file.split('.')
        if file.startswith(watchFileFormat) and pid[1].isdigit() and int(pid[1]) not in pids :
            print(file)
            os.remove(str.format(watchFileFormat + ".{0}", pid[1]))

def watchProcess(name, threshold = 50):
    util = []
    pids = getProcessIds(name)
    cleanWatch(pids)
    for pid in pids:
        try:
            p = psutil.Process(pid)
            file = str.format(watchFileFormat + ".{0}", str(pid))
            if os.path.exists(file):
                try:
                    util = pickle.load(open(file, "r"))
                except:
                    print(str.format("Unable to file for PID: {0}", str(pid)))
            util.append(p.get_cpu_percent(10.0))
            if (len(util) > 5):
                util.pop(0)
            pickle.dump(util, open(file, "wb"))
            if (len(util) == 5 and checkThreshold(util, threshold)):
                smtpObj = smtplib.SMTP('localhost')
                smtpObj.sendmail(from_addr, to_addrs, str.format("Process killed {0} with pid: {1}", name, pid))
                os.kill(pid, signal.SIGTERM)
        except:
            pass

if __name__ == '__main__':
    import sys
    if (len(sys.argv) > 2):
        watchProcess(sys.argv[1], sys.argv[2])
    else:
        watchProcess(sys.argv[1])
 

I think this script plus the crontab below should keep mono in check:

 

*   *    *   *   *   /usr/bin/python /root/watchProcess.py mono > /dev/null 2>&1

 

Updated: 4/3/2011 had issues with cleanup due to not casting the pid from the file name as an int also moved the isdigit so as to not catch files that end with non-numeric strings. In my case watchProcess.py ;P

The other day I noticed my custom made router's time was off by an hour when i ran the following command.

# date
Thu Mar  30 18:20:33 CST 2009

Immeadiately I realized I had the wrong timezone set as I generally use the Chicago timezone which displays as CDT. To fix this I copied the correct timezone into local time with the following command.

# cp /usr/share/zoneinfo/America/Chicago /etc/localtime

Now when I run the date command it displays the correct date time.


Ok so another password thing I just realized Subversion uses apache style hashed passwords. In any case they can be generate by the following command htpasswd -nbm <username> <password>.


I always forget how to generate passwords and apparently I am not alone as I have found many blogs that have perl and python scripts used just to perform this action. Anyway after about half an hour of searching I kinda remembered openssl can generate passwords. From this point it was just finding the right options so here it is openssl passwd -1 <mypassword>

One of the first things I realized I needed when I started using PostgreSQL was a way to automate backups. Originally I created a script that backed up the databases I specified but later I realized I didn't want to have to remember to add new databases to the list. Using Google I found a generic script, after some custom tweaks for FreeBSD I had the script I wanted. 

#!/bin/sh
# Location of binaries
bin="/usr/local/bin"
# Location of the backup logfile.
logfile="/data/backup/postgres/postgres.log"
# Location to place backups.
backup_dir="/data/backup/postgres"
username="pgsql"
database="template1"
touch $logfile
timeslot=`date +%H-%M`
databases=`$bin/psql -h localhost -d $database -U $username -q -c "\l" | sed -n 4,/\eof/p | grep -v rows\) | awk {'print $1'}`

for i in $databases; do
        echo "Backup and Vacuum complete at $timeinfo for time slot $timeslot on database: $i " >> $logfile
        $bin/vacuumdb -z -h localhost -U $username $i >/dev/null 2>&1
        j="60"
        while [ $j -ge 0 ]
        do
                temp0=`expr $j - 1`
                temp1=$j
                j=`expr $j - 1`
                if [ $temp0 -lt 0 ]
                then
                        suffix0=""
                        suffix1=".0"
                else
                        suffix0=".$temp0"
                        suffix1=".$temp1"
                fi
                if [ -f $backup_dir/postgresql-$i-database.gz$suffix0 ]
                then
                        echo "Renaming postgresql-$i-database.gz$suffix0 to postgresql-$i-database.gz$suffix1"
                        mv $backup_dir/postgresql-$i-database.gz$suffix0 $backup_dir/postgresql-$i-database.gz$suffix1
                fi
        done
        if [ $i != 'template0' ]
        then
                $bin/pg_dump -U $username $i -h 127.0.0.1 | gzip > "$backup_dir/postgresql-$i-database.gz"
        fi
done

This script creates 60 backups and when used in conjunction with a cron job you can make that weekly, daily, hourly, etc. I use this to create a daily backup. Now for some people 60 is probably more than they want to store. But that can be changed by changing the start value of the variable j. To make this work for your postgresql database you will most likely want to change the backup_dir and logfile variables. Once completed add a cron job and your done.

0 0 * * * /root/dbBackup.sh >> /dev/null 2>&1

This cron job will run daily at 12:00am and automatically backup all databases with the exception of the template0 database as I have yet to find a way to back this schema up.

 

 

By default PostgreSQL installed configuration only allows connections from the localhost. This is a good thing because the average user most likely has a LAMP style setup where all services are contained within the same server and this will limit its vulnerability to the outside.

The install I am using will need to support multiple connections from many different locations simultaneously. For example we have a couple ASP.NET applications that will access different databases on the server as well as a few Network appliances, for example snort, that will also access databases on the server.

First lets setup the server to start listening on the external interfaces. To do this we will need to access the posgres data directory. As I explained before I moved my data directory from the FreeBSD default location (/usr/local/pgsql/data) to /data/pgsql. In the postgres data directory you will need to edit the file postgresql.conf. Find the setting for listen_addresses and change it from localhost to * or a specific IP address or list of ip addresses.

listen_addresses = '*'                  # what IP address(es) to listen on;
                                        # comma-separated list of addresses;
                                        # defaults to 'localhost', '*' = all
                                        # (change requires restart)
#port = 5432                            # (change requires restart)
max_connections = 80                    # (change requires restart)

You will also notice that I changed the max_connections from 40 to 80, you could also change the port that postgres listens on if you chose but for this I will leave the default.

Next we need to edit the pg_hba.conf file to allow connections from an IP space other than the localhost (127.0.0.1). There may be more secure ways to setup this but between the firewall, ip restrictions, user accounts and md5 I feel pretty confident in my setup (until someone tells me otherwise). Anyway say I have a .NET application on my windows server(10.3.0.100), a database named testdb and the user i will be connection to the database with is named testuser if I wanted to set postgresql to allow this connection I would add the following line to the pg_hba.conf.

host    testdb       testuser 10.3.0.100/32   md5

Once I have made my changes to the configuration files I need to restart postgresql. To do this I run /usr/local/etc/rc.d/postgresql restart and in a few seconds I should be able to test the connection in my .NET application.

I my previous post I talked about rebuilding the kernel on a AMD64 machine with dual processors, what I didn't know at the time is the default installed kernel enables SMP but the GENERIC kernel profile does not. Enabling SMP in the FreeBSD kernel is quite easy all thats necessary is to add the SMP option prior to rebuilding the kernel.

options         SMP                     # Enable SMP kernel

Once this is done just follow the same steps I have described in previous posts. After you have completed the kernel install you can verify that the operating system sees all the processors by running one of the following commands.

[root@test] [/root]> cat /var/log/messages | grep cpu
Jan 30 10:33:11 test kernel: cpu0: <ACPI CPU> on acpi0
Jan 31 07:48:42 test kernel: cpu0 (BSP): APIC ID:  0
Jan 31 07:48:42 test kernel: cpu1 (AP): APIC ID:  1
Jan 31 07:48:42 test kernel: cpu0: <ACPI CPU> on acpi0
Jan 31 07:48:42 test kernel: cpu1: <ACPI CPU> on acpi0

[root@test] [/root]> ps -aux | grep cpu
root      10 99.0  0.0     0    16  ??  RL    7:48AM 783:07.85 [idle: cpu1]
root      11 99.0  0.0     0    16  ??  RL    7:48AM 781:42.93 [idle: cpu0]
root      42  0.0  0.0     0    16  ??  DL    7:48AM   0:00.60 [schedcpu]
kkikta 16847  0.0  0.1  5920  1256  p0  S+    8:52PM   0:00.00 grep cpu

[root@test] [/root]> top -n
last pid: 16900;  load averages:  0.00,  0.00,  0.00  up 0+13:05:43    20:53:44
27 processes:  1 running, 26 sleeping

Mem: 12M Active, 13M Inact, 31M Wired, 20M Buf, 1912M Free
Swap: 512M Total, 512M Free


  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
  635 pgsql       1   8    0 53580K  5776K nanslp 0   0:02  0.00% postgres
  624 pgsql       1  96    0 53580K  5480K select 1   0:02  0.00% postgres

As you probably noticed I colored the text in red that distinguishes between processors.

I know a lot of people use prefer the LAMP (Linux Apache MySQL PHP) stack but for me I like PostgreSQL. The main reason I prefer PostgreSQL to MySQL (recently purchased by Sun) is I come from a SQL Server/Oracle background and feel that PostgreSQL has a better implementation of the SQL (Standard Query Language) standard. That being said I also am a .NET/CF programmer so prefer Mono/Tomcat if when I host web based applications on non windows systems. So anyway when I got my new Dual-Opteron server I wanted to load PostgreSQL on it but its more complicated than just running a make install from ports in FreeBSD.

First being that I like to separate application data from the operating system I generally setup a separate partition for my apps to store data on. In this case I have a 18GB partition labeled /data upon which I will store the database files. Unfortunately this box is a 1U server and only has space for two drives (RAID-1) so I didn't have the option to put the database data on its own drives which I have found is helpful in windows systems.

Next I ran the make install from the postgresqlxx-server directory with options being the optimized build and 64-bit date support. Upon completion of the install I was notified that to support more connections I should rebuild my kernel with some options to increase shared memory.

  options         SYSVSHM
  options         SYSVSEM
  options         SYSVMSG
  options         SHMMAXPGS=65536
  options         SEMMNI=40
  options         SEMMNS=240
  options         SEMUME=40
  options         SEMMNU=120

I generally use a slightly modified GENERIC kernel, I use to heavily modify it but found that the performance gain was rather small, so the first three options were already in the kernel. After I added the bottom five options I ran buildkernel and installkernel which is covered in my previous post and rebooted the server.

After the server restarted I logged back in and proceeded to edit the rc.conf file in /etc to add the following lines:

postgresql_enable="YES"
postgresql_data="/data/pgsql"

The first line tells the operating system to start PostgreSQL when the server is started, the second tells PostgreSQL to use the /data/pgsql folder to store database data. At this point I thought all i needed to do to be up in running was run /usr/local/etc/rc.d/postgresql initdb and /usr/local/etc/rc.d/postgresql start. Well I was almost correct I forgot that the pgsql user did not have write access to the /data partition. So I created the folder /data/pgsql using the mkdir command and then gave the pgsql user ownership of the folder using chown pgsql:pgsql -R /data/pgsql. Next I re-ran the initdb command which executed successfully. Last I decided it would be a good idea to restart the server to make sure everything came up correctly which it did.

Now all i need to do is start the joyous of task of migrating databases, normally not a big deal but I have one thats rather large and will require me to schedule an outage. Undoubtedly this will mean I will have to do it at some awful hour in the morning, most likely on a weekend. Yeah me.

For a long time I use to see reports of brute force SSH attacks aganist my FreeBSD machines in mailbox every morning. Finnaly I got fed up not that they were even getting close to getting in but just tired of getting these huge reports. So I decided it was time to do something about it. First let me say I run PF (BSD Packet Filter) on all my FreeBSD machines. Its quite easy to setup so I will start there.

1. Rebuild your kernel to enable ALTQ being able to trottle bandwidth is pretty cool (optional)
1. Create a folder in /root called /kernels
   

   [root@test] [/usr/src/sys/i386/conf]# mkdir /root/kernels

2. Make a copy of the GENERIC kernerl profile and place it in the /root/kernels directory. Keep in mind that if your running say an AMD64 this directory will be slightly different.
   

   [root@test] [/usr/src/sys/i386/conf]# cp GENERIC /root/kernels/

3. Rename the file to something else like GENERIC-PF
   

   [root@test] [/usr/src/sys/i386/conf]# mv /root/kernels/GENERIC /root/kernles/GENERIC-PF

4. Link the new kernel file to directory where your kernel configuration files exist.
   

   [root@test] [/usr/src/sys/i386/conf]# ln -s /root/kernels/GENERIC-PF

5. Open the file in your favorite editor (vi for me)
   

   [root@test] [/usr/src/sys/i386/conf]# vi GENERIC-PF

   You may want to change the ident so that it reflects the changes you make to the kernel as well.
   

   ident         GENERIC-PF

   and add the following lines below the last line that starts with option and above the first line that beings with device.
   

   options         ALTQ
   options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
   options         ALTQ_RED        # Random Early Detection (RED)
   options         ALTQ_RIO        # RED In/Out
   options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
   options         ALTQ_PRIQ       # Priority Queuing (PRIQ)
   options         ALTQ_NOPCC      # Required for SMP build
   

6. Rebuild your kernel
   

   [root@test] [/usr/src/sys/i386/conf]# cd ../../../
   [root@test] [/usr/src]# make buildkernel KERNCONF=GENERIC-PF

7. If everything goes right install your new kernel
   

   [root@test] [/usr/src]# make installkernel KERNCONF=GENERIC-PF

8. Reboot
2. Enable PF in your /etc/rc.conf by adding the following lines to the end of the file
   

   pf_enable="YES"
   pflog_enable="YES"
   

3. Edit your /etc/pf.conf to setup some basic rules. Before doing this you should know what kind of network card you have you can find this info by running the command ifconfig. In the box I have at the house I have a VIA NIC so the driver is vr0. this
   
1. Edit the entry for ext_if="eth0" to contain your NIC driver ext_if="vr0". If your going to do ther filtering you will want to setup the ext_addr with your external ip address.
   
2. Let pf know your local interface is safe by telling it to skip filtering
   set skip on lo0
3. Its also not a bad idea to check malformed incoming packets this can be done by adding the line
   scrub in on $ext_if all
4. Setup a rule to block everything you don't explicitly want to allow and pass the good stuf
   block in log on $ext_if all
   block out log on $ext_if all
   pass  out on $ext_if inet proto tcp all flags S/SA keep state
   pass  out on $ext_if inet proto udp all keep state
   pass  out on $ext_if inet proto icmp all keep state
   
5. Enable ssh through the firewall
   pass  in  on $ext_if proto tcp from any to $ext_if port 22 keep state
   
6. Next you will probably want to setup your basic rules if you using any services such has http or ftp.
7. Add the table used to stop those script kiddies in their tracks.
   # sshd attacks
   table <ssh-violations> persist file "/etc/ssh-violations"
   block drop in from <ssh-violations> to any
4. You should end up with something that looks similar to this.
   

   ext_if="vr0"
   ext_addr="10.11.12.13"
   
   set skip on lo0
   scrub in on $ext_if all
   
   block in  log on $ext_if all
   block out log on $ext_if all
   
   pass  out on $ext_if inet proto tcp all flags S/SA keep state
   pass  out on $ext_if inet proto udp all keep state
   pass  out on $ext_if inet proto icmp all keep state
   
   #ssh
   pass  in  on $ext_if proto tcp from any to $ext_if port 22 keep state
   
   #http
   pass  in  on $ext_if proto tcp from any to any port 80 flags S/SA keep state
   
   # sshd attacks
   table <ssh-violations> persist file "/etc/ssh-violations"
   block drop in from <ssh-violations> to any
   
   antispoof for $ext_if

   
   
5. Next create a blank ssh-violations file.
   

   [root@test] [/etc]# touch ssh-violations

6. Enable PF
   

   [root@test] [/etc]# rc.d/pf start

7. Now that you have PF up and running comes the good part open up your favorite editor and create a file in the /root directory named sshd-fwscan.sh and paste the following.
   

   #!/bin/sh
   COMMAND="/sbin/pfctl"
   $COMMAND -t ssh-violations -T flush
   for ips in `cat /var/log/auth.log | grep sshd | grep "Illegal" | awk '{print $10}' | uniq -d` ; do
          $COMMAND -t ssh-violations -T add $ips
   done
   for ips in `cat /var/log/auth.log | grep sshd | grep "Invalid" | awk '{print $10}
   ' | uniq -d` ; do
          $COMMAND -t ssh-violations -T add $ips
   done
   cat /var/log/auth.log | grep sshd | grep "Failed" | rev  | cut -d\  -f 4 | rev | sort | uniq -c | \
   ( while read num ips; do
       if [ $num -gt 5 ]; then
            if ! $COMMAND -s rules | grep -q $ips ; then
                   $COMMAND -t ssh-violations -T add $ips
           fi
       fi
     done
   ) 
   

8. Change the permissions on that file to +x
   

   [root@test] [/root]# chmod +x sshd-fwscan.sh

9. Setup crontab so that the file is run say every 2 minutes.
   */2 * * * *     /root/sshd-fwscan.sh > /dev/null 2>&1
   

At this point, if you have done everything correctly, it should make be really hard for them to suceessfully a brute force attack SSH since they only have 120 seconds to figure out your password before they get dropped like a bad habit by the firewall.