Xen and the Art of Virtualization

I've installed Fedora Core 4 on one of my tinkering machines, and since it now ships with prebuilt Xen kernels I decided it was high time to give it a try. Having played with all kinds of machine virtualization from VMWare to Solaris 10 Zones to the Hypervisors on IBM pSeries servers (I'm still waiting for someone to give me time on a mainframe so I can tinker with z/VM though) I'd say I'm pretty familiar with the technology as a whole and the different approaches people take to it all.

So far I'm pretty impressed. Installation of the host (Domain0 in Xen speak) is easy as expected. Just install the supplied xen0 kernel, disable SELinux (and IPTables if you want to network it) and reboot. However, that's pretty much where the simple stuff ends. Unfortunately most of the pain and suffering I've had getting this to work has been caused by Red Hat, not Xen...

The Fedora guys have a Virtualization Quick Start page which describes what you need to do. It's pretty much accurate, except for when it comes to getting a working OS installation on your virtual disk image. The only thing I can think is that Yum has changed a bit since that document was written. After a bit of hacking I did manage to get a Fedora image booting under a virtual machine, but it was not pretty. So I decided to try a "build from almost scratch" distro I was familiar with - Gentoo.

I've run Gentoo on various machines of mine for almost 3 years now. It's fast, gives you a lot of control, but ultimately is a royal pain to maintain, especially for server environments. Installing it onto a Xen image is a dream though. It's quick, to the point, and let's you do what you want. All was going swimmingly until I tried to boot the virtual machine. It loaded the kernel (the supplied xenU kernel from Fedora) just fine, but when it came to mounting the file system, fsck.ext3 bombed with errors that there were unknown options in the filesystem metadata. It turns out that Red Hat have added an option to their e2fsprogs that allows online filesystem resizing, but this is not reflected in the version numbers they display. Luckily they are all statically linked, so I just copied the FC4 binaries over to my Gentoo partition and all is funky!

I now have a nice little playpen to install a new honeypot on. I've even got full IPv6 connectivity to it, so let's see what nastiness is spreading around the IPv6 side of the net these days...

How Privoxy ruined my day

I've been puzzled for the past few days about wierd things my browser has been doing. Sometimes things works nicely, other times it makes no sense! I was starting to wonder if there's a problem with the Firefox build that ships with FC4. One of the most painfull things it was doing was screwing with me trying to post here. I'd log in, type up something good, and then when I try "Save as Draft" it kicks me to the login page and loses my edits. And I kept on having to log into Slashdot...

I got so upset with losing posts, that I thought it was the blogger.com software. I then proceeded to download and install WordPress, which was quick and painless. Then I tried to tweak the config, and it bitched and screamed about my browser not sending any referer headers. Luckily it included a link in the error message, which took me to a nice little page that explained how to get your browser to send these headers. I checked my settings, and they were all correct! And then right at the bottom was a little note about Privoxy. And the light bulb clicked on. This great protector of my browser from such nasties as popups and double-click was also protecting me from blogging! So I added .blogger.com (and .slashdot.org and a few others) to the fragile list in the Privoxy configuration...

And now it all works....

Automated Abuse Reporting

For the past few days I've been tinkering with the idea of automated abuse reporting. I started out writing a little python script to grab the whois data for a specific IP address and parse out email addresses. Once more I'm moved by the power and simplicity of regular expressions, but I'm hugly frustrated by the fact that all the Regional Internet Registries don't use a common whois format! I know it's not realistic to have a totally common format, but a parsable abuse contact field should not be that hard. I'd rather not send a form letter to every email address in the whois record, as I'm not a fan of spam. Maybe I should start making enquiries...

Other than that, the filtering is working fine. Blocked a few more IP's. I've taken the limit down to 5/60 because slow scans were still getting too many attempts in.

So, my next quest is to start improving my IPv6 filter rules. I think the hard part is going to be finding a shell account on a machine on an IPv6 network that's not mine. Anyone out there feeling helpfull?

As if anyone reads this stuff anyway! I think I'll go directly to plan B and test drive the fair queing so I don't get lagged out playing WoW while downloading Fedora Core 4...

OpenBSD Network Protection - Summary

It works! Much more efficient and responsive than the first solution. Caught another kiddie yesterday, he only managed to get 13 attempts in a 25 second window before the system blocked him.

I changed the trigger rate after some additional testing after the previous post. 10/60 is much more realistic. Ideally it should be somewhere in the region of 5/60, but I'm being a bit conservative as that's how I connect to my home machines and I don't feel like locking myself out.

So, to summarise, the best way to tune your OpenBSD firewall to protect your systems from brute force dictionary attacks is to use the following rule format:

block quick from
pass in on $ext_if proto tcp to $target_ip_address port $port_to_protect flags S/SA keep state (max-src-conn-rate $connections_per_second_to_trigger, overload flush)

The trigger rate can be tuned to your level of paranoia, but I've found that having a long window to monitor in helps considerably.

Now, let's see if I can get automated abuse reporting going once I've caught these little buggers...

OpenBSD Protection - Part 2

It's been almost 24 hours since I put up the scan protection mentioned in the last article, and I've already blocked a scanner!

But having a look at it all, it's not a very good solution. Problems are:

1. Scanning is resource intensive. It put quite a notable load spike on the system when ever it ran, and that can be expected from shell scripts scanning log files.
   
2. Not granular enough. I was initially running the script every 5 minutes. And as luck would have it, the kiddie I caught started his scan at 15:24:05. If he had started his scan a minute later, there's a good chance I would not have picked it up. In the minute that his scan was allowed to run, he had already managed 97 connections!
   
3. Limited to single machine. As I'm getting all my info from ssh logs, and I'm blocking the packets coming in to the exposed host, when the do trip the wire I only end up protecting a single machine.

To get around these problems, I've decided to make use of the rate limiting features of pf in OpenBSD. I have a dedicated machine protecting my network and doing all my external routing. By putting the monitoring and blocking at this level, as soon as I pick up a scan I can block traffic from that IP address to my entire network.

To do this, I simply added the following rules to my pf.conf:

table <bad_hosts> persist file "/var/log/bad_hosts.list"
block quick from <bad_hosts>
pass in on tun0 proto tcp to (my IP address) port ssh flags S/SA keep state (max-src-conn-rate 5/10, overload <bad_hosts> flush)

Note that I load the bad IP addresses on start up from a file. To keep the file up to date, I run pfctl -t bad_hosts -Tshow > /var/log/bad_hosts.list every hour via cron.

Let's see how this works for the next few days. The next quest will be to automatically file an abuse report for the IP address once you pick it up...

OpenBSD SSH Protection

As may become apparent over time, I'm a Unix Geek. Each flavour is special in its own little way. My favourite for exposing to the 'Net is OpenBSD. But the problem I have with my current machines is that script kiddies keep trawling trying to get in over SSH, etc. So what I've done (inspired by DenyHosts) is write a small shell script to keep an eye on my authlogs and block offenders.

DenyHosts looks interest, and earns brownie points from me being written in Python. But I just need something to be small and specific. Also, DenyHosts only adds entries to your hosts.deny file, and I want to block all access from these little buggers!

OpenBSD ships with a good firewall. The approach I took was to parse my log files for people trying to get in via SSH and failing (with a bit of extra logic to stop it booting me if I do something silly) and then simply add offending IP addresses to the firewall as blocked hosts.

Here's the script that does the work: (it's messy, I know, but it works)

#!/bin/ksh
#
# Script to trawl logs for nastiness and log bad IP addresses
#

NUM_TRIES=3

SSH_INVALID_USERS=`grep 'Invalid user' /var/log/authlog | awk '{ print $10 }' | sort -u`

for iu in $SSH_INVALID_USERS; do
   num=`grep $iu /var/log/authlog | wc -l`
   if [ $num -gt $NUM_TRIES ]; then
     echo "$iu" >> /var/tmp/invalid_users.list
   fi
done

cat /var/tmp/invalid_users.list | sort -u > /var/tmp/invalid_users.list

SSH_FAILED_PASSWORD=`grep 'Failed password for' /var/log/authlog | grep -v 'invalid user' | awk '{ print $11 }' | sort -u`

for fp in $SSH_FAILED_PASSWORD; do
   num=`grep $fp /var/log/authlog | wc -l`
   if [ $num -gt $NUM_TRIES ]; then
     echo "$fp" >> /var/tmp/failed_passwords.list
   fi
done

cat /var/tmp/failed_passwords.list | sort -u > /var/tmp/failed_passwords.list

cat /var/tmp/invalid_users.list /var/tmp/failed_passwords.list | sort -u > /var/tmp/blockers.list

pfctl -t kiddies -vTadd -f /var/tmp/blockers.list

This script, run from cron as root, will update the kiddies table with bad IP addresses.

Here's the entries from the pf.conf file (firewall configuration) that does the work:

ext_if="vr0"
table <kiddies> persist
block in on $ext_if from <kiddies>

And that's all there is to it! If you want to see how effective you're being, you can run something like:

pfctl -t kiddies -vTshow

First Post

This is the first post! Lots of people at work blog, I'm still not 100% convinced it's worth the effort, but I've finally got something to say! I can't think of anywhere to say it other than at a blog type place, and I could not be arsed right now to set up my own blog system, so I'm gonna give Google a try and see how it goes...