NetNix

Moving...

2006-03-09T16:49:00.000Z

Well, it's been about 6 months since I last blogged. I've not had much interesting to say really, just had my head down working. Things have lightened up a bit since then, so hopefully I'll have a bit more time on my hands to order my thoughts and scribble them down. All this is going to happen at my new site, http://www.chris-read.net. See you there...

Why reading books is still important

2005-10-27T13:25:00.000+01:00

I finally got around to reading my copy of IPv6 Network Administration and in chapter 4 there's a section that deals with transition mechanisms which covers the kinds of problems I've been trying to solve with STFL. When I was first digging around the net to see what people had done about allowing IPv6 clients to connect to IPv4 only applications, I found some elusive references to something from the KAME project called faithd. All I could find out about it at the time was that it seemed to be half implemented in user space, and half in the kernel of FreeBSD, but it seems to have been lost in the mists of time. While the book also refers to it, there is also mention of RFC 3142 which describes something called TRT, or Transport Relay Translation. I now have a proper name for what I'm trying to do! That's why reading books is still important.

The solution described in the RFC is pretty much exactly what I've been thinking of, except for the following differences:

Name Resolution

Or the way the DNS is configured for the service. The approach I've been thinking of is pretty simple - assign an IPv6 address to the service you want to offer, add an AAAA record to DNS for that service, run STFL on that IPv6 address and forward the requests to the IPv4 address and port where the application server actually runs. While RFC 3142 does allude to doing it this way, it also talks about using a custom resolver library or DNS server to automatically convert the IPv4 address into an IPv6 address (the don't mention it specifically, but I guess they're talking about 6to4 which is described in RFC 3056).

Service Granularity

My philosophy is simple. You have one application (in my personal case it's Apache 1.3 on OpenBSD) that is not IPv6 compatible but you want to serve IPv6 clients with. So all you do is spark up an application on an equivalent port on an IPv6 address and just ferry the TCP packets back and forth. If you have more than one application you want to do this for, then you simply fire up additional STFL instances. The RFC seems to suggest making an entire IPv4 address and all its TCP and UDP services available on the same IPv6 address (hence the need for mangling resolver). I guess this is why the reference implementation need a kernel level component, and is probably why it never seemed to get off the ground - that's some pretty complicated stuff you need to do!

Introducing STFL - the Six to Four Layer

2005-08-30T11:41:00.000+01:00

It's been a long time since I last posted my ramblings on a simple way to allow legacy IPv4 services to serve IPv6 clients, but at last I have something to show for it! After playing around a bit more with my Python "proof of concept" code, I realized that while the idea was sound, implimenting it in Python may not be the best solution. The main problem I had was with the Python global interpreter lock. Even though I'd written the code using threads, it would not scale under heavy load. So, I decided to chip the layers of rust off my C skills. The result is the STFL project. It's functional, but needs polish and a bit of effort to get it working all the major Unix flavours. Currently it's only fully functional (and tested) on Linux and OpenBSD.

The easy way to IPv6 support for TCP apps

2005-07-17T15:15:00.000+01:00

When I initially got my static IPv6 address range from my ISP, I was running Linux as my firewall and Apache 2 on Solaris x86 as my web server. This meant that getting my site available over IPv6 was pretty straight forward. All I had to do was add new A6 and AAAA records to my DNS entries, configure the static IP address on the Solaris box and add an additional Listen directive to Apache. Life was good. Not that I ever got more than 10 hits a month on my site from IPv6 addresses, but that was not the point!

Then I switched the firewall and web server over to OpenBSD. One of the main factors in me deciding to do this was the great "Tinfoil Hat" approach the OpenBSD guys have to security. Sure, you may not be able to run the latest greatest version of your favourite app, but the stuff they do ship with the core distribution has been fairly well nit-picked by some pretty bright people. This means I don't have to worry so much about the script kiddies getting in...

However, this does mean that the Apache they ship and support and have tweaked to have run in its own little chrooted sandbox is pretty old - 1.3.29 at the time of writing this. Yes, all the security patches that come from the Apache group have been applied. But it does not do IPv6. There are patches out there from the Kame guys to enable this in the 1.3 branch, but it's old, experimental, and does not do SSL. I did for a few brief minutes contemplate seeing if I could at least try crafting my own patch to get mod_ssl to work, but then sanity kicked in. I am not a hard core developer. I am a Sys Admin who knows how to code.

I then decided to step back and look at the picture from another angle. All that's different really is the IP stack. The actual TCP protocol used to do all the chatting is the same. What I really needed was something to mess with the inbound IPv6 packets, send them to the required IPv4 host, and do the opposite on replies. Trying to write some code to do this somewhere in the network stack is more scary and presents more security bugs than trying to write a mod_ssl patch. So I decided to go for the quick and easy method - write a socket app that listens on an IPv6 port, and pumps all the TCP packets down a new IPv4 connection, and do the opposite with the replies.

A few hours later I had a small (about 200 lines at the moment) Python app to do this all for me. It's also fast handles multiple ports, and is protocol indepentant. I've tested HTTP, SSL, SSH, IMAP, POP3 and SMTP with it.

I'm busy working on the final touches before releasing it, but I will anounce it here when it's ready.

Xen and the Art of Virtualization

2005-06-23T09:23:00.000+01:00

I've installed Fedora Core 4 on one of my tinkering machines, and since it now ships with prebuilt Xen kernels I decided it was high time to give it a try. Having played with all kinds of machine virtualization from VMWare to Solaris 10 Zones to the Hypervisors on IBM pSeries servers (I'm still waiting for someone to give me time on a mainframe so I can tinker with z/VM though) I'd say I'm pretty familiar with the technology as a whole and the different approaches people take to it all.

So far I'm pretty impressed. Installation of the host (Domain0 in Xen speak) is easy as expected. Just install the supplied xen0 kernel, disable SELinux (and IPTables if you want to network it) and reboot. However, that's pretty much where the simple stuff ends. Unfortunately most of the pain and suffering I've had getting this to work has been caused by Red Hat, not Xen...

The Fedora guys have a Virtualization Quick Start page which describes what you need to do. It's pretty much accurate, except for when it comes to getting a working OS installation on your virtual disk image. The only thing I can think is that Yum has changed a bit since that document was written. After a bit of hacking I did manage to get a Fedora image booting under a virtual machine, but it was not pretty. So I decided to try a "build from almost scratch" distro I was familiar with - Gentoo.

I've run Gentoo on various machines of mine for almost 3 years now. It's fast, gives you a lot of control, but ultimately is a royal pain to maintain, especially for server environments. Installing it onto a Xen image is a dream though. It's quick, to the point, and let's you do what you want. All was going swimmingly until I tried to boot the virtual machine. It loaded the kernel (the supplied xenU kernel from Fedora) just fine, but when it came to mounting the file system, fsck.ext3 bombed with errors that there were unknown options in the filesystem metadata. It turns out that Red Hat have added an option to their e2fsprogs that allows online filesystem resizing, but this is not reflected in the version numbers they display. Luckily they are all statically linked, so I just copied the FC4 binaries over to my Gentoo partition and all is funky!

I now have a nice little playpen to install a new honeypot on. I've even got full IPv6 connectivity to it, so let's see what nastiness is spreading around the IPv6 side of the net these days...

How Privoxy ruined my day

2005-06-17T17:09:00.000+01:00

I've been puzzled for the past few days about wierd things my browser has been doing. Sometimes things works nicely, other times it makes no sense! I was starting to wonder if there's a problem with the Firefox build that ships with FC4. One of the most painfull things it was doing was screwing with me trying to post here. I'd log in, type up something good, and then when I try "Save as Draft" it kicks me to the login page and loses my edits. And I kept on having to log into Slashdot...

I got so upset with losing posts, that I thought it was the blogger.com software. I then proceeded to download and install WordPress, which was quick and painless. Then I tried to tweak the config, and it bitched and screamed about my browser not sending any referer headers. Luckily it included a link in the error message, which took me to a nice little page that explained how to get your browser to send these headers. I checked my settings, and they were all correct! And then right at the bottom was a little note about Privoxy. And the light bulb clicked on. This great protector of my browser from such nasties as popups and double-click was also protecting me from blogging! So I added .blogger.com (and .slashdot.org and a few others) to the fragile list in the Privoxy configuration...

And now it all works....

Automated Abuse Reporting

2005-06-13T21:46:00.000+01:00

For the past few days I've been tinkering with the idea of automated abuse reporting. I started out writing a little python script to grab the whois data for a specific IP address and parse out email addresses. Once more I'm moved by the power and simplicity of regular expressions, but I'm hugly frustrated by the fact that all the Regional Internet Registries don't use a common whois format! I know it's not realistic to have a totally common format, but a parsable abuse contact field should not be that hard. I'd rather not send a form letter to every email address in the whois record, as I'm not a fan of spam. Maybe I should start making enquiries...

Other than that, the filtering is working fine. Blocked a few more IP's. I've taken the limit down to 5/60 because slow scans were still getting too many attempts in.

So, my next quest is to start improving my IPv6 filter rules. I think the hard part is going to be finding a shell account on a machine on an IPv6 network that's not mine. Anyone out there feeling helpfull?

As if anyone reads this stuff anyway! I think I'll go directly to plan B and test drive the fair queing so I don't get lagged out playing WoW while downloading Fedora Core 4...

OpenBSD Network Protection - Summary

2005-06-09T11:48:00.000+01:00

It works! Much more efficient and responsive than the first solution. Caught another kiddie yesterday, he only managed to get 13 attempts in a 25 second window before the system blocked him.

I changed the trigger rate after some additional testing after the previous post. 10/60 is much more realistic. Ideally it should be somewhere in the region of 5/60, but I'm being a bit conservative as that's how I connect to my home machines and I don't feel like locking myself out.

So, to summarise, the best way to tune your OpenBSD firewall to protect your systems from brute force dictionary attacks is to use the following rule format:

block quick from
pass in on $ext_if proto tcp to $target_ip_address port $port_to_protect flags S/SA keep state (max-src-conn-rate $connections_per_second_to_trigger, overload flush)

The trigger rate can be tuned to your level of paranoia, but I've found that having a long window to monitor in helps considerably.

Now, let's see if I can get automated abuse reporting going once I've caught these little buggers...

OpenBSD Protection - Part 2

2005-06-07T10:52:00.000+01:00

It's been almost 24 hours since I put up the scan protection mentioned in the last article, and I've already blocked a scanner!

But having a look at it all, it's not a very good solution. Problems are:

Scanning is resource intensive. It put quite a notable load spike on the system when ever it ran, and that can be expected from shell scripts scanning log files.
Not granular enough. I was initially running the script every 5 minutes. And as luck would have it, the kiddie I caught started his scan at 15:24:05. If he had started his scan a minute later, there's a good chance I would not have picked it up. In the minute that his scan was allowed to run, he had already managed 97 connections!
Limited to single machine. As I'm getting all my info from ssh logs, and I'm blocking the packets coming in to the exposed host, when the do trip the wire I only end up protecting a single machine.

To get around these problems, I've decided to make use of the rate limiting features of pf in OpenBSD. I have a dedicated machine protecting my network and doing all my external routing. By putting the monitoring and blocking at this level, as soon as I pick up a scan I can block traffic from that IP address to my entire network.

To do this, I simply added the following rules to my pf.conf:

table <bad_hosts> persist file "/var/log/bad_hosts.list"
block quick from <bad_hosts>
pass in on tun0 proto tcp to (my IP address) port ssh flags S/SA keep state (max-src-conn-rate 5/10, overload <bad_hosts> flush)

Note that I load the bad IP addresses on start up from a file. To keep the file up to date, I run pfctl -t bad_hosts -Tshow > /var/log/bad_hosts.list every hour via cron.

Let's see how this works for the next few days. The next quest will be to automatically file an abuse report for the IP address once you pick it up...

OpenBSD SSH Protection

2005-06-06T12:15:00.000+01:00

As may become apparent over time, I'm a Unix Geek. Each flavour is special in its own little way. My favourite for exposing to the 'Net is OpenBSD. But the problem I have with my current machines is that script kiddies keep trawling trying to get in over SSH, etc. So what I've done (inspired by DenyHosts) is write a small shell script to keep an eye on my authlogs and block offenders.

DenyHosts looks interest, and earns brownie points from me being written in Python. But I just need something to be small and specific. Also, DenyHosts only adds entries to your hosts.deny file, and I want to block all access from these little buggers!

OpenBSD ships with a good firewall. The approach I took was to parse my log files for people trying to get in via SSH and failing (with a bit of extra logic to stop it booting me if I do something silly) and then simply add offending IP addresses to the firewall as blocked hosts.

Here's the script that does the work: (it's messy, I know, but it works)

#!/bin/ksh
#
# Script to trawl logs for nastiness and log bad IP addresses
#

NUM_TRIES=3

SSH_INVALID_USERS=`grep 'Invalid user' /var/log/authlog | awk '{ print $10 }' | sort -u`

for iu in $SSH_INVALID_USERS; do
   num=`grep $iu /var/log/authlog | wc -l`
   if [ $num -gt $NUM_TRIES ]; then
     echo "$iu" >> /var/tmp/invalid_users.list
   fi
done

cat /var/tmp/invalid_users.list | sort -u > /var/tmp/invalid_users.list

SSH_FAILED_PASSWORD=`grep 'Failed password for' /var/log/authlog | grep -v 'invalid user' | awk '{ print $11 }' | sort -u`

for fp in $SSH_FAILED_PASSWORD; do
   num=`grep $fp /var/log/authlog | wc -l`
   if [ $num -gt $NUM_TRIES ]; then
     echo "$fp" >> /var/tmp/failed_passwords.list
   fi
done

cat /var/tmp/failed_passwords.list | sort -u > /var/tmp/failed_passwords.list

cat /var/tmp/invalid_users.list /var/tmp/failed_passwords.list | sort -u > /var/tmp/blockers.list

pfctl -t kiddies -vTadd -f /var/tmp/blockers.list

This script, run from cron as root, will update the kiddies table with bad IP addresses.

Here's the entries from the pf.conf file (firewall configuration) that does the work:

ext_if="vr0"
table <kiddies> persist
block in on $ext_if from <kiddies>

And that's all there is to it! If you want to see how effective you're being, you can run something like:

pfctl -t kiddies -vTshow

First Post

2005-06-06T12:07:00.000+01:00

This is the first post! Lots of people at work blog, I'm still not 100% convinced it's worth the effort, but I've finally got something to say! I can't think of anywhere to say it other than at a blog type place, and I could not be arsed right now to set up my own blog system, so I'm gonna give Google a try and see how it goes...