Systems Administration is a lonely job. There is very little support for professionals and we're all very divided. There is, I find, a common desire to know what "the other guy" is doing or thinking, what tools they use, how they arrange their day, how they deal with stress, how they select one gig over another, how they provide for a wife and family in it all.

A podcast is a great way to do this, but there isn' t much out there. The closest was Kevin Devin's "From the Trenches", which appears to be dead, and always focused on the Windows/Cisco network segment which is frankly uninteresting to me.

Enter SAPro...

This new podcast will be interview and round-table focused, unlike my previous podcast(s) which were all in the "one guy with a mic" format. Even I didn't want to listen to them unless I was really bored.

I'm organizing several "admin roundtables" for diverse discussion and starting to line up several interviews with industry super-stars. If you want to participate please let me know. I am particularly interested in interviewing vendor backline support engineers, if you are one please contact me.

Keep your eyes open, I plan to roll out episodes soon.

This isn't new, but simply new to me, at least if I saw it before I didn't pay sufficient attention... Brocade Announces Definitive Agreement to Acquire Foundry Networks, press release dated 7/21/08. I mean, is commentary even required?

This, naturally, just adds a massive blow in the "fibre channel is dead" debate; assuming there is anyone that will continue to argue the point. Clearly 4Gb FC-Fabric has a useful place, but its clear that 10Gbps with is various RDMA add-ons will likely do what Infiniband had promised oh so long ago, and once the pricepoint of 10gigE hits the sweet spot in 1-2 years is all over for FC-Fabric.

But more interesting than the technology is the business practice of Brocade. McData was an upstart gunning against the more established Brocade... then McData shot like a rocket and made Brocade look like an aging dog.... solved by acquisition, bringing McData's lucrative chassis-based core switch business to Brocade who was being relgated more so toward edge switching and competing more and more with Qlogic. They've repositioned, trying to appear more like a data management company than storage networking company, including several attempts to popularize terms like FAN (File Area Network... you can hear the thud echo every time its said aloud).

We've all seen businesses who were so committed, stupidly, to dying products and services that they refused to diversify and grow with the trends for fear of loosing face. Brocade is, I think, simultaneously admitting defeat and refusing to die. Awesome business savy. A look at any business history shows that failure to commit to trends due to "core competency" only keeps you from becoming all that you can be. Steve Jobs said once, many moons ago, that if Xerox only realized what they had with the ALTO computer developed at XEROX PARC they, Xerox, would be become the giant to rival IBM in the computer market. Xerox had a core competency, it thought, in the copier business... in hind site we see that was absolutely insanely wrong and short sighted; Xerox's core competency was R&D from which it could spin off companies or divisions to execute on, 3M or Dow are good examples of this.

In that mindset, I can't wonder if Brocade will be a much bigger player in the future. Right now it looks like desperation to stay afloat, but who knows, in 10 years we may all feel very differently.

About 6 years ago or so I got tired of fixing problem with Tamarah Windows/Linux box and decided to pay the money for a 15" PowerBook. It was an excellent investment, she could work on the couch, no more lockups and reboots in Windows or mysterious "Bennnnnnn!" problems in Linux. Since then she's upgraded to a black MacBook, and when I joined Joyent they provided me with a MacBook Pro (which I'm typing on now). So far each of these 3 laptops has lost at least one drive. Since we've fallen in love with iTunes and iPhoto these drive failures have been a major blow, and prior to Leopard's TimeMachine we didn't do regular backups.

This post will refer solely to drives for personal use. In the datacenter you should be using RAID and/or backup or redundancy method in which case a single drive failure isn't something you waste time trying to analyze or fix.

I've run into 3 major types of drive failure:

1. PCB Failure: A case in which the PCB has been "fried". This happened dramtically once when connected an IDE drive to a system and let the disk rest, upside down, ontop of the case. It ran fine for aminute and then pop/spark there was a hole burned in a chip on the PCB. In this case the only solution is to go to eBay and buy an identical drive and swap the PCB.
2. Click of Death: This means catastrophic damage to a drive. The head is unable to position itself or read data and sweeps the platters in a sort of seizure. This is the sort of problem that likely requires you to open the drive or spend big bucks.
3. Damaged Cylinders: This is the kind of problem where the drive seems fine, mounts up and you can read for a bit and then hits some area of the platters where it freaks out and eventually spins up and down. This is most clearly seen when you image the drive with dd and it hits some point and exits on max retries.

Information on drive forensics and recovery is sparse. You tend to get one of three answers:

1. "d00d, totally put it in the freezer and then try it!" Variations come based on how you should protect against condensation, the best I've heard is to pack the drive in minute-rice.
2. "Send it to DriveSavers" (or other) This is super expensive, anywhere from $600 up beyond $2,000. You send them the failed disk and optionally a new drive to restore to. This can take weeks and is only for super extreme cases.
3. "Just download tool xyz.." There are lots of various software solutions for do-it-yourself drive recovery, most are old DOS based programs recommended on forums populated largely by Windows users.

In my most recent failure, the drive died one day for seemingly no reason. There was no impact or horror story, the OS just locked up, I rebooted and the OS would start to load and then just drift into an infinite slumber. I went through the painstaking process of replacing the drive in my MacBook Pro and re-installed everything from scratch. Once back up and running I put the old drive in a USB enclosure and attempted to image it using dd. Every attempt it would get 19GB into the drive and then give up.

This kind of problem is the easiest to deal with. There are special versions of dd, namely GNU ddrescue, which is just like dd, but instead of failing on bad blocks will track forward after a number of retries untill its read the whole disk, for better or worse.

In the case of my MacBook Pro drive I attached the USB enclosure to my OpenSolaris box, installed ddrescue, and imaged the drive to a file. Of the 80GB drive the tool reported that I lost about 250MB. I then created a ZFS ZVol of 80GB, used traditional dd to copy the image file into the volume, and then exported as an iSCSI target using iscsitadm. Using the globalSAN iSCSI Initiator for OS X I mounted the iSCSI Target, and used OS X "DiskUtility" to verify and repair the HFS+ Volume. All went well and I could then mount the volume and extract data. w00t!!! iSCSI Rules!

The tale of Tamarah's MacBook drive didn't end so happily. I had a backup of her laptop but it was really old. Glenn, our son, grabbed the laptop on the table sending it crashing to the tile floor below, hitting on the corner where the drive sits. The laptop was fine, but the drive was toast. After a Mac Genious showed us how to replace the drive I bought a new disk at Fry's and got things installed and running again, but the drive contained a lot of projects she wanted, and is commonly the case, when I showed her the data from the old backup she was uncertain as to whether it was enough. This is a big problem of the "unknown", when all your stuff is in one place you commonly forget what exactly is there.

I tried the USB enclosure trick but the drive wouldn't even spin up... click of death. Given the sensativity of the data I didn't want to go Rambo on the disk and so we sat down and had a serious discussion about whether or not it was worth having sent to a drive recovery company. The look on her face was enough to tell me what to do, and despite her guilt over the cost I sent it in. After a week and a half, the answer came back "nothing we can do". The tech was friendly and we had a good discussion about drive recovery, but long story short there was no hope and we were out $800. Frankly, for a lot of people that money is well spent because at least you exhausted all avenues, morn and get on with it.

When it comes to hardcore "swap the platters" style repair things get dicey. As simplistic as hard drives seem there are a lot of gotchas that you won't be aware of until its too late. This is where Scott Moulton of MyHardDriveDied.com comes in. Scott has done two presentations, both found on YouTube that provide a solid background for the black-art of hardcore drive recovery used by most of the big bucks recovery companies.

The first presentation is Hard Drive Recovery, available in 5 parts. This is followed up (a year later) by Advanced Hard Drive Data Recovery, again in 5 parts. Both presentations include excellent flash animations that illustrate his presentation perfected.

You can find even more videos by Scott on his SuperFlyFlippingA YouTube Channel, including an excellent presentation on SSD Flash vs Hard Drives.

Scott Moulton has done an amazing service to the community by providing detailed and experienced information regarding hard drive tinkering and recovery, including things you would never otherwise consider such as "Live PCB Swap"... watch and learn. :)

Of course, an ounce of prevention is blah blah blah. Technologies like OS X TimeMachine and ZFS make backup easier and more realistic than ever before and most importantly reduce data duplication significantly. Online backup solutions are good, but frankly are only feasible on very high speed lines in this era where a trip to the beach can result in 2GB's of new pictures. What I like best is the emergence of wi-fi USB Drive solutions that allow solutions like TimeMachine to backup whenever it likes without specially being hooked up to a drive... the more you back up the less there is to back up and the less hassle it is.

As a closing note... recovering the data is only part of the solution. I've found that some Apple apps like iPhoto and iTunes can be very unhappy when you attempt to import into a new system install. For instance, attempts to open my old iPhoto library have been unsuccessful. Thankfully I found iPhotoExtractor. As for iTunes, sadly iPods are not a backup solution... when attaching to a new system, even after authorizing, you may be told you need to delete and resync. In those cases, Sci-Fi Hi-Fi's PodWorks can come to the rescue allowing you to extract and import music from otherwise unusable iPods.

I am in no way qualified to speak with regard to whats going on around Blastwave, but given that there are no official statements and lots of rumors and speculation I'll share some info. Please note that I consider both Phil Brown and Dennis Clark as friends and am completely impartial, at least for this post.

If you've visited Blastwave.org recently you'll notice that parts of the site are "missing", you may also have noticed that Genunix was down for a bit. Threads regarding this issue have appeared both on OpenSolaris Discuss and comp.unix.solaris. A nice lengthy discussion was had with Dennis Clark today in #opensolaris, however I shall not quote that conversation directly, over 250 people were in the room at the time.

In the simplest possible terms, Dennis Clark and Phil Brown are in the process of parting ways. Phil Brown is the creator of the CSW (Community SoftWare) effort including the popular apt-get like pkg-get which has been the center of what people think of Blastwave. Dennis Clark is the man behind Blastwave which has provided the large infrastructure and pool of resources around CSW. These two have been intertwined for a very long time and, due to difference over future direction has escalated into separation. Because of this intemate relationship between CSW and Blastwave over so many years this disconnect does, in many ways, resemble a messy divorce.

As seen in Dennis Clarks email to OpenSolaris Discuss lawyers are involved and action is reportedly being taken against Phil Brown. Furthermore, Blastwave Inc is in progress of incorporating in Canada. CSW now has a new home suncsw.de in Germany.

Blastwave is still in operation for now, and thanks to the various Blastwave/CSW package mirrors users should have been minimally impacted, if at all.

For more information please read through the 3 message threads linked above so you can read from the participants themselves. Phil and Dennis are both active members of the community, if you have questions or concerns I encourage you to respectfully and courteously contact them directly via email, lists, or IRC.

In closing, I will remind everyone that both Dennis and Phil have done amazing things for the Solaris community for many years, and they both have intentions (it would appear) to continue doing so; albeit separately. For years and years they provided the Solaris community, unrecognized by Sun at the time, with high quality software on both SPARC and X86. Regardless of which side you take or which story you read never forget that both Dennis Clark and Phil Brown, together with countless maintainers and volunteers, deserve our lasting respect and appreciation. I'm personally saddened to see the division, but thankful for what they accomplished together.

The first annual OpenSolaris Storage Summit is coming on Sunday, September 21st, to San Jose, to be followed by SNIA's Storage Developer Conference. This is really exciting. SNIA SDC is one of the best storage conferences in the world (along with USENIX FAST), and OpenSolaris is undoubtedly the more powerful storage platform on earth... this is an excellent opportunity to get a lot of excellent and rewarding information in a week and get to meet the minds behind the technology in OpenSolaris.

Among the speaker lineup, myself and Mike Shapiro are the keynote speakers. If you have any interest in the future of storage you do not want to miss Mike Shapiro! Trust me. You don't.

As for my keynote, I would love to get feedback from everyone on what you'd like me to talk about. I have some ideas on what I could speak about, but I really want to provide as much value to the attendees as possible, so if you have topics you want discussed please send them my way.

Among the topics I will address is the impact of "cloud computing" on the modern storage infrastructure, how it relates to "Open Storage", and where things are heading in the future. Storage is, hands down, the most complex aspect of the cloud infrastructure and we'll dig our heals into it together.

The event is free, so you have no excuse not to attend! Just add your name to the registration list and your in! Pricing for SDC to follow is very reasonable and a very good investment in yourself and your company. If your in the Bay Area you've got to attend... if your outside of the area, tell your boss that Ben Rockwood said you need to be there, they'll give in right away. :)

Spencer Shepler is leaving Sun. I am saddened to see him go.

If you don't know Spencer's name, he's been an NFS giant for a long time and we all owe a debt of gratitude to him. Of his long list of accomplishments he is a a chair of the IETF NFSv4 Working Group and is the most prominent name on the NFSv4 RFC 3530.

Make sure to keep your eyes on his new blog.

Some time ago I wrote a big blog entry entitled Simplifying Zone Management with Kerberos. Since that time (3 years ago!?!) several improvements and simplifications have come along to make life far more pleasant. If your new to Kerberos, please refer to my previous entry before proceeding with this one.

Kerberos is still something I struggle to really wrap my head around. Thats probably due to the fact that I have never used it in a full production deployment... the problem with using something only in the lab is that you never are "done" and you never hit real world problems that push you beyond that which you dream up.

There are classicly two reasons people get interested in Kerberos: Single Sign On (SSO) and secure NFS. Once upon a time general purpose encryption support for tools like telnet, rlogin, and FTP were a great draw, but today between SSH and SSL/TLS support in almost everything there just isn't a reason to jump down the Kerberos path. SSO is a reason, but given that the defacto UNIX remote management protocol is SSH and SSH supports passwordless (authorized keys) login, its easy to mimic the benefits of SSO.... particularly if you use OpenSSH LPK, which allows you to store public keys in LDAP. As for NFS, well, there you probly still wanna stick with Kerberos unless you are willing to go the NFSv4/IPsec route.

The point here being, prior to SSH's rise to glory Kerberos was a vitally important enterprise solution, whereas today its a hard sell. When it comes down to it, there is really only one real standout reason to seriously consider Kerberos: passwords are never on the wire, period. That is indeed a kool thing, but I doubt it will convince most people enough to really get them to go at it. In the end, Kerberos is only really exciting when its implemented in a way such as ActiveDirectory, where you don't even know its there, you just turn it on and reap the benefits. So you can almost conclude that Kerberos is as important as ever, its just no longer worth the expenditure of effort.

To this end, I advise anyone interested in LDAP/Kerberos get familiar with ApacheDS, the work their doing is truly amazing. ...but more about that some other time.

As I was saying, Kerberos is now easier than ever to get up and running on Solaris, thanks to the kdcmgr and kclient tools.

Thanks to the kdcmgr command we no longer need to hand edit files and use the kinit command to initialize a Kerberos KDC. Here is an example of kdcmgr in action:

$ kdcmgr -a benr/admin -r CUDDLETECH.COM create master

Starting server setup
---------------------------------------------------

Setting up /etc/krb5/kdc.conf.

Setting up /etc/krb5/krb5.conf.

Initializing database '/var/krb5/principal' for realm 'CUDDLETECH.COM',
master key name 'K/M@CUDDLETECH.COM'                                            
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key: 
Re-enter KDC database master key to verify: 
 
Authenticating as principal root/admin@CUDDLETECH.COM with password.
WARNING: no policy specified for benr/admin@CUDDLETECH.COM; defaulting to no policy
Enter password for principal "benr/admin@CUDDLETECH.COM":
Re-enter password for principal "benr/admin@CUDDLETECH.COM":
Principal "benr/admin@CUDDLETECH.COM" created.

Setting up /etc/krb5/kadm5.acl.

---------------------------------------------------
Setup COMPLETE.

Done! If you look around you'll see this has created the necessary config and keytabs, as well as started the appropriate SMF services:

$ svcs -a | grep security
disabled       May_30   svc:/network/security/krb5_prop:default
online         May_30   svc:/network/security/ktkt_warn:default
online          3:11:34 svc:/network/security/krb5kdc:default
online          3:11:36 svc:/network/security/kadmin:default
$ ls -alh /etc/krb5/
total 40
drwxr-xr-x   2 root     sys          512 Jul 27 03:11 .
drwxr-xr-x  96 root     sys         4.5K Jul 25 14:58 ..
-rw-r--r--   1 root     sys           52 Jul 27 03:11 kadm5.acl
-rw-r--r--   1 root     root         998 Jul 27 03:11 kadm5.acl.sav
-rw-------   1 root     root        1.5K Jul 27 03:11 kadm5.keytab
-rw-r--r--   1 root     sys          430 Jul 27 03:11 kdc.conf
-rw-r--r--   1 root     root        1.3K Jul 27 03:11 kdc.conf.sav
-rw-r--r--   1 root     sys          968 Mar 22 10:43 kpropd.acl
-rw-r--r--   1 root     sys          367 Jul 27 03:11 krb5.conf
-rw-r--r--   1 root     root        2.0K Jul 27 03:11 krb5.conf.sav
-rw-------   1 root     root         383 Jul 27 03:11 krb5.keytab
-rw-r--r--   1 root     sys         1.1K Mar 22 10:43 warn.conf

As you can see, its all there and in order. You can immediately use kinit to get a ticket and use kadmin to create additional principles, no kadmin.local required. Very refreshing indeed.

As for the clients, manual configuration is so simple as to require very little improvement, never the less we can use the kclient command to standardize it:

# kclient -a benr/admin -k kdc1.cuddletech.com -R CUDDLETECH.COM

Starting client setup

---------------------------------------------------
Do you want to use DNS for kerberos lookups ? [y/n]: n
        No action performed.
kdc1.cuddletech.com

Note, this system and the KDC's time must be within 5 minutes of each other for Kerberos to function.  Both systems should run some form of time synchronization system like Network Time Protocol (NTP).

Setting up /etc/krb5/krb5.conf.
Obtaining TGT for benr/admin ...
Password for benr/admin@CUDDLETECH.COM:
localhost: RPC: Rpcbind failure - RPC: Success
kinit:  no ktkt_warnd warning possible

host/zone1.cuddletech.com entry ADDED to KDC database.
host/zone1.cuddletech.com entry ADDED to keytab.

---------------------------------------------------
Setup COMPLETE.

Easy as that. Please note that the RPC failure did suceed and didn't effect the configuration.

I want to note explicitly that in the above examples the hosts involved, namely the KDC, were present in DNS. Trying to do the same configuration above without DNS is not so easy and if you need to do so I recommend avoiding kdcmgr and using the manual procedure in my previous Kerberos post.

With regard to SSH, in Solaris 10 there is no need to make any changes to accomidate SSH. You do not need to edit /etc/pam.conf or even change the /etc/ssh/sshd_config file. Setup a KDC as above, setup a client as above (a zone for instance), get a ticket ("kinit") and SSH... your rockin' and rollin'.

If you need help with your installation or want to get connected with other Solaris Kerberos fans, visit the OpenSolaris Kerberos Project.

Today is everyones favorite day of the year: SysAdmin Day! To all my fellow admins, from myself and Joyent, a very warm pat on the back and "thank you" for all that hard work that no one else acknowledges. So kick back, enjoy a cold pint, and bask in the glory that is UNIX Systems Administration!

This is very important, WarGames, the most important geek film ever made, is in theaters TONIGHT at 7:30pm. ONE SHOWING ONLY.

This film has been hugely influential for me, and many others. That awesome unbuttoned shirt over tshirt look that we emulate to this day, decades of trying to get text-to-voice to sound like the film (which was an actor reading the lines word by word in reverse), and who didn't fall in love with Ally Sheedy!!!

Don't be dumb!!! Go! Tonight! 7:30PM!!!

In my previous post about the IP Provider I got the following comment: "There is nothing unpleasant about the wonderfulness that is tcpdump! You’ll need to put a lot of work in to match tcpdump’s usefulness with Dtrace…"

That just sounds like a challenge. Bring it on! Can snoop or tcpdump do this?

root@ultra ~$ ./ip_whosent.d 
Packet sent to 192.168.100.4: 88 byte packet on behalf of ssh (PID: 1075)
Packet sent to 192.168.100.4: 88 byte packet on behalf of ssh (PID: 1075)
Packet sent to 208.67.222.222: 56 byte packet on behalf of nscd (PID: 152)
Packet sent to 208.67.222.222: 71 byte packet on behalf of nscd (PID: 152)
Packet sent to 208.67.222.222: 56 byte packet on behalf of nscd (PID: 152)
Packet sent to 72.14.207.99: 52 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 8.12.32.9: 52 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 54 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 87 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 58 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 64 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 65 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 208.67.219.230: 644 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 208.67.219.230: 637 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 72.14.207.99: 660 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 208.67.219.230: 52 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 208.67.219.230: 664 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 8.12.32.9: 48 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 72.14.207.99: 40 byte packet on behalf of firefox-bin (PID: 1944)
^C

Here is the script:

#!/usr/sbin/dtrace -qs 



ip:ip:*:send
/execname != "sched"/
{ 
        printf("Packet sent to %s: %d byte packet on behalf of %s (PID: %d)n", 
                        args[2]->ip_daddr, args[4]->ipv4_length, execname, pid ); 
}

Oh but wait....... how about a full call stack on each sent packet? Just add a new line to the above script: stack();

root@ultra ~$ ./ip_sentstack.d 
Packet sent to 72.14.207.99: 84 byte packet on behalf of ping (PID: 2020)

              ip`ip_wput_ire+0x21f5
              ip`ire_send+0x1c9
              ip`ire_add_then_send+0x2b9
              ip`ip_newroute+0xa0a
              ip`ip_output_options+0x18c7
              ip`icmp_wput+0x44a
              unix`putnext+0x22b
              genunix`strput+0x1ad
              genunix`kstrputmsg+0x261
              sockfs`sosend_dgram+0x26e
              sockfs`sotpi_sendmsg+0x4a8
              sockfs`sendit+0x160
              sockfs`sendto+0x8e
              sockfs`sendto32+0x2d
              unix`sys_syscall32+0x101

Or check out one of the examples on the IP Provider wiki page (this is almost certainly by Brendan Gregg):

# ./ipio.d
 CPU  DELTA(us)          SOURCE               DEST      INT  BYTES
   1     598913    10.1.100.123 ->   192.168.10.75  ip.tun0     68
   1         73   192.168.1.108 ->     192.168.5.1     nge0    140
   1      18325   192.168.1.108 <-     192.168.5.1     nge0    140
   1         69    10.1.100.123 <-   192.168.10.75  ip.tun0     68
   0     102921    10.1.100.123 ->   192.168.10.75  ip.tun0     20
   0         79   192.168.1.108 ->     192.168.5.1     nge0     92

Here is the script:

#!/usr/sbin/dtrace -s

#pragma D option quiet
#pragma D option switchrate=10hz

dtrace:::BEGIN
{
        printf(" %3s %10s %15s    %15s %8s %6sn", "CPU", "DELTA(us)",
            "SOURCE", "DEST", "INT", "BYTES");
        last = timestamp;
}

ip:::send
{
        this->elapsed = (timestamp - last) / 1000;
        printf(" %3d %10d %15s -> %15s %8s %6dn", cpu, this->elapsed,
            args[2]->ip_saddr, args[2]->ip_daddr, args[3]->ill_name,
            args[2]->ip_plength);
        last = timestamp;
}

ip:::receive
{
        this->elapsed = (timestamp - last) / 1000;
        printf(" %3d %10d %15s <- %15s %8s %6dn", cpu, this->elapsed,
            args[2]->ip_daddr, args[2]->ip_saddr, args[3]->ill_name,
            args[2]->ip_plength);
        last = timestamp;
}

Can DTrace decrypt IPsec ESP payloads? No. Ok, so tcpdump isn't dead yet, but the capabilities offered by DTrace are far deeper. I've got a ton of ideas more that I could put here, but don't have time atm. DTrace for the win!

Recently introduced (snv_92) is the first piece of the DTrace Network Providers, the DTrace IP Provider. Here is a taste:

root@ultra include$ dtrace -qn 'ip:ip:*:receive{ printf("Packet recieved from %s: %d byte packetn", args[2]->ip_saddr, args[4]->ipv4_length ); }'
Packet recieved from 74.125.15.85: 40 byte packet
Packet recieved from 74.125.15.85: 40 byte packet
Packet recieved from 8.11.47.20: 88 byte packet
Packet recieved from 8.11.47.20: 216 byte packet
Packet recieved from 8.11.47.20: 200 byte packet
Packet recieved from 8.11.47.20: 136 byte packet
Packet recieved from 8.11.47.20: 104 byte packet
^C

Pretty soon snoop and tcpdump will be nothing more than unpleasant memories. :)

A big thank you to the DTrace Team!!!

In this entry we'll build on our our IPsec Basics discussed last time and actually create an IPsec connection.

IPsec can be used for direct system-to-system access known as "transport mode" or to create a virtual pipeline into which everything is encrypted, known as "tunnel mode". We're going to look at transport mode, which is an excellent solution for encrypting otherwise unencrypted protocols, such as SNMPv1/2 or telnet.

When encrypting and decrypting data we need keys. This can be done using PKI certificates or IKE generated one-time keys, but in this examples for simplicity sake we'll create our own "static" keys which will be used on both ends of the connection, thus said to be "pre-shared".

Creating Keys

Using the ipsecalgs command we can see the available algorithms, including DES, 3DES, AES, Blowfish, SHA and MD5. Different alogithms require different key lengths, for instance 3DES requires a 192 bit key, whereas Blowfish can use a key anywhere from 32bits up to 448 bits.

For interoperability reasons (such as OSX or Linux), you may with to create keys that are both ASCII and hex. This is done by choosing a string and converting it to hex. To know how long a string should be, divide the number of bits required by 8, this is the number of ASCII chars you need. The hex value of that ASCII string will be double the number of ASCII chars. Using the od utility we can convert ASCII-to-hex. Here I'll create 2 keys, one for AH which is a SHA1 160bit key (20 ASCII chars) and another for ESP which is a Blowfish 256bit key (32 ASCII chars):

benr@ultra ~$ echo "my short ah password" | od -t x1
0000000 6d 79 20 73 68 6f 72 74 20 61 68 20 70 61 73 73
0000020 77 6f 72 64 0a
0000025
benr@ultra ~$ echo "this is my long blowfish esp pas" | od -t x1
0000000 74 68 69 73 20 69 73 20 6d 79 20 6c 6f 6e 67 20
0000020 62 6c 6f 77 66 69 73 68 20 65 73 70 20 70 61 73
0000040 0a
0000041

To ensure proper length, I like using a little text-rule like you see below in vi:

         1         2    2    3 3       4         5         6   6     7  
1234567890 234567890 234567890 234567890 234567890 234567890 234567890
--------------------------------------------------------------------------------------------------------------------
my short ah password
6d792073686f72742061682070617373776f7264

this is my long blowfish esp pas
74686973206973206d79206c6f6e6720626c6f77666973682065737020706173

If you don't require interoperability by knowing the ASCII equivilent, just grab a random set of hex chars (head /dev/random | od -t x1).

Now that we have a key, lets use it.

Configuring IPsec Policies

IPsec policies are rules that the IP stack uses to determine what action should be taken. Actions include:

• bypass: Do nothing, skip the remaining rules if datagram matches.
• drop: Drop if datagram matches.
• permit: Allow if datagram matches, otherwise discard. (Only for inbound datagrams.)
• ipsec: Use IPsec if the datagram matches.

As you can see, this sounds similar to a firewall rule, and to some extent can be used that way, but you ultimately find IPFilter much better suited to that task. When you plan your IPsec environment consider which rules are appropriate in which place.

IPsec policies are defined in the /etc/inet/ipsecinit.conf file, which can be loaded/reloaded using the ipsecconf command. Lets look at a sample configuration:

benr@ultra inet$ cat /etc/inet/ipsecinit.conf 
##
##  IPsec Policy File:
##

# Ignore SSH
{ lport 22 dir both } bypass { }

# IPsec Encrypt telnet Connections to 8.11.80.5
{ raddr 8.11.80.5 rport 23 } ipsec { encr_algs blowfish encr_auth_algs sha1 sa shared }

Our first policy explicitly bypasses connections in and out ("dir both", as in direction) for the local port 22 (SSH). Do I need this here? No, but I include it as an example. You can see the format, the first curly block defines the filter, the second curly block defines parameters, the keyword in between is the action.

The second policy is what we're interested in, its action is ipsec, so if the filter in the first curly block matches we'll use IPsec. "raddr" defines a remote address and "rport" defines a remote port, therefore this policy applies only to outbound connections where we're telnet'ing (port 23) to 8.11.80.5. The second curly block defines parameters for the action, in this case we define the encryption algorithm (Blowfish), encryption authentication algorithm (SHA1), and state that the Security Association is "shared". This is a full ESP connection, meaning we're encrypting and encapsulating the full packet, if we were doing AH (authentication only) we would only define "auth_algs".

Now, on the remote side of the connection (8.11.80.5) we create a similar policy, but rather than "raddr" and "rport" we use "laddr" (local address) and "lport" (local port). We could even go so far as to specify the remote address such that only the specified host would use IPsec to the node. Here's that configuration:

##  IPsec Policy File:
##

# Ignore SSH
{ lport 22 dir both } bypass { }

# IPsec Encrypt telnet Connections to 8.11.80.5
{ laddr 8.11.80.5 lport 23 } ipsec { encr_algs blowfish encr_auth_algs sha1 sa shared }

To load the new policy file you can refresh the ipsec/policy SMF service like so: svcadm refresh ipsec/policy. I recommend avoiding the ipsecconf command except to (without arguments) display the active policy configuration.

So we've defined policies that will encrypt traffic from one node to another, but we're not done yet! We need to define a Security Association that will association keys with our policy.

Creating Security Associations

Security Associations (SAs) can be manually created by either using the ipseckeys command or directly editing the /etc/inet/secret/ipseckeys file, I recommend the latter, I personally find the ipseckeys shell very intimidating.

Lets look at a sample file and then discuss it:

add esp spi 1000 src 8.15.11.17 dst 8.11.80.5 auth_alg sha1 authkey 6d792073686f72742061682070617373776f7264 encr_alg blowfish encrkey 6d792073686f72742061682070617373
add esp spi 1001 src 8.11.80.5 dst 8.15.11.17 auth_alg sha1 authkey 6d792073686f72742061682070617373776f7264 encr_alg blowfish encrkey 6d792073686f72742061682070617373

It looks more intimidating that it is. Each line is "add"ing a new static Security Association, both are for ESP. The SPI is the "Security Parameters Index", is a simple numeric value that represents the SA, nothing more, pick any value you like. The src and dst define the addresses to which this SA applies, note that you have two SA's here, one for each direction. Finally, we define the encryption and authentication algorithms and full keys.

I hope that looking at this makes it more clear how policies and SA's fit together. If the IP stack matches a datagram against a policy who's action is "ipsec", it takes the packet and looks for an SA who's address pair matches, and then uses those keys for the action encryption.

Note that if someone obtains your keys your hosed. If you pre-shared keys in this way, change the keys from time-to-time or consider using IKE which can negotiate keys (and thus SAs) on your behalf.

To apply your new SA's, flush and then load using the ipseckeys command:

$ ipseckey flush
$ ipseckey -f /etc/inet/secret/ipseckeys

Is it working? How to Test

All this is for nothing if you don't verify that the packets are actually encrypted. Using snoop, you should see packets like this:

$ snoop -d e1000g0
Using device e1000g0 (promiscuous mode)
ETHER:  ----- Ether Header -----
ETHER:  
ETHER:  Packet 1 arrived at 9:52:4.58883
ETHER:  Packet size = 90 bytes
ETHER:  Destination = xxxxxxxxxxx, 
ETHER:  Source      = xxxxxxxxxx, 
ETHER:  Ethertype = 0800 (IP)
ETHER:  
IP:   ----- IP Header -----
IP:   
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:         .... ..0. = not ECN capable transport
IP:         .... ...0 = no ECN congestion experienced
IP:   Total length = 72 bytes
IP:   Identification = 36989
IP:   Flags = 0x4
IP:         .1.. .... = do not fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 61 seconds/hops
IP:   Protocol = 50 (ESP)
IP:   Header checksum = ab9c
IP:   Source address = XXXXXXXXX
IP:   Destination address = XXXXXXXXXXXX
IP:   No options
IP:   
ESP:  ----- Encapsulating Security Payload -----
ESP:  
ESP:  SPI = 0x3e8
ESP:  Replay = 55
ESP:     ....ENCRYPTED DATA....

And there you go. You can no encrypt communication transparently in the IP stack. Its a little effort to get going, but once its running your done... just remember to rotate those keys every so often!

Why do I care about this again?

In this modern era where SSH is the standard for communication its easy to get jaded. Either you can communicate via SSH or easily create a tunnel to get the job done. But lets face it, SSH is massively overused, and in many cases SSH tunneling is just downright ghetto. With IPsec we can as easily encrypt 100 ports as 1, whereas with SSH thats very ugly. Furthermore, there are many instances in which you want a secure communications channel thats as transparent as possible, such as a network database connection that doesn't offer native encryption or perhaps an SCM or even SNMPv1/2.

While most applications today provide some type of encryption capability it surprising how few people leverage them unless they are the default. In situations where its difficult or impractical to use encryption in the application, IPsec can be a really sweet solution.