Ian Murdock's distro formly known as "Indiana" will be birthed as "OpenSolaris" in less than a week, being debuted at CommunityOne on May 5th. This will be a major landmark even in the history of Solaris, right up there with the BSD-to-SysV transition and release of the code. There is no talk at Sun regarding Solaris 11, when pushed the only quote I get is "over my dead body", apparently coming from high within the organization. While no one will clarify on the situation, the current vibe seems to be that Solaris 10 will be with us for a very long time, in update purgatory, while the future revolves around the OpenSolaris distribution. Ultimately the decision will probly be made by Sun's attempts to get ISV's behind OpenSolaris... but this is only my hunch, I'll continue pushing Sun to clarify the roadmap, perhaps at CommunityOne will learn more.

The existing open source community is in my eyes at a current all time low. A lot of effort turned inward, likely because of a feeling that the community was slowing down the process. There are many who are pleased to see arguments and heated mail exchanges die off, but those things are a sign of passion. Internal staff seems to be reorganized monthly, and many of the people being replaced are being done so with less transparent replacements... exactly how many people are associated with Tonic, the community or anything related directly to "OpenSolaris" becomes a deeper mystery all the time. The lack of leadership is becoming ever more evident. High level the project has continued to close inward. The board is dominated by Sun and Simon Phipps agenda for the future. Even Solaris related blog entries from non-Sun employees has been steadily on the decline. Sun's dedicated to Open Source... they just don't seem to care much about including the community, who they seem to be highly mistrusting of.

The project is still open and moving, do not be mislead, projects like the Image Packaging System (IPS) have been model open development efforts for all to model and new OpenSolaris User Groups are continuing to spin up around the globe. Projects integrating into Nevada as the basis of Indiana, such as the Image Package System and the new installer, Caiman, will please users who previously tossed Solaris DVD's out the window in frustration. /usr/gnu has been added to bring even more GNU software to the system out of the box.

... Its time for a community reset. With the release of the OpenSolaris distro the last bits of the community started by Andy Tucker and Claire Giordano will be, in my view, gone away. The experiment in community official ended and replaced. Rather than the community being joint owners of Solaris it will be affirmed that Sun is firmly staying at the helm and we're free to board the train and pitch in if we choose. Those of us fighting against the tide are now presented with a choice... give up and try to re-invent our roles in the "new reality" or continue to fight the inevitable like so many of those in our community who still whine if an OS doesn't run on an i386 with 512K of RAM looking like a senile prick.

I am, first and foremost, a self proclaimed Solaris zealot. I love Solaris and frankly, nothing Sun does is going to change that. I find myself in the uncomfortable position of looking like a negative prick in the face of progress and I f**king hate it. I believed in something and I still do... most of my comrades have already given up or moved on, and I'm feeling awfully lonely. At this point, however, I have no interest in being a retarded martyr declaring my frustration with the whole situation and proclaiming an exodus to Linux... Solaris rules, long live Solaris.

So as for myself, I'm dropping the negativity and caving into the future. I'll still hold Sun responsible for releasing sub-par hardware (stop making me buy Dell!!!) but the brave new world of Solaris I shall embrace.

There is much yet to do. Solaris adoption is still low. There are still a lot of people who don't know that Solaris has been open sourced. Solaris innovations like DTrace, ZFS, and hopefully SMF, will increasingly see adoption on other platforms, especially with Jonathan constantly teasing licensing changes. I'm hearing more and more positions for Solaris 10+ sysadmins that are going unfilled because there "just aren't Solaris people out there". I recently taught a Solaris 10 training session because the training centers can't find competent Solaris trainers outside of Sun. The need for Solaris evangelists and mentors has never been greater. The OpenSolaris distribution is going to have a huge impact on Solaris adoption, we've got to ride that wave, no matter how big or small it is.

For those of us watching the day-to-day development of Nevada this new distro doesn't look like too big a change, but lets remember that to the majority of "Solaris SysAdmins" out there running Solaris8 on older SPARC systems this is a mindbogglingly revolutionary rebirth of the platform! We have to get out both the message that Solaris is still Solaris, at the same time that we get out the message about modernization. This is no small task, but it can be done, and I intend to fight to bring our wayward brethren into the light.

I raise a pint, my friends, to progress and pushing onward and upward. To being part of the solution and not the problem. To knowing when to fight and when to ride.

A frequent point of contention among sysadmins, the presence of a /root home directory for the root user is now settled by PSARC/2003/039 Alternate home directory for root user integrated into snv_87. Here is the new default passwd file, get used to it:

     1 root:x:0:0:Super-User:/root:/sbin/sh
      2 daemon:x:1:1::/:
      3 bin:x:2:2::/usr/bin:
      4 sys:x:3:3::/:
      5 adm:x:4:4:Admin:/var/adm:
      6 lp:x:71:8:Line Printer Admin:/usr/spool/lp:
      7 uucp:x:5:5:uucp Admin:/usr/lib/uucp:
      8 nuucp:x:9:9:uucp Admin:/var/spool/uucppublic:/usr/lib/uucp/uucico
      9 dladm:x:15:3:Datalink Admin:/:
     10 smmsp:x:25:25:SendMail Message Submission Program:/:
     11 listen:x:37:4:Network Admin:/usr/net/nls:
     12 gdm:x:50:50:GDM Reserved UID:/:
     13 mysql:x:70:70:MySQL Reserved UID:/:
     14 webservd:x:80:80:WebServer Reserved UID:/:
     15 postgres:x:90:90:PostgreSQL Reserved UID:/:/usr/bin/pfksh
     16 svctag:x:95:12:Service Tag UID:/:
     17 nobody:x:60001:60001:NFS Anonymous Access User:/:
     18 noaccess:x:60002:60002:No Access User:/:
     19 nobody4:x:65534:65534:SunOS 4.x NFS Anonymous Access User:/:

The default shell is still Borne, for now.

For anyone unfamiliar with the classic objection to the presence of /root, here's the recap. The pro-/root argument is that the root user needs a place to dump files other than /export or /tmp, and the presence of dot directories (eg: /.ssh) don't belong in the root (/) directory. The anti-/root argument is that the root user should not be doing these types of activities, rather a real user with RBAC (or sudo if you prefer) privs should be utilized for this purpose relegating the root user to basic system function and recovery only.

The reason for implementing this now is the flood of changes, many long debated, being integrated for the release of OpenSolaris (read: Indiana) on Monday (May 5th). If you don't normally browse the per-build changelogs I recommend you take some time to catch up: Index of Nevada Build Updates. Bookmark it, remember, new builds every 2 weeks, keep an eye on those changelogs to stay up on the changing landscape of Solaris.

If 10 years ago someone said "One day your wife will carry an extra hard drive in her purse", I'd have rolled my eyes. On a recent trip to pick up a hard drive (to replace the piece of crap that died in my MacBook Pro; so far every Apple laptop we've owned has had an OEM drive die) I saw, to my amazement, this:

CaseLogic, the folks that made those CD cases we all used to have in our cars, is now making neoprene sleeves for 2.5" hard drive enclosures. This is telling to me... CaseLogic decided that there was enough of a market to start peddling these. This says something about modern storage, says something about the expected reliability and mobility of spinning storage, and says something about the capacity of the ever more affordable flash storage in USB keys and such. And, the strange thing is, I just had to buy one.

But wait there's more! The wall of 3.5" enclosures had been pushed aside by a giant selection of 2.5" enclosures, most of them powered by the USB line alone, no need for an exteral DC plug. And in the corner of the rack was this interesting toy:

This is a Thermaltake BlacX HDD Docking Station, it accomidates 2.5" and 3.5" SATA drives.... like a damned Nintendo cartridge! And, the really funny thing is you'll find yourself blowing dust off the SATA paddle before inserting... oh the memories.

Most geeks, like myself, probly have a growing stack of SATA drives that aren't terribly old but have fallen by the way side as storage capacities have sky rocketed and prices plummeted in the last 3 years. Sure, there are lots of snazy USB/Firewire/eSATA enclosures out there, but generally the drives aren't worth it... but no longer is this a problem! Your old hard drives are now a very easy to use removable media for all your backup or temporary storage needs, no adapters or sleds required, just dust it off and slide it into the dock.

These two things, combined with the fact that your grandma's new Dell is probly going to have a 1TB drive, something that didn't seem possible in a 3.5" form factor just a couple of years ago, and some hope that aerial density will provide 2.5" with capacities well beyond 300GB in the future, as well as the coming wave of SSD solutions.... storage is looking to be at the peak of a wave thats going to crash out a lot of interesting things in the next couple of years.

Of course, what concerns me is that while bus speeds increase and capacities grow, throughput in real world situations is still low. 30MB/s is still considered pretty good in real-world usage because those poor little heads can only move so fast. Tiered storage combined with RAID is interesting considering the increases in arial density because the outer cylinders contain so much data, but with COW filesystems growing such as ZFS the data is increasingly spread around the platters if left unchecked which leads to slower transfer rates outside of the benchmarks. Bigger buffers can help, but in random workloads prefetch doesn't help as the drive doesn't know what sector to prefetch.

It wasn't log ago that I was begging a storage vendor to keep sending me 72GB drive because the rebuild times for a failed 167GB drive scared me. Gigabit speed networks increase the utilization of storage over the network, but again, those drive heads can only move so fast. I'm really interested to see what comes in the next couple years to try and catch up the random throughput of drives with the capacities. Will SSD be the solution or can spinning media vendors pull a rabbit out of their hats? Unless they do, my hunch is that in 10 years enterprise systems will be shipping with SAS SSD drives and relegate spinning media to secondary storage.

Any way you look at it, some kool stuff is coming; storage geeks stay vigilant!

Announced today on the OpenSolaris Announce list was this snazzy tool: easySMF. Its a simple to use web form that creates SMF Manifests that you can easily plug into your system (svccfg import my_manifest.xml). This is snazzy indeed and I'm very curious to know who wrote it, but its hosted by the OpenSolaris Hispano Project, nice work guys!

SMF Manifests are like Atari games, takes a couple tries to get the hang of things but you'll mellow into a groove, and the more you play around the more fun things you'll be able to do. Never the less, getting started can be a hill a little higher than you expect, easySMF should be a great tool to get your apps up quick and help you gain some confidence in manifest tweeking.

Last week amongst the fun at the MySQL Users Conference I woke up early enough one day to watch the keynote at which Jonathan Schwartz presented a 30 minute talk about the aquisition, Open Source software in general, and Sun's principles. I highly recommend watching this keynotes replay and I kindly ask that you resist the urge to flip through it, but rather get a cup of coffee (I need Illy as a sponsor) and set aside 30 minutes to watch it end-to-end.

If you can not see this embedded video above, click here: Jonathan Schwartz Keynote at MySQL

There are some very important things in this presentation...

Most strikingly is a comment regarding ZFS at the 15:00 mark, "its under a CDDL license but rumor has it there might be a change in its future." It would be hard to imagine ZFS getting re-licensed and not DTrace... and there has been GPL talk around Nevada in the past. We know OpenSolaris is coming in May. Draw your own conclusions.

The vibe in general here is that Sun is committed to Open Source software, more than ever before, and has big plans in the future. In many ways there are amusing parallel between the Open Systems or Open Standards talks that Scott McNealy used to do and Jonathan's Open Source talks.

.... after much debate, I'm not going to editorialize this anymore, but I recommend watching it to any Sun supporters specifically as they relate to the free software developers external to Sun.

I hate to be critical, but some times something has to be said so folks know we're paying attention.

New systems are available from Sun, the Sunfire X4140 and X4440... behold:

These systems are AMD Dual Core and presumably NVIDIA chipsets (an earlier comment by a read states its 4 GE ports are NGE). There is no press release that I can find! Joerg Moellenkamp blogs about it here, and the Sun "On the Record" blog mentioned it as well. I suppose this means I should dump the Sun.com RSS feed and instead use the "On The Record" feed instead.

Here's what I find really disturbing about the X4140 and X4440 releases. These systems were clearly designed as Quad-Core AMD systems to compliment the existing Intel Quad-Core systems of similar design. This is further confirmed by the many statements on the product page about being upgradable to Quad Core. The question is... why release them now? Why not wait untill the Quad Cores are available and release then? I just don't get it.

Another PR oddity was this: Sun Microsystems Announces MySQL 5.1, which includes this amusing phrase: "today announced the pending general availability of MySQL 5.1". Must. Make. MySQL Press Release. Why?

There are more examples but I'll stick with these two. The point is this... Sun has gone from a company that released in volume at the announcement, to a company that released prematurely (eg, the LOM debacle on the X4150's), to releasing software and systems pre-emptively!

What is going on here!? I don't want to slam Sun, heaven knows I'm still a fanboy, but honestly, does anyone know whats happening here? (I'll follow this entry with a ra-ra pro-Sun one to balance my SMI karma. I hate looking negative, but damnit people.)

I solved a problem today that I think others might find interesting and helpful, both in process and solution. I've posted it in the Joyent company blog: Joyeur: DTrace, MySQL, Ganglia, and Digging for Solutions. Check it out.

I presented my "MySQL & DTrace" talk at the MySQL Users Conf this morning. The presentation is more terse than I wanted because of the 45 minute per session slots. The presentation is focused on a quick primer on DTrace, followed by examples of tracing MySQL using the PID provider rather than waiting for static probes, as well as some notes about static probes and the current status on integration.

Download the PDF slides here. Keep your eye on the Joyuer, Joyent blog in the next week as both myself and Derek Crudgington will be talk more about MySQL & DTrace.

For those who attended, I hope it was useful, feedback so far has been very positive. I'll repeat here my special thanks to Joyent MySQL guru Derek Crudgington for his help, please check out his blog Blog of Derek Crudgington @ hell.jedicoder.net. Much more will be coming from the two of us soon, particularly in his DTrace MySQL Toolkit.

If you go get a cup of coffee at Starbucks you'll get it in this cup:

Its actually worse than the NetApp logo! Just horrible. But fear not, its only promotional. Apparently the idea is to go with something different for a short time to grab attention from people who are so used to the Starbucks logo that they mentally ignore it.

Whats with the continued rash of horrible logos?

In general I like the idea of distributed SCMs... but I've found that actually most of the time I use an SCM as much to distribute and archive as I do to actually do revision control. Over the last year or so I've really tried to become a big Hg fan, and honestly, its just one disappointment after another. For small local things its great, diffs are easy to create, etc. But pushing or pulling repositories takes an eternity, which is compounded by the atomic nature in which you don't see any files until its done. Trying to use a distributed SCM in a centralized way is a massive pita. And while I love the idea of pushing Hg changesets around rather than diff's, the reality is that unless all your developers are also using Hg you revert back to diff patches.

So I'm looking for feedback from others in this regard. Have you too been w00d by the advantages of a distributed SCM and just never become a fan, like myself? Have you made the switch and then reverted back to SVN/CVS/etc? Or, on the otherhand, have you been so wow'ed by Hg that you just won't go back? Post a comment and share your opinion.

BTW, I'm really interested in Hg experiences... please avoid with all your strength a "Hg sucks, use git" comment. :)

Every time a new application integrates static DTrace probes there is much rejoicing... and I'm glad for that, but I'm afraid a lot of people think that DTrace isn't useful in userland without probes being baked into the code. This couldn't be further from the truth, in fact, putting static probes in your code is, 90% of the time, just a convenience.

If you don't believe me, try this script on any application you like:

#!/usr/sbin/dtrace -s

#pragma D option flowindent

pid$target:a.out::entry
{
}
pid$target:a.out::return
{
}

Make that executable and run it like this:

root@ultra tmp$ ./pidflow.d -p `pgrep e16`
dtrace: script './pidflow.d' matched 3040 probes
CPU FUNCTION                                 
  1  -> EDebug                                
  1  <- EDebug                                
  1  -> TimersRun                             
  1    -> EDebug                              
  1    <- EDebug                              
  1    -> MenusTimeout                        
  1      -> ecore_list_goto_first             
  1      <- ecore_list_goto_first             
  1      -> _ecore_list_goto_first            
  1      <- _ecore_list_goto_first            
  1      -> ecore_list_next                   
  1      <- ecore_list_next                   
  1      -> _ecore_list_next   
...

In the above example your seeing functions within the Enlightenment DR16 window manager (best WM ever!!!!). There are no probes within E, I compiled this build months ago, there is no trickery involved. You could use this script on Apache2, MySQL, Oracle, you name it.

But how does this work? How does DTrace know what the internal functions are? Symbols my friend. If you've ever used nm you'll know that symbols (of which many are functions) are visible, and DTrace uses these as probe functions. Each function has an entry probe and return probe, plus a variety of offset probes although typically entry/return is all you care about.

The example above uses the PID provider (in the form "pid123", or in our case "pid$target" where $target is the PID fed in via the command line -p argument), and the "a.out" module. The module name "a.out" is actually a synonym for the binary name, so I could have used "e16" instead of "a.out", but I use the latter to be generic. If you we didn't specify the module name it would trace not only the e16 calls but also those of its shared libraries! This is handy if you want to watch, in the case of a window manger like Enlightenment, calls go in and out of E, X, or shared libraries. The probefunc isn't specified here, so we see them all. When the braces (action) are empty DTrace simply traces on probefunc name, and using the flowindent option DTrace formats the output with indents to indicate entry and return. (Prop's to Derek Crudgington for turning me into a flowindent fanatic.) Super handy!!!

But wait, there's more! DTrace can look inside arguments passed in to these functions! For simple arguments like a char pointer you can examine it using a copyin operation like "copyinstr(arg1)", assuming that arg1 (second argument to the function) is a char array. Things only get tricky when you examine structures, because DTrace doesn't know what the structure looks like, this is where "Translators" come in and you'll find good documentation on them in the Solaris Dynamic Tracing Guide.

The point here is, I hope, obvious. While embedded static probes in code is nice and definitely convenient, its by no means whatsoever a necessity. Scripts like the simple one above will amaze your friends and even if you don't care to learn DTrace yourself, run it on a process that you think is idle... you'll be surprised how many apps that "aren't doing anything" in fact are.

Happy tracing!

This is a call for all Solaris Evangelists in the SF Bay Area... LUGradio SF is happening this weekend at the Metreon. Sun/OpenSolaris will have a booth there, put together by the amazing Jesse Silver. I had planned to help man the booth on Saturday but other plans came up and I am forced to withdraw. If your an evangelist who wants to meet kool people and share the news about OpenSolaris I'd encourage you to go to the OpenSolaris Advocacy Community Group, signup on the list (if you aren't already) and send a mail saying you want to help out. I already requested that an Ultra40 be present for demo's (you can take the side panel off to show in innards, always a crowd pleaser).

If you don't consider yourself an evangelist, but like to see kool stuff and meet kool people I recommend going to the show as a participant... its only $10 to get in, and just about all the Linux heavy-weights are gonna be there. It'll be a good time for all... and don't forget to wear an OpenSolaris shirt. :)