If today is dragging, might want to refresh your memory of the great OddTodd... always a pick-me-up.

Teaching him has gotten me thinking back to my own youth. I had a box of Lego's but not a lot of sets. The one that I did get was in 1988, when my parents got me perhaps my favorite (but forgotten until recently) toy of youth: the Lego Technic 8865 "Test Car".

That set was amazing. I proudly displayed it on my shelf in my room, both because of my pride in building it as well as just how outright kool it is.

Since that time Technic has grown up as much as I have. Take a look at the Technic Lego 8421 Mobile Crane:

So tempted to buy that. I already have the Ferrari F1 set, which Tamarah bought for my birthday several years ago.

But most fun of all... this week Glenn is in a one-week Lego Pre-Engineering class. For 3 hours a day they geek out and build all manner of fun stuff.

One thing I'll throw out there for Dad's... Lego has an Education dept: Lego Education. Of particular interest to Tamarah and I, is that they have a complete Homeschool Curriculum and various kits, including robotics kits, for education. A really amazing resource for parents.

Whats interesting is that John & Damon really aware of an outcry from the community, that is: "How do all these devops shops do it!!" We want to emulate them, know what tools they have, how they use them, what works, what doesn't, etc. So to facilitate just that, they started a videocast sub-series called: Open Mic.

Open Mic 1: DevOps Metrics and Dashboards at Shopzilla from dev2ops.org on Vimeo.

In the first episode, they take us into Shopzilla, where Juan Paul Ramirez shows us their tools, metrics, and talks extensively about how they got to where they are. Excellent content!

If you haven't already seen, perhaps the most popular talk this year at Velocity, "A Day in the Life of Facebook", in which the Facebook Ops team introduces us to their tools and organization.

Whats really great here is that we're not share deeper information about how we're doing things, such that we can be a community of organizations. In the past, only a handful would really share and they were always far removed from useful pratice. I really hope this trend continues.

Big thanks to John and Damon for helping fuel that fire!

The OGB is keen to promote the uptake and open development of OpenSolaris and to work on behalf of the community with Oracle, as such the OGB needs Oracle to appoint a liaison by August 16, 2010, who has the the authority to talk about the future of OpenSolaris and its interaction with the OpenSolaris community otherwise the OGB will take action at the August 23 meeting to trigger the clause in the OGB charter that will return control of the community to Oracle.

That is to say, "start talking to us or we'll just shot ourselves in the head."

I made my opinion very clear via the IRC back-channel during the call. At least my call for a liaison was added into the resolution, but I am extremely opposed to this cowardly act.

What exactly do we have to gain or Oracle to loose? All Oracle does is runs out the clock, the entire OGB resigns, and then the one little bit of control the community has is gone. What motive, other than a benevolent act to garner press attention, does Oracle have to comply? We've just made their job easier.

I once advocated this kind of self-implosion tactic back in the Sun days. The reason was to re-organize the OpenSolaris leadership to be more engaged and industry focused. That was a good idea back in the days when I had faith that Sun would "do the right thing". However, those times have past. Oracle has made it clear that it either controls things or it doesn't... there is no give and take. I don't think we can demolish the structure and believe that Oracle will re-organize in such a way as to give the community more power. It was a long shot with Sun anyway.

Frankly, imho, this is just the OGB throwing its hands in the air. The body has been useless for a long time, but only because it has chosen to be. The majority of the OGB's life its wasted by trying to restrict its own authority by endlessly debating and re-writting the constitution. Its never lead anything, and it isn't now.

But the fact that its a wet rag doesn't mean we should simply throw in the towel. A weak seat of power is better than no seat at the table.

So where do we go from here? Who knows. At this point the die is cast and OGB is putting up their last stand. Maybe Oracle gets serious and does something, but I really doubt it. Not because they can't, but because its not in their best interest. Why kill something intent on killing itself.

My only concern as this point is to not loose regular code updates and access to the bug database. Yes, the existing code is "out there", but Oracle is still the biggest contributor, 99.999 to 1. Anyone can fork at any time right now, as is, so if your going to do that why would you risk cutting off the huge contributions continuously made by Oracle?

We're in no worse a position right now than we were during the Sun days. They didn't communicate, we had no visibility or impact on the OpenSolaris distribution, etc. Don't fall into the lie that things are now "worse" than they were... they aren't. Its status quo. The difference is that the OGB is no longer composed of Sun insiders who can get a sense of control from hallway conversations and are now as blind and weak as those of us in the community always have been.

The request for a liaison is a good one... I support it. But damnit, put the gun down. We don't need to act like irrational children having a tantrum. Ultimatums rarely workout the way you hope.

The bar is lower than the original resolution was, so we'll hope for the best and see.

UPDATE: OGB Member Peter Tribble has written a blog entry about this action, recommended reading. While I disagree with the action, Peter is a great guy whom I greatly respect.

I am partly responsible. Sorry to everyone that the blog has been so quiet lately. Given that state of Solaris right now, its unclear what is dead and what is alive. It feels futile to blog about features that may never really be viable. Couple that with OpenSolaris which still hasn't delivered and the fact that many of the features that need documenting are really pretty uninteresting to me (ie: IPS/AI).

The exodus still continues. Lots of engineers have left Sun and many more are considering leaving. I'm told by folks that its not a huge problem because while the big name guys are leaving, the real down in the trench do-ers are still there and working away. But it certainly is disheartening.

The most recent news out there was that Oracle yanked HP's OEM License, so if you run Solaris on HP Prolient servers, your hosed. See? Not a lot of positive stuff for me to blog about.

Personally, I've been more interested recently with the growing 'Devops' movement and IT standards. I've spent a lot of time in ITILv3, ISO 20K/27K, CobiT 4.1, COSO, NIST SP800-53, etc, etc, etc. A whole new and interesting world to me because I came to it instead of a company hoisting it on me against my will.

I have several articles to get out for SearchDataCenter which I'll plug here, and then will start rolling new content out here in a bit.

Simply it states the following:

On the "Solaris Near Term Roadmap", they state that Solaris 10 Update 9 will come some time in 2010 focusing on platform support and Oracle product integrations.

OpenSolaris is slated for an update in the "1st half CY2010". It will be based on snv_134 and be called OpenSolaris 2010.??.

No surprises here. But given the lack of information, no surprises isn't so bad.

This does cause me to think I might have been wrong in my belief that Solaris 11 would arrive this year. I still think it's not a bad assumption, but if it doesn't come I'll be really curious as to what the heck Solaris Engineering is so busy working on.

According to SVOSUG leader, the lovely Alta Elstad, Oracle policy does not permit user groups to be run by Oracle employees or to meet on Oracle properties. Therefore, this next meeting is likely to be the last meeting in the wonderful Santa Clara Campus Agnews Mansion.

At this meeting there will be discussion about the potential future of the group, in some form or another.

If you are in the Silicon Valley please attend! Even if only to pay your respects. Meeting is at 7:30PM.

I want to pay a special honor to Alan DuBoff who started the group and grew it so successfully. I remember approaching him in 2005 I think, with the idea for a Silicon Valley User Group... instead of pitching my idea, he shared his ideas on such a group and had already started to put it together... so at least I got to be one of the first members. :) Alan's done a lot of great things for the community and we owe him a debt.

Alta Elstad took over leadership of the group when Alan left Sun (ask Alan, he has great stories, but I'll leave it at that) and has done a great job of keeping the group going, even though attendance has continued to decline. Alta is a remarkable woman and someone whom I have great esteem for. She is truly one of a kind.

Finally, to all those who've presented, participated with, and attended SVOSUG over the years, props to you all. I really hope that SVOSUG or some form of a Sun Users Group perhaps can continue onwards in the Silicon Valley.

OpenSolaris is not an open source project in the traditional sense. It can not and should not be thought of as such. If you attempt to do so you will only be frustrated and confused.

We have today seen yet another example of such a misunderstanding, when a community member wished to have a community leader punished by the OGB for failing (in his mind) to appropriately act in the best spirit of open source ideals. The specifics of the incident are irrelevant.

"OpenSolaris" as an "open source community" is little more than scaffolding around Solaris Engineering. There is a community structure, but its not really used. There is a governance and governing board, but they have no power and do nothing of significance. You can't put back (ie: commit) to ON directly and must sign a contrib agreement to even submit code, which is only actually commited at the sole discretion of Sun/Oracle employees (typically your assigned "sponsor").

Consider my analogy: scaffolding. Scaffolding allows you to navigate the exterior of a structure, to look in to the exterior windows on each level, to build things onto the exterior, or repair cracks and such, so on and so forth. Scaffolding doesn't allow you to climb through the windows. You can knock on the glass and point at things, but you aren't inside that structure... your on the outside, looking in.

So it is with OpenSolaris. Oracle owns the structure. We can't see into its inner rooms, only those with windows to the exterior. We have the ability to look in, suggest or propose or create changes or additions, but we can only hand them to others who actually have access. We can repair the outside of the structure or create custom additions, but no one on the inside of that structure needs to be involved, since its no part of the primary structure itself. We can chat with other folks on the scaffolding, share ideas and fellowship, but we're still on the outside. Those on the inside may choose to open a window to communicate with us, but only if they choose to do so.

Compare this with what you hear people say about OpenSolaris. I'll include myself. We like to think we're inside that building, along side everyone else. And we're not. We are only as empowered as the good employees of Oracle choose to let us be, through their sponsorship. And this is never guaranteed anywhere... its purely based on their good will.

If you consider the constitution of OpenSolaris, it defines within it a social organization. The conventional word would be: club. And that's what OpenSolaris is... a club. Little more, little less.

Whats important to you, my faithful reader, is to embrace the reality of this and then leverage it. If you dilute yourself, as I did for many years, into believing that you have some power or ability beyond influence, you will simply frustrate yourself and become ineffective.

What OpenSolaris provides is an amazing amount of access. The developers are available to you (CG/Project forums), the code is available to you, parts of the decision making process are available to you (ie: ARC, CR's, etc), the support structure is available to you (bug databases), and more. You are not an Oracle employee and you are not in Solaris Engineering... but you can hang out with the folks who are. You can help them, you can talk to them, you can influence them, you can even send code their way. At the end of the day, they are going to do what they and their superiors (now that they have them, har har), instruct them to... but a little constructive and friendly influence can go a long long way in this world.

The harsh reality is, that if you don't like this, you need to consider your options. Many of us have fought hard against it already and failed again and again and again, and only made enemies along the way. If you don't like it, consider starting your own project based on the code. Build onto that structure all you wish. Nexenta has done it very successfully, so has Belenix, and Blastwave too. Those projects are successful and strong, but they don't penetrate into the structure, that is, into Solaris Engineering. They stand independently.

One of the advantages of IPS will be that once a repo is added to your publishers list, you no longer thing about where the package comes from. Blastwave has traditionally provided add on packages in /opt/csw... clearly seperated from the OS. In the future, they will overlay the OS seamlessly. Therefore, consider a situation, such as above, in which the Sun version of some package isn't what you wish it to be? Simply create your own version, publish it on your repo, and then go tell the community that a better version is available, please add it to their publisher list and install it. Viva la revolution. This, is the way to effect change, far more effectively than pounding and yelling through the glass.

I really have a desire to see the anger, bitterness, and cynicism die away from our great community. Consider the venerable Sun-Net-Mangers list... perhaps the best support list ever. Resources such as this have existed in the Solaris community for almost two decades, without Sun/Oracle involvement. We, as a community, have a lot of capability with or without Sun/Oracle. Lets not loose that can-do spirit. Re-kindle that passion, leverage the OpenSolaris community for what it really is, and make good things happen.

Zabbix won't win a beauty pageant... its ugly, lets face it. But once you get beyond that its all raw power. Its agent based making it easy to extend. Monitoring something new doesn't require writting some plugin or writing a complex XML description, its just a simple single line in the agent config. It supports alerting through email, jabber, or SMS natively and is capable of doing fine grained escalations. It graphs everything, so there is no need to have Cacti or Munin or some custom RRDtool setup in addition. It is the only monitoring tool/framework that I'm aware of that natively handles IPMI, both for monitoring and actions. It has proxy capabilities for monitoring into hard to reach places (such as a small branch office) and can be multi-teered to control several sites from one point of control. The list goes on and on. Zabbix truly is the state of the art in monitoring.... and its free!

But... its not entirely the most intuitive tool to use. Several core concepts must be fully understood to be effective with Zabbix or its a big confusing mess.. power that can't be harnessed.

So a very fortunate thing happened to me. I wanted to do a large proof of concept based on Zabbix 1.8.2 but needed to refresh on some basics, when it so happens that Packet Publishing tells me they'd like me to review Zabbix 1.8 Network Monitoring. While a strange coincidence, I wasn't sure if this was a blessing or not. Most books on monitoring tools are abstract for the first 4-5 chapters, then have a really crappy installation chapter, followed by several chapters on topics that never seem to be what you actually need to do. That is to say, useless.

Thankfully, Zabbix 1.8 Network Monitoring is perhaps the most practical book I've ever read. I'd dare say that if I wrote a book on Zabbix it'd be pretty much the same. There is no lengthy flow of abstract BS, its essentially a systematic walkthrough of Zabbix from beginning to end. The first chapter is how to preform a full installation, hitting on the various options and capabilities impacted by them. Then chapter two moves onto configuration, ending with getting your first alert. So, you've got Zabbix fully installed, configured, and alerting, and thats just the first 2 chapters! Thats the way technical books should be. :)

The book is laced with screenshots and CLI examples at all turns. It really is a walkthrough, and author Rihards Olups shows you ever step. This is especially important because most of the real configuration in Zabbix is via a web interface, and its confusing to navigate unless you have a picture of what you should be seeing. Its all there, which means you don't need to frantically flip pages in front of a screen trying to figure out how he did this or that.

It has a great chapter on reporting and another on graphing. I was really pleased that these we're lumped together or breezed over. They are key capabilities and are given plenty of space.

There is also a great chapter on troubleshooting (appendix actually), which will help you in any areas that cause you to stumble initially.

If you want to get going with Zabbix and don't want to waste time, this book will save you days. As I mentioned before, Zabbix is configured differently that the traditional tools out there, so you need a keen understanding of core concepts, such as "Hosts", "Items", and "Templates". You can piece it together from the (poor) Zabbix manual and experimentation, your you can just buy the book and get going.

Now.... that said, there are only two shortcomings to the book.

The first is that it can, at times, be a little too fluff-less. There are times a little up-front explanation could have been enhanced before just jumping into it and explaining as he goes along. But that will really be dependent upon your learning style. If you want to get Zabbix going, its great, but if you just want to read about it, its not so easy to just jumping in and out of the text to understand concepts. Again, its a walkthrough, not an overview (such as O'Reilly books, which tell you a lot about it but typically not enough about how to actually make it happen.)

My second nitpick is that distributed monitoring isn't really explored fully. There is a chapter on monitoring remote sites using proxies, but an additional chapter on complex mult-site installation featuring not just proxies, but also parent-child servers, would have been very welcome. I'm not sure if it was left out because of its complexity or some other reason. Perhaps he's setting himself up for a sequel covering advanced topics. May have even been due to the length, the book is 428 pages and is really dense material.

The book runs $45 which is pretty standard. The PDF ebook is $33, which is a little steep imho, but like I said, this book really will save you days... so it'll pay for itself in an hour or two. Incidentally, it looks great on my iPad. :) See the full list of contents and get a sample chapter here: Zabbix 1.8 Network Monitoring. Buy it direct or you can pick up direct from the publisher or at Amazon, or if you're in the Silicon Valley don't forget to help out the brothers at Digital Guru.

I'll throw out a personal thank you to Rihards Olups for the way he wrote this book. His approach was fantastic, and as a technical writer myself I really like the way he tackled it. It takes a special mindset to write so clearly and concisely, and I really appreciate that.

This post applies only to Nevada based installs post-snv_100. Or, more specifically, following the integration of "PSARC/2007/078 Hostid for X86 systems". For information about the sysinit module and how things worked on X86 prior to snv_100, please see the excellent post The dark side of the source - hostids by Frank Hofmann.

So the hostid is generated during installation and stored in /etc/hostid. This file contains 2 lines, a comment line and the encoded hostid. A valid hostid is 7 hex chars or less, padded to 8 hex numbers. Therefore, 0x0fffffff (zero followed by 7 f's) is valid, whereas 0xffffffff (zero followed by 8 f's) is not. To be clear again, these are hex numbers, not ASCII characters.

To set it we first edit /etc/hostid with vi to remove the second line, such that only the comment on line 1 remains. (Backup the hostid file if you think you might want it again later, or if your just playing around). Then we use a bit of PERL (based on an extraction from the method of the hostid service) to add in the encoded hostid. To make it effective update the boot archive (always do this manually! don't assume reboot will do it!) and reboot:

# echo "0x0ddb00b5" | /usr/perl5/bin/perl -e 'print("\""); while(<STDIN>){chop;tr/!-~/P-~!-O/;print $_;} print("\"\n"); exit 0;' >> /etc/hostid
# sync;sync
# bootadm update-archive
updating /platform/i86pc/amd64/boot_archive
updating /platform/i86pc/boot_archive
# sync;sync;reboot

When the box comes up you'll have your new hostid!

[root@am1-ja710-36 ~]# hostid
0ddb00b5

If for some reason you get a hostid of 00000000 or a warning that the hostid is invalid, you either got the value too large or encoded it wrong in /etc/hostid. Check that you pasted the code above properly and try it again.

Please note, I assume you're not doing this to be naughty. I only spent time to figure this out because I had several systems which for some reason got stuck with 00000000 hostid's (likely because something went wrong during jumpstart).

Hold on just one second! Lets get this clear first, IPS is all about network package repositories. There is no "local package" file format. Therefore, its a little confusing to say "create a package". So lets be crystal clear: we are going to send a bunch of files and meta-data as a package transaction to a package depot server. Fundamental concept.... moving on.

First things first, you can start an IPS Depot ("repository server", but the daemon is actually pkg.depotd) using the pkg/server SMF service, or start it manually in a shell (so you can watch it work, useful for your first time) like so:

root@quatro ~$  /usr/lib/pkg.depotd -d /export/pkg -p 80 --set-property publisher.prefix=cuddletech
[21/May/2010:00:40:19] INDEX Search Available
[21/May/2010:00:40:19] ENGINE Listening for SIGHUP.
[21/May/2010:00:40:19] ENGINE Listening for SIGTERM.
[21/May/2010:00:40:19] ENGINE Listening for SIGUSR1.
[21/May/2010:00:40:19] ENGINE Bus STARTING
[21/May/2010:00:40:19] ENGINE Started monitor thread '_TimeoutMonitor'.
[21/May/2010:00:40:19] ENGINE Serving on 0.0.0.0:80
[21/May/2010:00:40:19] ENGINE Bus STARTED
..

If you point your browser at http://localhost/ you'll see a web interface that looks identical to pkg.opensolaris.org. It has 0 packages. So lets add one.

There are many ways to add a package. The most popular way is to create a SysV package and then "convert it"... however I believe that to be extremely hypocritical, given that IPS is all about putting SysV packages to death. So we're going to avoid that method and assume that we're going to compile something from source and then package it.

I've decided to create a package for rsyslog. So I download the latest source and start to build it:

benr@quatro rsyslog-5.5.5$ ./configure --prefix=/usr
...
benr@quatro rsyslog-5.5.5$ gmake
...
benr@quatro rsyslog-5.5.5$ mkdir -p /tmp/staging_root
benr@quatro rsyslog-5.5.5$ DESTDIR=/tmp/staging_root gmake install
.....

Notice here that I built it to install into /usr, but before installing it I did a little switcharoo by over-riding the DESTDIR variable. The result is that gmake install installs into /tmp/staging_root, with all the links and configs as though it was in /usr.

Now that I've got my faked out installation, lets use pkgsend to import that directory structure:

benr@quatro rsyslog-5.5.5$ pkgsend open rsyslog@5.5.6
export PKG_TRANS_ID=1274430097_pkg%3A%2F%2Fcuddletech%2Frsyslog%405.5.6%2C5.11%3A20100521T082137Z
benr@quatro rsyslog-5.5.5$ export PKG_TRANS_ID=1274430097_pkg%3A%2F%2Fcuddletech%2Frsyslog%405.5.6%2C5.11%3A20100521T082137Z

benr@quatro rsyslog-5.5.5$ pkgsend import /tmp/staging_root

benr@quatro rsyslog-5.5.5$ pkgsend close
PUBLISHED
pkg://cuddletech/rsyslog@5.5.6,5.11:20100521T082137Z

And thats it. I first used pkgsend to open a new transaction to the depot server. This starts a new package. It returns to me a transaction ID which I export as a environmental variable used by future commands. Now I import the staging_root that I installed our tool into. Once done, I just close the transaction, and its done.

If I wanted to add meta-data, I could insert the following before closing the transaction:

$ pkgsend add set name=pkg.summary value="Rsyslog is an enhanced multi-threaded syslogd with a focus on security and reliability"
$ pkgsend add set name=pkg.description value="Rsyslog is an enhanced multi-threaded syslogd with a focus on security and reliability"
$ pkgsend add set name=description value="Rsyslog is an enhanced multi-threaded syslogd with a focus on security and reliability"
$ pkgsend add set name=packager value="Ben Rockwood (benr@cuddletech.com)"

Now go look at the repo in your browser again. Sure enough its there.

To install the new package, add your local repo to your IPS repo list and then install it:

benr@quatro ~$ pfexec pkg set-publisher -g http://localhost/ cuddletech
benr@quatro ~$ pkg publisher
PUBLISHER                             TYPE     STATUS   URI
opensolaris.org          (preferred)  origin   online   http://pkg.opensolaris.org/dev/
contrib.opensolaris.org               origin   online   http://pkg.opensolaris.org/contrib/
cuddletech                            origin   online   http://localhost/

benr@quatro ~$ pfexec pkg install rsyslog


DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1       33/33      0.7/0.7

PHASE                                        ACTIONS
Install Phase                                  41/41 

Pretty easy right? Creating packages can be really hard if you add all the files manually and nit-pick it, but personally I find this "import /some/dir" to work just fine. It also works on tar files! :)

Good luck with IPS.

For some time now you've had the option to download CD/DVD images of OpenSolaris (or SX:CE, or whatever), the source and tools themselves, or BFU images. When you combine BFU with the ACR (Automatic Conflict Resolution) tool, you could keep ugprading from build to build for a while before the works got so gummed up that you needed to reinstall.

You'll notice that snv_139 and snv_140 didn't include BFU's, making me wonder if this was a temporary problem or a permanent change.

Turns out BFU's are, essentially, gone for good. The source, supposedly, can still produce them, but they will no longer be provided with the source releases and very soon they won't be possible at all.

In March, Liane Praza announced in a Flag Day report, among other things, that:

"BFU is still available (though ACR is not), but pkg image-update is the preferred testing method. Support for BFU is expected to be removed in a few builds."

That was March, "in a few builds" is now.

I'm informed by James McPherson that following the integration of the IPS packaging changes, the onu ("Os/Net Update") tool will be replacing BFU, preforming essentially the same function.

If you want to keep up on all the deep changes that are happening to ON because of IPS, please subscribe to the on-ips-dev mailing list.

I hope you can start to understand just how major a change IPS really is... its not a topical cream for Solaris... its a heart transplant.