The Future of Init, Part I: SMF

18 Oct '05 - 19:03 by benr

Solaris 10 introduced more features and functionality into the UNIX world than any release by any vendor in the last 20 years. The most ground-breaking of these included DTrace, Zones and Predictive Self Healing. Whats interesting is that you never heard SMF mentioned in the marketing material, the reason for that is that "Predictive Self Healing" is the combination of SMF and FMA (the Fault Management Architecture), which work closely together in a unified fashion to manage software (SMF) and hardware (FMA) in an integrated fashion.

Of the three init systems, SMF is without a doubt the most robust, polished, and mature. Its enabled out of the box and aims to completely replace the existing RC scripts in the near future, despite the fact that RC scripts will continue to be supported if your so inclined.

On System V platforms, like Linux and Solaris, the kernel calls init when its finished loading. Init then looks at /etc/inittab and starts processing. Here is what the Solaris9 inittab looked like:

ap::sysinit:/sbin/autopush -f /etc/iu.ap
ap::sysinit:/sbin/soconfig -f /etc/sock2path
fs::sysinit:/sbin/rcS sysinit           >/dev/msglog 2<>/dev/msglog </dev/console
p3:s1234:powerfail:/usr/sbin/shutdown -y -i5 -g0 >/dev/msglog 2<>/dev/msglog
sS:s:wait:/sbin/rcS                     >/dev/msglog 2<>/dev/msglog </dev/console
s0:0:wait:/sbin/rc0                     >/dev/msglog 2<>/dev/msglog </dev/console
s1:1:respawn:/sbin/rc1                  >/dev/msglog 2<>/dev/msglog </dev/console
s2:23:wait:/sbin/rc2                    >/dev/msglog 2<>/dev/msglog </dev/console
s3:3:wait:/sbin/rc3                     >/dev/msglog 2<>/dev/msglog </dev/console
s5:5:wait:/sbin/rc5                     >/dev/msglog 2<>/dev/msglog </dev/console
s6:6:wait:/sbin/rc6                     >/dev/msglog 2<>/dev/msglog </dev/console
fw:0:wait:/sbin/uadmin 2 0              >/dev/msglog 2<>/dev/msglog </dev/console
of:5:wait:/sbin/uadmin 2 6              >/dev/msglog 2<>/dev/msglog </dev/console
rb:6:wait:/sbin/uadmin 2 1              >/dev/msglog 2<>/dev/msglog </dev/console
sc:234:respawn:/usr/lib/saf/sac -t 300
co:234:respawn:/usr/lib/saf/ttymon -g -h -p "`uname -n` console login: " -T sun -d /dev/console -l console -m ldterm,ttcompat

Look at inittab again, but this time from Solaris10:

ap::sysinit:/sbin/autopush -f /etc/iu.ap
sp::sysinit:/sbin/soconfig -f /etc/sock2path
smf::sysinit:/lib/svc/bin/svc.startd    >/dev/msglog 2<>/dev/msglog </dev/console
p3:s1234:powerfail:/usr/sbin/shutdown -y -i5 -g0 >/dev/msglog 2<>/dev/msglog

Just a little cleaner.

SMF manages services. Each service is configured by importing an XML manifest which specifies the various parameters of the service, such as its name, dependancies, how to start, stop, and restart the service, meta-data detailing the location of man pages and help documention, and more. Lets look at a simple SMF Manifest:

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<!-- OpenLDAP Manifest: Ben Rockwood - -->

<service_bundle type='manifest' name='openldap'>

        <create_default_instance enabled='true'/>

        <dependency name='config-file'
                <service_fmri value='file:///usr/local/etc/openldap/slapd.conf'/>

        <dependency name='loopback'
                <service_fmri value='svc:/network/loopback:default'/>

        <dependency name='physical'
                <service_fmri value='svc:/network/physical:default'/>

        <dependency name='fs-local'
                <service_fmri value='svc:/system/filesystem/local'/>

                timeout_seconds='60' />

                timeout_seconds='60' />

                timeout_seconds='60' />

        <stability value='Unstable' />

                        <loctext xml:lang='C'>OpenLDAP slapd</loctext>
                        <manpage title='slapd' section='8C' />
                        <doc_link name=''
                                uri='' />

The various directives in the manifest are pretty self explanitory, with the exception of "Stability", which doesn't indicate the stability of the service, but the stability of the API. Its a Solaris thing that you'll notice in all Sun man pages.

Sometimes a single line isn't enough for starting your service, and so our manifests can call methods, shell scripts that carry out actions for the 3 states above: start, stop, and restart. You could use one script and use an argument based on which state your refering to, in which case the method looks identical to a traditional init.rc script, or you could use a seperate script for each state, its all up to you.

Once we've created a manifest we can import in into SMF by using the svccfg tool. We can use svccfg to import new manifests, export existing manifests (dump back into XML), validate manifests, inventory them, etc. We can also use svccfg to manipulate manifests in any way we choose without having to export/import it. SMF keeps all manifest and associated information in its own private internal database, so once you've imported the manifest you can delete it, its not ever used again. The same, however, isn't true for methods, which obviously have to be someplace on disk to be executed.

Typically, manifests are stored in /var/svc/manifest (for reference) and methods are stored in /lib/svc/method.

Managing SMF services is easy using two tools: svcadm to control a service, and svcs to examine the state of a service. Here's and example:

$ svcs | grep nis
online         Oct_05   svc:/network/nis/update:default
online         Oct_05   svc:/network/nis/xfr:default
online         Oct_05   svc:/network/nis/server:default
online         Oct_05   svc:/network/nis/passwd:default
online         Oct_05   svc:/network/nis/client:default
$ svcadm disable svc:/network/nis/xfr:default
$ svcs *nis*
STATE          STIME    FMRI
disabled       Oct_05   svc:/network/rpc/nisplus:default
disabled       14:37:17 svc:/network/nis/xfr:default
online         Oct_05   svc:/network/nis/update:default
online         Oct_05   svc:/network/nis/server:default
online         Oct_05   svc:/network/nis/passwd:default

Both tools are flexable and provide a range of funcationlity while remaining easy to use. Using svcadm services can be put in any of the following states: online, disabled, or maintance. Additionally, services can be in an enabled (started but not up transition state) or offline (stopped but not disabled transition state). We can also restart (stop and start) and refresh (re-read the configuration, such as a BIND zone relead) services.

SMF uses a "contract" psuedo-filesystem (/system/contract") that is used to monitor the state of any given service. If the service stops for some reason, SMF will restart it. If a service is started but stops (such as a segv or error) on its own it will restart as many as three times, at which point it places the service in a maintance or offline state and logs the problem. Finding out why a service stopped running or failed to start is really easy. Just use the specia "-x" argument to svcs to list an explantation of what services aren't running and why:

$ svcadm enable svc:/network/dns/server:default
$ svcs dns/server
STATE          STIME    FMRI
offline        14:51:07 svc:/network/dns/server:default
$ svcs -x
svc:/network/dns/server:default (?)
 State: offline since Wed Oct 19 14:51:07 2005
Reason: Dependency file://localhost/etc/named.conf is absent.
   See: named(1M)
Impact: This service is not running.

svcs -x shows us what isn't running and why, as well as a URL that can offer an explanation as to what went wrong. (That url above is real, check it out!) By default only problematic services are shown, but if you pair "-a" with "-x" you can see the problem service as well as all the dependant services that can't start because the dependancy is down.

Another advantage is that SMF integrates with the Fault Management Architecture (FMA), which is why all services have those svc:// URI's. In this way, if an application has dependancies on a hardware device or vice versa the two facilities can work together to provide a true Predicitive-Self Healing solution.

It should be noted that a nice thing about enabling or disabling services in SMF is that they are permanant. When you enable a service, it will start at boot time. This solves the classic problem of starting services while the system is running but forgetting to make the change perminant. Likewise, if a service is disabled for some reason, it will not be started at boot time.

One last thing before we leave the discussion of SMF. Each service can have a variety of dependancies, on a file (such as a config file) or on another service. These dependancies form a chain for both deterining paralell boot order but also how to handle service failure or restarts. For instance, if a network adapter fails (assuming IPMP isn't in use) I might want my entire network services chain (Apache, BIND, NTP, etc) to reload configuration files, this is possible using the dependancy system in SMF. But we can also group services into a "milestone", which is the SMF equivilent to a run-level. Milestones are actually services but instead of starting or stopping something they simply trigger dependancies, so the if we created a "SAPP" milestone it might have dependancies on Apache and PostgreSQL. We can manipulate a this group of services using the milestone, in essence, we're pulling 2 dogs using one leash rather than each seperatly, in the same way that we've always used Init Run Levels, rc2 brought up certain services, and rc3 brought up others. Milestones are an extremely useful and powerful tool.

You can find the source to SMF here in the OpenSolaris source browser or in the full OpenSolaris source tarballs. SMF, like the rest of OpenSolaris, is licensed CDDL. If your interested in the future of SMF we welcome you to be a part of it! Visit the OpenSolaris SMF Community and get involved today!

Next on our list of Init systems is Apple/Darwin's "launchd". Stay tuned.

- - C O M M E N T S - -

Very good site! I like it! I just wanted to pass on a note to let you know what a great job you have done with this site..Thanks!

Kotova Olga (URL) - 19 November '05 - 12:12

The must see site about online poker is [[]]

online poker (Email) (URL) - 21 November '05 - 05:47


buy+Soma - 29 November '05 - 11:12


buy%2BSoma - 29 November '05 - 11:12


Phentermine+side+effects - 01 December '05 - 09:00

It is foolish to postpone enjoyment of your ordinary life until you are more successful, more secure, or more loved than you are today [[]]

Phentermine side effects (Email) (URL) - 02 December '05 - 09:55

You have some really cool stuff at your site. I’m sure gonna come back here. Standard Chair becomes Black Chips in final: [[]] , right Slot will Percieve Plane without any questions Lose Girl is very good Grass , Black, Central, Big nothing comparative to Full Player will Mistery unconditionally

Nicholas Smith (Email) (URL) - 20 January '06 - 08:05


Monica - 21 January '06 - 17:06

Holla and Happy Thanksgiving.

lloyd (Email) (URL) - 20 March '06 - 15:04

Can I share some resources with you?

mae (Email) (URL) - 22 March '06 - 00:06

Very well Done !!!

kristina (Email) (URL) - 03 April '06 - 10:32

Hey man…sorry I missed the party.

shelly (Email) (URL) - 06 April '06 - 07:35

you have to help me out with mine…

warren (Email) (URL) - 07 April '06 - 12:57

This site is a lot of fun very well designed.

hermelinda (Email) (URL) - 07 April '06 - 20:55

Good luck!

violet (Email) (URL) - 08 April '06 - 04:31

Very well Done !!!

laurette (Email) (URL) - 08 April '06 - 12:07

It looks like you really had a nice time.

teena (Email) (URL) - 08 April '06 - 19:55

It looks like you really had a nice time.

floyd (Email) (URL) - 09 April '06 - 06:33

Hope you come back soon!!

laurette (Email) (URL) - 10 April '06 - 14:39

Your pictures are great.

jenny (Email) (URL) - 11 April '06 - 00:35

you have to help me out with mine…

laurette (Email) (URL) - 11 April '06 - 11:41

Very interesting & professional site.

mirian (Email) (URL) - 11 April '06 - 22:46

Your pictures are great.

teena (Email) (URL) - 12 April '06 - 09:53

Hi there! Your site is cool!

pennie (Email) (URL) - 15 April '06 - 14:34

CT-Graphics – professional photography, royalty free photos, photo journalism, photo objects, action photography, images – San Diego, California royalty free photos.

Stock Photography – Royalty-Free Stock Photos – Photo Objects – Stock Images Low Cost High Resolution at CT-graphics Download royalty free stock photos, stock images, photo-objects at the best stock photography royalty free image site. Search our royalty-free image collections for stock photo objects and royalty free pics for business stock photography, dogs and animals photos, pictures of churches, medical photos, and more! Request pictures at CT – Graphics !!!

CT Graphics (Email) (URL) - 20 April '06 - 06:44

This site is a lot of fun very well designed.

micheal (Email) (URL) - 22 April '06 - 12:33

Very interesting and professional site! Good luck!

mae (Email) (URL) - 23 April '06 - 00:58

Follow your dreams, you can reach your goals.

Molli (Email) (URL) - 23 April '06 - 13:25

Very interesting & professional site. You done great work.

jerome (Email) (URL) - 24 April '06 - 01:39

Thanks for the special work and information!

teena (Email) (URL) - 24 April '06 - 12:16

Follow your dreams, you can reach your goals.

tommy (Email) (URL) - 24 April '06 - 23:29

I am here to say hello and you have a great site!

olga (Email) (URL) - 24 April '06 - 23:33

online casinos,
winning online casinos,
online casinos,
casinos online,
online casinos,
online casinos,

online poker rooms (Email) (URL) - 25 April '06 - 02:20

It looks like you really had a nice time.

orpha (Email) (URL) - 25 April '06 - 11:21

Nice blog! ;)

funny movies (Email) (URL) - 25 April '06 - 19:07

Great. Thanks!

meta (Email) (URL) - 26 April '06 - 00:40

Very interesting and professional site! Good luck!

bobbie (Email) (URL) - 26 April '06 - 17:55

Your site is amaizing. Can I share some resources with you?

Noris (Email) (URL) - 26 April '06 - 19:52

Hello and congratulations!

theodore (Email) (URL) - 27 April '06 - 05:49

This is the coolest La Cocina.

miguel (Email) (URL) - 27 April '06 - 12:25

Hello and congratulations!

jerome (Email) (URL) - 27 April '06 - 17:59

Very interesting & professional site. You done great work.

lloyd (Email) (URL) - 29 April '06 - 05:40

Holla and Happy Thanksgiving.

brandy (Email) (URL) - 29 April '06 - 16:36

I like your website alot…its lots of fun… you have to help me out with mine…

laurette (Email) (URL) - 30 April '06 - 04:09

I like this site!

lizbeth (Email) (URL) - 30 April '06 - 15:38

Follow your dreams, you can reach your goals.

brandy (Email) (URL) - 01 May '06 - 03:16

Holla and Happy Thanksgiving.

yolonda (Email) (URL) - 01 May '06 - 14:34

Very interesting & professional site. You done great work.

leona (Email) (URL) - 02 May '06 - 02:28

A very nice website !! Very well Done !!!

cherly (Email) (URL) - 02 May '06 - 14:02

hi there.

Nick (Email) (URL) - 05 May '06 - 13:21

I like your website alot…its lots of fun… you have to help me out with mine…

jay (Email) (URL) - 05 May '06 - 13:59

I like your website alot…its lots of fun… you have to help me out with mine…

Alexx (Email) (URL) - 05 May '06 - 18:54

Hope you come back soon!!

calvin (Email) (URL) - 12 June '06 - 23:52

Hope you come back soon!!

sonja (Email) (URL) - 13 June '06 - 01:04

Very interesting and professional site! Good luck!

alex (Email) (URL) - 13 June '06 - 15:54

I like your website alot…its lots of fun… you have to help me out with mine…

leola (Email) (URL) - 13 June '06 - 16:02

Follow your dreams, you can reach your goals.

izola (Email) (URL) - 13 June '06 - 18:48

Hey man…sorry I missed the party.

velma (Email) (URL) - 14 June '06 - 02:08

Personal information

Remember your information?

Small print: All html tags except <b> and <i> will be removed from your comment. You can make links by just typing the url or mail-address.