The Blog of Ben Rockwood

Solaris Extended Accounting

09 Feb '09 - 09:38 by benr

Extended Accounting is one of the many under appreciated features in Solaris, and quite possibly the worst documented to boot. So, it's time to fix that.

Accounting, in general, is a means of recording data about resource utilization, CPU in particular, with the intention to be used primarily for reporting and billing purposes. Conceptually it could be confused with Auditing (see I See You!: Solaris Auditing (BSM)), in that they both record activity, except that the scope is vastly different. With BSM you can see every syscall/process/etc. that is executed, by who, when, with what privileges, etc; but the purpose of auditing is to determine what happened, when it happened, how it happened, and by who in the context of security. By contrast, Accounting is interested in what ran, how long it ran, how much resource it consumed, and some basic related information in the context of resource utilization.

When used in conjunction with Solaris's array of observability tools, from BSM Auditing, to Kstats, to Dtrace, it provides a useful low impact way or maintaining a historical view of resource utilization more detailed than Kstats but more permanent than DTrace and more targeted than auditing. With kstats you can graph CPU usage over time, but with Extended Accounting you can determine the breakdown of CPU usage within the same period, for example.

The first thing you need to know about accounting on Solaris is that there are TWO separate accounting systems. The documentation did a poor job of making this clear for a long time. The UNIX Accounting system, we'll call it, has been around for over a decade and is a hodgepodge of ancient scripts. Its still there and documented in the Solaris Admin Guide. Like mainframes or NIS+, if you're not already using it you shouldn't start now. In this blog entry we focus purely on the newer Solaris specific Extended Accounting facility.

Extended Accounting was added to Solaris in Solaris 9 along with the excellent microstate accounting enhancements. Solaris maintains all sorts of amazing stats about processes that can be explored with proc tools, and the many *stat tools (mpstat, vmstat, etc.) via options uncommonly used. The extended accounting facility really just provides a framework in which these stats can be stored in a datafile for later use, rather than be discarded when a process terminated. When exit() is run a call is made to extended accounting, and if enabled for tasks or processes the data is dumped into a record.

The framework is extensible for use in a variety of ways. For instance, when using Solaris IPQoS you can use extended accounting to record network flow data. As part of Crossbow a wonderful suprise was a new non-IPQoS flow extended accounting resource which I previously discussed here: Crossbow for Christmas.

Getting start with Extended Accounting (exacct for short) is easily done using the acctadm command:

$ acctadm
            Task accounting: inactive
       Task accounting file: none
     Tracked task resources: none
   Untracked task resources: extended
         Process accounting: active
    Process accounting file: /var/adm/exacct/proc
  Tracked process resources: extended
Untracked process resources: host
            Flow accounting: inactive
       Flow accounting file: none
     Tracked flow resources: none
   Untracked flow resources: extended
            Net accounting: active
       Net accounting file: /var/adm/exacct/net
     Tracked net resources: extended
   Untracked net resources: none

Here you can see the various accounting types available: Task, Process, Flow (IPQoS), and Net (Crossbow). Each accounting type has a datafile to store records in which is user defined. A variety of different resources can be tracked for each accounting type, which can either individually selected or specified as either "extended" or "basic", which are groupings of resources. To see what they are use acctadm -r:

$ acctadm -r
process:
extended pid,uid,gid,cpu,time,command,tty,projid,taskid,ancpid,wait-status,zone,flag,memory,mstate
basic    pid,uid,gid,cpu,time,command,tty,flag
task:
extended taskid,projid,cpu,time,host,mstate,anctaskid,zone
basic    taskid,projid,cpu,time
flow:
extended saddr,daddr,sport,dport,proto,dsfield,nbytes,npkts,action,ctime,lseen,projid,uid
basic    saddr,daddr,sport,dport,proto,nbytes,npkts,action
net:
extended name,ehost,edest,vlan_pid,vlan_tci,sap,priority,bwlimit,devname,src_ip,dst_ip,src_port,dst_port,protocol,dsfield,curtime,ibytes,obytes,ipkts,opkts,ierrpkts,oerrpkts
basic    name,devname,ehost,edest,vlan_pid,vlan_tci,sap,priority,bwlimit,curtime,ibytes,obytes,ipkts,opkts,ierrpkts,oerrpkts

To enable a given accounting type specify the file, resources, and then enable it with the -E option, like so:

root@quadra ~$ acctadm -f /var/adm/exacct/task task
root@quadra ~$ acctadm -e extended task
root@quadra ~$ acctadm -E task
root@quadra ~$ acctadm task
            Task accounting: active
       Task accounting file: /var/adm/exacct/task
     Tracked task resources: extended
   Untracked task resources: none

All the above is well documented and straight forward. The tricky part is actually using the data thats collected and dealing with long running processes.

As I stated before, the accounting records are recorded at process termination. That means that if you start a long running process, such as a MySQL daemon that runs for 6 months, you won't see a record for it until its stopped. That sort of defeats the usefulness of accounting. Therefore, using the wracct command you can cause an individual process or task to write a "partial record". In this case, partial differentiates if from a full record written at exit. So if you want to get accounting data for such processes, you'll want a cron job that executes wracct against the process(es) at some given interval.

# wracct -i `pgrep arkeiad` -t partial process
#

There are two tools provided with Solaris to examine extended accounting data files. One is the lastcomm command which is like a shells "history" command, showing a history of commands run based on the process accounting data. The second is actually a demo C application found in /usr/demo/libexacct called exdump. This demo app is intended to help developers learn the extended accounting library (libexacct), but is used all too commonly by administrators to output the contents of an accounting file.

If you really want to take advantage of accounting data you'll need to build your own custom tools using either the Exacct PERL module (written by Sun, found in CPAN, and part of the Solaris PERL distribution) or the libexacct C library.

Getting started with the PERL modules can be confusing initially. You can find documentation for the module on CPAN and some examples in the System Administration Guide: Solaris Containers-Resource Management and Solaris Zones.

The problem with the examples in the manual is that they mask some of the inner workings via the convenience "dump" function. Thus I present to you a modified version that is a little more explicit:

#!/usr/bin/perl

##
## This is a modified version of the 'dumpexacct' example from the 
## 'System Administration Guide: Solaris Containers-Resource Management and Solaris Zones'
## In this version, rather than use the "dump()" convience function I instead navigate
## directly... this helps better illustrate how to actually use the Exacct PERL Module. 
#
## benr@cuddletech.com - 2/7/09

use strict;
use warnings;
use Sun::Solaris::Exacct qw(:EXACCT_ALL);

die("Usage is $0 n") unless (@ARGV == 1);

# Open the exact file and display the header information.
my $ef = ea_new_file($ARGV[0], &O_RDONLY) || die(error_str());
printf("Creator:  %sn", $ef->creator());
printf("Hostname: %snn", $ef->hostname());
 
my $obj_counter = 0;

# Dump the file contents
while (my $obj = $ef->get()) {                                                          ## Get an object from the EA File and reposition to next.
        print("---------------- OBJECT $obj_counter -----------------------n");
        $obj_counter++;

        my $objectType = $obj->type();                                                  ## Return the object type (group or item) 
        my $objectCatalogObj =  $obj->catalog();                                        ## Return a catalog object for this object
        my ($a, $b, $c) = $objectCatalogObj->value();                                   ## Breakout the catalog triplet

        printf("Object is: %s   -   Catalog: %s %s %sn", $objectType, $a, $b, $c );    ## Output the catalog and object type for the object at hand.

        if ( $objectType == EO_GROUP && $c == EXD_GROUP_PROC ) {                        ## If this is a PROC Group....
                my @objectList = $obj->value();                                         ## Return the sub-group which will contain actual items.

                foreach my $item (@objectList) {
                        my $itemsType = $item->type();                                  ## Get type for new object.. is it a EO_ITEM or EO_GROUP?

                        if ($itemsType == EO_ITEM){
                                my $itemCatalog = $item->catalog();                             ## Return the "Catalog Object" for this object.
                                my ($itemType, $itemGroup, $itemId) = $itemCatalog->value();    ## Breakout the catalog triple
                                my $itemValue = $item->value();                                 ## Get the item value itself.

                                print("ttId: $itemId tValue: $itemValuen");                 ## Now print just the catalog item id and item value.
                        } else {
                                print("ttERROR: Expected Item, got a group instead.n");
                        }
                }
        }

}

# Report any errors
if (ea_error() != EXR_OK && ea_error() != EXR_EOF)  {
        printf("nERROR: %sn", ea_error_str());
        exit(1);
}

The example opens an accounting file and returns a file object. Using that file object we can get some metadata and via the get() method we can walk the file, because after each get() it repositions to the next object. An accounting file is a collection of objects, which come in the form of groups and items. Groups contain yet other groups or items. Items contain data. Each object has an associated catalog which is a triplet that describe the object in the form: type, group, and id.

When implementing your accounting program you'll generally be walking the objects, then navigating groups and items to get to the data you want. To get your footing as to the structure of groups and types of data available in each item use the demo exdump tool and then start playing around. You'll find the /usr/include/sys/exacct_catalog.h header useful in deciphering the catalog values. The heavy use of double-typed variables can be confusing at first, but the header will help it make sense.

As you can see, utilizing extended accounting isn't for the casual user... but if you can find value in the data it collects you'll reap the benefits of your labor learning the PERL module. If you are serious, I'd recommend taking the example above to break out selected items and formatting or sorting them in a more palatable way. Here is a simple modified version that outputs one line per record for a small handful of items:

use strict;
use warnings;
use Sun::Solaris::Exacct qw(:EXACCT_ALL);

die("Usage is $0 n") unless (@ARGV == 1);

# Open the exact file and display the header information.
my $ef = ea_new_file($ARGV[0], &O_RDONLY) || die(error_str());
printf("Creator:  %sn", $ef->creator());
printf("Hostname: %snn", $ef->hostname());

## Print Header:
printf("%10s %4s %4s %5s %30sn", "Counter", "UID", "GID", "PID", "CMD");
my $idxCounter = 1;
 
# Dump the file contents
while (my $obj = $ef->get()) {                                                          ## Get an object from the EA File and reposition to next.
        $idxCounter++;  ## Count records.

        my $objectType = $obj->type();                                                  ## Return the object type (group or item) 
        my $objectCatalogObj =  $obj->catalog();                                        ## Return a catalog object for this object
        my ($a, $b, $c) = $objectCatalogObj->value();                                   ## Breakout the catalog triplet

        #printf("Object is: %s   -   Catalog: %s %s %sn", $objectType, $a, $b, $c );   ## Output the catalog and object type for the object at hand.

        if ( $objectType == EO_GROUP && $c == EXD_GROUP_PROC ) {                        ## If this is a PROC Group....
                my @objectList = $obj->value();                                         ## Return the sub-group which will contain actual items.

                my $xxPID = "";
                my $xxCMD = "";
                my $xxUID = "";
                my $xxGID = "";

                foreach my $item (@objectList) {
                        my $itemsType = $item->type();                                  ## Get type for new object.. is it a EO_ITEM or EO_GROUP?

                        if ($itemsType == EO_ITEM){
                                my $itemCatalog = $item->catalog();                             ## Return the "Catalog Object" for this object.
                                my ($itemType, $itemGroup, $itemId) = $itemCatalog->value();    ## Breakout the catalog triple
                                my $itemValue = $item->value();                                 ## Get the item value itself.

                                #print("ttId: $itemId tValue: $itemValuen");                        ## Now print just the catalog item id and item value.

                                $xxPID = $itemValue if ( $itemId == EXD_PROC_PID );
                                $xxUID = $itemValue if ( $itemId == EXD_PROC_UID );
                                $xxGID = $itemValue if ( $itemId == EXD_PROC_GID );
                                $xxCMD = $itemValue if ( $itemId == EXD_PROC_COMMAND );

                        } else {
                                print("ttERROR: Expected Item, got a group instead.n");
                        }
                } 

                ## Now output pretty formated data:
                printf("%10d %4d %4d %5d %30sn", $idxCounter, $xxUID, $xxGID, $xxPID, $xxCMD);
        }

}

# Report any errors
if (ea_error() != EXR_OK && ea_error() != EXR_EOF)  {
        printf("nERROR: %sn", ea_error_str());
        exit(1);
}

Here is what the above looks like when run:

$ ./prettyproc.pl  /var/adm/exacct/proc | head
Creator:  SunOS
Hostname: quadra

   Counter  UID  GID   PID                            CMD
         2    0    0  6878                        acctadm
         3    0    0  6879                        acctadm
         4 1004  999  6883                         db2set
         5 1004  999  6882                             sh
         6 1004  999  6881                          db2fm
         7    0    0  6880                             sh

Please note that there should a lot of additional checking in the above, and you should differentiate between items that are full versus partial, but this should help you get started down the extended accounting road. :)

- - C O M M E N T S - -
I like to use the extended accounting system too.
It could be improved, if you don’t have to trigger the wracct by an external process, but by the scheduler at programmable times.

If you have a list of processes, you can’t do wracct on all of them at a exact time. But you don’t have to, if the process does not run.
So with a little help from a scheduler, which saves the data, if a process gets scheduled after accounting trigger time, no data is lost and all processes get accounted at the same time.

To my knowledge, this feature is not there, but should be implementable.

I should have patented that. :)

Knut Grunwald (Email) (URL) - 09 February '09 - 13:48

There’s also getacct(2) system call which lets you get in-memory exacct record w/o writing it to a file. This is why extended accounting can be enabled without logging into a file.

Andrei Dorofeev (Email) - 09 February '09 - 20:48

Good explanation on extended accounting. Just the thing I was looking for. You were right, SUN doc just gives you the highlight and nothing in detail.

I took your second script, and expanded to include more objects. It worked like a charm.

Thanks!!!

Rupen - 23 April '09 - 15:11

So with a little help from a scheduler, which saves the data, if a process gets scheduled after accounting trigger time, no data is lost and all processes get accounted at the same time.

Chris (Email) (URL) - 10 May '09 - 07:39

Excellent,

Nice collection of top bookmarking site. Good thought and great idea. Thanks for the information.

Accounting Homework (Email) (URL) - 25 June '09 - 13:28

cheap ugg
ugg outlet
ugg boots
cheap uggs online
cheap ugg online
cheap ugg boots online
[[http://www.cheapuggsonline.net]]

[[http://www.uggbootshome.net]]

cheap uggs online (Email) (URL) - 17 December '09 - 06:37

[[http://www.uggbootsstore.net/]] uggs outlet
[[http://www.uggbootsstore.net/]] ugg outlet

uggs on sale (Email) (URL) - 30 December '09 - 06:17

Welcome to have a look!
ugg fashion shoes—classic,cove,nightfall,sundance,mini…

UGGBOOTS-free shipping! Discount40%-50% off (Email) (URL) - 30 December '09 - 07:40

[[http://www.emocuk.com]] this post is really awesome

porno izle (Email) (URL) - 07 January '10 - 22:24

I found your website about a month ago and check it several times a day. It is by far one of the funniest sites that I have come across. I just wanted to say thank you for the laughs and keep up the great work!
[[http://www.louboutinshoesonline.com/]]

christian louboutin shoes (Email) (URL) - 11 January '10 - 08:37

I find a good website, they are wholesale nike shoes,wholesale nike shoes,wholesale jordan shoes,nike shox,air jordan boot,puma shoes,if you want to wholesale nike shoes,wholesale jordan shoes, nike shox,air jordan boots,puma shoes, you can check the [[[http://www.goods-in-china.com]]]]

wholesale nike shoes (Email) (URL) - 14 January '10 - 03:44

Good post! Thanks you for your information! China Wholesale Wholesale China Wholesalers Wholesale Game Accessories Wholesale Iphone Accessories Video Game Accessories Wholesale Wholesale Wii Accessories Wholesale Xbox 360 Accessories Wholesale Xbox 360 Games Wholesale Video Games Cheap Video Games Cheap Ps3 Games Cheap Xbox 360 Games Wholesale Computers Wholesale Laptop Computers Wholesale Laptops Discount Computers Cheap Computers Wholesale Iphones Wholesale Iphone Wholesale Iphones 3g Hiphones Wholesale Hiphone Wholesale Hiphones Wholesale Nokia Wholesale Nokia 8800 Wholesale Nokia n97 wholesale blackberry wholesale blackberry phones wholesale blackberry 9700 wholesale blackberry 9600 wholesale blackberry 9500

raging bull (Email) (URL) - 15 January '10 - 06:33

Good post! Thanks for your information! ed hardy ed hardy ed hardy clothing ed hardy clothing ed hardy swimwear ed hardy swimwear ed hardy jeans ed hardy jeans ed hardy hoodies ed hardy hoodies ed hardy shoes ed hardy shoes ed hardy uk ed hardy uk ed hardy bags ed hardy bags ed hardy shirts ed hardy shirts christian audigier christian audigier ed hardy mens ed hardy mens ed hardy womens ed hardy womens ed hardy kids ed hardy kids ed hardy

ed hardy (Email) (URL) - 15 January '10 - 07:38

uggs (Email) (URL) - 19 January '10 - 09:17

Good post! Thanks for your information!

brand clothing (Email) (URL) - 20 January '10 - 06:37

thank you !!

ed hardy (Email) (URL) - 20 January '10 - 06:38

[[http://www.buykamagra.com]] buy kamagra
[[http://www.viagracialis.com]] viagra cialis

M65 Jacket (Email) (URL) - 21 January '10 - 02:05

[[http://www.brand-clothing.com]] brand clothing

brand clothing (Email) (URL) - 21 January '10 - 02:50

good post

abercrombie clothing (Email) (URL) - 22 January '10 - 08:34

Good post! Thanks for your information!

As Seen On TV (Email) (URL) - 27 January '10 - 06:43

It is my great pleasure to visit your website and to enjoy your excellent post here. I like that very much. I can feel that you paid much attention for

those articles, as all of them make sense and are very useful. Thanks so much for sharing. I can be very good reader&listener if you are same searching for all to be good. Appreciate for your time!
Happy New Year!
[[http://www.uggboots-home.net]]

uggs (Email) (URL) - 28 January '10 - 10:44

Nothing is able to fulfill, as long as confidence. Everything is difficult at the beginning, is now at the beginning of the show that you have succeeded in half.

iphone (Email) (URL) - 29 January '10 - 07:54

Thank u for sharing,very good and useful for me.I will have eyes on your post,and i am Looking forward to more useful info!

uggs on sale-ugg outlet store (Email) (URL) - 03 February '10 - 09:19

nice post here, if you want to know more about best mbts and ugg boots, just click here
[[http://www.bestmbtshoes.com]]
[[http://www.uggbootsroom.com]]

ugg (Email) (URL) - 06 February '10 - 03:26

Nothing is able to fulfill, as long as confidence. Everything is difficult at the beginning, is now at the beginning of the show that you have succeeded in half.

puma boots (Email) (URL) - 06 February '10 - 07:54

Your article is very useful!Thank you for sharing.Nice post.

Tiffany Accessories (Email) (URL) - 08 February '10 - 08:45

dsfsdfsdfsdfjdsfsdfsdfsdfuıfsdfsd

zayÄ±flama hapÄ± (Email) (URL) - 23 February '10 - 12:04

It is by far one of the funniest sites that I have come across. I just wanted to say thank you for the laughs and keep up the great work

porno izle (Email) (URL) - 11 March '10 - 01:20

It is by far one of the funniest sites that I have come across. I just wanted to say thank you for the laughs and keep up the great work

Technology (URL) - 11 March '10 - 12:05

This is really a great stuff for sharing.keep it up .Thanks for sharing.
[url= [[http://www.airmax-shox.com/en/category..]]]]

nike airmax (Email) - 12 March '10 - 07:27

nice artwork thanks for sharing

acente (Email) (URL) - 12 March '10 - 09:24

nice websites and your comment

porno izle (Email) (URL) - 12 March '10 - 13:03

nice artwork thanks

cheap clothes (Email) (URL) - 16 March '10 - 04:43

for sharing

air jordan (Email) (URL) - 16 March '10 - 04:43

Interesting blog. Actually google made searching of information easy on any topic. Well keep it up and post more interesting blogs.

Accounting Homework Help (Email) (URL) - 24 March '10 - 14:22