Solaris Extended Accounting in the Real World
21 Oct '09 - 06:36 by benrRecently we talked about Solaris Auditing (BSM) in the Real World. Like BSM, Extended Accounting is a fantastic feature of Solaris that is utterly useless without tools. Solaris goes so far as giving you the capability but not so far as to hand you the rest of the solution on a silver platter. On one hand this means that the technology isn't pigeon holed due to the capabilities of a single tool, but at the same time it creates a barrier to entry that causes many people to simply ignore it all together. So, yet again, let me provide a simple tool to fill some of that void.
In a previous post, Solaris Extended Accounting, I described Extended Accounting and provided two scripts to get you started, one was a PERL script to dump Extended Accounting ("exacct") data files and the other was called "prettyproc" which output Proccess Accounting files in a more human friendly way. This post should be viewed as Part 2 of that post.
When & How to use Extended Accounting
The most basic explanation of Extended Accounting is this: a facility that records certain events upon completion for later analysis. Those certain events depend on which of the four accounting types we're using. For processes, the cumulative data maintained by Solaris microstate accounting is written into a single record as process termination. For tasks, which are groups of processes within a single project, the same applies but recorded on each task termination rather than process. For (Crossbow) net, aggregate network utilization is written out on regular intervals (15 seconds). We'll ignore IPQoS "flow" Accounting entirely for the time being.
So the first thing we should say is that Extended Accounting is not a monitoring facility. If you want to know how much CPU or Memory is being used at some given time you should rely upon Kstats or /proc statistics on a polling schedule.
What Extended Accounting is good for is reporting. Consider 'net' accounting; every 15 seconds a record is created for each data link (dladm show-link). You could easily create a report at some interval (hour, day, week, month?) for both total bytes/packets sent/recieved on each link or great a graph or perhaps most likely calculate 95th percentile on the links. Now, in this case of 'net' accounting you could also use an external system to poll the data remotely via SNMP or locally via kstats, but this might serve as a better "definiative" local record.
Proc accounting is fuzzy ground though. The best way I can explain process accounting is to imagine that every time you executed a command Solaris was secretly running "time(1M)" and then storing the output on your behalf.
benr@quadra Downloads$ time tar xfj flash_player_10_solaris_x86.tar.bz2 real 0m0.763s user 0m0.705s sys 0m0.070s
This is, essentially, whats happening! Solaris maintains a lot of detail on what processes are doing (known as "microstate accounting"). Normally, when a process terminates that data is simply discarded, however if Process Extended Accounting is enabled its dumped out as a record! From this record we can see interesting stats such as when the process started, when it finished (real time), how long it spent cpu time in kernel-land (sys time), how long it spent cpu time in user-land (user time), how many context switches it made, how much swapping it did, what its average RSS memory usage was, etc, etc, etc.
But as wonderful as this is, I have to make it crystal clear that this data isn't written out untill a process terminates! If MySQL runs for 4 months, it outputs a single record when it was finally shut down, and that record is the accumulation of that full 4 months of running!
Here is the exception. Proc and Task records can be "full" or "partial". When a process/task terminates and creates a record, that's a "full record". However, using "wracct" we can force a process or task to create a "partial record", which is essentially a way of saying "Just tell me what you've got so far!" The rub is that, in the proc case, that data is cumulative, so if you wanted to report on what a process has done in the last 24 hours you need to write a partial record every 24 hours and the find the difference between the partial record yesterday and partial record today. Talk about fun.
Now, besides all that, who actually bills users or reports usage based on total CPU time? Total context switches? This isn't the 1970's nor is this likely to be a Super Computer reporting computational time. In short, the data probably isn't terribly useful as a basis for billing in this day and age without some creative thought.
So then lets think... what can we determine from the data. Based on CPU usage we could determine what the top 5 CPU consuming processes were. Based on average RSS usage we could determine what the top memory consumers were. So on and so forth. Interesting perhaps... but worth it?
Go back to what I said about running "time" on every command. This data could be of used for capacity planning or, with some intelligence, behavior monitoring. Are your users complaining about commands taking too long to run, but when you ask how long they give you a bogus number or simply shrug? Extended Accounting can tell you. Are batch jobs running at night but want a record of when they started and how resource hungry they were? Here is a way that doesn't involve writing wrappers!
In short, Extended Accounting is a pretty lousy billing system on todays mulit-core systems, but it can provide useful historical statistics to questions that might be otherwise difficult to answer.
Practical Tools
The first tool I'll provide you with is a PERL replacement for the Solaris included /usr/demo/libexacct/exdump.c: exdebug.pl. This tool offers the following advantages:
- exdump.c hasn't been updated for the new Crossbow provided 'net' accounting data; exdebug.pl is module agnostic and works with them all.
- The output is just much cleaner and intuitive for exploring what ExAcct can do for you.
- Its implemented in PERL making it easier to get in there and build something, rather than dealing with the libexacct learning curve in C. If nothing else you can quickly prototype and then re-implement in C.
Here is an example from an 'net' record:
benr@quadra exacct$ acctadm net
Net accounting: active
Net accounting file: /var/adm/exacct/net
Tracked net resources: extended
Untracked net resources: none
benr@quadra exacct$ pfexec ./exdebug.pl /var/adm/exacct/net | more
Creator: SunOS
Hostname: quadra
---------------- OBJECT 0 -----------------------
Object is: EO_GROUP - Catalog: EXT_GROUP EXC_DEFAULT EXD_GROUP_NET_LINK_DESC
Id: EXD_NET_DESC_NAME Value: testzone0
Id: EXD_NET_DESC_EHOST Value:
Id: EXD_NET_DESC_EDEST Value:
Id: EXD_NET_DESC_VLAN_TPID Value: 0
Id: EXD_NET_DESC_VLAN_TCI Value: 0
Id: EXD_NET_DESC_SAP Value: 0
Id: EXD_NET_DESC_PRIORITY Value: 0
Id: EXD_NET_DESC_BWLIMIT Value: 0
Id: EXD_NET_DESC_DEVNAME Value: testzone0
Id: EXD_NET_DESC_V4SADDR Value: 0
Id: EXD_NET_DESC_V4DADDR Value: 0
Id: EXD_NET_DESC_SPORT Value: 0
Id: EXD_NET_DESC_DPORT Value: 0
Id: EXD_NET_DESC_PROTOCOL Value: 0
Id: EXD_NET_DESC_DSFIELD Value: 0
...
---------------- OBJECT 67 -----------------------
Object is: EO_GROUP - Catalog: EXT_GROUP EXC_DEFAULT EXD_GROUP_NET_LINK_STATS
Id: EXD_NET_STATS_NAME Value: e1000g1
Id: EXD_NET_STATS_CURTIME Value: 1256033841
Id: EXD_NET_STATS_IBYTES Value: 2411692067
Id: EXD_NET_STATS_OBYTES Value: 202604900
Id: EXD_NET_STATS_IPKTS Value: 2005669
Id: EXD_NET_STATS_OPKTS Value: 1265178
Id: EXD_NET_STATS_IERRPKTS Value: 0
Id: EXD_NET_STATS_OERRPKTS Value: 0
---------------- OBJECT 68 -----------------------
Object is: EO_GROUP - Catalog: EXT_GROUP EXC_DEFAULT EXD_GROUP_NET_FLOW_STATS
Id: EXD_NET_STATS_NAME Value: inbound_ssh
Id: EXD_NET_STATS_CURTIME Value: 1256033841
Id: EXD_NET_STATS_IBYTES Value: 93958770
Id: EXD_NET_STATS_OBYTES Value: 106077944
Id: EXD_NET_STATS_IPKTS Value: 238395
Id: EXD_NET_STATS_OPKTS Value: 321977
Id: EXD_NET_STATS_IERRPKTS Value: 0
Id: EXD_NET_STATS_OERRPKTS Value: 0
---------------- OBJECT 69 -----------------------
Object is: EO_GROUP - Catalog: EXT_GROUP EXC_DEFAULT EXD_GROUP_NET_LINK_STATS
Id: EXD_NET_STATS_NAME Value: testzone0
Id: EXD_NET_STATS_CURTIME Value: 1256033861
Id: EXD_NET_STATS_IBYTES Value: 4528169
Id: EXD_NET_STATS_OBYTES Value: 0
Id: EXD_NET_STATS_IPKTS Value: 64405
Id: EXD_NET_STATS_OPKTS Value: 0
Id: EXD_NET_STATS_IERRPKTS Value: 0
Id: EXD_NET_STATS_OERRPKTS Value: 0
In the above example you'll see the variety of objects offered by the net accounting module, including link descriptions, link statistics ('testzone0' is a VNIC and 'e1000g1' is a physical interface), and flow statistics (inbound_ssh is a flowadm defined flow).
The second tool is exacctly, a human friendly Proc Extended Accounting dumper. It is also implemented in PERL and in fact was derived from the exdebug app above.
benr@quadra exacct$ acctadm proc
Process accounting: active
Process accounting file: /var/adm/exacct/proc
Tracked process resources: extended
Untracked process resources: host
benr@quadra exacct$ pfexec ./exacctly /var/adm/exacct/proc | more
Creator: SunOS
Hostname: quadra
ZONE UID GID PID CMD | Real User Sys | Start Date | RSS AVG RSS MAX SysCalls Swaps
----------------------------------------------------+--------------------------+--------------------------+--------------------------------------------------
global 0 0 1922 acctadm | 0.07 0.00 0.01 | Tue Oct 20 03:10:01 2009 | 524 K 12904 K 450 0 | FULL
global 0 0 1920 sh | 0.07 0.00 0.00 | Tue Oct 20 03:10:01 2009 | 2036 K 12904 K 103 0 | FULL
global 25 25 1924 sendmail | 0.10 0.01 0.01 | Tue Oct 20 03:10:01 2009 | 1912 K 12904 K 543 0 | FULL
global 0 0 1927 sendmail | 0.01 0.00 0.01 | Tue Oct 20 03:10:01 2009 | 2288 K 13172 K 267 0 | FULL
global 0 0 1923 mail | 0.10 0.00 0.00 | Tue Oct 20 03:10:01 2009 | 504 K 12904 K 169 0 | FULL
global 0 0 1921 sh | 0.11 0.00 0.00 | Tue Oct 20 03:10:01 2009 | 920 K 12904 K 102 0 | FULL
The output is really wide, but everyone should have a big ol' screen these days. Notice the depth of information here. For each terminated process we see the zone it was in, user and group, PID and command name itself (ExAcct doesn't record arguments), then we see real/sys/user time in seconds (ExAcct actually has nanosecond granularity, so these are rounded numbers), the start time and other goodness. The last column reports whether the record is full or partial.
This tool is, in and of itself, useful for many administrators to start using Extended Accounting that might otherwise have ignored it. Even more so, I hope it sparks your interest and imagination as to the possibilities! Just think of all the ways to amaze your boss and fellow admins!
Data File Rotation
Like any log, don't be lazy and forget to rotate those files or you'll have a mess on your hands. Rotating your extended accounting data files will make them easier to dissect and consume less disk. Here are some examples lines you can drop into /etc/logadm.conf, Solaris's default log rotation tool:
/var/adm/exacct/proc -N -p 1d -C 7 -b '/usr/sbin/acctadm -x process' -a '/usr/sbin/acctadm -e extended -f /var/adm/exacct/proc process' /var/adm/exacct/net -N -p 1d -C 7 -b '/usr/sbin/acctadm -x net' -a '/usr/sbin/acctadm -e extended -f /var/adm/exacct/net net' /var/adm/exacct/task -N -p 1d -C 7 -b '/usr/sbin/acctadm -x task' -a '/usr/sbin/acctadm -e extended -f /var/adm/exacct/task task'
These examples will rotate each day (-p 1d) and keep 7 logs (-C 7) before destroying. The important bit is that you can't just mv the file, you need to stop accounting, rotate, then resume it.
Remember to ensure that logadm isn't commented out in the root crontab.
Parting Thoughts & Cautions
Before I wrap up, I want to note something about Process records. Here is one as seen with exdebug:
---------------- OBJECT 0 -----------------------
Object is: EO_GROUP - Catalog: EXT_GROUP EXC_DEFAULT EXD_GROUP_PROC
Id: EXD_PROC_PID Value: 1922
Id: EXD_PROC_UID Value: 0
Id: EXD_PROC_GID Value: 0
Id: EXD_PROC_PROJID Value: 1
Id: EXD_PROC_TASKID Value: 39949
Id: EXD_PROC_CPU_USER_SEC Value: 0
Id: EXD_PROC_CPU_USER_NSEC Value: 2047013
Id: EXD_PROC_CPU_SYS_SEC Value: 0
Id: EXD_PROC_CPU_SYS_NSEC Value: 6237135
Id: EXD_PROC_START_SEC Value: 1256033401
Id: EXD_PROC_START_NSEC Value: 311640743
Id: EXD_PROC_FINISH_SEC Value: 1256033401
Id: EXD_PROC_FINISH_NSEC Value: 380283918
Id: EXD_PROC_COMMAND Value: acctadm
Id: EXD_PROC_TTY_MAJOR Value: 4294967295
Id: EXD_PROC_TTY_MINOR Value: 4294967295
Id: EXD_PROC_FAULTS_MAJOR Value: 0
Id: EXD_PROC_FAULTS_MINOR Value: 0
Id: EXD_PROC_MESSAGES_SND Value: 0
Id: EXD_PROC_MESSAGES_RCV Value: 0
Id: EXD_PROC_BLOCKS_IN Value: 0
Id: EXD_PROC_BLOCKS_OUT Value: 0
Id: EXD_PROC_CHARS_RDWR Value: 20100
Id: EXD_PROC_CONTEXT_VOL Value: 102
Id: EXD_PROC_CONTEXT_INV Value: 0
Id: EXD_PROC_SIGNALS Value: 0
Id: EXD_PROC_SWAPS Value: 0
Id: EXD_PROC_SYSCALLS Value: 450
Id: EXD_PROC_ACCT_FLAGS Value: 2
Id: EXD_PROC_ANCPID Value: 1920
Id: EXD_PROC_WAIT_STATUS Value: 0
Id: EXD_PROC_ZONENAME Value: global
Id: EXD_PROC_MEM_RSS_AVG_K Value: 524
Id: EXD_PROC_MEM_RSS_MAX_K Value: 12904
Okey, lots of data, lots of goodness. Notice EXD_PROC_BLOCKS_IN, OUT, and CHARS_RDWR? They are useless. I can't go into why here, but don't get excited about them or bother doing anything, the values are crap. If your a veteran Kstat diver you'll recognize similar values in the Kstat cpu_stat class... same story.
Hopefully this post as helped provide you with a more practical understanding of Extended Accounting and provided you with some resources to get in there and use the data. There is a wealth of possibilities if you just avail yourself of them. :)
In this site is very nice , useful for me, i bookmarked your blog
Thanks
tax jobs (Email) (URL) - 28 October '09 - 23:46
Thank you very much!cheap links of london (Email) (URL) - 07 November '09 - 00:53
Thanks!!Tiffany jewellery (Email) (URL) - 14 November '09 - 02:38
Great post! Hope to be better. Better means more features.good post,I think so!
Thanks for your information, i have read it, very good！
Bing is a really overlord!! support Bing~~
This is great news. Best of luck for the future and keep up the good work.
links of london (Email) (URL) - 17 November '09 - 03:18
[[http://www.cheapuggsonline.net/UGG_Ult..]] UGG Ultra Short boots[[http://www.cheapuggsonline.net/UGG_Cla..]] UGG Classic Tall Boots
cheap uggs online (Email) (URL) - 17 December '09 - 06:42
[[http://www.buykamagra.com]] buy kamagra[[http://www.viagracialis.com]] viagra cialis
M65 Jacket (Email) (URL) - 21 January '10 - 02:37
[[http://www.uggbootshome.net]] Ugg Boots[[http://www.uggbootshome.net]] Uggs Outlet
uggs outlet (Email) (URL) - 21 January '10 - 10:16
Thanks for sharing your good blog with us!gucci handbags (Email) (URL) - 20 February '10 - 09:02
1234568links of london (Email) (URL) - 24 February '10 - 04:03
Links of London( [[http://www.linksgiftstore.com]] )Links of London( [[http://www.toplinkslondon.com]] )
links of london (Email) (URL) - 24 February '10 - 04:05
good one!suresh (Email) - 03 March '10 - 09:38
I took my first loan when I was a teenager and it helped my business very much. But, I need the car loan again.HarrietGARDNER (Email) (URL) - 19 March '10 - 19:07
Especially now Air Jordan Force 5 are released in today market .And the Air Jordan Force 6 will be released recentlyair jordan (Email) (URL) - 09 April '10 - 08:07
.Especially now Air Jordan Force 5 are released in today market .And the Air Jordan Force 6 will be released recently . So do not miss the chance to get it .We welcome you to visit our website for more information [[http://www.jordanmvp.com]] .Hope hear from you soon .Thank you !air jordan (Email) (URL) - 12 April '10 - 09:42
It’s just timing,but time is limited…Replica handbags (Email) (URL) - 27 April '10 - 02:36
[[http://www.gucci-shoes-bags.com]][[http://www.guccinewshop.com]]
Dear friends welcome to our store: we have a specail offer now which is once you buy any product you can get a free gift as well, these gifts are in the gift area, you can free to choose and please add to the shopping cart, we will send it together with your purchased product. Thanks!
gucci (Email) (URL) - 27 April '10 - 07:51
buy ugg bootscheap ugg boots
ugg cardy boots
women ugg boots
buy UGG Classic Tall
ugg boots for sale
ugg Classic Tall
ugg boots on sale
discount ugg boots
UGG Classic
UGG Bailey Button Boots
Welcome to [[http://www.fbib2b.com]]
ukjean (Email) (URL) - 29 April '10 - 00:55
I’m a little late finding this article, but I loved it.As we know, now sunglasses are loved by more and more people, and everyone would like to have a pair of comfortable sunglasses,save up to 47%.welcome to [[http://www.eyewear-rayban.com]]
rayban sunglasses (Email) (URL) - 29 April '10 - 07:19
You could buy one item from the website, and you would get a free Links of London free cham and a free chain. We think that you must deeply content with your items. if you are not satisfy your items, you could return to links of london charm us, we would like to change for you or refund you at your earliest convience. If you buy goods from our website, we will endeavor our best to help you solve your matters not only before purchase but also after purchase. links bracelet You must be content with your goods and service. The core of Links of London is There is no such thing as the best, for the better is yet to come.! we hope that you could enjoy your shooping!wangrongjiao (Email) (URL) - 29 April '10 - 08:36
The Supra Strapped NS shoes are one of the top skateboarding shoes on the block. It is a good chance for you to buy now .Nowadays ,we published a new kind of Air Jordan ,that is the Air Jordan Force line .They are so good that when you see it , you will like it and are willing to own one .Supra (Email) (URL) - 04 May '10 - 02:44
Bienvenue à notre[url=[[http://www.wowgold4europe.fr/SiteWeb/G..]]]] wow europe [/url]site web.xiao (Email) - 04 May '10 - 04:04
P90x .It really is not expensive if you factor in the costof a gym membership,P90x workout . The cost for P90X is
about three months of a paid gym membership but you get to
keep the program foreverP90x . You can try many of the
online sites, but it will be the same as buying from the
company or a Beachbody Coach. Make sure you are getting
original DVD’s. People are selling copies all over. The
problem is how long will they last, P90x workout ,and you
truly need the exercise and nutrition guide to even follow
the program. You can go to any site or you can go to
www.p90xmall.com and click on products. P90x dvd You can
order directly from the site,P90x dvd.
p90x (Email) (URL) - 05 May '10 - 07:45
She did not have money for the rent, so she went to the pawnshop with all of her. The boss asked links of london rings what she wanted to do with so many coppers, but after carefully looking he picked up one and said this was a gold. And asked how much she wanted.wangrongjiao (Email) (URL) - 11 May '10 - 02:25
Do you want buy the beautiful shoes .We can provide you many style of Air Jordan Shoes. Here we can offfer you traditional Air Jordan 1 shoes, beautiful Air Jordan 2 shoes and some other cheap Air Jordan Shoes, such as Air Jordan 10 , Air Jordan 23and other shoes .Both are in its superior quality .Also we can offer you with our best services for discount jordan shoes . It is a good chance for you to buy now .Nowadays ,we published a new kind of Air Jordan ,that is the Air JordanForce line .They are so good that when you see it , you will like it and are willing to own one . Like the Air Jordan Force 3 , colorful Air Jordan Force 4 ,you may never miss it .Especially now Air Jordan Force 5 are released in today market .And the Air Jordan Force 6 will be released recently . So do not miss the chance to get it . For more information [[http://www.jordanmvp.com.We]] look forward hearing from you soon.cheap Air Jordan Shoes (Email) (URL) - 13 May '10 - 06:32
Immediately catch of Joe Smith, Joe Smith is to buy only the birds, not kill, officials say he is not honest, the Joe Smith beat him, then links of london someone is forced admissions, Joe Smith was sentenced to death, autumn Wenzhan. pandora rings jewelry people have reported seeing his son, returned home. And accompanied by John Doe to see Joe Smith Joe Smith was links of london sentenced to death, thinking clearly is to buy pandora beads and charms ， how to kill has become a pick up? John Doe 1 came to a small town, shop around to find Miejiang found Miejiang whereabouts immediately report, the magistrate received a report and track down the Miejiang, pandora jewelry done a quiet conscience, fear.wangrongjiao (Email) (URL) - 14 May '10 - 09:15
Dell inspiron 640m Battery [[http://www.adapterlist.com/dell/inspir..]]laptop battery (Email) (URL) - 14 May '10 - 09:40
Panasonic VW-VBG130 Battery [[http://www.globallaptopbattery.co.uk/c..]]laptop batteries (Email) (URL) - 14 May '10 - 09:42
it is good!sto cardits (Email) (URL) - 20 May '10 - 05:23
We think that you must deeply content with your items. if you are not satisfy your items, you could return to links of london charm us, we would like to change for you or refund you at your earliest convience.car dvd (Email) (URL) - 23 May '10 - 09:47
afsdfsdfafdafadwulijun (Email) - 24 May '10 - 08:59
Buy Nike Air Max 90 Shoes just $45-55 USD in[[http://www.iofferitems.com,]], 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!nike air max 90 shoes (Email) (URL) - 28 May '10 - 02:49
Buy Nike Air Max 90 Shoes just $45-55 USD in [[http://www.iofferitems.com,]], 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!nike air max 90 shoes (Email) (URL) - 28 May '10 - 05:49
wholesale hats [[http://www.aj-wholesale.com]]wholesale hats (Email) (URL) - 29 May '10 - 05:48
[[http://www.mbtshoeslatest.com]][[http://www.nikeairmaxshoe.com]]
Nike air max shoe (Email) (URL) - 24 June '10 - 07:51
The athletic shoes which makes using this technology may the very good local constable convoy mobilization body, Air Max 2009.[[http://www.allhotshoes.com/]]
air max shoes (URL) - 09 July '10 - 05:31
Mbt zapatos como una tormenta se extendió por Europa y el mundo. Mbt zapatos cómo tener tanto poder? Según la investigación, Mbt zapatos no sólo su propio estilo de diseño único, sino también porque hay magia buena forma física, así Mbt zapatos no es coincidencia que es tan popular. Esta es una forma de cambio histórico Mbt zapatos son.yll (Email) - 31 July '10 - 06:46
[url=[[http://www.cheapjerseyschina.com]disco..]] nfl jerseys from china[/url][url=[[http://www.cheapjerseyschina.com]disco..]] nfl jerseys wholesale[/url]
tanglnog (Email) (URL) - 03 August '10 - 02:08
Según la investigación, Mbt zapatos no sólo su propio estilo de diseño único, sino también porque hay magia buena forma física, así Mbt zapatos no es coincidencia que es tan popular. Esta es una forma de cambio histórico Mbt zapatos son.save fuel (Email) (URL) - 24 August '10 - 14:59
excellent article , I added you to my [[http://www.china-wholesale-directory.c..]] Top China Wholesalers category.. thanks for sharing the article!China Wholesale Directory (Email) (URL) - 25 August '10 - 08:13
very cool article ,thanks for sharing the article!like my cool stuff .very useful.uCoolStuff is the leading China wholesaler for [[http://www.ucoolstuff.com]] cool stuff [[http://www.ucoolstuff.com]] cool gifts , unusual gadgets and other unique gift ideas. We provide the very latest cool stuff and cool gifts for you
cool stuff (Email) (URL) - 25 August '10 - 08:19
The True Religion RICKY SUPER BIG T MEN JEANS are some of the most uniquely hand-crafted jeans on the market for guys!Every pair of True Religion Jeans is made in America and designed with a 1970’s feel.Kylie Bissutti in True Religion Jeans Outlet Contest winning Victoria’s Secret model, Kylie Bisutti was spotted stopping by Anastasia Salon to have her eyebrows shaped wearing a pair True Religion Brand.Even though women own an average of Seven Jeans they tend to only wear four of them on a regular basis.[[http://www.jean-jeans.com]]
True Religion Jeans Outlet (Email) - 28 August '10 - 10:43
The best Coach Outlet and Coach Purses supplier on the Internet here. We are proud to offer our customers high quality and low price Coach Bags.About 70% discount now.Here has been dedicated in the business of best quality Coach Bag.[[http://www.handbags-onsale.com]]
Coach Bags (Email) - 28 August '10 - 10:44
[[http://www.macvobconverter.net]] VOB Converter mac is 2010 the hottest software which is very popular among mac users who want to convert VOB to other video formats on mac like MP4, AVI, WMV, MPG, MPEG, MOV, TS, FLV, etc. With super fast conversion speed and high output quality, it can ensure you a fluent process to convert VOB mac. And in the same way, this Mac VOB Converter can convert various video formats to VOB for enjoyment.[[http://www.macvobconverter.net/mac_vob..]]