The Blog of Ben Rockwood

Crashing Solaris for Fun and Profit

09 June '09 - 23:00 by benr

Crashing is the wrong title actually. We're talking about panics. Its sort of like saying "hacking" when you mean "cracking". A "crash" is when an OS preforms some operation that typically causes the system to reboot. Solaris is very unique from rival Linux in that 99% of the time such an event will be caught by the OS and handled as a "panic" instead of an uncontrolled crash.

While its not a sexy feature of Solaris, panics are extremely important things. The reality is that sh*t happens, it just does. When it does happen you want to collect as much information about the event as possible to fix the problem that resulted in badness. Therefore, the advantage of a panic over a crash is a crash dump which can be analyzed for the cause. This is why panics are a very good thing indeed. As I like to say, "If you've gotta crash, fine, but you'd better give me a reason!"

This blog entry is interested in the 1% of Solaris issues that don't result in a panic, but you wish would. In between the concepts of a "crash" and a "panic" is that dreaded situation we call "hung". A "hung" or "locked" or "wedged" system is one that is still running, technically, but is otherwise unusable. Typically this is the result of the kernel being ok but the userland stack being trashed beyond repair. Perhaps the two most common causes of this are abuse cases such as memory/swap exhaustion or fork bombs, where the kernel is still running but processes can't spawn to let you see whats happening. The normal way to deal with this is to reboot the system, either via IPMI or sending some poor soul (typically, you) down to the data center to unceremoniously press the big button.

So here's the problem with rebooting a system in a bad way.... it doesn't panic, meaning you might get the system back up post-reboot but you have no idea what happened unless something in the logs tips you off. If there are no relevant logs entries you have to simply shrug and pray that it doesn't happen again. So what can you do about it? On SPARC systems you'd hit "STOP-A" and sync the box. But, what about Solaris/X86.... what do you do then?

Thankfully there are three very handy ways of dealing with such situations. They can be combined or used individually. Lets take them in turn.

Please note! This entry applies only to Solaris/X86!

Panic on NMI

Intel introduced a concept of Non-Maskable Interrupts a long while back. These NMI's are extremely high priority and can not be blocked by the OS. While I've had trouble fully researching them, the most common use is to kick an otherwise unresponsive system into a "diagnostic" mode. On some systems its implemented as a jumper, others a button, yet other IPMI or SP commands. In the case of Solaris/X86 there are two tunables that can cause the OS to react to NMI; one causes a panic, yet another causes the system to drop into a kmdb session for live debugging. By default Solaris will simply output a message to console saying an NMI was received but otherwise do nothing.

The first of these two tunables can be added to /etc/system, it is not dynamic. You must reboot for it to take effect, it is:

set pcplusmp:apic_panic_on_nmi=1

Once this is added and the system rebooted the receipt of an NMI will cause an immediate panic followed by reboot. The most common way to invoke this behavior would be via the IPMI command: ipmitool -I lanplus -H somehost -U root chassis power diag. Remember, if you get a seriously stuck system your going to reboot the box anyway, typically via IPMI, so instead of using "power cycle" you choose "power diag" and have some tasty data to dig through (or send to Sun).

I like to call this feature "panic on demand". :)

Enter kmdb On NMI

In addition to, or instead of, the panic option above, we can use the following tunable to drop the system into the kmdb kernel debugger on NMI receipt:

set pcplusmp:apic_kmdb_on_nmi=1

If it follow the panic option above it'll panic and then drop into the debugger, which is the best option. But please know that this only will work if you have kmdb loaded at boot time by adding the -k kernel argument via your bootloader (ie: GRUB). If your working with a production system this might not be something you want hanging over your head all the time and thus a more developer oriented solution in my mind. If your writting drivers you'll certainly want to keep kmdb loaded, but everyone else will more likely prefer the "panic and reboot" option above.

Snooping; The Deadman Watchdog Timer

Ever watch or read The Abyss? There was a mini-sub that would automatically float to the surface with surveillance tapes and such, if a timer wasn't reset every 12 hours, in hopes that if anything went wrong there would at least be a partial record of events. Watch dog timers are a similar concept... the kernel uses some means to determine function and if it ceases to function the white flag is waived and it panics.

While its rare, the comment in code (line 163) is the best explanation: "Setting "snooping" to a non-zero value will cause a deadman panic if snoop_interval microseconds elapse without lbolt increasing. The default snoop_interval is 50 seconds."

So we simply add the following tuning(s) to /etc/system:

set snooping=1
set snoop_interval=90000000

The first line enables snopping. The second line changes the default 50 second interval to 90 seconds. I don't see any reason that 50 seconds isn't long enough, but if you want to be paranoid and use a 5 minute interval you can, just change 300 seconds to microseconds and reboot to lock it in.

Snooping has been safe in all my testing to date, but obviously it will feel risky to the casual sysadmin, so this is not something I'd enable by default. If, however, you have a system that mysteriously goes into a dead hang in the middle of the night, this is a better option than being woken up just to testify that it did actually reboot and you still have no idea why. :)

Poking NMI

As stated, the best way to poke NMI once your ready for it is via IPMI (tested on Sun and Dell):

# ipmitool -I lanplus -H somebox -U root chassis power diag

If you have a moderately recent version of ILOM you can poke the /SP/diag/generate_host_nmi value like so:

-> cd /SP/diag
/SP/diag

-> show

 /SP/diag
    Targets:
        snapshot

    Properties:
        generate_host_nmi = (Cannot show property)
        state = disabled

    Commands:
        cd
        set
        show

-> set generate_host_nmi=true
Set 'generate_host_nmi' to 'true'

What Next?

One you've got a dump things are on the uptick, but given that you may not be a kernel developer, and if your wise have no desire to become one, you have some options. The first is to send the panic to Sun and hope they come back with a good answer. The second is to use this as your opportunity to dig deep into the guts of Solaris and learn something interesting. I recommend reading my blog entry some time ago: Solaris Core Analysis, Part 1: mdb and Part 2: Solaris CAT, reading The Solaris Operating System on x86 Platforms: Crashdump Analysis Operating System Internals (PDF), getting the latest SolarisCAT, and if possible attending one of Max Bruning's courses.

Finally, a shout out to my p33ps in #opensolaris IRC for their help in assisting with my testing and research on NMI, in particular "LONGCAT".

- - C O M M E N T S - -
I’m a little puzzled by your comment about Linux—I haven’t seen many uncommanded reboots of Linux that weren’t accompanied by kernel panics, except in cases of major hardware issues. (In my experience, these are often the result of power supply faults, which the OS obviously can’t do anything about.)

Linux can produce dumps on kernel panics, although it doesn’t seem to be common practice to do so.

Maybe I’m just missing the point.

David Brodbeck (Email) - 11 June '09 - 17:27

You said it, “although it doesn’t seem to be common practice to do so”. If a panic doesn’t result in a dump its unfruitful.

benr - 11 June '09 - 17:54

Fair enough. It’s really a configuration issue—the kernel is fully capable of producing dumps, but most distributions don’t configure it to do so. I suspect this is because most Linux users aren’t kernel hackers and wouldn’t know what to do with a dump if they had one. It’s a bit like the situation with “core” files, which on probably 90% of systems do nothing but sit around and take up disk space. ;)

David Brodbeck (Email) - 11 June '09 - 22:38

Thanks for posting the article, was certainly a great read!

jordan shoes (Email) (URL) - 12 June '09 - 02:37

This describes exactly the symptoms one of our production boxes is having. And it is handling a lot of tasks with not much memory.

david stahl - 12 June '09 - 14:24

Everything ought to be beautiful in a human being: face, and dress, and soul, and ideas.

runescape gold (Email) (URL) - 23 June '09 - 09:21

You’ve got your deep-sea underwater science-fiction films mixed up. “Sphere” was the film where they had to reset the mini-sub every 12 hrs.

John (Email) - 26 June '09 - 16:10

Thanks for your nice sharing first.I don’t know if you are shopping online or not.I always do this.
I think it is a very convenience and save time way to buy goods.I often buy jordan shoes , UGG Boots, Tiffany jewelry on the Internet.you can seach Air Jordan ,Nike Shoes to find the jordan shoes, and I also like Nike Shoes, they are good wear. Anyway,I like Tiffany jewelry best.It can make me more confident and buty.It’s really a good shopping way,if you haven’t do that,try it.You will enjoy yourself.

air jordan (Email) (URL) - 01 July '09 - 09:28

Your article very interesting, I have introduced a lot of friends look at this article, the content of the articles there will be a lot of attractive people to appreciate, I have to thank you such an article.

runescape accounts (URL) - 08 July '09 - 03:57

Thanks for all of this great information. I can see you worked hard and did a great job. The techies of the world appreciate it.

How Old (Email) (URL) - 12 July '09 - 00:19

Very thought provoking. I like the sort of ending that leaves it opn to personal input. Makes it work for just about everyone I think. Nicely done! I’ll subscribe.

supra shoes (Email) (URL) - 13 July '09 - 09:36

Hey, your posts have inspired me! – I love the way you directly get to the point, and then work outwards. I’ve been trying to do figure out what I want to say about ,that would allow me to do exactly the same thing.

chi flat iron (Email) (URL) - 24 July '09 - 05:58

Thanks for your post.

Gucci Bags (Email) (URL) - 02 August '09 - 05:00

I recently came accross your blog and have been reading along. I thought I would leave my first comment. I don’t know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.

air jordan shoes (Email) (URL) - 06 August '09 - 09:54

nike shoes jordan shoes air jordans air jordan jordan shoes air jordans

nike shoes (URL) - 14 September '09 - 07:20

We offer wholesale sport jerseys(NFL,NBA,NHL,MLB),brand-name shoes, T-Shirt, bags, jeans, etc at a low price. If you

are interested in our products,Please kindly visit our website .”Best Service, Highest Quality and Special Price”is our company principle
welcome to our website:

www.tradebeneficiary.com

linadams (Email) (URL) - 18 September '09 - 07:22

thanks.

nike dunks (Email) (URL) - 18 September '09 - 20:33

thanks for your post

ugg 5815 (Email) (URL) - 28 September '09 - 07:10

I agree, Linux can produce dumps on kernel panics, although it doesn’t seem to be common practice to do so.

discount furniture (Email) (URL) - 05 October '09 - 07:17

However, they do not only for the fleece actually but the primary purpose is the mutton for table eating.

ugg boots uk (Email) (URL) - 18 October '09 - 00:40

[[http://www.usa-jordan.com]]

air jordan (Email) (URL) - 26 October '09 - 07:32

It was a very nice idea! Just wanna say thank you for the information you have shared. Just continue writing this kind of post. I will be your loyal reader. Thanks again.

Christian louboutin shoes (Email) (URL) - 27 October '09 - 01:28

This is what I look for a long time information, are very grateful.CHI Flat Iron Blue In Love

newshopstyle (Email) (URL) - 05 November '09 - 06:28

I think all of the woman must love jewelry,but not all of them can buy it for it’s high price,even the top brand like tiffany and gucci,but ,now,you can buy it from our website for lower price,come on.

newshopstyle (Email) (URL) - 05 November '09 - 06:29

Thank you very much!

cheap links of london (Email) (URL) - 07 November '09 - 01:08

Air Max shoes

ugg (Email) (URL) - 10 November '09 - 09:29

It is a contoured base that conforms,memoryfoam-like,to your foot so that you almost feel like the shoes was made for you.It is then reinforced to provide extra support.The bottom tier is a combination of th “Classis” and the “Pods” that are seperated in the flat version of the shoes.

wholesale gucci handbags (Email) (URL) - 12 November '09 - 14:14

Retail UGG australia boots on sale ，we are retail ugg boots seller,we offer original australia ugg boots spend less money to got a comfortable and warm winter.UGG classic cardy,ugg classic mini,UGG nightfall are still sales hot the years.for a fashion life style,uggs are worth to buy!

bailey button uggs (Email) (URL) - 16 November '09 - 07:54

Great post! Hope to be better. Better means more features.
good post,I think so!
Thanks for your information, i have read it, very good！
Bing is a really overlord!! support Bing~~
This is great news. Best of luck for the future and keep up the good work.

links of london (Email) (URL) - 17 November '09 - 03:25

However, they do not only for the fleece actually but the primary purpose is the mutton for table eating. [[http://www.ugg-mall.net]] ugg boots on sale

ugg boots on sale (Email) (URL) - 18 November '09 - 09:12

There are several Gucci shoes that are considered to be elegant and classic designs.When it comes to footwear for wearingout on the town they have a great collection of suede sandals,satin mules,Gucci pumps and more.

gucci man shoes (Email) (URL) - 21 November '09 - 04:08

Ugg online
Cheap ugg boots
Discount uggs boots
Ugg ultra short
Ugg classic

Cheap ugg
Ugg sale
Ugg classic tall
Shopping for ugg botts

ugg online (Email) (URL) - 23 November '09 - 06:00

I think I will try to recommend this post to my friends and family, cuz it’s really helpful.ugg classic tall
Thank you so much my mother is going to love the ugg classic mini.

ertry (Email) (URL) - 25 November '09 - 05:08

good

air jordan shoes (Email) (URL) - 01 December '09 - 07:51

[url=[[http://www.uggbootshome.net]Ugg]] Boots[/url]

ugg boots (Email) (URL) - 02 December '09 - 07:32

[[http://www.uggbootshome.net]] Wholesale Ugg Boots

wholesale ugg boots (Email) (URL) - 02 December '09 - 07:32

This is what I look for a long time information, are very grateful.ps:welcome to my site
ugg knightsbridge boots

ugg (Email) (URL) - 02 December '09 - 10:13

Gucci is considered one of the most influential handbag and shoes designers in the fashion industry. The label is synonymous with opulence and quality, and is widely regarded as a status symbol.

cheap gucci shoes (Email) (URL) - 05 December '09 - 01:58

[[http://www.cheapuggsonline.net/]]
ugg boots

[[http://www.nikestarsonsale.com/]]
basketball shoes

JUN (Email) (URL) - 07 December '09 - 02:20

I like this ugg ultra short, What is your refund/returns policy?
Look at this ugg ultra tall that i bought last week, Pretty! Fast service. Good Seller

ere (Email) (URL) - 10 December '09 - 15:26

hair products in the market and 100% Quality Guarantee

chi hair tools (Email) (URL) - 14 December '09 - 00:59

[[http://www.cheapuggsonline.net]] cheap ugg

[[http://www.cheapuggsonline.net]] ugg outlet

cheap uggs online (Email) (URL) - 17 December '09 - 06:15

The sale of fake Gucci shoes is now a common event. But, the online shopping platform adds a fresh layer of deception. Individuals buying the fashionable shoes do not understand that the products they are buying are fake.

gucci man shoes (Email) (URL) - 19 January '10 - 04:19

[[http://www.buykamagra.com]] buy kamagra
[[http://www.viagracialis.com]] viagra cialis

M65 Jacket (Email) (URL) - 21 January '10 - 02:13

Ugg Boots,also called Ug or Ugg boots,are nowadays one of the most popular boots.They existed already in 1930’s and became popular in 1960’s when Australian surfers found out,that they can warm their feet after coming out from cold water.They have everything you need-comfort and warmth.You will never be cold in those boots because 100% sheepskin will warm you even in the coldest days.The natural thermal properties of sheepskin make possible that Ugg Boots will warm you in the winter but in the summer your feet will not be wet.It is true that Ugg is shorter from ugly,but if you know even little bit about fashion you will know that Ugg boots are not ugly.We can see famous people like Pamela Anderson or Kate Hudson wearing Ugg Boots.

Nowadays we have new improved Ugg boots for sale.You can get them for anybody.We have women boots,children boots,slippers and casuals.

Our UGG Australia Boots are all authentic.Their properties and features as below:

*In cold temperatures,the fleece works as an insulating layer of warmth and keeps the warmth of body heat inside of the boot.
*In the heat,the wool fibers keep the air circulating; thus keeping your fee dry and pulling the perspiration away from your skin.Sheepskin also contains lanolin,which moisturizes your feet when worn barefoot.
*ugg boots have reinforced toes,which are double stitched and extra durable heels.
*feature a natural,thick fleece on a removable in-sole for arch support.
*The sole is usually durable in most terrain,lightweight and provides traction as well.
welcome to [[http://www.bootscardy.com]]

bootscardy (Email) (URL) - 24 February '10 - 13:50

gemahrv0225
Replica Louis Vuitton
Replica Watches

replica handbags (Email) (URL) - 25 February '10 - 08:51

All people deserve good life and loans or just auto loan will make it better. Just because freedom is based on money.

Anita28Brown (Email) (URL) - 05 March '10 - 06:21

heyI appreciate the concern which is been jobs. The things need to be sorted out because it is about the individual but it can be with everyone.
Job Sites

Job Sites (Email) (URL) - 10 March '10 - 07:56

Are you want to buy uggs?There are many good ugg boots

gucci shoes (Email) (URL) - 11 March '10 - 05:24

Well in this situation farming for the jordan dmp can only be made easy if Discount UGG Boots jordan dmp saleyou have right World of Warcraft tools in your hand. One of the options open before you is to.258479852

air jordan shoes (Email) (URL) - 13 March '10 - 03:14

Louzhu you write the article is my favorite model

cheap mbt shoes (URL) - 20 March '10 - 07:14

there are so many people like ugg boots, but the price for them are too high , it is a pity, if one can not get one, but now, I would like to recomend you one site where Wholesale Ugg Boots
Discount Ugg Boots

cheap, discount Ugg Bailey Button Boots 5803 for sale (Email) (URL) - 21 March '10 - 04:27

if you want to buy here is some good Website
for another you can see this any more

mbt shoes (Email) (URL) - 06 April '10 - 01:51

we sell cheap christian louboutin shoes ,such as louboutin boots,loubouitin sandals,louboutin pumps,louboutin evening,louboutin wedges and so on,all at good design and free shipping.
[[http://www.mychristianlouboutinshoes.c..]]

louboutin sandals (Email) (URL) - 08 April '10 - 03:23

This is just a theory but I think it’s pretty sound.[[http://www.bigtimekicks.com]]

air jordans (Email) (URL) - 09 April '10 - 02:19

There you can find many different kinds of cheap Air Jordan Shoes .We are a wholesaler of the Air Jordan Shoes

air jordan (Email) (URL) - 09 April '10 - 08:27

Supra Shoes are always one of the top skateboarding shoes on the block. Overload Skateshop carries a wide range of Supra Shoes to fit your 9-stair kickflips. If you are looking for a pair of Supra Shoes…you have came to the right place. I will introduce supra shoes at next time.cheap supra sneakers.

Supra (URL) - 12 April '10 - 03:10

we supply cheap jordan shoes, we believe that to bring you a lot of surprises.All our products are free shipping,non-tax,7days to your home,incredible discounted price for top quality products.

jordans2010 (Email) (URL) - 19 April '10 - 07:57

Hello – I must say, I’m impressed with your site. I had no trouble navigating through all the tabs and information was very easy to access. I found what I wanted in no time at all. [[http://www.mbtsalestore.com]]

MBT Shoes (Email) (URL) - 20 April '10 - 09:00

Dear friend,

Weclome to our website at [[http://www.aj-family.com]]

We supply various fashionable and comfortable A.J. 1 to A.J.23 shoes with A1 quality and original packing at good prices.

Wholesale and sample order are both accepted.We also do drop shipping.

Shipping methods are EMS,TNT and DHL etc and the delivery time is approximately 5 days.

We’ll do our best to meet your needs according to your request and also guarantee fast and safe ship door to door to you.

If you have any interest of them,just feel free to contact us at any time.

Sincerely hope we can start a good business relationship on the basis of equality and mutual benefit.Thank you in advance.

nike jordan (Email) (URL) - 20 April '10 - 09:23

it is interesting,god bless you!

Solstice Kron (Email) (URL) - 22 April '10 - 03:11

it is interesting,god bless you

Wholesale Handbags (Email) (URL) - 22 April '10 - 07:23

I very intersted in the article

cheap mbt shoes (Email) (URL) - 23 April '10 - 03:34

Index futures contracts have been traded actively since their debut in Shanghai on April 16. But analysts said the current high turnover rate and small holding size indicated that the market was dominated by retail investors.

wholesale supras shoe (Email) (URL) - 27 April '10 - 02:29

Reaching the Chinese consumer has been something of a holy grail for international businesses, even when the country had only 400 million people in the 1930s.

cheap supra shoe (Email) (URL) - 27 April '10 - 03:05

Most of the students have been discharged from hospital, but eight remained for further medical observation, according to the spokesman.

real supra shoe (Email) (URL) - 27 April '10 - 03:40

Japan has no such ability, while Russia has no such motivation and India is more worried about China, Yang explained.

sale supra shoe (Email) (URL) - 27 April '10 - 06:17

An Ultra-orthodox Jewish family look at a construction site in the east Jerusalem neighborhood of Ramat Shlomo, Monday, April 26, 2010. [Agencies]

buy supras shoe (Email) (URL) - 27 April '10 - 06:38

“The government ordered the Interior Ministry immediately after the Biden incident to not even talk about new construction for Jewish homes in east Jerusalem,” said Meir Margalit, a city councilman who said the information came from top Jerusalem officials intimately involved with construction projects.

wholesale supras shoe (Email) (URL) - 27 April '10 - 06:58

[[http://www.gucci-shoes-bags.com]]
[[http://www.guccinewshop.com]]
Dear friends welcome to our store: we have a specail offer now which is once you buy any product you can get a free gift as well, these gifts are in the gift area, you can free to choose and please add to the shopping cart, we will send it together with your purchased product. Thanks!

ymtina (Email) (URL) - 27 April '10 - 09:06

If you want a great couple of comfortable shoes, accept up the casual and sophisticated supra shoes. These amazing Supra shoes will be a sensible asset for those with orthopedic issues and bottom problems. Pick up these designs for comfort all the way.

Supra (Email) (URL) - 04 May '10 - 03:19

P90x .It really is not expensive if you factor in the cost

of a gym membership,P90x workout . The cost for P90X is

about three months of a paid gym membership but you get to

keep the program foreverP90x . You can try many of the

online sites, but it will be the same as buying from the

company or a Beachbody Coach. Make sure you are getting

original DVD’s. People are selling copies all over. The

problem is how long will they last, P90x workout ,and you

truly need the exercise and nutrition guide to even follow

the program. You can go to any site or you can go to

[[http://www.p90xmall.com]] and click on products. P90x dvd You can

order directly from the site,P90x dvd.

p90x (Email) (URL) - 05 May '10 - 05:55

These two colorways of the Air Max LeBron VII are inspired by two of the biggest idols zoom lebron had as a child. The red pair is inspired by Deion Sanders, while the other pair honors Penny Hardaway. The pair that is inspired after Penny Hardaway borrows the colors from the Orlando Magic.

nikecorner (Email) - 05 May '10 - 13:44

Do you knowtales of pirates gold?if you play the online game,you will knowbuy tales of pirates goldis the game gold.

tales of pirates gold (Email) (URL) - 07 May '10 - 05:33

thank you ,I have learned a lot.

MBT Shoes (URL) - 14 May '10 - 15:12

Toshiba satellite a100 Battery [[http://www.adapterlist.com/toshiba/sat..]]

laptop battery (Email) (URL) - 15 May '10 - 03:43

Dell rn873 Battery [[http://www.globallaptopbattery.co.uk/d..]]

laptop batteries (Email) (URL) - 15 May '10 - 03:44

As you may get to know something more about the jordan retro iii

Shoes
jordan retro iii

fv (Email) (URL) - 18 May '10 - 05:22

As you may get to know something more about the jordan retro iii

Shoes
eg jordan retro iii [[http://www.mydiscountjordanshoes.com/a..]] jordan retro iii

fv (Email) (URL) - 18 May '10 - 05:22

First of all, do not hurry when buying Coach Bags, Footwear is the matter of fashion and style.

Coach Bag (Email) (URL) - 19 May '10 - 01:54

it is good!

bright shadow gold (Email) (URL) - 20 May '10 - 05:35

the blog is informative.

UGG sale (Email) (URL) - 27 May '10 - 02:12

the article is useful.

chi Hair straightener (Email) (URL) - 27 May '10 - 02:14

that is great.

fitness (Email) (URL) - 27 May '10 - 02:15

Buy Nike Air Max 90 Shoes just $45-55 USD in[[http://www.iofferitems.com,]], 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!

nike air max 90 shoes (Email) (URL) - 28 May '10 - 02:09

Buy Nike Air Max 90 Shoes just $45-55 USD in [[http://www.iofferitems.com,]], 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!

nike air max 90 shoes (Email) (URL) - 28 May '10 - 05:28

wholesale hats [[http://www.aj-wholesale.com]]

wholesale hats (Email) (URL) - 29 May '10 - 05:49

wholesale hats [[http://www.aj-wholesale.com]]

wholesale hats (Email) (URL) - 03 June '10 - 06:37

[[http://www.mbtshoeslatest.com]]
[[http://www.nikeairmaxshoe.com]]

Nike air max shoe (Email) (URL) - 24 June '10 - 07:53

Do you like the ugg ? Ugg Classic Short Boots are UGG Australia’s original heritage styles. Ugg Classic Tall will keep your feet dry and the ultimate comfortable.
[[http://www.ugg-boots-london.com/ugg-cl..]]

Ugg Classic Short (Email) (URL) - 07 July '10 - 01:21

UGG created fashion lines such as the Carolinas and Boots Stella. These boots have a sheepskin found the same as in UGG classic boots other hardships of winter, but with the heel provide a touch of fashion. Other varieties are in the style of UGG classic boots Cove and Knightsbridge, the soles have secure grip and top with unique and elegant.

cheap ugg boots (Email) (URL) - 21 July '10 - 08:53

[[http://www.cheapjerseyschina.com]]
[[http://www.jerseyscloset.com]]

tanglnog (Email) (URL) - 31 July '10 - 02:45

[[http://www.jerseyscloset.com]]

tanglnog (Email) (URL) - 31 July '10 - 02:46

I really liked reading your post!. Quallity content. With such a valuable blog i believe you deserve to be ranking even higher in the search engines :). Check out the link in my name. That links to a tool that really helped me rank high in google. This way even more people can enjoy your posts and nothing beats a big audiance ;)

Michael Jordan Shoes (Email) (URL) - 03 August '10 - 13:18

Amazing write-up! This could aid plenty of people find out more about this particular issue. Are you keen to integrate video clips coupled with these? It would absolutely help out. Your conclusion was spot on and thanks to you; I probably won’t have to describe everything to my pals. I can simply direct them here!

Air jordan 1 (Email) (URL) - 03 August '10 - 13:19

that is very kind of you to do this for us, thank you very much. you can have a look at the [url][[http://www.monclershop.org]] [/url]
[url][[http://www.monclerjacke.org]] [/url]

moncler jacket (Email) (URL) - 04 August '10 - 01:31

that is very kind of you to post this for us, that is very good. you can have a look at the [url][[http://www.uggbootsoutlet.biz]] [/url]
[url][[http://www.ugg-sale.biz]] [/url]

uggs outlet (Email) (URL) - 04 August '10 - 01:34

ugg clothing
ugg clothing boots
women’s ugg clothing boots – free shipping [[http://www.uggclothingboots.org]]

ugg outlet (Email) (URL) - 23 August '10 - 13:39

ugg outlet,ugg boots outlet,ugg boots store [[http://www.ugg-boots-outlet.org]]

ugg clothing (Email) (URL) - 23 August '10 - 13:40

new ugg
new uggs
new ugg boots – free shipping [[http://www.new-ugg-boots.org]]

new ugg (Email) (URL) - 23 August '10 - 13:41

We always look for something can’t reach, walk around the real truth and can’t achieve the goals we expect, therefore, we still not know how to get it, do it and make it.

ugg boots uk (Email) (URL) - 24 August '10 - 07:50

Thanks for the valuable information. This was just the thing I was looking for; I really like how it includes the actual curved shape flight paths. Keep posting. I will be visiting back soon.

cheap uggs (Email) (URL) - 24 August '10 - 07:55

excellent article , I added you to my [[http://www.china-wholesale-directory.c..]] Top China Wholesalers category.. thanks for sharing the article!

China Wholesale Directory (Email) (URL) - 25 August '10 - 08:14

very cool article ,thanks for sharing the article!like my cool stuff .very useful.
uCoolStuff is the leading China wholesaler for [[http://www.ucoolstuff.com]] cool stuff [[http://www.ucoolstuff.com]] cool gifts , unusual gadgets and other unique gift ideas. We provide the very latest cool stuff and cool gifts for you

cool stuff (Email) (URL) - 25 August '10 - 08:17

Five Fingers,Vibram Five Fingers,Vibram Five Fingers Shoes
[[http://www.cheapvibramfivefingersshoes..]]

Five Fingers (Email) (URL) - 28 August '10 - 16:38

vibram 5 fingers,5 fingers shoes,vibram 5 fingers shoes [[http://www.ibram-5-fingers.org]]

vibram 5 fingers (Email) (URL) - 28 August '10 - 16:42

thank you.is good
[[http://www.airjordan-sale.com/michael-..]] Michael Jordan Shoes

air jordan sale (Email) (URL) - 06 September '10 - 16:35

Top Online Stores is a SEO Friendly [[http://www.toponlinestores.org]] free directory where you can find the best online shopping stores selected by hand and sorted by category [[http://www.china-wholesale-directory.c..]] china wholesale .

link directory (Email) (URL) - 14 September '10 - 07:24

that is very kind of you to do this for us, thank you very much. you can have a look at the cheap ugg boots.

uggs sale (Email) (URL) - 15 September '10 - 09:38

百度
[url=[[http://www.sina.com]sina[/url]]]]

Tashard Choice JerseysPatrick Crayton Jerseys (Email) (URL) - 16 September '10 - 00:53

In probing and examining the oceanic surface of the world named Solaris from a hovering research station the oblivious human scientists are, in turn, being studied by the sentient planet itself. In due course, Solaris scientifically probes for and examines the secret, guilty thoughts of the human beings who are analyzing it. Solaris has the ability to manifest their secret, guilty concerns in human form, for each scientist to personally confront, while the self-aware planet studies their responses to its psychological experiments.

Solaris is pervaded by a powerful, poetic sense of the physical remoteness of outer space. The sense of loneliness that this engenders is among Lem’s philosophic explorations of man’s anthropomorphic limitations. First published in Warsaw in 1961, the 1970 Polish-to-French-to-English translation of Solaris is the best-known of Lem’s English-translated works.

wow accounts (Email) (URL) - 20 October '10 - 07:55

Very happy to see your article, I very much to like and agree with your point of view. Thank you for sharing. At the same time,i love

offshore oil rig jobs (Email) (URL) - 27 October '10 - 03:30

00zoe

MBT shoes[[http://www.mbtshoes4sale.co.uk]]
Timberland boots[[http://www.timberlandbootsale.co.uk]]

MBT shoes clearance (Email) (URL) - 01 November '10 - 03:06

Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with more information? It is extremely helpful for me.

beat making software (Email) (URL) - 18 November '10 - 03:01

Its nice to read some post about this nowadays. Its meaningful and somehow sharing this to readers like me, makes me want to surf the web to be able to get more wonderful ideas.Thank youfor posting this…

how to get your ex girlfriend back (Email) (URL) - 24 November '10 - 04:20

[[http://www.uggonlinebootssale.com/]]
Days are getting colder.Are you looking for a pair of great boots to keep your feet warm?The answer for you is ugg boots.

Ugg boots sale (Email) (URL) - 24 November '10 - 06:11

Love is frail at birth, but it grows stronger with age if it is properly fed.

cheap uggs uk (Email) (URL) - 29 November '10 - 04:02

[[http://www.jacket-buy.com]]

adidas shoes (Email) (URL) - 05 December '10 - 10:33

11zoe
cheap ugg boots[[http://www.cheapuggbootssale.co.uk]]
Timberland boots sale[[http://www.timberlandboots4sale.co.uk]]

cheap ugg boots (Email) (URL) - 13 December '10 - 03:12

[[http://www.cheapuggukboots.com/]]

cheap ugg (Email) (URL) - 04 January '11 - 03:36

However, how to distinguish ugg boots? This is a problem, if you want to learn how to do it, you’ll remember some tips and be careful when you choose ugg boots.

ugg boots au (Email) (URL) - 20 January '11 - 03:08