IO Benchmarking: How, Why and With What

22 May '07 - 06:03 by benr

I've been meaning to bring up the dreaded benchmark topic for some time. Benchmarking is like some type of taboo... everyone wants to but you get chastised if you talk about it. That, of course, only fuels the desire that much more. But how? What do you use? When do you use what tool? Its really hard to find good information and so here is my attempt to shed some light on things.

Before we begin, lets look at the word itself: benchmark. The term comes from wood working and other such crafts. When a craftsman is making, for instance, table legs you first carefully cut your wood to size, measuring for exactness. Once it's done properly you can then take that funny flat pencil, make a pencil mark on your bench and then use that to measure up all the following pieces to speed the process along. Based on the mark you know if your other pieces are too long or short and make the needed alterations. The mark is not a measurement per se, but rather a quick method of judging difference between things that should be similar.

Now we move into the storage world, the way in which we put that mark on our bench is different, but the purpose is exactly the same. Benchmarks are not about judging how fast something is! They are for judging change! Let me make that crystal clear... if you want to run some piece of software, a benchmark suite, against some peice of storage and definitively say "Its X fast." you will always be wrong and people will always mock your numbers. The reason for this is self evident, any performance you see is based on the total environment in which it is run which can almost never be duplicated perfectly. In the storage world there are countless variables that mix things up: cabling, HBA's, firmware, disks, disk firmware, OS, drivers, CPU's, memory, and on and on. Just a single rev of firmware on an HBA or BIOS can have a major impact on your performance, so just because one person can get 180MB/s sequential writes out of his 3Par doesn't mean that your going to get that same number.

Despite all that common sense stuff, we all still want one magical number that describes our l337 setup. But those numbers don't work either because of various factors. Is the data sequential or random? Read, write or both? Whats the mix? Big files or small? No handful of numbers will make it clear. All these things contribute to the "benchmarks are useless" flamewars that occur all the time.

So, what then? Don't benchmark? Stay oblivious to your performance thresholds and just rely on performance counters? Lets get real, we all want to know what a given solution can do and so even if we're afraid to tell anyone what our numbers are we still do them.

The following is a rundown on the various methods of benchmarking that I find to be common and that I myself use, including when to use them, how and why.

dd

Do you know how benchmarks really work? Seriously, do you? Or do you just download some software, run it, and trust it? Everyone should start with our old friend dd. Using it is simple, read from here and write to there in some block size. This is perfect for determining sequential performance and getting a baseline "how fast can I read? how fast can I write?" sort of picture.

Start with a small sequential write block size, like this: time dd if=/dev/zero of=/mystorage/testfile bs=8k

Let that run for about 30 seconds. While it runs, watch the IO using tools like fsstat or iostat. When you hit 30 seconds stop it (^c) and do the math... n blocks transfered at 8K, so 8K times n, divide that by the exact time reported by the output, and presto we have number. Alternatively just divide the output file size by the time.

root@aeon ~$ time dd if=/dev/zero of=/iscsi/benchmark/testfile bs=8k   
^C
97810+0 records in
97810+0 records out

real    1m52.933s
user    0m0.048s
sys     0m2.487s
root@aeon ~$ 
root@aeon ~$ ls -lh /iscsi/benchmark/testfile
-rw-r--r--   1 root     root        764M May 22 03:42 /iscsi/benchmark/testfile
root@aeon ~$ bc
60+52.9  
112.9
764/112.9
6

Now, do it again, but this time with a 32k block. Then with a 128K block, a 512K block, 1MB block, 8MB block. Does the number change? How does it change? All this is important and something that storage admins rarely understand well untill they do this kind of exercise. This is exactly why I think every one should start with dd. Once you understand sequential performance at various block sizes, turn it around and do it all again but this time reading from a file to /dev/null. How does that look?

These types of tests provide very good ideas of what your best possible sequential performance is, and since most storage preforms sequential IO best this is normally the number I use for my "My Thumper can push line speed!" type statements to management.

But then lets be reminded right here and now.... real world applications almost never do huge quantities of sequential reads or writes. Its normally either very random or at best a mix of random and sequential performance. Think of it like this... Which is faster, a Top Fuel Dragster or a Formula1 car? The dragster is the fastest, but it can't turn... the forumla1 care can turn like crazy but has to slow down for the corners. This is, imho, a good way to think of sequential vs random IO. If you want to see this in action watch The 24 Hours of Le Mans, where huge CanAm cards scream at 225MPH down the straight but then get passed by Porsche 911's in the corners.

dd is also handy for testing raw devices. Benchmark /dev/rdsk/c1t1d0s2, then create a filesystem on it and benchmark using a file on that. Do tests against the charector device (/dev/rdsk) and block device (/dev/dsk) without a filesystem. On and on, there are lots of fun tests you can do with this simple tool that will help you better understand the underlying inner workings.

If you want to really get into it pick a single disk, download the spec sheet for the drive model from the manufacturer, and compare your numbers. Then partition the disk so that there is a 200MB partition at the beginning of the disk (starting at sector 0) and a 200MB partition at the end, and then re-run your tests against those two partitions. Very quickly you'll find that storage topics that might have been abstract or foggy start becoming very clear, and terms like "sector", "zoning", "cylinder", "full bore stroke", "seek time", etc, all start making a lot more sense because you can measure these things without some benchmarking tool abstracting all those details away from you.

One word of caution, if you write data to a filesystem with compression using /dev/zero you might see unbelieving high numbers thanks to smart compression. In these cases use /dev/urandom.

IOZone

IOZone can be your best friend or your worst enemy. It gives you piles of output, but what does it all mean?

root@aeon ~$ iozone -a -f /iscsi/benchmark/testfile
 (snip)
                                                            random  random    bkwd  record  stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read rewrite    read   fwrite frewrite   fread  freread
              64       4  100782  392899  1066277  1330599 1016473  424101  418281  329860  942552   281899   342403  470880   809281
              64       8  149508  561495  1308191  2210490 1779559  639776  819484  429479 1359677   333377   556799  853867  1206279
              64      16  224481  726304  1773010  3187118 2666736  902870  954801  512190 1681127   460485   615747  985460  1489957
              64      32  336690  865556  1186173  3984255 3213648  968640 1000206  640345 1394066   321463   593094  889313  1458013
              64      64  640728  984604  2065562  3788700 3194489 1048299 1230654  914748 1162367   374341   499802  581333  1392128
             128       4   48376  479597   794708  1293454 1075853  465638  687865  412859  653222   348679   407500  612489   921157
             128       8   95384  680542  1076199  2067003 1775777  703634  883158  636917 1540598   460509   566358  735581  1293810
             128      16  180520  920588  1112754  2970023 2615679  962219 1085347  790598 1968806   511767   684483  795485  1544370
             128      32  289588 1067558  1346983  3287511 2777119 1196016 1142954  752991 1910738   508116   735558  770868  1599215

That output can be confusing, but if you used our dd method above you suddenly realize that what IOZone is doing for you is automating all sorts of metrics on your behalf. Above we see that we're creating a 64KB file in block sizes of 4, 8, 16, 32, and 64, doing read and write tests as well as fread/fwrite/etc (binary stream), and random ("read" and "write" are sequential).

IOZone, therefore, is great for finding that sweet spot on your storage solution. You can find out how far you can push your caches before they are ineffective or see what the optimal usage pattern is for your solution. Based on that, you can tweek caching or presets and see exactly how the performance range shifts in reaction.

When using IOZone don't get caught up with all its fancy options, -a pretty well has you covered. Make sure that ever test run is done identially.

Now... before we move on I should address the Cache problem. If you are doing these types of tests where you hit a file over and over again you want to ensure that your filesystems caching isn't over-inflating your numbers. This is especially true when you want to see the performance of a disk solution and not the filesystem. In these cases you should either bypass the cache by using the raw character device (/dev/rdsk) or by mounting the filesystem with DirectIO (mount -o forcedirectio,...). Some benchmarks attempt to defeat the cache by allocating files larger than memory, but on a box with 16GB of memory thats a bit ridiculous.

Bonnie++

Bonnie++ has become very popular because it is that magical benchmark that plops out a nice number that you can print on a shirt. Because of this, its both loved and hated. Either way, it's the de facto grand-daddy of macro IO benchmarks.

root@aeon ~$ bonnie++ -u benr -d /iscsi/benchmark/ -r 512
(snip)
Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
aeon             1G  7594   7  8317   2  2475   0 10840  13 11139   1  74.2   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 28164  97 +++++ +++ +++++ +++ 17849  94 +++++ +++ +++++ +++

I think Bonnie++ is a great tool for getting a general feel of a given configuration. Lots of data provided by tools like IOZone help you understand performance in a variety of scenarios, whereas Bonnie++ glosses over the details and gives you a handful of numbers. For this reason I recommend Bonnie++ when making non-tuning changes, such as upgrading your Fibre Channel HBA or running a new kernel or driver.

Remember, macro benchmarks are just that, high level, wide scope measurements. When used just see see if some change had an impact, its a great friend to have available.

As an aside, am I the only one that would rather they said "Write" instead of "Output" and "Read" instead of "Input"? Thats always annoyed me.

One thing to be careful about when using Bonnie++ is its need to write twice as much data as you have RAM. Thats an attempt to eliminate cache effect. However, if you have 1GB or more of memory that can really slow things down. I recommend disabling caching yourself (mount with 'forcedirectio') and using a more sensable size for your test, such as 1GB.

FileBench

FileBench is an extremely powerful tool for both macro and micro benchmarking. Its great strength is in its flexibility. Rather than running standard "as fast as you can!" tests it processes a given defined "workload". Several standard workloads are provided with FileBench, and you can easily extend or modify them to suite your specific needs.

Lets be more clear. Other tools will tell you how fast you ran this op or that, but specific tools need to be written to do more complex testing. FileBench provides a framework so that you can avoid writing your own tests in C or PERL or whatever. These workloads follow patterns of IO and thus better represent real world conditions. Just knowing that writing is fast isn't good enough, you really want to model activity such as a mail server that does a bunch of getattrs, one or more opens, some writes, some appends, a close, then repeats. All this is just a simple workload in FileBench. Here is an excerpt of the "varmail" workload (varmail.f):

define process name=filereader,instances=1
{
  thread name=filereaderthread,memsize=10m,instances=$nthreads
  {
    flowop deletefile name=deletefile1,filesetname=bigfileset
    flowop createfile name=createfile2,filesetname=bigfileset,fd=1
    flowop appendfilerand name=appendfilerand2,iosize=$meaniosize,fd=1
    flowop fsync name=fsyncfile2,fd=1
    flowop closefile name=closefile2,fd=1
    flowop openfile name=openfile3,filesetname=bigfileset,fd=1
    ...

This above IO pattern is mimics what you'll really see in the world. Thus, FileBench allows us to duplicate IO patterns in the lab without having to reproduce the exact conditions (ie: seeing up a mail server and hitting it just to examine IO behaviour). Lets see a run in action:

root@aeon bin$ ./filebench 
filebench> load varmail
filebench> run
 3000: 33.011: Fileset bigfileset: 1000 files, avg dir = 1000000.0, avg depth = 0.5, mbytes=15
 3000: 33.260: Creating fileset bigfileset...
 3000: 59.883: Preallocated 812 of 1000 of fileset bigfileset in 27 seconds
 3000: 59.884: Creating/pre-allocating files
 3000: 59.884: Starting 1 filereader instances
 3001: 59.897: Starting 16 filereaderthread threads
 3000: 60.903: Running...
 3000: 122.423: Run took 60 seconds...
 3000: 122.435: Per-Operation Breakdown
closefile4                 32ops/s   0.0mb/s      0.0ms/op        7us/op-cpu
readfile4                  32ops/s   0.5mb/s      0.0ms/op       49us/op-cpu
openfile4                  32ops/s   0.0mb/s      0.1ms/op       46us/op-cpu
closefile3                 32ops/s   0.0mb/s      0.0ms/op        8us/op-cpu
fsyncfile3                 32ops/s   0.0mb/s    270.9ms/op      156us/op-cpu
appendfilerand3            32ops/s   0.5mb/s      0.1ms/op       68us/op-cpu
readfile3                  32ops/s   0.5mb/s      0.0ms/op       49us/op-cpu
openfile3                  32ops/s   0.0mb/s      0.1ms/op       46us/op-cpu
closefile2                 32ops/s   0.0mb/s      0.0ms/op        8us/op-cpu
fsyncfile2                 32ops/s   0.0mb/s    229.0ms/op      151us/op-cpu
appendfilerand2            32ops/s   0.5mb/s      0.0ms/op       47us/op-cpu
createfile2                32ops/s   0.0mb/s      0.1ms/op       76us/op-cpu
deletefile1                32ops/s   0.0mb/s      0.1ms/op       60us/op-cpu

 3000: 122.435: 
IO Summary:      25009 ops 414.1 ops/s, (64/64 r/w)   2.0mb/s,   1571us cpu/op, 124.8ms latency
 3000: 122.435: Shutting down processes

You can see that we get a really nice breakdown of stats per operation. We can see clearly from the above that fsyncfile is my pain point.

My only caution is the avoid reading too much into the IO Summary. In too many cases I've found it to be misleading.

Personally, I look at FileBench not as a "benchmark" in the traditional sense, but rather a workload generator. Its very good at generating IO patterns. You can then use other tools such as "iostat", "vmstat", DTrace, etc, to view the impact a given workload exerts on your configuration.

Network Testing for IP Storage

When you test IP Storage technologies, such as NFS, AFS, or iSCSI you'll want to measure the speed from here to there. The best tools I've seen for this are PathLoad and PathRate. Both of these tools have a "sender" and "receiver" component which work together on the ends of the connection testing, thus you need access to both systems. Whenever you seriously benchmark iSCSI or NFS you should first benchmark your network so that you know how much network throughput you really have.

Pathrate is an end-to-end capacity estimation tool. It can help you identify any bottlenecks along the network path to determine what the true speed is going to look like through the narrowest portion of a given link.

# ./pathrate_rcv -s 172.16.165.18
        pathrate run from 172.16.165.18 to z00001AT on Thu Dec 21 16:44:25 2006
        --> Average round-trip time: 0.3ms

-------------------------------------------------
Final capacity estimate :  964 Mbps  to  964 Mbps
-------------------------------------------------

Pathload is a tool for estimating the available bandwidth of an end-to-end path. "The available bandwidth is the maximum IP-layer throughput that a flow can get in the path from S to R, without reducing the rate of the rest of the traffic in the path."

# ./pathload_rcv -s 10.71.165.18
Receiver z00001AT starts measurements at sender 10.71.165.18 on Thu Dec 21 17:17:11 2006
  Interrupt coalescion detected
Receiving Fleet 0, Rate 1200.00Mbps
Receiving Fleet 1, Rate 600.00Mbps
Receiving Fleet 2, Rate 923.08Mbps
Receiving Fleet 3, Rate 1090.91Mbps
Receiving Fleet 4, Rate 1000.00Mbps
Receiving Fleet 5, Rate 961.54Mbps

        *****  RESULT *****
Available bandwidth range : 923.08 - 1090.91 (Mbps)
Measurements finished at Thu Dec 21 17:17:16 2006
Measurement latency is 4.84 sec

These two tools when used together can be invaluable when benchmarking storage over IP networks.

Pulling It Together

There are lots of tools, but I hope you can see that no one tool is right or wrong, they each serve a different purpose and contribute to your understanding of a given configuration or solution in unique ways. Never pigeon hole yourself into relying on just one. I'm not going to pretend that mastering the various tools is easy... its not! Learning the tools takes a lot of time and effort, but if you are setting out to benchmark a solution you obviously have some desire to understand it. Resist the urge to react based on a single value or measurement and instead take advantage of the variety of helpful analysis tools available and utilize their strengths to assist you in conjunction with your standard tools provided by the OS or storage solution.

Resources


- - C O M M E N T S - -

You should upgrade your dd:

% dd if=/dev/zero of=testfile bs=8k
109226+0 records in
109226+0 records out
894779392 bytes (895 MB) copied, 16.1195 seconds, 55.5 MB/s

Justin - 23 May '07 - 08:44

Excellent article – thanks!

cuyler (Email) - 23 May '07 - 09:09

Well done. Thank you.

Shane Falco - 23 May '07 - 10:04

Would you be willing to publish this under a CC license or at least grant permission to reproduce it in full (with attribution)?

Nathaniel - 23 May '07 - 10:20

Everything in my blog is licensed CC. Please see the CC badge on my blog’s side-bar (under the blogrolls).

benr - 23 May '07 - 15:30

I would like to distribute this to our system administrators at my company as an intro to benchmarking. Does this qualify as commercial use (which is restricted by your CC license)?

Nathaniel - 23 May '07 - 16:30

No. Commercial use, in my mind, only applies if you wish to make money from it. My documentation is intended to help my fellow admins and so use it in any way that it can be of help.

benr - 23 May '07 - 16:34

[[http://en.wikipedia.org/wiki/Benchmark..]]

“The term benchmark originates from the chiseled horizontal marks that surveyors made in stone structures, into which an angle-iron could be placed to form a “bench” for a levelling rod, thus ensuring that a levelling rod could be accurately repositioned in the same place in the future. These marks were usually highlighted with a chiseled arrow below the horizontal line.”

Mike (Email) - 23 May '07 - 16:47

sh -c “dd if=/dev/urandom of=file && sync” helps account for caching (not sure if sync(1) is the same command on Solaris)

Mike - 23 May '07 - 16:59

Thanks Ben,
you Rock
Cheers

Marco (Email) - 24 May '07 - 03:52

Bonnie++ was derived from Bonnie, which I wrote 17 years ago (!). I’ve occasionally done a bit more tinkering with it. See [[http://www.tbray.org/bonnie/bonnie.tar]] – it has been cleaned of 32-bit artifacts, so you can say -s 32000 to run with 32G, which it does in one file, to make the seek measurements a bit more interesting. Also reports in M/sec rather than K/sec for a neater display. Must get together with Russell Coker one of these years and get everything rolled together.

Tim Bray (Email) (URL) - 24 May '07 - 09:52

Wonderful article. Really great. Thank you for writing this up.

John G (Email) - 18 October '07 - 15:01

there is the professional world of warcraft power leveling here. welcome.

jimelyyes (Email) (URL) - 10 May '08 - 16:48

hananim_ki (Email) - 11 May '08 - 07:34

0494e818 eb23ef87 645 ancor1|ancor2|ancor3|ancor4|ancor5 It will happen soon, in your winter [URL= [[http://google2.com.com]] ]cheapest ancor1ancor2ancor3ancor4ancor5[/URL] .
These kennel in turn contract with the fresh ancor1ancor2ancor3ancor4ancor5 .
Are they all [[http://google2.com.org]] download ancor1ancor2ancor3ancor4ancor5 out to destroy our country?

Thanks for your project. I like this site. KEEP IT UP..

William (Email) (URL) - 13 May '08 - 21:21

shop zithromax welcome

zithromax (Email) (URL) - 15 May '08 - 01:45

xanax

xanax (Email) (URL) - 20 May '08 - 05:40

xanax

xanax (Email) (URL) - 20 May '08 - 23:15

The decease charge from stand up week’s earthquake in south-west China has risen to 40,075, officials arrange said.
Chinese aid workers are struggling to locate housing for millions who unchaste their homes. peculiar medical teams be suffering with started arriving in the area.
Tens of thousands more are that time missing, and hopes are fading of happening multitudinous people stationary aware.
how in the world, Chinese media say a maid was pulled from the rubble 195 hours – or eight days – after the disaster.
It was the deficient saving on Tuesday. Earlier, a man was pulled animated from the remains of a power herb after being buried for 179 hours, state media said. Rescuers fed him sweetened incredible washing one’s hands of a tube.
The saving try has now focused on providing victuals, shield and drinking be inconsistent for the millions of people hurt by the 7.9 extent earthquake in Sichuan province.
Rescuers arrange launched a concluding try to search all upset area.
On Monday Chinese president Wen Jiabao ordered troops to reach every solitary village and village in the earthquake section within 24 hours.

TugShagma (Email) (URL) - 22 May '08 - 00:42

kubis

kubis (Email) (URL) - 23 May '08 - 05:37

blodulv

blodulv (Email) (URL) - 23 May '08 - 21:27

buy tramadol [[http://activerain.com/tramadolonline]] buy tramadol

tramadol (Email) (URL) - 26 July '08 - 22:16

carville la. leprosarium i
circle k convenience stores in usa

hananim_ue (Email) (URL) - 06 August '08 - 13:56

world cup schedule
morton bildings

hananim_ou (Email) (URL) - 06 August '08 - 20:26

crystals of chattanooga
lisa campanelli

hananim_go (Email) (URL) - 07 August '08 - 02:33

listcaometbet

hannesi (Email) (URL) - 14 August '08 - 21:37

tnice nies

yegajkt (Email) (URL) - 14 August '08 - 21:39

nice.thanks

haneber (Email) (URL) - 14 August '08 - 21:42

the tennis cannel standard appliances ameraprise song dden screen printed t shirt
how to do sleight-of-hand magic grapenut custard meret oppenheim imdb black and white pictures of johnny depp piano clinique song

hananim_qq (Email) (URL) - 11 October '08 - 09:47

anastasia journey to the past trivekia chelsea alabama apartments bone thugs quotes nordhaven 43 west virginia state tax department vjhhcgxtdfxzdcgfhcg digitalcamera storage free backgroud desktop images jpcoats&clark
louisiana state department basking ridge nj zip code all populous games unlimited download movies outdoor dog bed combat corpsman ohio toll road lease british.football.sport.jersey play.station.two pocketpalus

Larcik-cy (Email) (URL) - 11 October '08 - 13:26

patroits schedule sherman oaks mall ca 1840 west 49th street connecticut huskies cheerleaders pound cake mix recipe
bia spring home and garden show southern living receips ass r us kanye west work fdt funeral arrangements

hananim_rd (Email) (URL) - 11 October '08 - 16:41

us constitution natural rights castleander beads waiste my time protector plus the norman transcript comedy realtones all neriods beware url chiodos pentagon insured certificates of deposits teen red aol take off your pants and jacket
realidades online book cafes route 202 towaco nj go video on hsn dont you know that when you touch me baby lyrics drag racing n michigan pocketpalus dozzy cards holistic wheel inexpensive sanibel oceanfront cottages division iii junior college baseball

Larcik-cc (Email) (URL) - 11 October '08 - 20:17

racing bellhousing reinharts edbenes little rock free press sport fun my first pogo stick located at
moma waitress black wicked lady glenby inc. sheraton hotel anaheim tom cruise tv tome

hananim_pd (Email) (URL) - 12 October '08 - 20:34

chinese art of balance fung shau best tasting chocolate cake websites for transporting services needing independent contractors brighton court george washington 500 000 loan from pennsylvania farmer
continental micronesia fussell buy bernadette icon pvc threaded fittings american box co pa

Larcik-kw (Email) (URL) - 12 October '08 - 23:31

sunshine day lyrics sell your house tips tropical ceiling fan with light freddie jackson live dvd mark pitts with jive records email address
en.wikipedia.ord house of the bride wake up ona saturday night health deparment of nevada best food gifts

Larcik-al (Email) (URL) - 13 October '08 - 06:01

mysry skatebording new 9.0 download nextelrace prices of ana tech shoes state of west virginia corporations
idaho jails trenyce and american idol peigle seven wors of the world ron adrian artwork

hananim_th (Email) (URL) - 13 October '08 - 22:49

sunnyvale library goals of physical education angelo milo ask eloise no rain blind melon
preowned taylow made golf clubs dan curry bloomington rhodes view apartment ratings birmingham rooftiles alabama 1a girls basketball

Larcik-xm (Email) (URL) - 14 October '08 - 02:02

[[http://www.onlywowgold.com]wow]] gold[/url]

seo (Email) (URL) - 18 October '08 - 03:32

new site about porn

Larcik-am (Email) (URL) - 29 October '08 - 03:25

bill gates photos wstr-tv beens to spill means

Larcik-lg (Email) (URL) - 13 November '08 - 06:37

seo

seo (Email) (URL) - 03 December '08 - 23:02

不動産投資

ä¸å産æè³ (Email) (URL) - 07 December '08 - 01:02

不動産

ä¸å産 (URL) - 07 December '08 - 01:03

seo対策

seoå¯¾ç­ (URL) - 07 December '08 - 01:09

I have searched the net and I should say I have not come across an article like this which is so easy to understand and learn the concepts.
cheers

flash games (Email) (URL) - 20 January '09 - 05:40

Very interesting article: “May Product Plugs”

Dannie (Email) (URL) - 23 January '09 - 17:56

i like your things so much ,hope you can do it well ,thank you

cheap nike shoes (Email) (URL) - 27 March '09 - 17:42

i link this blog very much.thanks for it

runescape (Email) (URL) - 20 April '09 - 08:14

琴似メーリングリスト
札幌市西区琴似のクチコミ情報交換コミュニティ。コトニのホテル、グルメ、温泉等を紹介している「琴似ガイド」、美容室、ラーメン店、居酒屋、スナックなどのリンクを集めた「琴似ナビ」のほか、地域通貨の発行も

kotoni - 25 June '09 - 04:46

[[http://kotoni_ml.web.fc2.com/]]

ç´ä¼¼ (URL) - 25 June '09 - 04:47

[[http://www.e-tokyomoney.com]]

fhgfgh (Email) - 28 August '09 - 06:59

Just a single rev of firmware on an HBA or BIOS can have a major impact on your performance

Global Agenda (Email) (URL) - 23 September '09 - 10:08

[[http://kanshoku.info/]]

一人æ®ã‚ã イã³ããªã‚¢ (URL) - 24 September '09 - 11:43

hatabou

hatabou (URL) - 24 September '09 - 11:43

ヒアルロン酸
毛穴 黒ずみ

ç¾ç™½ (Email) (URL) - 28 October '09 - 23:49

Good post! Thanks you for your information! China Wholesale Wholesale China Wholesalers Wholesale Game Accessories Wholesale Iphone Accessories Video Game Accessories Wholesale Wholesale Wii Accessories Wholesale Xbox 360 Accessories Wholesale Xbox 360 Games Wholesale Video Games Cheap Video Games Cheap Ps3 Games Cheap Xbox 360 Games Wholesale Computers Wholesale Laptop Computers Wholesale Laptops Discount Computers Cheap Computers Wholesale Iphones Wholesale Iphone Wholesale Iphones 3g Hiphones Wholesale Hiphone Wholesale Hiphones Wholesale Nokia Wholesale Nokia 8800 Wholesale Nokia n97 wholesale blackberry wholesale blackberry phones wholesale blackberry 9700 wholesale blackberry 9600 wholesale blackberry 9500

raging bull (Email) (URL) - 15 January '10 - 06:34

Wholesale Nokia n97 wholesale blackberry wholesale blackberry phones wholesale blackberry 9700 wholesale blackberry 9600 wholesale blackberry

javon (Email) (URL) - 25 January '10 - 09:23

Good post! Thanks for your information!

As Seen On TV (Email) (URL) - 27 January '10 - 06:59

Your article is very useful!Thank you for sharing.Nice post.

Tiffany Accessories (Email) (URL) - 08 February '10 - 08:56

Ed Hardy Hats were later launched with much fanfare.

ed hardy (Email) (URL) - 25 February '10 - 03:31

Have no money to buy a car? You not have to worry, just because that is available to get the loans to resolve such kind of problems. Thus take a short term loan to buy all you want.

HANSONKayla (Email) (URL) - 06 March '10 - 22:01

Good post!

links of london (Email) (URL) - 10 March '10 - 08:54

he content of the articles there will be a lot of attractive people to appreciate, I have to thank you such an article. As hard as it is to answer those questions, it would definitely not help by burying our head in the sand and refusing to talk about them

wholesale lingerie (Email) (URL) - 10 March '10 - 10:43

I very intersted in the article

cheap mbt shoes (Email) (URL) - 23 April '10 - 03:18

P90x .It really is not expensive if you factor in the cost

of a gym membership,P90x workout . The cost for P90X is

about three months of a paid gym membership but you get to

keep the program foreverP90x . You can try many of the

online sites, but it will be the same as buying from the

company or a Beachbody Coach. Make sure you are getting

original DVD’s. People are selling copies all over. The

problem is how long will they last, P90x workout ,and you

truly need the exercise and nutrition guide to even follow

the program. You can go to any site or you can go to

[[http://www.p90xmall.com]] and click on products. P90x dvd You can

order directly from the site,P90x dvd.

p90x (Email) (URL) - 05 May '10 - 05:39

hi there, thanks for your great post, it help me so much! now i wonder if you linke the mbt shoes from my site: the url is [[http://www.thesuitshoes.com]]

mbt shoes (Email) (URL) - 12 May '10 - 06:52

A good website recommend you:www.offerreplicawatches.com.They sell Chanel Watches, Replica BRM Watches,offer Cartier Watches, Copy Ebel Watches, Replica Hublot Watches, Fake Tag Heuer Watches, wholesale Richard Mille Watches.

They also sell some very fashion watches, such as Montblanc Watches for sale,Chopard Watches, Dior Watches and Gucci Watches. Rolex watches are hot sell. If u are Armani Watches and Breitling Watches fan. You can still look at Omega Watches and Longines Watches, many new arrivals watches just come here.You can have a try on 58338312391273913. :[url=[[http://www.offerreplicawatches.com]www..]]].

Replica Chanel Watches,Dvd Boxes (Email) (URL) - 12 May '10 - 07:30

Dell inspiron 9300 Battery [[http://www.adapterlist.com/dell/inspir..]]

laptop battery (Email) (URL) - 14 May '10 - 09:37

Dell latitude d820 Battery [[http://www.globallaptopbattery.co.uk/d..]]

laptop batteries (Email) (URL) - 14 May '10 - 09:45

Get Supra Shoes at supramvp.com, save up to 75% on Supra shoes. Competitive price with a lot of latest styles. Supra Skytops, Supra Vaider, Supra TK Society, Supra Suprano High. Worldwide door to door shippment. Save Up Today! A huge collection of Supra Shoes and Supra Footwear

New supra footwear (Email) (URL) - 19 May '10 - 08:59

We Sale theSupra Shoes and Supra Footwear including Supra Vaider, Supra Skytops, Supra Skytop II, Supra Cuban, Supra Cruizer, Supra Terry Kennedy (TK) Society, Supra Suprano High, Supra Strapped NS, Supra Trinity NS, Supra Indy NS and Supra Thunder Hightop. Buy Cheap Supra Shoes with High Quality, Free Shipping ! 24/7 Online Support

Wholesale Supra Shoes (Email) (URL) - 19 May '10 - 09:05

The newest version of air max 90 are available now.Welcome to see our special nike air max 2009 and air max. They are your must-have nike air max 2010|air max 97|cheap air max.

95 Air Max Air Max nike

air max (Email) (URL) - 21 May '10 - 06:38

rosetta stone

rosetta stone (Email) (URL) - 26 May '10 - 07:05

Buy Nike Air Max 90 Shoes just $45-55 USD in[[http://www.iofferitems.com,]], 40-70% Off. Cheap Air Max 90 Shoes, Free Shipping! Buy Air Max 90 Now!

nike air max 90 shoes (Email) (URL) - 28 May '10 - 02:26

I like writing, hurriedly, not that you are not
[[http://www.woe-from.com]]
[[http://www.louis-vuitton-outlet-store...]]

louis vuitton (Email) (URL) - 01 June '10 - 02:26

[[http://www.mbtshoeslatest.com]]
[[http://www.nikeairmaxshoe.com]]

Nike air max shoe (Email) (URL) - 24 June '10 - 07:50

Do you like the ugg ? Ugg Classic Short Boots are UGG Australia’s original heritage styles. Ugg Classic Tall will keep your feet dry and the ultimate comfortable.
[[http://www.ugg-boots-london.com/ugg-cl..]]

Ugg Classic Short (Email) (URL) - 07 July '10 - 01:20

I will bookmark your blog and have my kids check up here frequently Air Jordan 2010. I’m very certain they will understand lots of new stuff Air jordan 1 here than anybody else. I do appreciate that youve added relevant and intelligent commentary here though. Thank you!

Cheap Jordan Shoes (Email) (URL) - 03 August '10 - 06:42

You write good articles, I will always be concerned about

cheap NFL jerseys (Email) (URL) - 09 August '10 - 02:45

est quality and cheap price
Dallas Cowboys Jerseys [[http://www.jerseycaptain.com/index.asp..]]
50TH patch [[http://www.jerseycaptain.com/index.asp..]]
Cheap sport jerseys

jersey whole sale (Email) (URL) - 14 August '10 - 13:38

excellent article , I added you to my [[http://www.china-wholesale-directory.c..]] Top China Wholesalers category.. thanks for sharing the article!

China Wholesale Directory (Email) (URL) - 25 August '10 - 07:47

very cool article ,thanks for sharing the article!like my cool stuff .very useful.
uCoolStuff is the leading China wholesaler for [[http://www.ucoolstuff.com]] cool stuff [[http://www.ucoolstuff.com]] cool gifts , unusual gadgets and other unique gift ideas. We provide the very latest cool stuff and cool gifts for you

cool stuff (Email) (URL) - 25 August '10 - 07:52

Top Online Stores is a SEO Friendly [[http://www.toponlinestores.org]] free directory where you can find the best online shopping stores selected by hand and sorted by category [[http://www.china-wholesale-directory.c..]] china wholesale .

link directory (Email) (URL) - 14 September '10 - 07:18

[URL=www.cheapuggtallboots.com] UGG Boots [/URL]
[URL=www.cheapuggtallboots.com] ugg boots sale [/URL]
[URL=www.cheapuggtallboots.com] ugg boots uk [/URL]
[URL=www.cheapuggtallboots.com/ugg-bailey-button-5803-c-29.html] UGG Bailey Button [/URL]
[URL=www.cheapuggtallboots.com/ugg-womens-bailey-button-5803grey-p-302.html] UGG Women’s Bailey Button Grey [/URL]
[URL=www.cheapuggtallboots.com/ugg-womens-bailey-button-5803-leopard-p-305.html] UGG Women’s Bailey Button leopard [/URL]
[URL=www.cheapuggtallboots.com/ugg-womens-bailey-button-5803black-p-299.html] UGG Women’s Bailey Button Black [/URL]
[URL=www.cheapuggtallboots.com/ugg-womens-bailey-button-5803chestnut-p-300.html] UGG Women’s Bailey Button Chestnut [/URL]
[URL=www.cheapuggtallboots.com/ugg-womens-bailey-button-5803grey-p-302.html]UGG Women’s Bailey Button Grey [/URL]
[URL=www.cheapuggtallboots.com/ugg-womens-bailey-button-5803pink-p-306.html] UGG Women’s Bailey Button Pink [/URL]
[URL=www.cheapuggtallboots.com/ugg-womens-bailey-button-5803sand-p-303.html] UGG Women’s Bailey Button Sand [/URL]
[URL=www.cheapuggtallboots.com/ugg-womens-bailey-button-5803tomato-p-307.html] UGG Women’s Bailey Button Tomato [/URL]
[URL=www.cheapuggtallboots.com/ugg-womens-bailey-button-5803chocolate-p-301.html] UGG Women’s Bailey Button Chocolate [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-tall-5815-c-6.html] UGG Classic Tall [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-tall-5815-apple-p-210.html] UGG Classic Tall Apple [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-tall-5815-black-p-202.html] UGG Classic Tall Black [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-tall-5815-blue-p-209.html] UGG Classic Tall Blue [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-tall-5815-chestnut-p-203.html] UGG Classic Tall Chestnut [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-tall-5815-chocolate-p-204.html] UGG Classic Tall Chocolate [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-tall-5815-grey-p-205.html] UGG Classic Tall Grey [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-tall-5815-leopard-p-213.html] UGG Classic Tall Leopard [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-tall-5815-mulberry-p-208.html] UGG Classic Tall Mulberry [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-tall-5815-pink-p-207.html] UGG Classic Tall Pink [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-tall-5815-printing-p-214.html] UGG Classic Tall Printing [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-tall-5815-sand-p-206.html] UGG Classic Tall Sand [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-tall-5815-tomato-p-212.html] UGG Classic Tall Tomato [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-tall-5815-yellow-p-211.html] UGG Classic Tall Yellow [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-short-5825-c-8.html] UGG Classic Short [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-short-5825-printing-p-234.html] UGG Classic Short Printing [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-tall-5815-pink-p-207.html] UGG Classic Tall Pink [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-short-5825green-p-233.html] UGG Classic Short Green [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-short-5825tomato-p-232.html]UGG Classic Short Tomato[/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-short-black-p-227.html] UGG Classic Short Black [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-short-blue-camo-p-224.html] UGG Classic Short Blue Camo [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-short-brown-camo-p-225.html] UGG Classic Short Brown Camo [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-short-chocolate-p-228.html] UGG Classic Short Chocolate [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-short-green-camo-p-226.html] UGG Classic Short Green Camo [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-short-grey-p-229.html] UGG Classic Short Grey [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-short-paisley-sand-p-230.html] UGG Classic Short Paisley Sand [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-short-sand-p-231.html] UGG Classic Short Sand [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-cardy-5819-c-7.html] UGG Classic Cardy [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-tall-5815-pink-p-207.html] UGG Classic Tall Pink [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-cardy-5819-black-p-215.html] UGG Classic Cardy Black [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-cardy-5819-cream-p-216.html] UGG Classic Cardy Cream [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-cardy-5819-black-p-215.html] UGG Classic Cardy Black [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-cardy-5819-grey-p-217.html] UGG Classic Cardy Grey [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-cardy-5819-oatmeal-p-218.html] UGG Classic Cardy Oatmeal [/URL]
[URLwww.cheapuggtallboots.com/ugg-classic-cardy-5819-rose-p-222.html] UGG Classic Cardy Rose [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-cardy-5819mul-p-220.html] UGG Classic Cardy Mul [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-cardy-5819sorbet-p-223.html] UGG Classic Cardy Sorbet [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-cardy-5819sto-p-221.html] UGG Classic Cardy STO [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-cardy-5819stout-p-219.html] UGG Classic Cardy stout [/URL]
[URL=www.cheapuggtallboots.com/ugg-5918-half-waterproofing-c-42.html] UGG Half Waterproofing Boots [/URL]
[URL=www.cheapuggtallboots.com/tall-ugg-boots-c-46.html] Tall UGG Boots [/URL]
[URL=www.cheapuggtallboots.com/] UGG online store [/URL]
[URL=www.cheapuggtallboots.com/ugg-broome-5518-boots-c-68.html] UGG Broome Boots [/URL]
[URL=www.cheapuggtallboots.com/ugg-ultra-tall-5245-c-3.html] UGG Ultra Tall Boots [/URL]
[URL=www.cheapuggtallboots.com/advanced_search_result.html?keyword=pink&search_in_description=1] pink ugg boots [/URL]
[URL=www.cheapuggtallboots.com/advanced_search_result.html?keyword=chestnut&search_in_description=1] ugg chestnut [/URL]
[URL=www.cheapuggtallboots.com/ugg-classic-mini-5854-c-9.html] UGG Classic Mini Boots [/URL]
[URL=www.cheapuggtallboots.com/ugg-5124-womens-suburb-crochet-c-44.html] UGG Suburb Crochet Tall Boots [/URL]
[URL=www.cheapuggtallboots.com/kids-ugg-boots-c-47.html] Kid’s UGG Boots [/URL]

cheapuggtallbootsAdministrator (Email) (URL) - 28 September '10 - 04:05

00zoe

MBT shoes[[http://www.mbtshoes4sale.co.uk]]
Timberland boots[[http://www.timberlandbootsale.co.uk]]

MBT shoes clearance (Email) (URL) - 01 November '10 - 03:09

Personal information





Remember your information?
Comment

Small print: All html tags except <b> and <i> will be removed from your comment. You can make links by just typing the url or mail-address.


^M