The Blog of Ben Rockwood

zdb: Examining ZFS At Point-Blank Range

01 Nov '08 - 08:13 by benr

ZFS is an amazing in its simplicity and beauty, however it is also deceivingly complex. The chance that you'll ever be forced to peer behind the veil is unlikely outside of the storage enthusiast ranks, but as it proliferates more questions will come up regarding its internals. We have been given a tool to assist us investigate the inner workings, zdb, but it is, somewhat intentionally I think, undocumented. Only two others that I know have had the courage to talk about it publicly, Max Bruning who is perhaps the single most authoritative voice regarding ZFS outside of Sun, and Marcelo Leal.

In this post, we'll look only at the basics of ZDB to establish a baseline for its use. Running "zdb -h" will produce a summary of its syntax.

In its most basic form, zdb poolname, several bits of information about our pool will be output, including:

Cached pool configuratino (-C)
Uberblock (-u)
Datasets (-d)
Report stats on zdb's I/O (-s), this is similar to the first interval of zpool iostat

Thus, zdb testpool is the same as zdb -Cuds testpool. Lets look at the output. The pool we'll be using is actually a 256MB pre-allocated file with a single dataset... as simple as it can come.

root@quadra /$ zdb testpool
    version=12
    name='testpool'
    state=0
    txg=182
    pool_guid=1019414024587234776
    hostid=446817667
    hostname='quadra'
    vdev_tree
        type='root'
        id=0
        guid=1019414024587234776
        children[0]
                type='file'
                id=0
                guid=6723707841658505514
                path='/zdev/disk002'
                metaslab_array=23
                metaslab_shift=21
                ashift=9
                asize=263716864
                is_log=0
Uberblock

        magic = 0000000000bab10c
        version = 12
        txg = 184
        guid_sum = 7743121866245740290
        timestamp = 1225486684 UTC = Fri Oct 31 13:58:04 2008

Dataset mos [META], ID 0, cr_txg 4, 87.0K, 49 objects
Dataset testpool/dataset01 [ZPL], ID 30, cr_txg 6, 19.5K, 5 objects
Dataset testpool [ZPL], ID 16, cr_txg 1, 19.0K, 5 objects
                      capacity   operations   bandwidth  ---- errors ----
description          used avail  read write  read write  read write cksum
testpool             139K  250M   638     0  736K     0     0     0     0
  /zdev/disk002      139K  250M   638     0  736K     0     0     0     0

And so we see a variety of useful information, including:

Zpool (On Disk Format) Version Number
State
Host ID & Hostname
GUID (This is that numberic value you use when zpool import doesn't like the name)
Children VDEV's that make up the pool
Uberblock magic number (read that hex value as "uba-bloc", get it, 0bab10c, its funny!)
Timestamp
List of datasets
Summary of IO stats

So this information is interesting, but frankly not terribly useful if you already have the pool imported. This would likely be of more value if you couldn't, or wouldn't, import the pool, but those cases are rare and 99% of the time zpool import will tell you want you want to know even if you don't actually import.

There are 3 arguments that are really the core ones of interest, but fefore we get to them, you absolutely must understand something unique about zdb. ZDB is like a magnifying glass, at default magnification you can see that its tissue, turn up the magnification and you see that it has veins, turn it up again and you see how intricate the system is, crank it up one more time and you can see blood cells themselves. With zdb, each time we repeat an argument we increase the verbosity and thus dig deeper. For instance, zdb -d will list the datasets of a pool, but zdb -dd will output the list of objects within the pool. Thus, when you really zoom in you'll see commands that look really odd like zdb -ddddddddd. This takes a little practice to get the hang of, so please toy around on a small test pool to get the hang of it.

Now, here are summaries of the 3 primary arguments you'll use and how things change as you crank up the verbosity:

- -bb: Outputs a breakdown of space (block) usage for various ZFS object types.
- -bbb: Same as above, but includes breakdown by DMU/SPA level (L0-L6).
- -bbbb: Same as above, but includes line line per object with details about it, including compression, checksum, DVA, object ID, etc.
- -bbbbb...: Same as above.
- -d: Output list of datasets, including ID, cr_txg, size, and number of objects.
- -dd: Output concise list of objects within the dataset, with object id, lsize, asize, type, etc.
- -ddd: Same as dd.
- -dddd: Outputs list of datasets and objects in detail, including objects path (filename), a/c/r/mtime, mode, etc.
- -ddddd: Same as previous, but includes indirect block addresses (DVAs) as well.
- -dddddd....: Same as above.
zdb -R pool:vdev_specifier:offset:size[:flags]: Given a DVA, outputs object contents in hex display format. If given the :r flag it will output in raw binary format. This can be used for manual recovery of files.

So lets play with the first form above, block traversal. This will sweep the blocks of your pool or dataset adding up what it finds and then producing a report of any leakage and how the space breakdown works. This is extremely useful information, but given that it traverses all blocks its going to take a long time depending on how much data you have. On a home box this might take minutes or a couple hours, on a large storage subsystem is could take hours or days. Lets look at both -b and -bb for my simple test pool:

root@quadra ~$ zdb -b testpool

Traversing all blocks to verify nothing leaked ...

        No leaks (block sum matches space maps exactly)

        bp count:              50
        bp logical:        464896        avg:   9297
        bp physical:        40960        avg:    819    compression:  11.35
        bp allocated:      102912        avg:   2058    compression:   4.52
        SPA allocated:     102912       used:  0.04%

root@quadra ~$ zdb -bb testpool

Traversing all blocks to verify nothing leaked ...

        No leaks (block sum matches space maps exactly)

        bp count:              50
        bp logical:        464896        avg:   9297
        bp physical:        40960        avg:    819    compression:  11.35
        bp allocated:      102912        avg:   2058    compression:   4.52
        SPA allocated:     102912       used:  0.04%

Blocks  LSIZE   PSIZE   ASIZE     avg    comp   %Total  Type
     3  12.0K   1.50K   4.50K   1.50K    8.00     4.48  deferred free
     1    512     512   1.50K   1.50K    1.00     1.49  object directory
     1    512     512   1.50K   1.50K    1.00     1.49  object array
     1    16K      1K   3.00K   3.00K   16.00     2.99  packed nvlist
     -      -       -       -       -       -        -  packed nvlist size
     1    16K      1K   3.00K   3.00K   16.00     2.99  bplist
     -      -       -       -       -       -        -  bplist header
     -      -       -       -       -       -        -  SPA space map header
     3  12.0K   1.50K   4.50K   1.50K    8.00     4.48  SPA space map
     -      -       -       -       -       -        -  ZIL intent log
    16   256K   18.0K   40.0K   2.50K   14.22    39.80  DMU dnode
     3  3.00K   1.50K   3.50K   1.17K    2.00     3.48  DMU objset
     -      -       -       -       -       -        -  DSL directory
     4     2K      2K   6.00K   1.50K    1.00     5.97  DSL directory child map
     3  1.50K   1.50K   4.50K   1.50K    1.00     4.48  DSL dataset snap map
     4     2K      2K   6.00K   1.50K    1.00     5.97  DSL props
     -      -       -       -       -       -        -  DSL dataset
     -      -       -       -       -       -        -  ZFS znode
     -      -       -       -       -       -        -  ZFS V0 ACL
     1    512     512     512     512    1.00     0.50  ZFS plain file
     3  1.50K   1.50K   3.00K      1K    1.00     2.99  ZFS directory
     2     1K      1K      2K      1K    1.00     1.99  ZFS master node
     2     1K      1K      2K      1K    1.00     1.99  ZFS delete queue
     -      -       -       -       -       -        -  zvol object
     -      -       -       -       -       -        -  zvol prop
     -      -       -       -       -       -        -  other uint8[]
     -      -       -       -       -       -        -  other uint64[]
     -      -       -       -       -       -        -  other ZAP
     -      -       -       -       -       -        -  persistent error log
     1   128K   4.50K   13.5K   13.5K   28.44    13.43  SPA history
     -      -       -       -       -       -        -  SPA history offsets
     -      -       -       -       -       -        -  Pool properties
     -      -       -       -       -       -        -  DSL permissions
     -      -       -       -       -       -        -  ZFS ACL
     -      -       -       -       -       -        -  ZFS SYSACL
     -      -       -       -       -       -        -  FUID table
     -      -       -       -       -       -        -  FUID table size
     1    512     512   1.50K   1.50K    1.00     1.49  DSL dataset next clones
     -      -       -       -       -       -        -  scrub work queue
    50   454K   40.0K    101K   2.01K   11.35   100.00  Total

Here we can see the "zooming in" effect I described earlier. Here "BP" stands for "Block Pointer". The most common "Type" you'll see is "ZFS plain file", that is, a normal data file like an image or textfile or something... the data you care about.

Moving on to the second form, -d to output datasets and their objects. This is where introspection really occurs. With a simple -d we can see a recursive list of datasets, but as we turn up the verbosity (-dd) we zoom into the objects within the dataset, and then just get more and more detail about those objects.

root@quadra ~$ zdb -d testpool/dataset01
Dataset testpool/dataset01 [ZPL], ID 30, cr_txg 6, 18.5K, 5 objects

root@quadra ~$ zdb -dd testpool/dataset01
Dataset testpool/dataset01 [ZPL], ID 30, cr_txg 6, 18.5K, 5 objects

    Object  lvl   iblk   dblk  lsize  asize  type
         0    7    16K    16K    16K  14.0K  DMU dnode
         1    1    16K    512    512     1K  ZFS master node
         2    1    16K    512    512     1K  ZFS delete queue
         3    1    16K    512    512     1K  ZFS directory
         4    1    16K    512    512    512  ZFS plain file

So lets pause here. We can see the list of objects in my testpool/dataset01 by object id. This is important because we can use those id's to dig deeper on an individual object later. But for now, lets zoom in a little bit more (-dddd) on this dataset.

root@quadra ~$ zdb -dddd testpool/dataset01
Dataset testpool/dataset01 [ZPL], ID 30, cr_txg 6, 18.5K, 5 objects, rootbp [L0 DMU objset] 400L/200P DVA[0]=<0:12200:200> DVA[1]=<0:3014c00:200> fletcher4 lzjb LE contiguous birth=8 fill=5 cksum=a525c6edf:45d1513a8c8:ef844ac0e80e:22b9de6164dd69

    Object  lvl   iblk   dblk  lsize  asize  type
         0    7    16K    16K    16K  14.0K  DMU dnode

    Object  lvl   iblk   dblk  lsize  asize  type
         1    1    16K    512    512     1K  ZFS master node
        microzap: 512 bytes, 6 entries

                casesensitivity = 0 
                normalization = 0 
                DELETE_QUEUE = 2 
                ROOT = 3 
                VERSION = 3 
                utf8only = 0 

    Object  lvl   iblk   dblk  lsize  asize  type
         2    1    16K    512    512     1K  ZFS delete queue
        microzap: 512 bytes, 0 entries


    Object  lvl   iblk   dblk  lsize  asize  type
         3    1    16K    512    512     1K  ZFS directory
                                 264  bonus  ZFS znode
        path    /
        uid     0
        gid     0
        atime   Fri Oct 31 12:35:30 2008
        mtime   Fri Oct 31 12:35:51 2008
        ctime   Fri Oct 31 12:35:51 2008
        crtime  Fri Oct 31 12:35:30 2008
        gen     6
        mode    40755
        size    3
        parent  3
        links   2
        xattr   0
        rdev    0x0000000000000000
        microzap: 512 bytes, 1 entries

                testfile01 = 4 (type: Regular File)

    Object  lvl   iblk   dblk  lsize  asize  type
         4    1    16K    512    512    512  ZFS plain file
                                 264  bonus  ZFS znode
        path    /testfile01
        uid     0
        gid     0
        atime   Fri Oct 31 12:35:51 2008
        mtime   Fri Oct 31 12:35:51 2008
        ctime   Fri Oct 31 12:35:51 2008
        crtime  Fri Oct 31 12:35:51 2008
        gen     8
        mode    100644
        size    21
        parent  3
        links   1
        xattr   0
        rdev    0x0000000000000000

Now, this output is short because the dataset include only a single file. In the real world this output will be gigantic and should be redirected to a file. When I did this on the dataset containing my home directory the output file was 750MB... its a lot of data.

Look specifically at Object 4, a "ZFS plain file". Notice that I can see that files pathname, uid, gid, a/m/c/crtime, mode, size, etc. This is where things can get really interesting!

In zdb's 3rd form above (-R) we can actually display the contents of a file, however we need its Device Virtual Address (DVA) and size to do so. In order to get that information, we can zoom in using -d little further, but this time just on Object 4:

root@quadra /$ zdb -ddddd testpool/dataset01 4
Dataset testpool/dataset01 [ZPL], ID 30, cr_txg 6, 19.5K, 5 objects, rootbp [L0 DMU objset] 400L/200P DVA[0]=<0:172e000:200> DVA[1]=<0:460e000:200> fletcher4 lzjb LE contiguous birth=168 fill=5 cksum=a280728d9:448b88156d8:eaa0ad340c25:21f1a0a7d45740

    Object  lvl   iblk   dblk  lsize  asize  type
         4    1    16K    512    512    512  ZFS plain file
                                 264  bonus  ZFS znode
        path    /testfile01
        uid     0
        gid     0
        atime   Fri Oct 31 12:35:51 2008
        mtime   Fri Oct 31 12:35:51 2008
        ctime   Fri Oct 31 12:35:51 2008
        crtime  Fri Oct 31 12:35:51 2008
        gen     8
        mode    100644
        size    21
        parent  3
        links   1
        xattr   0
        rdev    0x0000000000000000
Indirect blocks:
               0 L0 0:11600:200 200L/200P F=1 B=8

                segment [0000000000000000, 0000000000000200) size   512

Now, see that "Indirect block" 0? Following L0 (Level 0) is a tuple: "0:11600:200". This is the DVA and Size, or more specifically it is the triple: vdev:offset:size. We can use this information to request its contents directly.

And so, the -R form can display and individual blocks from a device. To do so, we need to know the pool name, vdev/offset (DVA) and its size. Given what we did above, we now know that, so lets try it:

root@quadra /$ zdb -R testpool:0:11600:200
Found vdev: /zdev/disk002

testpool:0:11600:200
          0 1 2 3 4 5 6 7   8 9 a b c d e f  0123456789abcdef
000000:  2073692073696854  6620747365742061  This is a test f
000010:  0000000a2e656c69  0000000000000000  ile.............
000020:  0000000000000000  0000000000000000  ................
000030:  0000000000000000  0000000000000000  ................
000040:  0000000000000000  0000000000000000  ................
000050:  0000000000000000  0000000000000000  ................
...

w00t! We can read the file contents!

You'll notice in the zdb syntax ("zdb -h") that this syntax above accepts flags as well. We can find these in the ZDB source. The most interesting is the "r" flag which rather than display the data as above, actually dumps the data in raw form to STDERR.

So why is this useful? Try this on for size:

root@quadra /$ rm /testpool/dataset01/testfile01
root@quadra /$ sync;sync
root@quadra /$ zdb -dd testpool/dataset01
Dataset testpool/dataset01 [ZPL], ID 30, cr_txg 6, 18.0K, 4 objects

    Object  lvl   iblk   dblk  lsize  asize  type
         0    7    16K    16K    16K  14.0K  DMU dnode
         1    1    16K    512    512     1K  ZFS master node
         2    1    16K    512    512     1K  ZFS delete queue
         3    1    16K    512    512     1K  ZFS directory

 ....... THE FILE IS REALLY GONE! ..........

root@quadra /$ zdb -R testpool:0:11600:200:r  2> /tmp/output
Found vdev: /zdev/disk002
root@quadra /$ ls -lh /tmp/output 
-rw-r--r-- 1 root root 512 Nov  1 01:54 /tmp/output
root@quadra /$ cat /tmp/output 
This is a test file.

How sweet is that! We delete a file, verify with zdb -dd that it really and truely is gone, and then bring it back out based on its DVA. Super sweet!

Now, before you get overly excited, some things to note... firstly, if you delete a file in the real world you probly don't have its DVA and size already recorded, so your screwed. Also, notice that the origonal file was 21 bytes, but the "recovered" file is 512... its been padded, so if you recovered a file and tried using an MD5 hash or something to verify the content it wouldn't match, even though the data was valid. In other words, the best "undelete" option is snapshots.. they are quick, easy, use them. Using zdb for file recovery isn't practical.

I recently discovered and used this method to deal with a server that suffered extensive corruption as a result of a shitty (Sun Adaptec rebranded STK) RAID controller gone berzerk following a routine disk replacement. I had several "corrupt" files that I could not read or reach, if I tried to do so I'd get a long pause, lots of errors to syslog, and then a "I/O Error" return. Hopeless, this is a "restore from backups" situation. Regardless, I wanted to learn from the experience. Here is an example of the result:

[root@server ~]$ ls -l /xxxxxxxxxxxxxx/images/logo.gif
/xxxxxxxxxxxxxx/images/logo.gif: I/O error

[root@server ~]$  zdb -ddddd pool/xxxxx 181359
Dataset pool/xxx [ZPL], ID 221, cr_txg 1281077, 3.76G, 187142 objects, rootbp [L0 DMU objset] 400L/200P DVA[0]=<0:1803024c00:200> DVA[1]=<0:45007ade00:200> fletcher4 
lzjb LE contiguous birth=4543000 fill=1
87142 cksum=8cc6b0fec:3a1b508e8c0:c36726aec831:1be1f0eee0e22c

    Object  lvl   iblk   dblk  lsize  asize  type
    181359    1    16K     1K     1K     1K  ZFS plain file
                                 264  bonus  ZFS znode
        path    /xxxxxxxxxxxxxx/images/logo.gif
        atime   Wed Aug 27 07:42:17 2008
        mtime   Wed Apr 16 01:19:06 2008
        ctime   Thu May  1 00:18:34 2008
        crtime  Thu May  1 00:18:34 2008
        gen     1461218       
        mode    100644
        size    691  
        parent  181080
        links   1
        xattr   0
        rdev    0x0000000000000000
Indirect blocks:
               0 L0 0:b043f0c00:400 400L/400P F=1 B=1461218

                segment [0000000000000000, 0000000000000400) size    1K

[root@server ~]$ zdb -R pool:0:b043f0c00:400:r 2> out
Found vdev: /dev/dsk/c0t1d0s0
[root@server ~]$ file out   
out: GIF file, v89

Because real data is involved I had to cover up most of the above, but you can see how the methods we learned above were used to gain a positive result. Normal means of accessing the file failed miserably, but using zdb -R I dumped the file out. As a verification I opened the GIF in an image viewer and sure enough it looks perfect!

This is a lot to digest, but this is about as simple a primer to zdb as your going to find. Hopefully I've given you a solid grasp of the fundamentals so that you can experiment on your own.

Where do you go from here? As noted before, I recommend you now check out the following:

Max Bruning's ZFS On-Disk Format Using mdb and zdb: Video presentation from the OpenSolaris Developer Conference in Prague on June 28, 2008. An absolute must watch for the hardcore ZFS enthusiast. Warning, may cause your head to explode!
Marcelo Leal's 5 Part ZFS Internals Series. Leal has tremendous courage to post these, he's doing tremendous work! Read it!

Good luck and happy zdb'ing.... don't tell Sun. :)

- - C O M M E N T S - -
Nice post Ben!
keep going the excellent work, and let’s dig deeper into ZFS!

Leal.

Leal (Email) (URL) - 03 November '08 - 01:04

Hi there!
My first post at this great blog!
I wanna show u my dayly updated blog: Black Amateur Fuck Video
Have a nice day!
BB!

P.S. if you don’t want to see this message please write me to no.ads08@gmail.com with subject “NO ADS” and URL of your forum
Thank you for cooperation!

KurmanAhlabm (Email) (URL) - 07 November '08 - 16:08

Бесспорно, бессмысленная новость

cicypebep (Email) (URL) - 07 November '08 - 17:25

Excellent blog! Interesting article and very informative! I will necessarily subscribe for this blog. [[http://lowsalego.com/map.html]]

bopdilly (Email) (URL) - 13 November '08 - 02:43

Excellent blog! Interesting article and very informative! I will necessarily subscribe for this blog. [[http://lowsalego.com/map.html]]

bopdilly (Email) (URL) - 13 November '08 - 02:44

Hello Ben!
Just for update my blog address:
[[http://www.eall.com.br/blog]]
In case someone want to follow the link, and do not get an error.
The link for the post in your article would be:
[[http://www.eall.com.br/blog/?p=312]]

Leal (Email) (URL) - 13 November '08 - 11:09

Ben, enjoyed reading your blog. zdb is sweet. wondering if there is way to import a duplicate copy of a pool created using metro-mirror kind of setup?

Jay (Email) - 03 February '09 - 22:49

I like the gold.

aoc money (URL) - 27 February '09 - 08:14

Enjoyed reading your blog. zdb is sweet. wondering if there is way to import a duplicate copy of a pool created using metro-mirror kind of setup?

John (Email) (URL) - 30 March '09 - 14:04

Nice post.

Gary Winnick (Email) (URL) - 20 April '09 - 06:52

Good Job.

used plastic shredder (Email) (URL) - 05 May '09 - 11:54

Enjoyed reading your blog. zdb is sweet. wondering if there is way to import a duplicate copy of a pool created using metro-mirror kind of setup?

Chris (Email) (URL) - 10 May '09 - 07:42

Het was nooit zo gemakkelijk aangezien het nieuwe idee voor de uw grote benedenlijnbouw met de procedures betreffende rejse werkt.

girlteam (Email) (URL) - 21 October '09 - 07:29

For all the internet business people targeted traffic is very important which you can only expect if you have some clear ideas about [url=[[http://www.altidbulgarien.dk]billige]] rejser[/url] watches.

girlteam (Email) - 21 October '09 - 07:29

It was a very nice idea! Just wanna say thank you for the information you have shared. Just continue writing this kind of post. I will be your loyal reader. Thanks again.

Christian louboutin shoes (Email) (URL) - 27 October '09 - 01:29

Thank you very much!

cheap links of london (Email) (URL) - 07 November '09 - 01:24

Great post! Hope to be better. Better means more features.
good post,I think so!

links of london (Email) (URL) - 17 November '09 - 04:53

Great post! Hope to be better. Better means more features.
good post,I think so!
Thanks for your information, i have read it, very good！
Bing is a really overlord!! support Bing~~
This is great news. Best of luck for the future and keep up the good work.

tiffany and co (Email) (URL) - 19 November '09 - 01:30

Just one question: how to add your blog into my rrs reader, thanks so much.

christian louboutin (Email) (URL) - 03 January '10 - 07:05

Hey Ben, excellent info, I’ve tried it also but what do you do when you get multiple indirect blocks for a huge file ?
Indirect blocks:
0 L0 0:b043f0c00:400 400L/400P F=1 B=1461218

In my case I have multiple lines of output instead of only one, how does the full vdev/dva uri apllies here ? I’m trying to restore a huge file (.iso 650 mb) and my pool cannot be reimported (i/o error whatever that is, pool was peachy until dead one morning after power cut) I would like a more detailed tut on restoring files manually. I’m also thinking or writing a custom script on that. Thx

ruessavelm (Email) - 12 January '10 - 05:41

Good post! Thanks you for your information! China Wholesale Wholesale China Wholesalers Wholesale Game Accessories Wholesale Iphone Accessories Video Game Accessories Wholesale Wholesale Wii Accessories Wholesale Xbox 360 Accessories Wholesale Xbox 360 Games Wholesale Video Games Cheap Video Games Cheap Ps3 Games Cheap Xbox 360 Games Wholesale Computers Wholesale Laptop Computers Wholesale Laptops Discount Computers Cheap Computers Wholesale Iphones Wholesale Iphone Wholesale Iphones 3g Hiphones Wholesale Hiphone Wholesale Hiphones Wholesale Nokia Wholesale Nokia 8800 Wholesale Nokia n97 wholesale blackberry wholesale blackberry phones wholesale blackberry 9700 wholesale blackberry 9600 wholesale blackberry 9500

raging bull (Email) (URL) - 15 January '10 - 06:42

thanks

kamagra (Email) (URL) - 21 January '10 - 01:26

good read thanks

M65 Jacket (Email) (URL) - 21 January '10 - 01:26

many thanks

viagra cialis (Email) (URL) - 21 January '10 - 01:55

Hey man…sorry I missed the party.

javon (Email) (URL) - 25 January '10 - 09:11

Good post! Thanks for your information!

As Seen On TV (Email) (URL) - 27 January '10 - 07:16

Ed Hardy Hats were later launched with much fanfare. Today, they are perceived as status symbols and extremely popular among the young and old, and even celebrities.

ed hardy (Email) (URL) - 25 February '10 - 03:02

Ed Hardy

ed hardy (Email) (URL) - 25 February '10 - 03:04

You probably spent quite a bit of money on your ugg boots, so it’s important bailey button ugg boots take care of them properly to ensure they last for a long time. uggs No one wants to run around in a pair of boots that look classic tall ugg boots and ugg sale, so here are a few classic short ugg boots you can do to make sure your ugg boots sale keep that brand new look for as long as possible.classic cardy ugg boots
Do NOT wash your cheap ugg boots. If you should find metallic ugg boots been stained you should attempt ugg australia spot treat them before you do anything metallic ugg boots . You may need to scrub the entire winter boots if spot treatments do not work, but this is not something you should do on nightfall ugg boots regular basis.
Are you looking a discount ugg boots online? There are various places an individual can purchases sundance ii ugg boots. The most popular is to purchase ugg boots 2010 online. There are a ultra tall ugg boots of popular websites that cheap uggs preferred amongst users due to its reliability.
ultra short ugg boots can also could be worn by older children. Many kinds of colours of uggs on sale are available for kids, and their stylish design makes sure kids will like to wear the australia ugg boots as they will be seen as the most fashionable kids in the playground, while you can rest assured that your children are warm enough.

ugg boots (Email) (URL) - 26 February '10 - 03:05

Ed Hardy Hats were later launched with much fanfare. Today, they are perceived as status symbols and extremely popular among the young and old, and even celebrities.
ed hardy

ed hardy (Email) (URL) - 03 March '10 - 08:00

one day i went shopping outside,and in an ed hardy store,I found some kinds of ed hardy i love most they are Your website is really good Thank you for the information

bt lami shoes for sale (Email) (URL) - 09 March '10 - 03:23

Follow your dreams, you can reach your goals.

jju (Email) (URL) - 10 March '10 - 02:14

Don’t waste your time on a man/woman, who isn’t willing to waste their time on you.

links of london (Email) (URL) - 10 March '10 - 08:12

cool ver very good

Hiphop (Email) (URL) - 04 April '10 - 20:59

Here have a variety of advanced technology to produce electronic products,
[[http://www.trademic.com]]
Most wholesale electronics today use semiconductor apparatus to accomplish electron control. The abstraction of

semiconductor wholesale electronics and accompanying technology is advised a annex of physics, admitting the architecture and

architecture of cyberbanking circuits to break applied problems appear beneath go to wholesale electronics,engineering. This

commodity focuses on engineering aspects of electronics.wholesale electronics is a able manufacturer, specializes in

development, manufacture. We accept broad Cheap Consumer Electronics, China wholesale Electronics go!: Like Cheap Unlocked

Cell Phones, Hiphone, MP3 Players, MP4 Players, MP5 Players, Computer Accessories, Car Electronics, Bluetooth Products, USB

Flash Drive, Memory Cards, Health & Beauty, iPod & iPhone Accessories, Digital Photo Frames,Car Audios, Car DVD Players,

Digital Cameras, Digital Camcorders, Video Game Accessories, Spy Cameras, Security Products and Electronic Gadgets from

China.
We accept over 2500 accomplished and accomplished advisers and our accessories of 7,300 aboveboard meters are able with

avant-garde productions and testing machines. We accept the absolute superior ascendancy arrangement and strive to

accommodated and beat our customer’s expectations,wholesale electronics.
Do you find mbt lami shoes for a long time. Here is the richest in the world and you can find your needs bring you harvest.

yes,
[[http://www.mbtsonsale.com/mbt-lami.htm..]] mbt lami shoes Purple Shoes is the first physiological

footwear that has a positive effect on the whole body. buy mbt lami shoes ’s sole is curved, not flat. Anti

-shoe is unstable, not stable. Swissmasai activates your muscles instead of undermining them. The mbt lami

shoes Birch Shoes Chapa feature feature rich nubuck leather with single-layer mesh upper bodies for strong but

breathable wear,mbt lami.

wholesale electronics (Email) (URL) - 19 April '10 - 05:45

agwfeature rich nubuck leather with single-layer mesh upper bodies for strong but

aee (Email) (URL) - 19 April '10 - 08:11

I very intersted in the article

cheap mbt shoes (Email) (URL) - 23 April '10 - 03:02

good site have a look
[[http://www.trademic.com/cheap-wholesal..]]

wholesale dvd (Email) (URL) - 05 May '10 - 02:19

Enjoyed reading your blog. zdb is sweet. wondering if there is way to import a duplicate copy of a pool created using metro-mirror kind of setup?
[[http://www.trademic.com/product/index...]]

wholesale china (Email) (URL) - 05 May '10 - 02:24

hi there, thanks for your great post, it help me so much! now i wonder if you linke the mbt shoes from my site: the url is [[http://www.thesuitshoes.com]]

mbt shoes (Email) (URL) - 12 May '10 - 06:45

I love air jordan shoes

air jordan shoes (Email) (URL) - 12 May '10 - 13:08

you have come to the right place. We provide the highest quality Battery at the lowest price with the highest level of service, all in a secure and convenient platform. [[http://www.adapterlist.com/hp/pavilion..]] Hp pavilion dv9700 laptop battery

laptop battery (Email) (URL) - 14 May '10 - 09:14

HP pavilion dv2000 Battery [[http://www.globallaptopbattery.co.uk/h..]]

laptop batteries (URL) - 14 May '10 - 09:27

The newest version of air max 90 are available now.Welcome to see our special nike air max 2009 and air max. They are your must-have nike air max 2010|air max 97|cheap air max.

air max (Email) (URL) - 20 May '10 - 06:26

This is a good,common sense article.Very helpful to one who is just finding the resouces about this part.It will certainly help educate me.

Bell & Ross (Email) (URL) - 20 May '10 - 08:35

[[http://www.okayoutlet.com]]

gucci outlet (Email) (URL) - 24 May '10 - 02:23

thank you for sharing the article.
The easy-to-use Video Converter for Mac lets you to enjoy your videos on all sorts of palyback including PSP, iPod, Mobile Phone, Zune, iPhone, Apple TV and MP4/MP3 player.
Free download supported. [[http://www.videoconverterformac.com]]

video converter for mac (Email) (URL) - 28 May '10 - 05:54

This is all very new to me and this article really opened my eyes.Thanks for sharing with us your wisdom.

Replica Tag Heuer Watches (Email) (URL) - 31 May '10 - 07:05

I like writing, hurriedly, not that you are not
[[http://www.woe-from.com]]
[[http://www.louis-vuitton-outlet-store...]]

louis vuitton (Email) (URL) - 01 June '10 - 02:21

cheap supra shoes

cheap supra shoes (Email) (URL) - 02 June '10 - 06:57

Thanks for sharing your article.

cheap jordan shoes (Email) (URL) - 21 June '10 - 02:16

cheap nike shox shoes [[http://www.livedealmall.net/]]

nike shox shoes (Email) (URL) - 22 June '10 - 09:33

[[http://www.mbtshoeslatest.com]]
[[http://www.nikeairmaxshoe.com]]

Nike air max shoe (Email) (URL) - 24 June '10 - 07:55

Do you like the ugg ? Ugg Classic Short Boots are UGG Australia’s original heritage styles. Ugg Classic Tall will keep your feet dry and the ultimate comfortable.
[[http://www.ugg-boots-london.com/ugg-cl..]]

Ugg Classic Short (Email) (URL) - 07 July '10 - 01:25

Doing homework means getting a tattoo machine([[http://www.idealhere.com/]])
10 Wrap Coils([[http://www.idealhere.com/Tattoo-Machin..]]) complete and thorough knowledge about the market, its trends, its requirements, its pros and cons and a lot more other aspects.
www.idealhere.com

Wholesale Accessories (Email) (URL) - 07 July '10 - 05:36

Your article are great, will visit very soon. there is a site: [[http://www.cheapinsanitydvds.com/]]

Insanity (Email) (URL) - 07 August '10 - 13:45

The first layer of MBT Shoes means a high quality MBT Footwear cowhide after processing of machine. The main feature of MBT first layer cowhide: Masai Shoes surface layer are made of grain materials with close woven fibers; MBT Sandals feature smooth feeling, good strength and great abrasive resistance. MBT Medical Shoes lining are composed of suede with thick fiber, big diameter and flocking wool on the surface. Comparing with second layer of Discount MBT Sheos, first layer of MBT Shoes Cheap with more smooth surface and better quality, can be worn longer time. Our website launches MBT Outlet with first layer cowhide, so you can feel free to shop here and get satisfied MBT Shoes Clearance, including MBT Sawa, MBT Sirima, MBT Kaya, MBT Changa and so on .
Welcome to our website: [[http://www.mbtshoesmasai.com]]

mbtshoesmasai (Email) (URL) - 09 August '10 - 02:52

[url=[[http://www.cheapjerseyschina.com]disco..]] nfl jerseys from china[/url]
[url=[[http://www.cheapjerseyschina.com]disco..]] nfl jerseys wholesale[/url]

tanglnog (Email) (URL) - 10 August '10 - 02:17

Buy a piece of ghd for yourself. Come and join us [[http://www.ghdiron-outlet.com/]] to win the cheap ghd.

ghd outlet (Email) (URL) - 13 August '10 - 03:27

As we know, now GHD are loved by more and more people, which will save up to 45%.welcome to [[http://www.ghdoutlet-au.com/.]].

ghd straighteners (Email) (URL) - 13 August '10 - 03:33

Come and join us [[http://www.ghdoutlet-uk.com/index.php]] to win the ghd iv styler.

ghd hair straighteners (Email) (URL) - 13 August '10 - 03:34

One Sunday womens timberland boots morning, I was told that someone was waiting for me in timberland 6 boots office. The young person who answered the rectory timberland boots door said that it was “the woman who said she left all the notes.” When I saw her mens timberland boots I was shocked, since I immediately recognized her from church but had no timberland boots outlet idea that it was she who wrote the notes. She was sitting in a chair in timberland roll top boots office with her hands folded in her lap. Her head was bowed and when she raised white timberland boots it to look at me, she could barely smile without pain.

timberland pro boots (Email) (URL) - 25 August '10 - 06:21

very cool article ,thanks for sharing the article!like my cool stuff .very useful.
uCoolStuff is the leading China wholesaler for [[http://www.ucoolstuff.com]] cool stuff [[http://www.ucoolstuff.com]] cool gifts , unusual gadgets and other unique gift ideas. We provide the very latest cool stuff and cool gifts for you

cool stuff (Email) (URL) - 25 August '10 - 07:29

excellent article , I added you to my [[http://www.china-wholesale-directory.c..]] Top China Wholesalers category.. thanks for sharing the article!

China Wholesale Directory (Email) (URL) - 25 August '10 - 07:37

Top Online Stores is a SEO Friendly [[http://www.toponlinestores.org]] free directory where you can find the best online shopping stores selected by hand and sorted by category.

link directory (Email) (URL) - 14 September '10 - 07:08

There are various kinds of ghd hair straighteners on hot sell. GHD styler specialized in ghd australia market and ghd outlet. It provides many cheap ghd, ghd iv styler, pink ghd, purple ghd, which save up to 50%, with free shipping!

ghd outlet (Email) (URL) - 25 October '10 - 05:19

Hey dear your blog is very much rocking and stunning. This blog has main attraction and high quality features.

newest jordan shoes (Email) (URL) - 08 November '10 - 06:26

i have been pleased after reading this blog the knowledge which has been provided via this blog is simply tremendous.
i really appreciate the blogger for doing this

refurbish used computer (Email) (URL) - 07 February '11 - 06:25