Understanding ZFS: Compression

06 Nov '08 - 09:31 by benr

One of the most appealing features ZFS offers is built in compression capabilities. The tradeoffs are self evident, consume additional CPU but conserve disk space. If your running an OLTP database then compression probly isn't for you, however if you are doing bulk data archiving this could be a huge win.

ZFS is built with the realization that in modern systems we typically have large amounts of memory and CPU available, and we should be provided with the means to put those resources to work. Contrast this with the traditional logic that compression slows things down, because we stop and compress the data before flushing it out to disk, which takes time. Consider that in some situations, you may have significantly faster CPU and Memory than you have IO throughput, in which case it may in fact be faster to read and write compressed data because your reducing the quanity of IO through the channel! Compression isn't just about saving disk space... keep an open mind.

The first important point about ZFS Compression is that its granular. Within ZFS we create datasets (some people call them "nested filesystems", but I find that confusing terminology), each of which has inherited properties. One of those properties is compression. Therefore, if we create a "home" dataset which mounts to "/home", and then create a "home/user" dataset for each user, we can do interesting things, such as apply per-user quotas (disk usage limits) or reservations (set aside space) or, in this context, enable, disable or specify differing types of compression. Some users may want compression, others may not, or you may wish all to use it by default. ZFS gives us a wide range of flexible options. Most importantly, if we change our mind at some point we can change the setting and all new data is compressed, the old uncompressed data is still used as expected. This means that changes are no disruptive, however this does mean that if you really want to later conserve all the disk you can you'd need to enable compression and then slosh all the data off and then back in.

So, how do we enable compression? Simple, use the zfs set compression=on some/dataset command. If we then "get all" properties from a dataset we'll see some interesting information. Here is an example (pruned for length) of my home directory:

root@quadra ~$ zfs get all quadra/home/benr
NAME              PROPERTY         VALUE                  SOURCE
quadra/home/benr  type             filesystem             -
quadra/home/benr  creation         Thu Oct  9 11:33 2008  -
quadra/home/benr  used             122G                   -
quadra/home/benr  available        432G                   -
quadra/home/benr  referenced       122G                   -
quadra/home/benr  compressratio    1.19x                  -
quadra/home/benr  mounted          yes                    -
quadra/home/benr  quota            none                   default
quadra/home/benr  reservation      none                   default
quadra/home/benr  recordsize       128K                   default
quadra/home/benr  mountpoint       /quadra/home/benr      default
quadra/home/benr  checksum         on                     default
quadra/home/benr  compression      on                     inherited from quadra/home
...

Here we see that compression is "on", and was inherited automatically from its parent dataset "quadra/home". We can also see the compression ratio above: 1.19x.

But what are our options? Just on or off? Many ZFS properties have simplistic "defaults", in this case "on" means that we use the "lzjb" compression algorithm. We can instead specify the exact algorithm. Current, in fairly modern releases of Nevada/OpenSolaris we have available the default LZJB (a lossless compression algorithm created by Jeff Bonwick, which is extremely fast) and gzip at compression levels 0-9. If you set "compression=gzip" you'll get GZIP level 6 compression, however you can explicitly "set compression=gzip-9". More compression algorithms may be added in the future. (The source is out there, feel free to give us another!)

But how can you see the effect? Did you know the "du" command will show you the on-disk (compressed) size of a file? Lets experiment!

root@quadra ~$ zfs create quadra/test
root@quadra ~$ zfs get compression quadra/test
NAME         PROPERTY     VALUE        SOURCE
quadra/test  compression  off          default

Ok, we have a dataset to play with. I've downloaded Moby Dick and combined into a single text file.

root@quadra test$ ls -lh moby-dick.txt 
-rw-r--r-- 1 root root 1.8M Nov  6 01:38 moby-dick.txt
root@quadra test$ du -h moby-dick.txt 
1.8M    moby-dick.txt
root@quadra test$ head -4 moby-dick.txt 
.. < chapter I 2  LOOMINGS >

     Call me Ishmael.  Some years ago--never mind how
long precisely --having little or no money in my purse, and nothing particular

Alright, so here is Moby Disk in text, weighing in at 1.8M uncompressed. Lets not enable compression (LZJB), copy the file and see how much benefit we get:

root@quadra test$ zfs set compression=on quadra/test
root@quadra test$ cp moby-dick.txt moby-dick-lzjb.txt
root@quadra test$ sync

root@quadra test$ ls -lh
total 3.5M
-rw-r--r-- 1 root root 1.8M Nov  6 01:40 moby-dick-lzjb.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:38 moby-dick.txt
root@quadra test$ du -ah 
1.7M    ./moby-dick-lzjb.txt
1.8M    ./moby-dick.txt
3.5M    .

Nice, we're saving some space. Now lets repeat with gzip.

root@quadra test$ zfs set compression=gzip quadra/test
root@quadra test$ cp moby-dick.txt moby-dick-gzip.txt
root@quadra test$ ls -lh
total 4.6M
-rw-r--r-- 1 root root 1.8M Nov  6 01:44 moby-dick-gzip.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:40 moby-dick-lzjb.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:38 moby-dick.txt
root@quadra test$ du -ah 
1.7M    ./moby-dick-lzjb.txt
1.8M    ./moby-dick.txt
1.1M    ./moby-dick-gzip.txt
4.6M    .

Ahhhh. Nice gain there. Remember that this is gzip-6 really, lets crank it up to gzip-9!

root@quadra test$ zfs set compression=gzip-9 quadra/test
root@quadra test$ cp moby-dick.txt moby-dick-gzip9.txt
root@quadra test$ ls -lh
total 4.6M
-rw-r--r-- 1 root root 1.8M Nov  6 01:44 moby-dick-gzip.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:46 moby-dick-gzip9.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:40 moby-dick-lzjb.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:38 moby-dick.txt
root@quadra test$ du -ah 
1.7M    ./moby-dick-lzjb.txt
1.8M    ./moby-dick.txt
1.1M    ./moby-dick-gzip.txt
512     ./moby-dick-gzip9.txt
4.6M    .

Wow! Thats savings. Just to put this in context, I'll test gzip'ing the file like your used to (using tmpfs, not zfs):

root@quadra test$ cd /tmp
root@quadra tmp$ ls -alh moby-dick.txt 
-rw-r--r-- 1 root root 1.8M Nov  6 01:47 moby-dick.txt
root@quadra tmp$ gzip moby-dick.txt 
root@quadra tmp$ ls -alh moby-dick.txt.gz 
-rw-r--r-- 1 root root 1.1M Nov  6 01:47 moby-dick.txt.gz

And so we see here that just gzip'ing the file matches the compression I got with gzip (gzip-6) enabled.

But before you get too excited, remember that this is consuming system CPU time. The more compression you do, the more CPU you'll consume. If this is a dedicated storage system your working on, then consuming a ton of CPU for compression may well be worth it (many appliances have fast CPU's for just this reason), however if your running critical apps and CPU really counts, then notch it down or even turn it off. I highly recommend you dry-run your application workload and then load test it hard to see whether or not that extra CPU will be a problem. Whenever possible, try to determine these things before you deploy, not after.

To follow up that idea, remember that we can set differing compression levels on different datasets. You may want to put your application data on an uncompressed dataset, but store less commonly used data or backups on a separate dataset where you've cranked compression up. Get creative!

ZFS is an amazing technology and compression is certainly one of its big attractions for the common user. Workstation always low on disk? Compression to the rescue, no stupid FUSE or loopback tricks required. :)


A Word Of Warning

At this point I do want to warn you of something. Notice that du displays actual disk consumption, not true file size. Now consider the way in which most admins actually use the command... to total up cumulative file sizes. On a typical file system, "du -sh ." will nicely total up all the files, which would be the same as if I tar'ed up the files and looked at the tarball's filesize. When using compression you can not use "du" in this way because the files are larger than the actual disk usage. So you get into potentially confusing situations like this:

root@quadra test$ ls -alh
total 5.6M
drwxr-xr-x 2 root root    6 Nov  6 01:46 .
drwxr-xr-x 8 root root    8 Nov  6 01:33 ..
-rw-r--r-- 1 root root 1.8M Nov  6 01:44 moby-dick-gzip.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:46 moby-dick-gzip9.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:40 moby-dick-lzjb.txt
-rw-r--r-- 1 root root 1.8M Nov  6 01:38 moby-dick.txt
root@quadra test$ du -h .
5.6M    .
root@quadra test$ tar cfv md.tar moby-dick*
moby-dick-gzip.txt
moby-dick-gzip9.txt
moby-dick-lzjb.txt
moby-dick.txt
root@quadra test$ ls -lh md.tar  
-rw-r--r-- 1 root root 7.0M Nov  6 02:11 md.tar

In the real world, this could come to you as a shock if you wanted to rsync a bunch of data, totalled it up using "du" to estimate the bits that need to move, and then got nervous when you moved way more bits than you initially expected because your forgot to take compression into the equitation. So hopefully some of you can learn here not just how ZFS works, but appreciate "du" in a new way as well. :)


- - C O M M E N T S - -

wow, this is a great post! I’d heard a lot of good things about ZFS, and the more I hear, the more I hope Sun releases it on Linux. I have my doubts about how soon that will happen, though.

If it weren’t for the lack of a Solaris brain trust here, I’d be tempted to switch. I just don’t have the experience with it. I just keep getting more and more incentive to work with it.

Matt Simmons (Email) (URL) - 06 November '08 - 12:46

What the freak at logo of blog?

aZ - 06 November '08 - 14:47

Is there any way to determine which files have been compressed on the filesystem and which have not? I can see how you can determine the properties of the ZFS filesystems but I don’t see a way of determining the properties associated with a particular object (file, directory, etc.). If you could tell that a file was or was not created with compression, you could write a script which would only “slosh off and back on” files which needed their compression status changed. Similar considerations might apply when you need to adjust other properties (encryption, copies, etc.) as well.

Rand Huntzinger - 06 November '08 - 15:08

Good post as usual Ben!

Dave Pickens (Email) (URL) - 06 November '08 - 16:46

Ben,

Thanks for such a comprehensive post. I’ve mainly been using ZFS in-house, using it in anger for non-critical stuff so I feel more comfortable when I jump to deploying business critical apps on it.

But your post lead me to thinking – I’ve also been considering throwing together a fat-ass drive array + mobo for HTPC storage accessed via iSCSI [that way I can have a huge quantity of video and audio content, and the noisy box can get stashed somewhere far away from the home theatre. Do you think that ZFS+compression would cope with that – provided the CPU had enough capacity? From the above, I can’t see why not…thus giving me even more “space” for my data!

Also – do you think (Open)Solaris needs to add an legacy option to du to “help” those of us sysadmins who regularly jump between say Solaris, Ubuntu and Mac OS X? And might just forget that we’re not necessarily looking at the actual total file size, which as you say is what we’re accustomed to?

Mark Glossop - 06 November '08 - 16:54

Mark,

I’m pretty sure that’s and ideal usage because the CPU would otherwise be pretty idle. It’s where the processing and the storage occur in the same place that you’d get into trouble, such as if you were decoding/encoding data directly on the system with ZFS compression. It all depends on whether you’re I/O bound or CPU bound.

I’m glad to hear about the options to compression. I’ve previously only been using on/off, but I’m interested in trying to get some more space out of backup boxes for which space is much more critical than storage.

One question, how granular is the compression, really? That is, is compression less of an issue with large sequential accesses over small random accesses? Is the compression over each block?

Drew (Email) - 07 November '08 - 18:15

Hi there!
My first post at this great blog!
I wanna show u my dayly updated blog: Black Amateur Fuck Video
Have a nice day!
BB!

P.S. if you don’t want to see this message please write me to no.ads08@gmail.com with subject “NO ADS” and URL of your forum
Thank you for cooperation!

KurmanAhlabm (Email) (URL) - 08 November '08 - 19:28

I have a hypothesis that compression performance in filesystems isn’t intuitive, and it’s always the case that if one tries it, the performance will “feel” better than one expects.

There are two reasons for this:

1. It is natural to think of compression in simple space/time tradeoff terms. However, every block of data that doesn’t have to be transferred also saves the CPU time required to manage the transfer. Furthermore, in terms of CPU time accounting, that time is often attributed to general system overhead costs and does not get credited to a specific user process, so accounting data will often show the additional CPU cost, but not the related CPU savings.

2. It is natural to think of files which aren’t human readable as random data, and hence not compressible or less compressible than they actually are.

Dave Hamaker (Email) - 04 December '08 - 21:19

>When using compression you can not use “du” in this way because the
>files are larger than the actual disk usage.

i don`t have solaris around here at home, but why not using gnu`s “du” ? that one has ”—apparent-size” as an option and iirc, it gave me the real size of compressed files on zfs-fuse.

here`s the manpage: [[http://www.gnu.org/software/coreutils/..]]

roland (Email) - 01 January '09 - 18:03

Hey, uh, I downloaded moby.zip from the URL you gave, concatenated all the files inside except for moby.0 (which isn’t in your head output) and README, but the resulting file was only 1202893 bytes. Where are your other 600k coming from?

Also, what’s wrong with LZJB that it could only compress that file by 7% or so? I don’t have an LZJB implementation handy, but LZF (which is usually lightning fast but for some reason takes 100ms on this file) compresses it by 40% down to 737181 bytes, and gzip compresses it by 53% to 59% (569776 to 487367 bytes, -1 and -9), not the mere 49% you got. And what’s up with gzip compressing the file to 512 bytes the second time?

Kragen Javier Sitaker (Email) (URL) - 24 January '09 - 06:02

Very good opinions. I am sure, will be useful for everyone .thank you very much for informing to me .good work.i will try this just after my 70-290 for managing and maintaining a Microsoft windows server 2003 environment about which I am confident to pass in first attempt because I have already pass 70-649 which is for Microsoft certification as well as your page concern I must say that you have done a great job I must return on your page to read you more.

TestKing (Email) (URL) - 17 March '09 - 12:41

I downloaded moby.zip from the URL you gave, concatenated all the files inside except for moby.0 (which isn’t in your head output) and README, but the resulting file was only 1202893 bytes. Where are your other 600k coming from?

Chris (Email) (URL) - 10 May '09 - 07:08

I downloaded moby.zip from the URL you gave, concatenated all the files inside except for moby.0 (which isn’t in your head output) and README, but the resulting file was only 1202893 bytes. Where are your other 600k coming from?

Same ask.

rJr (Email) (URL) - 07 July '09 - 07:15

It was a very nice idea! Just wanna say thank you for the information you have shared. Just continue writing this kind of post. I will be your loyal reader. Thanks again.

Christian louboutin shoes (Email) (URL) - 27 October '09 - 00:43

Thank you very much!

cheap links of london (Email) (URL) - 07 November '09 - 00:50

Thanks!!

Tiffany jewellery (Email) (URL) - 14 November '09 - 02:38

Great post! Hope to be better. Better means more features.
good post,I think so!
Thanks for your information, i have read it, very good!
Bing is a really overlord!! support Bing~~
This is great news. Best of luck for the future and keep up the good work.

links of london (Email) (URL) - 17 November '09 - 03:18

Just one question: how to add your blog into my rrs reader, thanks so much.

christian louboutin (Email) (URL) - 03 January '10 - 07:06

links of london links of london blog

links of london (Email) (URL) - 12 January '10 - 11:22

Good post! Thanks you for your information! China Wholesale Wholesale China Wholesalers Wholesale Game Accessories Wholesale Iphone Accessories Video Game Accessories Wholesale Wholesale Wii Accessories Wholesale Xbox 360 Accessories Wholesale Xbox 360 Games Wholesale Video Games Cheap Video Games Cheap Ps3 Games Cheap Xbox 360 Games Wholesale Computers Wholesale Laptop Computers Wholesale Laptops Discount Computers Cheap Computers Wholesale Iphones Wholesale Iphone Wholesale Iphones 3g Hiphones Wholesale Hiphone Wholesale Hiphones Wholesale Nokia Wholesale Nokia 8800 Wholesale Nokia n97 wholesale blackberry wholesale blackberry phones wholesale blackberry 9700 wholesale blackberry 9600 wholesale blackberry 9500

raging bull (Email) (URL) - 15 January '10 - 06:41

thanks

kamagra (Email) (URL) - 21 January '10 - 01:21

good read thanks

M65 Jacket (Email) (URL) - 21 January '10 - 01:22

many thanks

viagra cialis (Email) (URL) - 21 January '10 - 01:55

I have enjoyed reading, I will make sure and bookmark this page and be back to follow you.

hickory nc furniture (Email) (URL) - 23 January '10 - 22:52

[[http://www.linkslife.uk.com]] links of london
[[http://www.linkslife.uk.com]] links of london jewellery

links of london (Email) (URL) - 03 March '10 - 07:44

If you want to buy real estate, you would have to get the loans. Furthermore, my mother all the time utilizes a financial loan, which is really reliable.

Sheree31Mayer (Email) (URL) - 10 April '10 - 01:49

In a woman’s dress, the jewelry is unique; there is only one role— let a woman be more elegant, links of london friendship bracelet attractive and beautiful. But, one thing I can make a sure: An elegant woman, even if she loved jewelry, but also must not be too obsessed with this, so that confused his whole body from top to bottom are hanging ornaments, like a Christmas tree.

wangrongjiao (Email) (URL) - 30 April '10 - 07:11

I like you post. [[http://www.offerreplicawatches.com.The..]] sell Chanel Watches, Replica BRM Watches,offer Cartier Watches, Copy Ebel Watches, Replica Hublot Watches, Fake Tag Heuer Watches, wholesale Richard Mille Watches.

They also sell some very fashion watches, such as Montblanc Watches for sale,Chopard Watches, Dior Watches and Gucci Watches. Rolex watches are hot sell. If u are Armani Watches and Breitling Watches fan. You can still look at Omega Watches and Longines Watches, many new arrivals watches just come here.You can have a try on 58338312391273913. :[url=[[http://www.offerreplicawatches.com]www..]]].

Replica Chanel Watches,Dvd Boxes (Email) (URL) - 12 May '10 - 07:33

If you are looking for the best brand new laptop batteries at the most preferred price, you have come to the right place. [[http://www.adapterlist.com/toshiba/sat..]] We provide the highest quality Battery at the lowest price with the highest level of service, all in a secure and convenient platform.

laptop battery (Email) (URL) - 14 May '10 - 08:51

thank you for this outstanding article.I thought certino was the best technologh for laptop battery performance.We specialize in [[http://www.globallaptopbattery.co.uk/s..]] ,laptop AC adapters. All our products are brand new, with the excellent service from our laptop battery of customer service team.

laptop batteries (Email) (URL) - 14 May '10 - 08:54

thank you for sharing the article.
The easy-to-use Video Converter for Mac lets you to enjoy your videos on all sorts of palyback including PSP, iPod, Mobile Phone, Zune, iPhone, Apple TV and MP4/MP3 player.
Free download supported. [[http://www.videoconverterformac.com]]

video converter for mac (Email) (URL) - 28 May '10 - 05:53

[[http://www.airjordanshoescheap.com/]]

air jordan shoes (Email) (URL) - 02 June '10 - 02:13

[[http://www.cuddletech.com/blog/pivot/e..]]

vibram five fingers (Email) (URL) - 10 June '10 - 08:44

It’s very nice! I love what you wrote.
I think we can make friends.

vibram five fingers (Email) (URL) - 22 June '10 - 08:20

[[http://www.mbtshoeslatest.com]]
[[http://www.nikeairmaxshoe.com]]

Nike air max shoe (Email) (URL) - 24 June '10 - 06:19

I totally love this article.

ed hardy (Email) (URL) - 02 July '10 - 08:49

I will bookmark your blog and have my kids check up here frequently Air Jordan 2010. I’m very certain they will understand lots of new stuff Air jordan 1 here than anybody else. I do appreciate that youve added relevant and intelligent commentary here though. Thank you!

Cheap Jordan Shoes (Email) (URL) - 03 August '10 - 06:42

Your article are great, will visit very soon. there is a site: [[http://www.cheapinsanitydvds.com/]]

Insanity review (Email) (URL) - 07 August '10 - 13:40

The first layer of MBT Shoes means a high quality MBT Footwear cowhide after processing of machine. The main feature of MBT first layer cowhide: Masai Shoes surface layer are made of grain materials with close woven fibers; MBT Sandals feature smooth feeling, good strength and great abrasive resistance. MBT Medical Shoes lining are composed of suede with thick fiber, big diameter and flocking wool on the surface. Comparing with second layer of Discount MBT Sheos, first layer of MBT Shoes Cheap with more smooth surface and better quality, can be worn longer time. Our website launches MBT Outlet with first layer cowhide, so you can feel free to shop here and get satisfied MBT Shoes Clearance, including MBT Sawa, MBT Sirima, MBT Kaya, MBT Changa and so on .
Welcome to our website: [[http://www.mbtshoesmasai.com]]

mbtshoesmasai (Email) (URL) - 09 August '10 - 02:52

Buy a piece of ghd for yourself. Come and join us [[http://www.ghdiron-outlet.com/]] to win the cheap ghd.

ghd outlet (Email) (URL) - 13 August '10 - 03:20

As we know, now GHD are loved by more and more people, which will save up to 45%.welcome to [[http://www.ghdoutlet-au.com/.]].

ghd straighteners (Email) (URL) - 13 August '10 - 03:20

Come and join us [[http://www.ghdoutlet-uk.com/index.php]] to win the ghd iv styler.

ghd hair straighteners (Email) (URL) - 13 August '10 - 03:21

It is very beautiful. Thank you for sharing. There is a site: [[http://www.lovemypursemall.com]]

designer bags (Email) (URL) - 14 August '10 - 14:35

You say well, I am agree with you very much.

hair straighteners (Email) (URL) - 17 August '10 - 08:39

I know this stuff.It’s one of the best I’m sure!
ugg boots
accessorize

christian louboutin (Email) (URL) - 21 August '10 - 05:54

excellent article , I added you to my [[http://www.china-wholesale-directory.c..]] Top China Wholesalers category.. thanks for sharing the article!

China Wholesale Directory (Email) (URL) - 25 August '10 - 07:16

very cool article ,thanks for sharing the article!like my cool stuff .very useful.
uCoolStuff is the leading China wholesaler for [[http://www.ucoolstuff.com]] cool stuff , [[http://www.ucoolstuff.com]] cool gifts , unusual gadgets and other unique gift ideas. We provide the very latest cool stuff and cool gifts for you

cool stuff (Email) (URL) - 25 August '10 - 07:16

The town’s poor seem to me often to live the most independent air max classic womens shoes lives of any. May be they are simply great nike air max enough to receive without misgiving. [[http://www.sellnikeairmax.com/]] LIJ

nike air max (URL) - 14 September '10 - 03:18

Top Online Stores is a SEO Friendly [[http://www.toponlinestores.org]] free directory where you can find the best online shopping stores selected by hand and sorted by category.

link directory (Email) (URL) - 14 September '10 - 06:58

Instead, we see people with ugly timberland shoes store problems in whom God is working so they can bloom beautifully — just like I now see in my orchids whether they are “lumps” or blossoms! On a certain timberland work shoes day at a certain hour, we will pull into the station. Bands will be playing and flags waving. Once we get there, so many wonderful dreams will come true and the pieces of our timberland wheat shoes lives will fit together like a completed jigsaw puzzle. How restlessly we pace the aisles, *ing the minutes for timberland for you loitering—waiting, waiting, waiting for the station. It is a good thing that the God Who made us is not impatient with people like me with my timberland shoe company orchid. He lovingly thinks about us all the time timberland boots and he even temporarily gave up His heavenly home and his timberland traditional handsewn hold on diety to die for the sins of timberland mens custom world nearly 2,000 years ago. His death back then covered the sins of every cheap timberland boots person who will ever live. He then rose to life after three mens timberland chukka? days and ever since has wanted us to accept His gracious act by loving Him and also those around us. When we do this, we will see people in a totally different timberland womens premium boots light — one that redefines what it means to be ugly or different or weird or strange. [[http://www.timberland4you.co.uk/]]  YQ

timberland for you (Email) (URL) - 15 September '10 - 03:34

There are various kinds of ghd hair straighteners on hot sell. GHD styler specialized in ghd australia market and ghd outlet. It provides many cheap ghd, ghd iv styler, pink ghd, purple ghd, which save up to 50%, with free shipping!

ghd outlet (Email) (URL) - 25 October '10 - 05:19

A sense of acceptance moves the relationship with full of joy. Women want to be with that man who is affirmative with his career, goals, ambitions and relationship. They prefer these types of men and they are so rare these days.Here was someone immersed in a search for truth and beauty. Words had been cheap timberland boots treasured, words that were beautiful. And I felt as if the words timberland waterproof boots somehow delighted in being discovered, for they were obviously very generous to the as yet anonymous timberland work boots writer of the notes. And now this person was in turn learning the secret of sharing them. Beauty so shines when given away. There were timberland 6 inch sensitivity and a beauty to her that have nothing to do with timberland 6 inch boots looks. She was one to be listened to, whose words were so easy to take to timberland boots sale heart.I used to find notes left in the collection basket, beautiful timberland boots outlet notes about my homilies and about the writer’s black timberland boots thoughts on the daily scriptural readings. The person who penned the notes mens timberland boots would add reflections to my thoughts and would always include some timberland pro boots quotes from poets and mystics he or she had white timberland boots read and remembered and loved. The notes fascinated black timberland shoes uk me.The only truth that exists timberland hiking boots is, in that sense, free. [[http://www.timberland6inch.com/]] CLQ

cheap timberland boots (Email) (URL) - 25 October '10 - 06:02

Personal information





Remember your information?
Comment

Small print: All html tags except <b> and <i> will be removed from your comment. You can make links by just typing the url or mail-address.


^M