First Look at ZFS Deduplication10 Nov '09 - 09:57 by benr
ZFS Deduplication was recently putback (Sun terminology for "commit") to ON (Solaris's primary codebase). That means it should go out at snv_128 (Build 128) due later this week.
Unable to wait for the BFU archives I resorted to actually building the code myself to play; something I've not felt the burning need to do for at least 2 years (I'll blog about that shortly). Here's the initial review...
In typical fashion putting ZFS Dedup to work is a trivial task. Zpools are created in the normal way, the dedup feature is enabled on a per-dataset basis and therefore is a simple matter of turning it on:
root@quadra ~$ zpool create stick c4t0d0 root@quadra ~$ zpool get all stick NAME PROPERTY VALUE SOURCE stick size 3.75G - stick capacity 0% - stick altroot - default stick health ONLINE - stick guid 12142487970365036186 default stick version 21 default stick bootfs - default stick delegation on default stick autoreplace off default stick cachefile - default stick failmode wait default stick listsnapshots off default stick autoexpand off default stick dedupratio 1.00x - stick free 3.75G - stick allocated 76.5K -
Notice that there is no option to enable dedup for the pool, however there is a read-only "dedupratio" key. Because ZFS properties are inherited by child datasets we'll enable dedup on the root dataset, in this case "stick":
root@quadra ~$ zfs set dedup=on stick
Done! That's it. Really, you're done! Stop reading this now. :)
... ok, maybe I'll go into it a bit more.
As with many ZFS Dataset Properties, there can be more than one setting. The default value of the "dedup" properties is "off". It can also be set to "on", "sha256", "verify", or "fletcher4,verify". "on" is simply a pseudonym for "sha256". "verify" is a pseudonym for "sha256,verify" and enables an ability to detect and correct hash collisions, however this is very system intensive and is not recommended for casual use, if you require absolute integrity at all costs, go for it, but test your workload first. Phrases like "hash collision" can cause a panic, but remember that the odds are astronomical. For details on this see Jeff Bonwick's post on ZFS Dedup.
So, now for some testing. I've created my "stick" pool on a new 4GB micro-USB stick and enabled dedup. Lets copy in a bunch of JPEG's to several directories and see what happens:
root@quadra ~$ zfs list stick NAME USED AVAIL REFER MOUNTPOINT stick 73.5K 3.69G 21K /stick root@quadra ~$ mkdir /stick/userA root@quadra ~$ mkdir /stick/userB root@quadra ~$ mkdir /stick/userC root@quadra ~$ cd img root@quadra img$ time cp * /stick/userA real 0m15.395s user 0m0.005s sys 0m0.174s root@quadra img$ time cp * /stick/userB real 0m15.952s user 0m0.004s sys 0m0.112s root@quadra img$ time cp * /stick/userC real 0m2.347s user 0m0.004s sys 0m0.125s root@quadra img$ zfs list stick NAME USED AVAIL REFER MOUNTPOINT stick 203M 3.62G 203M /stick root@quadra img$ cd /stick/userA/ root@quadra userA$ du -sh . 74M .
OK, notice that I'm copying in 74MB of data, 3 times, each to a different directory. (Its slow because its a crappy USB stick.) If we run du it registers the proper size, if we look at zfs list it shows the full size of 203MB. In fact, if I look at the dataset properties I have no indication at all of its on-disk size:
root@quadra userA$ zfs get all stick NAME PROPERTY VALUE SOURCE stick type filesystem - stick creation Tue Nov 10 0:07 2009 - stick used 220M - stick available 3.62G - stick referenced 220M - stick compressratio 1.00x - stick mounted yes - stick quota none default stick reservation none default ...
So here's the magic... look at the pool size:
root@quadra ~$ zpool list stick NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT stick 3.75G 72.5M 3.68G 1% 3.06x ONLINE -
Beautiful. 72.5MB allocated and we correctly see the dedup ratio of 3 (less than the file sizes, leading me to believe there are some duplicate images, which I don't doubt).
Yet again, ZFS makes it "just work". And you don't need a big huge expensive peice of gear, I'm deduping on this:
Suck it Data Domain. :)
For the elite ZFS Internals hackers out there, you can get a closer look at dedup using zdb -S (thanks to Jeff Victor for the tip):
root@quadra ~$ zdb -S stick Simulated DDT histogram: bucket allocated referenced ______ ______________________________ ______________________________ refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE ------ ------ ----- ----- ----- ------ ----- ----- ----- 2 623 69.9M 69.9M 69.9M 1.83K 210M 210M 210M 4 14 1.63M 1.63M 1.63M 84 9.8M 9.8M 9.8M Total 637 71.5M 71.5M 71.5M 1.91K 219M 219M 219M dedup = 3.07, compress = 1.00, copies = 1.00, dedup * compress / copies = 3.07
So now its time to really beat on this thing and see if and where it breaks. Dedup for the masses is coming in the mail!!!
Now, whet happens with zfs send | zfs receive? Does it send deduped blocks only or undeduped as usual? Would be very nice to know.
Thanks for a great posts,
Dmitry Sorokin (Email) - 10 November '09 - 14:27Ben, nice post.
Dmitry, zfs send/receive dedup was recent integrated. See [[http://www.c0t0d0s0.org/archives/6096-..]]
Interesting way of implementing ZFS dedupe in the send/receive streams. I’m not too sure how useful that is however, if I understand the concept correctly.
Let’s take Ben’s example:
1. copy set of files to ZFS data set with dedupe enabled, then take a snapshot A.
2. zfs send -D | zfs receive the set based on snapshot A
3. copy the same set of files to different subdirectory and take snapshot B
4. zfs send -I -D | zfs receive the A to B delta
5. copy the same set of files to yet another different subdirectory and take snapshot C
6. zfs send -I -D | zfs receive the B to C delta
If I understand the current implementation of ZFS dedupe for send/receive streams, every time the snapshot delta is sent, the full size will be sent, as oppose to really sending nothing, as the data was seeded originally in step 2. Doesn’t really help with replication over WAN. Why not keep metadata for every checksummed block and really send over only blocks that don’t exist at the remote end?
This approach is much better and scalable, as oppose to deduplicating only blocks within one stream.
Again, I might misinterpreted the info, but that’s the way I understood it.
Anyone can confirm what’s really happening?
Dmitry Sorokin (Email) - 11 November '09 - 18:09Great post! Hope to be better. Better means more features.
good post,I think so!
Thanks for your information, i have read it, very good！
Bing is a really overlord!! support Bing~~
This is great news. Best of luck for the future and keep up the good work.
zfscurious - 14 December '09 - 23:34Just want to tell you that your blog is like having a whole banana split for dessert – fantastical.
[[http://www.webroyalty.com]] [[http://www.buykamagra.com]] buy kamagra
[[http://www.viagracialis.com]] viagra cialis [[http://www.mobilershop.com]]
[[http://www.beautycasa.com]] [[http://www.salelouboutins.com]] [[http://www.salelouboutins.com]]
of a gym membership,P90x workout . The cost for P90X is
about three months of a paid gym membership but you get to
keep the program foreverP90x . You can try many of the
online sites, but it will be the same as buying from the
company or a Beachbody Coach. Make sure you are getting
original DVD’s. People are selling copies all over. The
problem is how long will they last, P90x workout ,and you
truly need the exercise and nutrition guide to even follow
the program. You can go to any site or you can go to
[[http://www.p90xmall.com]] and click on products. P90x dvd You can
order directly from the site,P90x dvd. [[http://www.edhardyau.com/]] [[http://www.thesuitshoes.com]] [[http://www.offerreplicawatches.com.The..]] sell Chanel Watches, Replica BRM Watches,offer Cartier Watches, Copy Ebel Watches, Replica Hublot Watches, Fake Tag Heuer Watches, wholesale Richard Mille Watches.
They also sell some very fashion watches, such as Montblanc Watches for sale,Chopard Watches, Dior Watches and Gucci Watches. Rolex watches are hot sell. If u are Armani Watches and Breitling Watches fan. You can still look at Omega Watches and Longines Watches, many new arrivals watches just come here.You can have a try on 58338312391273913. :[url=[[http://www.offerreplicawatches.com]www..]]]. [[http://www.uggretailers.org]]
Ugg Boots UK (URL) - 13 May '10 - 01:33Toshiba pa3421u-1brs Battery [[http://www.adapterlist.com/toshiba/pa3..]] [[http://www.globallaptopbattery.co.uk/a..]] [[http://www.mbts-mbtshoes.com/]] [[http://zentaistore.co.uk]] [[http://www.airjordanshoescheap.com]]
susan (Email) - 25 May '10 - 01:53Please trust us. Choose us! spider man costume.We will bring you more exciting zentai unitard and better service.T [[http://www.fivefingersrunning.com]] [[http://www.uggofficial.nl]] [[http://www.uggtall.com]] [[http://www.uggsonline.us]] [[http://www.uggbaileybutton.org]] [[http://www.donedhardy.name]] [[http://www.abercrombieofficial.org]] [[http://www.uggretailers.org]] [[http://www.bedbugsbully.com/]] bed bug bite ] [ [[http://www.plantar-fasciitistreatment...]] plantar fasciitis cure ]
[[http://www.triathlonshoes.org]] [[http://www.uggofficial.de]] [[http://www.iofferitems.com]] [[http://www.sxhgz.com]] : nike shoes [[http://www.mbtshoeslatest.com]]
[[http://www.ugg-boots-london.com/ugg-cl..]] [[http://www.mbt-mbt-shoe.com]] you not only can find various
different styles fashion MBT shoe makeup but also can get the products earlier and faster. We stock everything we sell. No one can make delivery to you quicker than us, much
more than this, we offer you the most reasonable price. Also, our online support is the best. It is our aim to treat our customers with our best service!
cheap nfl jerseys [[http://www.cheapjerseyschina.com]disco..]] nfl jerseys from china[/url]
[url=[[http://www.cheapjerseyschina.com]disco..]] nfl jerseys wholesale[/url]
Christian audiger:http://www.samyibiz.com [[http://www.ugg5498.com/]]
welcome to ugg5498.com !
You’ve found the world’s most comfortable ugg boots. The thicker merino sheepskin used in our ugg boots cools your toes in summer and warms in winter to produce a supremely comfortable 22 degree constant temperature. Unlike other footwear, ugg boots are made from a natural insulator. Constantly circulating air prevents heat exchange keeping you comfortable regardless of the temperature. With thicker sheepskin we offer greater insulation, comfort and strength.
welcome to visit our website,you will have a wonderful business trip~~~!!! [[http://www.ugg4ok.com/]]
As we all known, UGG Australia Boots are legendary. 100% pure Australia lambs wool used in all our foot wear products. But 2010 summer UGG Sandals more pop as well as ugg boots in winter. Our ugg sandals sale provides series of women & men UGG Sandals Collection:UGG Sandals 1759 Gypsy, UGG Sandals Halendi, UGG Tasmina Braid Sandals all you want to be found at our online store.
“High quality, Best service” is our pursuit. Fashion design & High quality UGG Sandals 2010 Sale for customers. Top Services and Latest Fashion Information are from Ugg Sandals Sale Online Store. Buy Ugg Sandals from ugg4ok enjoy Fast shipping! The exclusive Ugg Sandal is worthy of own. [[http://www.ugg4womens.com/]]
A series of fashion UGG boots items diversified latest designs here: UGG Bailey Button boots, UGG Classic Cardy Boot, UGG Classic Crochet Boots and so on. It’s perfect for your breathable comfort. And a great chance for you to buy your favorite UGG boots! The UGG will create curiosity and interest for your colourful life when you experience them yourself. Exquisite design and high quality are very perfect for any occasions. Take it if you like it. There are numerous listings of UGG boots for sale here. May you have a wonderful business trip! [[http://www.china-wholesale-directory.c..]] Top China Wholesalers category.. thanks for sharing the article!
uCoolStuff is the leading China wholesaler for [[http://www.ucoolstuff.com]] cool stuff [[http://www.ucoolstuff.com]] cool gifts , unusual gadgets and other unique gift ideas. We provide the very latest cool stuff and cool gifts for you [[http://www.toponlinestores.org]] free directory where you can find the best online shopping stores selected by hand and sorted by [[http://www.china-wholesale-directory.c..]] china wholesale category. [[http://www.uggsbailey.co.uk/sitemap.ht..]]
marathon training visit: [[http://www.marathontrainingschedule.ne..]]
[ running training program ]
Timberland boots[[http://www.timberlandbootsale.co.uk]] [[http://www.uggsalebootsuk.com/ugg-sale..]]
tory burch shoes sale (Email) - 07 December '10 - 17:57i appreciate your viewpoint.that is great!
tory burch wholesale (Email) - 09 December '10 - 13:12thanks !! very helpful post!
tory burch shoes (Email) - 10 December '10 - 06:32This is a great story. Thanks!
tory burch shoes sale (Email) - 10 December '10 - 15:53As a Newbie, I am always searching online for articles that can help me. Thank you.
tory burch (Email) - 11 December '10 - 08:53Great articles & Nice a site….
tory burch shoes sale (Email) - 11 December '10 - 09:32Great site,A lot of useful information here .GOOD!
tory burch sale (Email) - 11 December '10 - 13:3111zoe
cheap ugg boots[[http://www.cheapuggbootssale.co.uk]]
Timberland boots sale[[http://www.timberlandboots4sale.co.uk]]
tory burch flats (Email) - 13 December '10 - 14:19Thank you for wonderful blog post .
tory burch boots (Email) - 13 December '10 - 14:34ohhh nice info VRy interesting to read it .
tory burch wholesale (Email) - 14 December '10 - 12:17Such a usefule blog…wow !!!!
tory burch flats (Email) - 14 December '10 - 13:27Every people work hard to earn money to live a good life.me too.As a guys of ugg australia.I want to tell people that more about info from inner of Ugg Australia marketing department.the profit of one pair of ugg boots is very high!if you cost 300 USD to buy a Ugg lady boots,in fact,the most of cost is only about 60-100 usd.So what is the best way to get a “ugly boots” in a lowest price? The right answer is get it from a [[http://www.ugg-official-website.com]] ,this is a only one online shop who get right of selling genuine ugg boots from Ugg Australia. But this site is not know by lots of people,becoz it is only face to inner worker of company.Now I come here and share it with every one who want to have a genuine ugg boots in a lower price than buy from shop.
The most cold winter is coming this years.Go and own a Ugg boots to keep our foots warm.Let us have a wonderful time in this winter ! Go for 1 pair now -> [[http://www.ugg-official-website.com]]
tory burch shoes (Email) - 14 December '10 - 16:17Such a usefule blog…wow !!!!
tory burch (Email) - 16 December '10 - 15:59Interesting article. Were did you got all the information from..
tory burch shoes sale (Email) - 16 December '10 - 16:29Great articles & Nice a site….
tory burch sale (Email) - 17 December '10 - 07:45I read a few of your other posts and wanted to know if you would be interested in exchanging blogroll links?
tory burch discount (Email) - 17 December '10 - 08:39Keep working ,great job!
tory burch flats (Email) - 18 December '10 - 07:04Hi there I like your post.
tory burch boots (Email) - 18 December '10 - 07:44your post looks very interesting for me.
tory burch wholesale (Email) - 20 December '10 - 06:22You are a very smart person!
tory burch flats (Email) - 20 December '10 - 07:38That’s Too nice, when it comes in india hope it can make a Rocking place for youngster.. hope that come true.
tory burch sale (Email) - 21 December '10 - 07:51In searching for sites related to web hosting and specifically comparison hosting linux plan web, your site came up.
tory burch shoes sale (Email) - 21 December '10 - 08:36I usually don’t post in Blogs but your blog forced me to, amazing work.. beautiful …
tory burch sale (Email) - 22 December '10 - 07:11Your blog is so informative … keep up the good work!!!!
tory burch wholesale (Email) - 22 December '10 - 08:01I wish getting over a broken heart can be so easy as following a few steps.. but its not…
tory burch shoes sale (Email) - 22 December '10 - 09:21I haven’t much to say on this time but you do great job and appreciate your blog.
tory burch sale (Email) - 23 December '10 - 13:44I read books for a while but now I changed it to great blogs, your place is one of the examples why I began to pay attention to internet resources. Good luck!
tory burch wholesale (Email) - 23 December '10 - 14:27Keep focusing on your blog. I love how we can all express our feelings. This is an extremely nice blog here
tory burch shoes sale (Email) - 23 December '10 - 15:41Hi thanks for yet another funny and interesting post.
tory burch discount (Email) - 23 December '10 - 16:58your traffic increases, keep adding more and more money making programs to your site.
tory burch sale (Email) - 24 December '10 - 14:51I don’t usually reply to posts but I will in this case.
tory burch wholesale (Email) - 24 December '10 - 15:43thanks !! very helpful post
tory burch shoes sale (Email) - 24 December '10 - 16:44Such a usefule blog…wow !!!!
tory burch discount (Email) - 24 December '10 - 17:05The article is so nice that I love it .
tory burch sale (Email) - 27 December '10 - 16:37This was a really fabulous website! I was luck to search for your theme in blog.
tory burch wholesale (Email) - 27 December '10 - 17:22I learn a lot in your topics , thanks .
tory burch shoes sale (Email) - 27 December '10 - 19:03Very good you say, as I thought.
tory burch discount (Email) - 27 December '10 - 19:10Keep working ,great job!
tory burch wholesale (Email) - 02 January '11 - 15:50thanks !! very helpful post! amazing stuff thanx
tory burch shoes sale (Email) - 02 January '11 - 16:17Thanks for your blog ,it is beautiful and useful , I very appreciation.
tory burch wholesale (Email) - 03 January '11 - 15:05The article is best usefully, I love it very much at any time.
tory burch discount (Email) - 03 January '11 - 17:07Thank you for your blog, I benefit a lot from it.
tory burch wholesale (Email) - 05 January '11 - 16:24So good to get more informed on this and again , thanks
tory burch sale (Email) - 05 January '11 - 17:30I agree with totally, please write more.
tory burch shoes sale (Email) - 05 January '11 - 17:44Hi thanks for yet another funny and interesting post. Where do you receive your inspiration for all this?
tory burch wholesale (Email) - 06 January '11 - 13:44So good to get more informed on this and again , thanks
MAC cosmetics sale (Email) - 10 January '11 - 12:54I accede with totally, please address more.
tory burch sale (Email) - 18 January '11 - 16:27I don’t usually reply to posts but I will in this case.
tory burch sale (Email) - 20 January '11 - 16:45Such a usefule blog…wow !!!!
tory burch shoes (Email) - 20 January '11 - 17:14Hello, I found your blog in a new directory of blogs. I dont know how your blog came up, must have been a typo, Your blog looks good. Have a nice day.
tory burch sale (Email) - 09 February '11 - 13:17