The Blog of Ben Rockwood

Explore Your ZFS Adaptive Replacement Cache (ARC)

29 Oct '08 - 19:07 by benr

Some time ago I wrote a tool which I call arc_summary. It is a kstat based PERL application that runs some calculations and presents you with a pretty report of ZFS ARC. The idea is to help you interpret the data more appropriately. Lets look at some output:

benr@quadra ~$ ./arc_summary.pl 
System Memory:
         Physical RAM:  4083 MB
         Free Memory :  130 MB
         LotsFree:      63 MB

ZFS Tunables (/etc/system):

ARC Size:
         Current Size:             1530 MB (arcsize)
         Target Size (Adaptive):   1555 MB (c)
         Min Size (Hard Limit):    382 MB (zfs_arc_min)
         Max Size (Hard Limit):    3062 MB (zfs_arc_max)

ARC Size Breakdown:
         Most Recently Used Cache Size:          100%   1555 MB (p)
         Most Frequently Used Cache Size:         0%    0 MB (c-p)

ARC Efficency:
         Cache Access Total:             2090556
         Cache Hit Ratio:      71%       1493620        [Defined State for buffer]
         Cache Miss Ratio:     28%       596936         [Undefined State for Buffer]
         REAL Hit Ratio:       70%       1472533        [MRU/MFU Hits Only]

         Data Demand   Efficiency:    98%
         Data Prefetch Efficiency:     1%

        CACHE HITS BY CACHE LIST:
          Anon:                        0%        12854                  [ New Customer, First Cache Hit ]
          Most Recently Used:         55%        825782 (mru)           [ Return Customer ]
          Most Frequently Used:       43%        646751 (mfu)           [ Frequent Customer ]
          Most Recently Used Ghost:    0%        3619 (mru_ghost)       [ Return Customer Evicted, Now Back ]
          Most Frequently Used Ghost:  0%        4614 (mfu_ghost)       [ Frequent Customer Evicted, Now Back ]
        CACHE HITS BY DATA TYPE:
          Demand Data:                60%        900081 
          Prefetch Data:               0%        9547 
          Demand Metadata:            38%        572127 
          Prefetch Metadata:           0%        11865 
        CACHE MISSES BY DATA TYPE:
          Demand Data:                 2%        12318 
          Prefetch Data:              87%        524549 
          Demand Metadata:             9%        55222 
          Prefetch Metadata:           0%        4847 
---------------------------------------------

First notice that you do not need to be root to run the report. I ran this on my personal workstation, an Intel QuadCore with 4GB of RAM.

So looking at the report, first I output some general memory data for you, then output any ZFS tunable you placed in /etc/system (this will not utilize MDB, but may one day).

Then we look at the ARC Sizing data. The "Current Size" is how large the ARC really is, whereas the "Target Size" is a constantly changing number, like a stock price, of what the ARC thinks it should be. I also include the Min and Max hard sizes for ARC, these are tunable.

The ARC Size Breakdown shows the division of the cache between "Most Recently Used" (MRU) and "Most Frequently Used". More on this later. When you see parenthesis in the output those are the variable names, for those using DTrace or getting into the code.

I wrote arc_summary to answer the question "Why is my ARC so damned big!?!" The ARC Efficiency section helps you determine where or not your getting bang for your buck. ZFS ARC stats presented by Kstat provide several breakdowns of cache hits, so this section really runs computation on those values to present them in a sane manner.

In the report above, we have a cache hit ratio of 70%. The "Cache Hit Ratio" and "REAL Hit Ratio" differ in that ZFS considers an Anon buffer as a hit... but its not really, so I remove those for the "REAL" ratio.

We then breakdown Demand vs Prefetch hits. ZFS does aggressively caches data via prefetch; in the example above, only 1% of hits are a result of prefetched data already being in the cache, 98% of it was hit in the cache because it was explicitly asked for. So in this case, pre-fetch isn't helping me.

Perhaps the most interesting information here is the "Cache Hits by Cache List". ARC maintains multiple cache lists, including:

Most Recently Used: Cache hit once.
Most Frequently Used: Cache hit multiple times.
Most Recently Used Ghost: Objects that were in MRU cache, but removed to save space.
Most Frequently Used Ghost: Same but for MFU.

These Ghosts lists are magic. If you get a lot of hits to the ghost lists, it means that ARC is WAY too small and that you desperately need either more RAM or an L2 ARC device (likely, SSD). Please note, if you are considering investing in L2 ARC, check this FIRST.

So by looking at the spread between MRU and MFU hits, we get an idea of how much the cache is rolling over.... 55% of hits are MRU (1 hit), whereas 43% are MFU (multiple hits), so, not bad. The higher MFU is, the better, but this all depends on your workload. Workstations will tend toward MRU, but on one of my servers, for instance, the ARC was 99% MFU.

Looking on we can see what type of requests are hits or misses. Notice that cache hits are 60% demand (explicitly requested) data blocks, and 38% demand metadata.... prefetch isn't helping on this box.

Its all really pretty self explanatory, if its not, let me know and I'll do an ARC expose, but I tried to create the output report to be pretty intuitive.

Please note, this report is cumulative since boot. It should compliment the arcstats tool. arcstat can tell you want is happening, arc_summary can tell you what has been happening. Any serious ZFS deployment should have both of these bad boys around in your toolbag.

On a closing note... I hope this tool helps you appreciate the innovation of the ARC. Designed at IBM (IBM ARC page) and then implemented and improved for ZFS, ARC is elegant, powerful and extremely efficient! I hope my tool can help you better appreciate this amazing innovation in some small way.

- - C O M M E N T S - -
hello, thanks authors. exellent post

cialis (Email) (URL) - 30 October '08 - 09:36

Thanks for the post, again. I love your descriptive posts on the neat features of Solaris. It reminds me why it’s a great operating system at times.

When using your arc_summary.pl script (just downloaded), I get “Attempt to access disallowed key ‘zfs’ in a restricted hash at arc_summary.pl line 41.” when it first tries to do ${Kstat}->{zfs}. I don’t have time to investigate it now, but I thought I’d give you the heads up that the hash isn’t always accessible. The test system is SunOS 5.10 Generic_118855-33

Drew (Email) - 30 October '08 - 17:29

Drew: I do everything on Nevada… I several S10 releases likely do not include some or all the ARC kstats. I’ll add a check to the next release to gracefully catch it.

Thanks

benr - 30 October '08 - 17:41

good site. Continue also

fuepamacrague (Email) (URL) - 31 October '08 - 09:58

Ben,

While arc_summary does not work with 6/06, it does work with 8/07. Nice work!

Robert Escue (Email) - 31 October '08 - 12:24

The good convenient site is made.
hp laptop

poibiapoulk (Email) (URL) - 01 November '08 - 09:36

The good convenient site is made.
hp laptop

poibiapoulk (Email) - 01 November '08 - 09:36

The good convenient site is made.
hp laptop

poibiapoulk (Email) - 01 November '08 - 09:36

Good site, comfortable navigation.
used laptop sale

Juillafluch (Email) - 01 November '08 - 16:24

good site. it was very interestingly to wande
digital music laptop

abserbali (Email) (URL) - 01 November '08 - 18:13

Hello! Very good site and stylish design
dollhouse and miniatures supplies

amingontien (Email) - 02 November '08 - 21:46

Hello, excellent site, very rich in content and correctly carefully thought out
pizza

duddifsCruich (Email) - 03 November '08 - 21:47

Hello! Very good site and stylish design

AREKPOUPRINIA (Email) (URL) - 04 November '08 - 03:50

Thank you for such good site. It is sorry that before him did not find

exosteDed (Email) (URL) - 05 November '08 - 02:51

What is a “Cache Miss: Prefetch Data”? I imaine if ARC is asked “give me Data or give me Metadata” and that is not in a cache, its a Miss. Is somebody else above ARC prefetching and telling that to ARC?

Bernd Eckenfels (Email) (URL) - 05 November '08 - 04:48

good site. Continue also

loummauploabe (Email) (URL) - 05 November '08 - 19:45

good site. Continue also

loummauploabe (Email) - 05 November '08 - 19:45

good site. Continue also

loummauploabe (Email) - 05 November '08 - 19:45

Hello! good site, thank you!

rispabomism (Email) (URL) - 08 November '08 - 01:12

Hello! Very good site! All is done neatly, beautifully

couniombich (Email) (URL) - 08 November '08 - 07:28

Hello! Very good site! All is done neatly, beautifully

couniombich (Email) - 08 November '08 - 07:28

Hello! Very good site! All is done neatly, beautifully

couniombich (Email) - 08 November '08 - 07:28

Hello! Very good site! All is done neatly, beautifully

couniombich (Email) - 08 November '08 - 07:28

Hello, excellent site, very rich in content and correctly carefully thought out

DeebraliarMob (Email) (URL) - 08 November '08 - 23:54

god site. Continue also

inceselve (Email) (URL) - 09 November '08 - 03:03

god site. Continue also

inceselve (Email) - 09 November '08 - 03:03

god site. Continue also

inceselve (Email) - 09 November '08 - 03:03

In actual fact very good site…successes are in advancement

brealsfreemia (Email) (URL) - 09 November '08 - 06:25

The good convenient site is made.

glaceague (Email) (URL) - 09 November '08 - 18:53

Hello! Good site, much useful

acinuicainc (Email) (URL) - 10 November '08 - 01:49

Hello! Good site, much useful

acinuicainc (Email) - 10 November '08 - 01:49

Hello! Good site, much useful

acinuicainc (Email) - 10 November '08 - 01:49

Hello! Good site, much useful

acinuicainc (Email) - 10 November '08 - 01:49

Hello! good site, thank you!

Urgettearracy (Email) (URL) - 10 November '08 - 04:56

Hello! good site, thank you!

BiareHience (Email) (URL) - 10 November '08 - 08:11

Hello! good site, thank you!

BiareHience (Email) - 10 November '08 - 08:11

Hello I am a soviet that I called on this site I very like here

Incuclipurl (Email) (URL) - 12 November '08 - 03:23

Hello I am a soviet that I called on this site I very like here

Incuclipurl (Email) - 12 November '08 - 03:23

Hello I am a soviet that I called on this site I very like here

Incuclipurl (Email) - 12 November '08 - 03:23

Hello, excellent site, very rich in content and correctly carefully thought out

Irrinilaw (Email) (URL) - 12 November '08 - 07:11

Hello, excellent site, very rich in content and correctly carefully thought out

Irrinilaw (Email) (URL) - 12 November '08 - 07:11

Hello, excellent site, very rich in content and correctly carefully thought out

Irrinilaw (Email) - 12 November '08 - 07:12

Thank you for such good site. It is sorry that before him did not find

aspehealf (Email) (URL) - 12 November '08 - 22:20

Hallo everybody. By chance barged into this site and understood that finally got there, where it is nece
ssary

beriengariE (Email) (URL) - 13 November '08 - 03:47

excellent site, very rich in content and correctly carefully thought out ..

Chris (Email) (URL) - 10 May '09 - 07:13

I am Very thank full the owner of this blog. Becouse of this blog is very imformative for me.. And I ask u some thiing You make more this type blog where we can get more knowledge. and any one tell me how can I find this type blog. [[http://www.penisenlargementz.com]]

Penis Enlargement (Email) (URL) - 27 May '09 - 06:01

Penis size always concern to men’s every where in every culture. In most cultures a large penis is considered the sign of virility, stamina and power. An average penis size is 5.2-6 inches. In this age of modern science various penis enlargement techniques offer penis enlargement for men. Natural Penis Enlargement

Penis Enlargement (Email) (URL) - 27 July '09 - 07:50

It was a very nice idea! Just wanna say thank you for the information you have shared. Just continue writing this kind of post. I will be your loyal reader. Thanks again.

Christian louboutin shoes (Email) (URL) - 27 October '09 - 00:46

Thank you very much!

cheap links of london (Email) (URL) - 07 November '09 - 00:56

haha , thanks all the time, These Christian louboutin shoes was one of fashion’s best-kept secrets:means Christian louboutin has attracted a growing clientele for whom the fact that he is not a household name is all part of the appeal.

christian louboutin (Email) (URL) - 15 November '09 - 07:35

Great post! Hope to be better. Better means more features.
good post,I think so!
Thanks for your information, i have read it, very good！
Bing is a really overlord!! support Bing~~
This is great news. Best of luck for the future and keep up the good work.

links of london (Email) (URL) - 17 November '09 - 03:25

Just one question: how to add your blog into my rrs reader, thanks so much.

christian louboutin (Email) (URL) - 03 January '10 - 07:03

This may be a perfect example of information asymmetry and adverse selection in insurance. I hope all textbook authors and legislators notice.

christian louboutin shoes (Email) (URL) - 06 January '10 - 08:50

Keep up the good work, Your article is really great and I truly enjoyed reading it. Waiting for some more great articles like this from you in the coming days.

christian louboutin (Email) (URL) - 08 January '10 - 09:03

links of london charm links of london

links of london (Email) (URL) - 12 January '10 - 05:35

Effective and fast penis enlargement is entirely possible with the right techniques used consistently and correctly, and you can even see real results within two to three weeks! I used specific yet very simple and easy-to-duplicate techniques to increase

Home Improvement Resources (Email) (URL) - 13 January '10 - 07:03

Good post! Thanks you for your information! China Wholesale Wholesale China Wholesalers Wholesale Game Accessories Wholesale Iphone Accessories Video Game Accessories Wholesale Wholesale Wii Accessories Wholesale Xbox 360 Accessories Wholesale Xbox 360 Games Wholesale Video Games Cheap Video Games Cheap Ps3 Games Cheap Xbox 360 Games Wholesale Computers Wholesale Laptop Computers Wholesale Laptops Discount Computers Cheap Computers Wholesale Iphones Wholesale Iphone Wholesale Iphones 3g Hiphones Wholesale Hiphone Wholesale Hiphones Wholesale Nokia Wholesale Nokia 8800 Wholesale Nokia n97 wholesale blackberry wholesale blackberry phones wholesale blackberry 9700 wholesale blackberry 9600 wholesale blackberry 9500

raging bull (Email) (URL) - 15 January '10 - 06:42

Cheap Video Games Cheap Ps3 Games Cheap Xbox 360 Games Wholesale Computers Wholesale Laptop Computers Wholesale Laptops Discount

Weight Loss Reviews (Email) (URL) - 20 January '10 - 08:58

thanks

kamagra (Email) (URL) - 21 January '10 - 01:26

good read thanks

M65 Jacket (Email) (URL) - 21 January '10 - 01:26

many thanks

viagra cialis (Email) (URL) - 21 January '10 - 01:55

I will make sure and bookmark this page, I will come back to follow you more.

hickory nc furniture (Email) (URL) - 22 January '10 - 21:24

New battery review module surface is black, not because of the positive electrode exists, the module is likely to be integrated back contact silicon solar cells, silicon, where the role is often to extend the service life of equipment.Mitsubishi motors in October 2009 28-30 at the yokohama exhibition center held the Pacific “GreenDevice2009″ displayed on the high efficiency solar camera battery 19.1% respectively. In the square 15cm in polysilicon solar cells, realized the highest efficiency. The research achievements in 2009 September 2009 and the application of physics to the international society in October EUPVSEC “, “said in a statement.

battery review (Email) (URL) - 30 January '10 - 03:42

Good post! Thanks for your information! As Seen On TV

As Seen On TV (Email) (URL) - 30 January '10 - 06:04