On May 9, 2011, at 12:29 PM, Chris Forgeron wrote: > Hello, > > I'm on FreeBSD 9 with ZFS v28, and it's possible this combination is causing > my issue, but I thought I'd start here first and will cross-post to the > FreeBSD ZFS threads if the Solaris crowd thinks this is a FreeBSD problem. > > The issue: From carefully watching my ARC/L2ARC size and activity when I set > primarycache=metadata and secondarycache=all, the secondarycache isn't acting > like "all", it's just metadata. > > I've read a lot on the functionality of the ARC/L2ARC, and I know that L2ARC > is filled by a scanning the ARC for soon to expire objects, and then copying > them to the L2ARC. > > There is my first question: > > Q1 - Doesn't this behavior mean that the L2ARC can never get data objects if > the ARC doesn't hold them?
Yes. > Is setting primary to metadata and secondary to all an impossible request? Today, this is not how it is implemented. However, it is open source :-) The current architecture allows the L2ARC fill to not impact the normal ARC operations. If you implement a policy that says "only use the L2ARC to cache data" then you are implicitly requiring the write bandwidth of the L2ARC to be faster than the read bandwidth of the pool -- not likely to be a winning combination. This isn't a problem for the ARC because it has memory bandwidth, which is, of course, always greater than I/O bandwidth. > I believe the user data still goes through the ARC, it's just not kept when > primarycache=metadata. Correct. > I should back up and explain what I have, and what I'm trying to do. > > I'm running a 24 gig ZFS system with a 20 TB pool, roughly 12 TB full. It's > serving NFS data to an ESX server for my various VM's and it's running great, > albeit a bit slow. I have 4 120 gig SSD's as my L2ARC. > > I'm estimating my DeDeuplicationTable (DDT) needs to be well over 24 gigs, > and with the default primary and secondary cache settings of "all" there is a > lot of churning of both the ARC and L2ARC. I can see it on my L2ARC SSD > drives, as they all fill up within a day, and stay chock full as the server > continues to run 50+ VM's. However, there is so much user data to be > accessed, that it keeps a constant pressure on the metadata, and eventually > the user data flowing in erodes most of the metadata. > > I'm trying to make sure my DDT is as available as possible without putting > more RAM in this server. If I can dedicate my ARC to holding as much of the > DDT as possible, and then use my L2ARC for any DDT overflow, and then lastly > for data, I'd be happy with the performance. I understand there's a > performance hit here over RAM. > > When I set primarycache and secondarycache both to metadata, I find that my > L2ARC drives fill up to around 20 Gig Each, for a total of 80 gigs on my > L2ARC, and it won't go any further than this, no matter how hard I beat on > it. > > However, when I set primarycache=metadata and secondarycache=all, I find the > same behavior as setting both to metadata. My L2ARC's doesn't budge past 20 > Gigs each. > > I'm familiar with arc_meta_used and arc_meta_limit. I've increased my > arc_meta_limit to be 75% of my ARC, as I'm not interested in caching data in > the ARC. I can watch arc_meta_used drop when I set primarycache=all, due to > the pressure of the data objects pushing out the metadata, and setting > primarycache=metadata allows the arc_meta_used to grow back to it's limit. > > Q2 is basically: What's the best way to keep as much of the DDT live in ARC > and L2ARC without buying more RAM? > > Hopefully I'm not dismissed with "Just buy more RAM", which I can see as a > valid point in a lot of situations where people are running on 4 and 8 gig > systems and trying to access 12+ TB of data in a DeDupe situation. :-) > > Keeping my RAM at 24 gigs isn't just a budget request, but it also the max > RAM you can get in most workstation boards these days, and it's moving in the > direction of better energy usage and heat generation, as my 120 Gig SSD's > burn a fraction of the power than another 24 gigs of DDR3 will. Perhaps use the tool designed for the task? -- richard > > Even if I added more RAM, I'd still have this problem. Say I have 196 Gigs of > RAM in this server, and I want to dedicate the ARC just to metadata, and the > L2ARC to user data. From my experiments, this wouldn't work. As I keep > scaling up this server, it will keep eroding the metadata as userdata pours > through the cache system. > > I think the ultimate would be cache priority - Setting metadata to be the > most important, and then user data as secondary importance, so that it never > evicts any metadata from ARC/L2ARC for user data, however I don't think this > is possible today, and I'm unsure if it's on the drawing board. > > Your input is appreciated. > > Thanks. > _______________________________________________ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
