> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
> boun...@opensolaris.org] On Behalf Of Edward Ned Harvey
>
> New problem:
>
> I'm following all the advice I summarized into the OP of this thread, and
> testing on a test system. (A laptop). And it's just not working. I am
> ju
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
> boun...@opensolaris.org] On Behalf Of Edward Ned Harvey
>
> New problem:
>
> I'm following all the advice I summarized into the OP of this thread, and
> testing on a test system. (A laptop). And it's just not working. I am
> ju
Op 10-05-11 06:56, Edward Ned Harvey schreef:
>> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
>> boun...@opensolaris.org] On Behalf Of Edward Ned Harvey
>>
>> BTW, here's how to tune it:
>>
>> echo "arc_meta_limit/Z 0x3000" | sudo mdb -kw
>>
>> echo "::arc" | sudo mdb -k | gre
Op 09-05-11 15:42, Edward Ned Harvey schreef:
>> > in my previous
>> > post my arc_meta_used was bigger than my arc_meta_limit (by about 50%)
> I have the same thing. But as I sit here and run more and more extensive
> tests on it ... it seems like arc_meta_limit is sort of a soft limit. Or it
>
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
> boun...@opensolaris.org] On Behalf Of Edward Ned Harvey
>
> BTW, here's how to tune it:
>
> echo "arc_meta_limit/Z 0x3000" | sudo mdb -kw
>
> echo "::arc" | sudo mdb -k | grep meta_limit
> arc_meta_limit= 76
Op 09-05-11 15:42, Edward Ned Harvey schreef:
>> > in my previous
>> > post my arc_meta_used was bigger than my arc_meta_limit (by about 50%)
> I have the same thing. But as I sit here and run more and more extensive
> tests on it ... it seems like arc_meta_limit is sort of a soft limit. Or it
>
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
> boun...@opensolaris.org] On Behalf Of Frank Van Damme
>
> in my previous
> post my arc_meta_used was bigger than my arc_meta_limit (by about 50%)
I have the same thing. But as I sit here and run more and more extensive
tests on i
Op 09-05-11 14:36, Edward Ned Harvey schreef:
>> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
>> boun...@opensolaris.org] On Behalf Of Edward Ned Harvey
>>
>> So now I'll change meta_max and
>> see if it helps...
>
> Oh, know what? Nevermind.
> I just looked at the source, and i
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
> boun...@opensolaris.org] On Behalf Of Edward Ned Harvey
>
> So now I'll change meta_max and
> see if it helps...
Oh, know what? Nevermind.
I just looked at the source, and it seems arc_meta_max is just a gauge for
you to use, so
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
> boun...@opensolaris.org] On Behalf Of Frank Van Damme
>
> Otoh, I struggle to see the difference between arc_meta_limit and
> arc_meta_max.
Thanks for pointing this out. When I changed meta_limit and re-ran the
test, there was no
Op 08-05-11 17:20, Edward Ned Harvey schreef:
>> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
>> boun...@opensolaris.org] On Behalf Of Edward Ned Harvey
>>
>> But I'll go tune and test with this knowledge, just to be sure.
>
> BTW, here's how to tune it:
>
> echo "arc_meta_limit
Just another data point. The ddt is considered metadata, and by default the
arc will not allow more than 1/4 of it to be used for metadata. Are you still
sure it fits?
Erik Trimble wrote:
>On 5/7/2011 6:47 AM, Edward Ned Harvey wrote:
>>> See below. Right around 400,000 blocks, dedup is su
On May 8, 2011, at 7:56 AM, Edward Ned Harvey wrote:
>> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
>> boun...@opensolaris.org] On Behalf Of Edward Ned Harvey
>>
>> That could certainly start to explain why my
>> arc size arcstats:c never grew to any size I thought seemed reaso
On 05/08/11 09:22, Andrew Gabriel wrote:
Toby Thain wrote:
On 08/05/11 10:31 AM, Edward Ned Harvey wrote:
...
Incidentally, does fsync() and sync return instantly or wait? Cuz
"time
sync" might product 0 sec every time even if there were something
waiting to
be flushed to disk.
The
Toby Thain wrote:
On 08/05/11 10:31 AM, Edward Ned Harvey wrote:
...
Incidentally, does fsync() and sync return instantly or wait? Cuz "time
sync" might product 0 sec every time even if there were something waiting to
be flushed to disk.
The semantics need to be synchronous. Anything
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
> boun...@opensolaris.org] On Behalf Of Edward Ned Harvey
>
> But I'll go tune and test with this knowledge, just to be sure.
BTW, here's how to tune it:
echo "arc_meta_limit/Z 0x3000" | sudo mdb -kw
echo "::arc" | sudo mdb -k
> From: Garrett D'Amore [mailto:garr...@nexenta.com]
>
> It is tunable, I don't remember the exact tunable name...
Arc_metadata_limit
> or some such.
There it is:
echo "::arc" | sudo mdb -k | grep meta_limit
arc_meta_limit= 286 MB
Looking at my chart earlier in this discussion,
It is tunable, I don't remember the exact tunable name... Arc_metadata_limit or
some such.
-- Garrett D'Amore
On May 8, 2011, at 7:37 AM, "Edward Ned Harvey"
wrote:
>> From: Garrett D'Amore [mailto:garr...@nexenta.com]
>>
>> Just another data point. The ddt is considered metadata, and by
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
> boun...@opensolaris.org] On Behalf Of Edward Ned Harvey
>
> That could certainly start to explain why my
> arc size arcstats:c never grew to any size I thought seemed reasonable...
Also now that I'm looking closer at arcstats, it
On 08/05/11 10:31 AM, Edward Ned Harvey wrote:
>...
> Incidentally, does fsync() and sync return instantly or wait? Cuz "time
> sync" might product 0 sec every time even if there were something waiting to
> be flushed to disk.
The semantics need to be synchronous. Anything else would be a horribl
On 06/05/11 9:17 PM, Erik Trimble wrote:
> On 5/6/2011 5:46 PM, Richard Elling wrote:
>> ...
>> Yes, perhaps a bit longer for recursive destruction, but everyone here
>> knows recursion is evil, right? :-)
>> -- richard
> You, my friend, have obviously never worshipped at the Temple of the
> Lam
> From: Garrett D'Amore [mailto:garr...@nexenta.com]
>
> Just another data point. The ddt is considered metadata, and by default the
> arc will not allow more than 1/4 of it to be used for metadata. Are you
> still
> sure it fits?
That's interesting. Is it tunable? That could certainly star
> From: Erik Trimble [mailto:erik.trim...@oracle.com]
>
> (1) I'm assuming you run your script repeatedly in the same pool,
> without deleting the pool. If that is the case, that means that a run of
> X+1 should dedup completely with the run of X. E.g. a run with 12
> blocks will dedup the fi
On 5/7/2011 6:47 AM, Edward Ned Harvey wrote:
See below. Right around 400,000 blocks, dedup is suddenly an order of
magnitude slower than without dedup.
40 10.7sec 136.7sec143 MB 195
MB
80 21.0sec 465.6sec287 MB 391
> See below. Right around 400,000 blocks, dedup is suddenly an order of
> magnitude slower than without dedup.
>
> 4010.7sec 136.7sec143 MB 195
MB
> 8021.0sec 465.6sec287 MB 391
MB
The interesting thing is
New problem:
I'm following all the advice I summarized into the OP of this thread, and
testing on a test system. (A laptop). And it's just not working. I am
jumping into the dedup performance abyss far, far eariler than predicted...
My test system is a laptop with 1.5G ram, c_min =150M, c_max
On 5/6/2011 5:46 PM, Richard Elling wrote:
On May 6, 2011, at 3:24 AM, Erik Trimble wrote:
Casper and Richard are correct - RAM starvation seriously impacts snapshot or
dataset deletion when a pool has dedup enabled. The reason behind this is that
ZFS needs to scan the entire DDT to check t
On May 6, 2011, at 3:24 AM, Erik Trimble wrote:
> On 5/6/2011 1:37 AM, casper@oracle.com wrote:
>>> Op 06-05-11 05:44, Richard Elling schreef:
As the size of the data grows, the need to have the whole DDT in RAM or
L2ARC
decreases. With one notable exception, destroying a data
One of the quoted participants is Richard Elling, the other is Edward Ned
Harvey, but my quoting was screwed up enough that I don't know which is which.
Apologies.
>> >zdb -DD poolname
>> This just gives you the -S output, and the -D output all in one go. So I
>Sorry, zdb -DD only works f
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
> boun...@opensolaris.org] On Behalf Of Edward Ned Harvey
>
> > zdb -DD poolname
> This just gives you the -S output, and the -D output all in one go. So I
Sorry, zdb -DD only works for pools that are already dedup'd.
If you wa
> From: Richard Elling [mailto:richard.ell...@gmail.com]
>
> > --- To calculate size of DDT ---
> zdb -S poolname
Look at total blocks allocated. It is rounded, and uses a suffix like "K,
M, G" but it's in decimal (powers of 10) notation, so you have to remember
that... So
On 06 May, 2011 - Erik Trimble sent me these 1,8K bytes:
> If dedup isn't enabled, snapshot and data deletion is very light on RAM
> requirements, and generally won't need to do much (if any) disk I/O.
> Such deletion should take milliseconds to a minute or so.
.. or hours. We've had problem
On 5/6/2011 1:37 AM, casper@oracle.com wrote:
Op 06-05-11 05:44, Richard Elling schreef:
As the size of the data grows, the need to have the whole DDT in RAM or L2ARC
decreases. With one notable exception, destroying a dataset or snapshot requires
the DDT entries for the destroyed blocks to
>Op 06-05-11 05:44, Richard Elling schreef:
>> As the size of the data grows, the need to have the whole DDT in RAM or L2ARC
>> decreases. With one notable exception, destroying a dataset or snapshot
>> requires
>> the DDT entries for the destroyed blocks to be updated. This is why people
>> can
Op 06-05-11 05:44, Richard Elling schreef:
> As the size of the data grows, the need to have the whole DDT in RAM or L2ARC
> decreases. With one notable exception, destroying a dataset or snapshot
> requires
> the DDT entries for the destroyed blocks to be updated. This is why people can
> go for
On May 4, 2011, at 7:56 PM, Edward Ned Harvey wrote:
> This is a summary of a much longer discussion "Dedup and L2ARC memory
> requirements (again)"
> Sorry even this summary is long. But the results vary enormously based on
> individual usage, so any "rule of thumb" metric that has been bouncing
> From: Karl Wagner [mailto:k...@mouse-hole.com]
>
> so there's an ARC entry referencing each individual DDT entry in the L2ARC?!
> I had made the assumption that DDT entries would be grouped into at least
> minimum block sized groups (8k?), which would have lead to a much more
> reasonable ARC re
so there's an ARC entry referencing each individual DDT entry in the L2ARC?! I
had made the assumption that DDT entries would be grouped into at least minimum
block sized groups (8k?), which would have lead to a much more reasonable ARC
requirement.
seems like a bad design to me, which leads to
> From: Erik Trimble [mailto:erik.trim...@oracle.com]
>
> Using the standard c_max value of 80%, remember that this is 80% of the
> TOTAL system RAM, including that RAM normally dedicated to other
> purposes. So long as the total amount of RAM you expect to dedicate to
> ARC usage (for all ZFS us
Good summary, Ned. A couple of minor corrections.
On 5/4/2011 7:56 PM, Edward Ned Harvey wrote:
This is a summary of a much longer discussion "Dedup and L2ARC memory
requirements (again)"
Sorry even this summary is long. But the results vary enormously based on
individual usage, so any "rule o
40 matches
Mail list logo
