Re: [zfs-discuss] L2ARC & Workingset Size

[Richard Elling]

On Apr 8, 2010, at 3:23 PM, Tomas Ögren wrote:
> On 08 April, 2010 - Abdullah Al-Dahlawi sent me these 12K bytes:
> 
>> Hi Richard
>> 
>> Thanks for your comments. OK ZFS is COW, I understand, but, this also means
>> a waste of valuable space of my L2ARC SSD device, more than 60% of the space
>> is consumed by COW !!!. I do not get it ?
> 
> The rest can and will be used if L2ARC needs it. It's not wasted, it's
> just a number that doesn't match what you think it should be.

Another way to look at it is: all cache space is "wasted" by design.  If the 
backing 
store for the cache were performant, there wouldn't be a cache.  So caches waste
space to gain performance. Space, dependability, performance: pick two.
 -- richard

ZFS storage and performance consulting at http://www.RichardElling.com
ZFS training on deduplication, NexentaStor, and NAS performance
Las Vegas, April 29-30, 2010 http://nexenta-vegas.eventbrite.com 





___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] L2ARC & Workingset Size

[Tomas Ögren]

On 08 April, 2010 - Abdullah Al-Dahlawi sent me these 12K bytes:

> Hi Richard
> 
> Thanks for your comments. OK ZFS is COW, I understand, but, this also means
> a waste of valuable space of my L2ARC SSD device, more than 60% of the space
> is consumed by COW !!!. I do not get it ?

The rest can and will be used if L2ARC needs it. It's not wasted, it's
just a number that doesn't match what you think it should be.

/Tomas
-- 
Tomas Ögren, st...@acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Umeå
`- Sysadmin at {cs,acc}.umu.se
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] L2ARC & Workingset Size

[Abdullah Al-Dahlawi]

Hi Richard

Thanks for your comments. OK ZFS is COW, I understand, but, this also means
a waste of valuable space of my L2ARC SSD device, more than 60% of the space
is consumed by COW !!!. I do not get it ?

On Sat, Apr 3, 2010 at 11:35 PM, Richard Elling wrote:

> On Apr 1, 2010, at 9:41 PM, Abdullah Al-Dahlawi wrote:
>
> > Hi all
> >
> > I ran a workload that reads & writes within 10 files each file is 256M,
> ie,  (10 * 256M = 2.5GB total Dataset Size).
> >
> > I have set the ARC max size to 1 GB  on  etc/system file
> >
> > In the worse case, let us assume that the whole dataset is hot, meaning
> my workingset size= 2.5GB
> >
> > My SSD flash size = 8GB and being used for L2ARC
> >
> > No slog is used in the pool
> >
> > My File system record size = 8K , meaning 2.5% of 8GB is used for L2ARC
> Directory in ARC. which ultimately mean that available ARC is 1024M - 204.8M
> = 819.2M Available ARC  (Am I Right ?)
>
> this is worst case
>
> > Now the Question ...
> >
> > After running the workload for 75 minutes, I have noticed that L2ARC
> device has grown to 6 GB !!!   
>
> You're not interpreting the values properly, see below.
>
> > What is in L2ARC beyond my 2.5GB Workingset ?? something else is has been
> added to L2ARC 
>
> ZFS is COW, so modified data is written to disk and the L2ARC.
>
> > Here is a 5 minute interval of Zpool iostat
>
> [snip]
> > Also, a FULL  Kstat ZFS for 5 minutes Interval
>
> [snip]
> > module: zfs instance: 0
> > name:   arcstatsclass:misc
> > c   1073741824
> > c_max   1073741824
>
>  Max ARC size is limited to 1GB
>
> > c_min   134217728
> > crtime  28.083178473
> > data_size   955407360
> > deleted 966956
> > demand_data_hits843880
> > demand_data_misses  452182
> > demand_metadata_hits68572
> > demand_metadata_misses  5737
> > evict_skip  82548
> > hash_chain_max  18
> > hash_chains 61732
> > hash_collisions 1444874
> > hash_elements   329553
> > hash_elements_max   329561
> > hdr_size46553328
> > hits978241
> > l2_abort_lowmem 0
> > l2_cksum_bad0
> > l2_evict_lock_retry 0
> > l2_evict_reading0
> > l2_feeds4738
> > l2_free_on_write184
> > l2_hdr_size 17024784
>
> size of L2ARC headers is approximately 17MB
>
> > l2_hits 252839
> > l2_io_error 0
> > l2_misses   203767
> > l2_read_bytes   2071482368
> > l2_rw_clash 13
> > l2_size 2632226304
>
> currently, there is approximately 2.5GB in the L2ARC
>
> > l2_write_bytes  6486009344
>
> total amount of data written to L2ARC since boot is 6+ GB
>
> > l2_writes_done  4127
> > l2_writes_error 0
> > l2_writes_hdr_miss  21
> > l2_writes_sent  4127
> > memory_throttle_count   0
> > mfu_ghost_hits  120524
> > mfu_hits500516
> > misses  468227
> > mru_ghost_hits  61398
> > mru_hits412112
> > mutex_miss  511
> > other_size  56325712
> > p   775528448
> > prefetch_data_hits  50804
> > prefetch_data_misses7819
> > prefetch_metadata_hits  14985
> > prefetch_metadata_misses2489
> > recycle_miss13096
> > size1073830768
>
> ARC size is 1GB
>
> The best way to understand these in detail is to look at the source which
> is nicely commented. L2ARC design is commented near
>
> http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/fs/zfs/arc.c#3590
>
>  -- richard
>
> ZFS storage and performance consulting at 
> http://www.RichardElling.com
> ZFS training on deduplication, NexentaStor, and NAS performance
> Las Vegas, April 29-30, 2010 http://nexenta-vegas.eventbrite.com
>
>
>
>
>
>


-- 
Abdullah Al-Dahlawi
PhD Candidate
George Washington University
Department. Of Electrical & Computer Engineering

Check The Fastest 500 Super Computers Worldwide
http://www.top500.org/list/2009/11/100
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.or

Re: [zfs-discuss] L2ARC & Workingset Size

[Richard Elling]

On Apr 1, 2010, at 9:41 PM, Abdullah Al-Dahlawi wrote:

> Hi all
> 
> I ran a workload that reads & writes within 10 files each file is 256M, ie,  
> (10 * 256M = 2.5GB total Dataset Size).
> 
> I have set the ARC max size to 1 GB  on  etc/system file
> 
> In the worse case, let us assume that the whole dataset is hot, meaning my 
> workingset size= 2.5GB
> 
> My SSD flash size = 8GB and being used for L2ARC
> 
> No slog is used in the pool
> 
> My File system record size = 8K , meaning 2.5% of 8GB is used for L2ARC 
> Directory in ARC. which ultimately mean that available ARC is 1024M - 204.8M 
> = 819.2M Available ARC  (Am I Right ?)

this is worst case

> Now the Question ...
> 
> After running the workload for 75 minutes, I have noticed that L2ARC device 
> has grown to 6 GB !!!   

You're not interpreting the values properly, see below.

> What is in L2ARC beyond my 2.5GB Workingset ?? something else is has been 
> added to L2ARC 

ZFS is COW, so modified data is written to disk and the L2ARC.

> Here is a 5 minute interval of Zpool iostat 

[snip]
> Also, a FULL  Kstat ZFS for 5 minutes Interval

[snip]
> module: zfs instance: 0 
> name:   arcstatsclass:misc
> c   1073741824
> c_max   1073741824

Max ARC size is limited to 1GB

> c_min   134217728
> crtime  28.083178473
> data_size   955407360
> deleted 966956
> demand_data_hits843880
> demand_data_misses  452182
> demand_metadata_hits68572
> demand_metadata_misses  5737
> evict_skip  82548
> hash_chain_max  18
> hash_chains 61732
> hash_collisions 1444874
> hash_elements   329553
> hash_elements_max   329561
> hdr_size46553328
> hits978241
> l2_abort_lowmem 0
> l2_cksum_bad0
> l2_evict_lock_retry 0
> l2_evict_reading0
> l2_feeds4738
> l2_free_on_write184
> l2_hdr_size 17024784

size of L2ARC headers is approximately 17MB

> l2_hits 252839
> l2_io_error 0
> l2_misses   203767
> l2_read_bytes   2071482368
> l2_rw_clash 13
> l2_size 2632226304

currently, there is approximately 2.5GB in the L2ARC

> l2_write_bytes  6486009344

total amount of data written to L2ARC since boot is 6+ GB

> l2_writes_done  4127
> l2_writes_error 0
> l2_writes_hdr_miss  21
> l2_writes_sent  4127
> memory_throttle_count   0
> mfu_ghost_hits  120524
> mfu_hits500516
> misses  468227
> mru_ghost_hits  61398
> mru_hits412112
> mutex_miss  511
> other_size  56325712
> p   775528448
> prefetch_data_hits  50804
> prefetch_data_misses7819
> prefetch_metadata_hits  14985
> prefetch_metadata_misses2489
> recycle_miss13096
> size1073830768

ARC size is 1GB

The best way to understand these in detail is to look at the source which 
is nicely commented. L2ARC design is commented near
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/fs/zfs/arc.c#3590

 -- richard

ZFS storage and performance consulting at http://www.RichardElling.com
ZFS training on deduplication, NexentaStor, and NAS performance
Las Vegas, April 29-30, 2010 http://nexenta-vegas.eventbrite.com 





___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] L2ARC & Workingset Size

[Abdullah Al-Dahlawi]

Hi Tomas

Thanks for the clarification. If I understood you right ,  you mean that 6
GB (including my 2.5GB files)  has been written to the device and still
occupy space on the device !!!

This is fair enough for this case since most of my files ended up in L2ARC
 Great ...

But this brings two related questions

1. What are really in L2ARC ... is it my old workingset data files that have
been updated but still in L2ARC ? or something else ? Metadata ?

2. More importantly, what if my workingset was larger that 2.5GB (Say 5GB),
I guess my L2ARC device will be filled completely before all my workingset
transfer to the L2ARC device !!!

Thanks 

On Sat, Apr 3, 2010 at 4:31 PM, Tomas Ögren  wrote:

> On 02 April, 2010 - Abdullah Al-Dahlawi sent me these 128K bytes:
>
> > Hi all
> >
> > I ran a workload that reads & writes within 10 files each file is 256M,
> ie,
> > (10 * 256M = 2.5GB total Dataset Size).
> >
> > I have set the ARC max size to 1 GB  on  etc/system file
> >
> > In the worse case, let us assume that the whole dataset is hot, meaning
> my
> > workingset size= 2.5GB
> >
> > My SSD flash size = 8GB and being used for L2ARC
> >
> > No slog is used in the pool
> >
> > My File system record size = 8K , meaning 2.5% of 8GB is used for L2ARC
> > Directory in ARC. which ultimately mean that available ARC is 1024M -
> 204.8M
> > = 819.2M Available ARC  (Am I Right ?)
>
> Seems about right.
>
> > Now the Question ...
> >
> > After running the workload for 75 minutes, I have noticed that L2ARC
> device
> > has grown to 6 GB !!!   
>
> No, 6GB of the area has been touched by Copy on Write, not all of it is
> in use anymore though.
>
> > What is in L2ARC beyond my 2.5GB Workingset ?? something else is has been
> > added to L2ARC 
>
> [ snip lots of data ]
>
> This is your last one:
>
> > module: zfs instance: 0
> > name:   arcstatsclass:misc
> > c   1073741824
> > c_max   1073741824
> > c_min   134217728
> [...]
> > l2_size 2632226304
> > l2_write_bytes  6486009344
>
> Roughly 6GB has been written to the device, and slightly less than 2.5GB
> is actually in use.
>
> > p   775528448
>
> /Tomas
> --
> Tomas Ögren, st...@acc.umu.se, 
> http://www.acc.umu.se/~stric/
> |- Student at Computing Science, University of Umeå
> `- Sysadmin at {cs,acc}.umu.se
>



-- 
Abdullah Al-Dahlawi
PhD Candidate
George Washington University
Department. Of Electrical & Computer Engineering

Check The Fastest 500 Super Computers Worldwide
http://www.top500.org/list/2009/11/100
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] L2ARC & Workingset Size

[Tomas Ögren]

On 02 April, 2010 - Abdullah Al-Dahlawi sent me these 128K bytes:

> Hi all
> 
> I ran a workload that reads & writes within 10 files each file is 256M, ie,
> (10 * 256M = 2.5GB total Dataset Size).
> 
> I have set the ARC max size to 1 GB  on  etc/system file
> 
> In the worse case, let us assume that the whole dataset is hot, meaning my
> workingset size= 2.5GB
> 
> My SSD flash size = 8GB and being used for L2ARC
> 
> No slog is used in the pool
> 
> My File system record size = 8K , meaning 2.5% of 8GB is used for L2ARC
> Directory in ARC. which ultimately mean that available ARC is 1024M - 204.8M
> = 819.2M Available ARC  (Am I Right ?)

Seems about right.

> Now the Question ...
> 
> After running the workload for 75 minutes, I have noticed that L2ARC device
> has grown to 6 GB !!!   

No, 6GB of the area has been touched by Copy on Write, not all of it is
in use anymore though.

> What is in L2ARC beyond my 2.5GB Workingset ?? something else is has been
> added to L2ARC 

[ snip lots of data ]

This is your last one:

> module: zfs instance: 0
> name:   arcstatsclass:misc
> c   1073741824
> c_max   1073741824
> c_min   134217728
[...]
> l2_size 2632226304
> l2_write_bytes  6486009344

Roughly 6GB has been written to the device, and slightly less than 2.5GB
is actually in use.

> p   775528448

/Tomas
-- 
Tomas Ögren, st...@acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Umeå
`- Sysadmin at {cs,acc}.umu.se
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss