> > Patching zfs_prefetch_disable = 1 has helped
> It's my belief this mainly aids scanning metadata. my
> testing with rsync and yours with find (and seen with
> du & ; zpool iostat -v 1 ) pans this out..
> mainly tracked in bug 6437054 vdev_cache: wise up or die
> http://www.opensolaris.org/jive/thread.jspa?messageID=42212
>
> so to link your code, it might help, but if one ran
> a clean down the tree, it would hurt compile times.
I think the slowdown that I'm observing is due to the changes
that have been made for 6542676 "ARC needs to track meta-data
memory overhead".
There is now a limit of 1/4 of arc size ("arc_meta_limit")
for zfs meta-data.
On a 32-bit x86 platform with > 512MB physical memory,
the arc size is limited to 3/4 of the size of the kernel heap
arena, which is 3/4 * ~ 650MB => ~ 500MB.
1/4 of that 500MB is ~ 125MB for zfs meta data.
When more than 1/4 of arc is used for meta-data,
meta-data allocations steal space from arc mru/mfu list.
When more than 1/4 of arc is used for meta-data and
arc_reclaim_needed() returns TRUE, entries from the
dnlc cache are purged and arc data is evicted.
Apparently, before 6542676 it was possible to use a lot
more meta-data if we compare it to what is possible now
with 6542676.
void
arc_init(void)
{
...
/* limit meta-data to 1/4 of the arc capacity */
arc_meta_limit = arc_c_max / 4;
...
}
static int
arc_evict_needed(arc_buf_contents_t type)
{
if (type == ARC_BUFC_METADATA && arc_meta_used >= arc_meta_limit)
return (1);
...
}
static void
arc_get_data_buf(arc_buf_t *buf)
{
/*
* We have not yet reached cache maximum size,
* just allocate a new buffer.
*/
if (!arc_evict_needed(type)) {
...
goto out;
}
/*
* If we are prefetching from the mfu ghost list, this buffer
* will end up on the mru list; so steal space from there.
*/
...
if ((buf->b_data = arc_evict(state, size, TRUE, type)) == NULL) {
...
}
static void
arc_kmem_reap_now(arc_reclaim_strategy_t strat)
{
...
if (arc_meta_used >= arc_meta_limit) {
/*
* We are exceeding our meta-data cache limit.
* Purge some DNLC entries to release holds on meta-data.
*/
dnlc_reduce_cache((void *)(uintptr_t)arc_reduce_dnlc_percent);
}
...
}
The Tecra-S1 (32-bit Solaris x86) has
> arc_meta_limit::print
0x7380000 <<<<<<<<<<<<<<<<<<<
> arc_meta_limit::print -d
0t121110528
> ::arc
{
anon = -735455552
mru = -735455488
mru_ghost = -735455424
mfu = -735455360
mfu_ghost = -735455296
size = 0x131dae70
p = 0xb10983e
c = 0x1330105e
c_min = 0x4000000
c_max = 0x1ce00000 <<<<<<<<<<<<<<<<<<<<
hits = 0x2e405
misses = 0x9092
deleted = 0x5f
recycle_miss = 0x45bf
mutex_miss = 0
evict_skip = 0x6e4b0
hash_elements = 0x54dd
hash_elements_max = 0x54de
hash_collisions = 0x398e
hash_chains = 0x1887
hash_chain_max = 0x7
no_grow = 0
> 0x1ce00000%4=X
7380000
Patching arc_meta_limit to 1/2 of arc size improves find performance.
Another problem:
In dbuf.c, dbuf_read_impl() arc_meta_used accounting appears to be
broken, the amount of meta-data used ("arc_meta_used") is inflated:
db->db.db_data = zio_buf_alloc(DN_MAX_BONUSLEN);
arc_space_consume(512);
Why 512? Apparently, we zio_buf_alloc DN_MAX_BONUSLEN = 0x140 bytes
but consume 0x200 bytes of meta-data?
(When these buffers are freed, only DN_MAX_BONUSLEN = 0x140 bytes
are returned to arc meta-data)
I'm currently using the following changes, which seem to
restore the zfs performace to what it has been before
6542676 - more or less:
diff -r bec4e9eb1f01 usr/src/uts/common/fs/zfs/arc.c
--- a/usr/src/uts/common/fs/zfs/arc.c Fri Jun 01 08:24:48 2007 -0700
+++ b/usr/src/uts/common/fs/zfs/arc.c Sat Jun 02 22:09:33 2007 +0200
@@ -2781,10 +2781,10 @@ arc_init(void)
arc_c = arc_c_max;
arc_p = (arc_c >> 1);
- /* limit meta-data to 1/4 of the arc capacity */
- arc_meta_limit = arc_c_max / 4;
- if (arc_c_min < arc_meta_limit / 2 && zfs_arc_min == 0)
- arc_c_min = arc_meta_limit / 2;
+ /* limit meta-data to 1/2 of the arc capacity */
+ arc_meta_limit = arc_c_max / 2;
+ if (arc_c_min < arc_meta_limit / 4 && zfs_arc_min == 0)
+ arc_c_min = arc_meta_limit / 4;
/* if kmem_flags are set, lets try to use less memory */
if (kmem_debugging())
diff -r bec4e9eb1f01 usr/src/uts/common/fs/zfs/dbuf.c
--- a/usr/src/uts/common/fs/zfs/dbuf.c Fri Jun 01 08:24:48 2007 -0700
+++ b/usr/src/uts/common/fs/zfs/dbuf.c Sat Jun 02 22:09:52 2007 +0200
@@ -470,7 +470,7 @@ dbuf_read_impl(dmu_buf_impl_t *db, zio_t
if (db->db_blkid == DB_BONUS_BLKID) {
ASSERT3U(db->db_dnode->dn_bonuslen, ==, db->db.db_size);
db->db.db_data = zio_buf_alloc(DN_MAX_BONUSLEN);
- arc_space_consume(512);
+ arc_space_consume(DN_MAX_BONUSLEN);
if (db->db.db_size < DN_MAX_BONUSLEN)
bzero(db->db.db_data, DN_MAX_BONUSLEN);
bcopy(DN_BONUS(db->db_dnode->dn_phys), db->db.db_data,
This message posted from opensolaris.org
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
