I've recently set up gjournal on top of gmirror on FreeBSD 8.2. I understand
that this setup has a lot of redundant writing. It is working, but I'm not
sure I've set it up as efficiently as I should.
During prolonged writes, such as copying large files to the file system
across the network or producing very large logfiles, the "low_mem" and
"skipped_bytes" statisics rise rapidly and the system becomes less
responsive. "top" always reports free memory, so I don't think that's the
issue. "journal_full" and "wait_for_copy" have never occurred. Here's a
sample of what happens after a couple hours of intense writing...
kern.geom.journal.stats.low_mem: 5379
kern.geom.journal.stats.journal_full: 0
kern.geom.journal.stats.wait_for_copy: 0
kern.geom.journal.stats.switches: 7543
kern.geom.journal.stats.combined_ios: 265318
kern.geom.journal.stats.skipped_bytes: 935712768
"low_mem" sounds like a bad thing. What could I do to remedy that? Did I
make the journal too small? The stats say that "journal_full" has never
happened, so maybe not? Is there a setting I should tweak?
The Handbook says 1GB is good enough most of the time, but it also says that
3x the amount of physical memory is a good size as well. I compromised
between the two and made an 8GB journal for this system that has ~4GB of
memory.
Replying to myself for posterity... I think I figured it out.
I believe "low_mem" means its running out of gjournal cache, defined by
kern.geom.journal.cache.limit, which is initially set by
kern.geom.journal.cache.divisor, which is initially 2, which means half of
the kernel memory, which is vm.kmem_size, which is 330MB on my i386 8.2
system by default. I don't know where this default comes from, but it
seems common. This gives me about 167MB of cache for journalling.
During big file transfers, I can watch the "journal_data" statistic in
"vmstat -m". It climbs close to the 167MB cache size. That's when I see
kern.geom.journal.stats.low_mem tick over and the stutter happens. So I
think I understand what I'm seeing now.
I could try increasing vm.kmem_size and possibly the KMEM_SIZE option
that's compiled into my kernel as described in the ZFS tuning documents,
since it's a similar issue. This would allow the kernel to have more
memory and thus allow gjournal to use more memory for caching. If I'm
consistently filling the cache faster than I can empty it, making the
cache bigger probably won't help. This journal is on top of a mirror, so
I'm always expecting it to read faster than it writes... Copying large
files locally might always max out the buffer? No harm in trying, I
guess.
Or I could just live with it. It's doing what I told it to do.
_______________________________________________
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
