kern.geom.journal.stats.low_mem refers to what?
I've recently set up gjournal on top of gmirror on FreeBSD 8.2. I understand that this setup has a lot of redundant writing. It is working, but I'm not sure I've set it up as efficiently as I should. During prolonged writes, such as copying large files to the file system across the network or producing very large logfiles, the "low_mem" and "skipped_bytes" statisics rise rapidly and the system becomes less responsive. "top" always reports free memory, so I don't think that's the issue. "journal_full" and "wait_for_copy" have never occurred. Here's a sample of what happens after a couple hours of intense writing... kern.geom.journal.stats.low_mem: 5379 kern.geom.journal.stats.journal_full: 0 kern.geom.journal.stats.wait_for_copy: 0 kern.geom.journal.stats.switches: 7543 kern.geom.journal.stats.combined_ios: 265318 kern.geom.journal.stats.skipped_bytes: 935712768 "low_mem" sounds like a bad thing. What could I do to remedy that? Did I make the journal too small? The stats say that "journal_full" has never happened, so maybe not? Is there a setting I should tweak? The Handbook says 1GB is good enough most of the time, but it also says that 3x the amount of physical memory is a good size as well. I compromised between the two and made an 8GB journal for this system that has ~4GB of memory. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: kern.geom.journal.stats.low_mem refers to what?
I've recently set up gjournal on top of gmirror on FreeBSD 8.2. I understand that this setup has a lot of redundant writing. It is working, but I'm not sure I've set it up as efficiently as I should. During prolonged writes, such as copying large files to the file system across the network or producing very large logfiles, the "low_mem" and "skipped_bytes" statisics rise rapidly and the system becomes less responsive. "top" always reports free memory, so I don't think that's the issue. "journal_full" and "wait_for_copy" have never occurred. Here's a sample of what happens after a couple hours of intense writing... kern.geom.journal.stats.low_mem: 5379 kern.geom.journal.stats.journal_full: 0 kern.geom.journal.stats.wait_for_copy: 0 kern.geom.journal.stats.switches: 7543 kern.geom.journal.stats.combined_ios: 265318 kern.geom.journal.stats.skipped_bytes: 935712768 "low_mem" sounds like a bad thing. What could I do to remedy that? Did I make the journal too small? The stats say that "journal_full" has never happened, so maybe not? Is there a setting I should tweak? The Handbook says 1GB is good enough most of the time, but it also says that 3x the amount of physical memory is a good size as well. I compromised between the two and made an 8GB journal for this system that has ~4GB of memory. Replying to myself for posterity... I think I figured it out. I believe "low_mem" means its running out of gjournal cache, defined by kern.geom.journal.cache.limit, which is initially set by kern.geom.journal.cache.divisor, which is initially 2, which means half of the kernel memory, which is vm.kmem_size, which is 330MB on my i386 8.2 system by default. I don't know where this default comes from, but it seems common. This gives me about 167MB of cache for journalling. During big file transfers, I can watch the "journal_data" statistic in "vmstat -m". It climbs close to the 167MB cache size. That's when I see kern.geom.journal.stats.low_mem tick over and the stutter happens. So I think I understand what I'm seeing now. I could try increasing vm.kmem_size and possibly the KMEM_SIZE option that's compiled into my kernel as described in the ZFS tuning documents, since it's a similar issue. This would allow the kernel to have more memory and thus allow gjournal to use more memory for caching. If I'm consistently filling the cache faster than I can empty it, making the cache bigger probably won't help. This journal is on top of a mirror, so I'm always expecting it to read faster than it writes... Copying large files locally might always max out the buffer? No harm in trying, I guess. Or I could just live with it. It's doing what I told it to do. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: kern.geom.journal.stats.low_mem refers to what?
On Wed, Mar 30, 2011 at 9:07 PM, Luke Dean wrote: > > The Handbook says 1GB is good enough most of the time, but it also says >> that 3x the amount of physical memory is a good size as well. I compromised >> between the two and made an 8GB journal for this system that has ~4GB of >> memory. >> > > I've never understood why that article states that calculation. The amount of RAM a system has little to do with what gjournal needs. That formula will in most cases allocate far more journal space than is really necessary. Disk speed and latency are far more important factors. Back when I was using gjournal, I used a 2GB journal on 7200rpm and was never able to panic the system unlike with a 1 GB journal but that was with all other settings at default. Allocating more to gjournal cache would increase the need for space so perhaps it's good you've made such a large one. As you've discovered, gjournal's performance isn't so great when large files are involved. It still helps on multi-threaded access but those big sequential operations destroy most the other optimizations in place. You might want to consider placing your journal on an SSD. -- Adam Vande More ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
