Re: 11.2-STABLE kernel wired memory leak

2019-03-29 Thread Paul Mather
On Mar 29, 2019, at 5:52 AM, Robert Schulze wrote: > Hi, > > I just want to report a similar issue here with 11.2-RELEASE-p8. > > The affected machine has 64 GB ram and does daily backups from several > machines in the night, at daytime there a parallel runs of clamav on a > specific dataset.

Re: 11.2-STABLE kernel wired memory leak

2019-03-29 Thread Robert Schulze
Hi, I just want to report a similar issue here with 11.2-RELEASE-p8. The affected machine has 64 GB ram and does daily backups from several machines in the night, at daytime there a parallel runs of clamav on a specific dataset. One symtom is basic I/O-performance: After upgrading from 11.1 to

Re: 11.2-STABLE kernel wired memory leak

2019-02-20 Thread Mike Tancsa
On 2/20/2019 3:14 AM, Eugene Grosbein wrote: > 20.02.2019 3:49, Mike Tancsa wrote: > >> On 2/19/2019 2:35 PM, Mike Tancsa wrote: >>> dd if=/dev/zero of=/tanker/test bs=1m count=10 >> The box has 32G of RAM. If I do a >> >> # sysctl -w vfs.zfs.arc_max=12224866304 >> vfs.zfs.arc_max: 32224866304

Re: 11.2-STABLE kernel wired memory leak

2019-02-20 Thread Eugene Grosbein
20.02.2019 3:49, Mike Tancsa wrote: > On 2/19/2019 2:35 PM, Mike Tancsa wrote: >> dd if=/dev/zero of=/tanker/test bs=1m count=10 > > The box has 32G of RAM. If I do a > > # sysctl -w vfs.zfs.arc_max=12224866304 > vfs.zfs.arc_max: 32224866304 -> 12224866304 > # > > after WIRED memory is at

Re: 11.2-STABLE kernel wired memory leak

2019-02-19 Thread Mike Tancsa
On 2/19/2019 2:35 PM, Mike Tancsa wrote: > dd if=/dev/zero of=/tanker/test bs=1m count=10 The box has 32G of RAM. If I do a # sysctl -w vfs.zfs.arc_max=12224866304 vfs.zfs.arc_max: 32224866304 -> 12224866304 # after WIRED memory is at 29G, it doesnt immediately reclaim it and there is

Re: 11.2-STABLE kernel wired memory leak

2019-02-19 Thread Mike Tancsa
On 2/12/2019 11:49 AM, Eugene Grosbein wrote: > 12.02.2019 23:34, Mark Johnston wrote: > >> I suspect that the "leaked" memory is simply being used to cache UMA >> items. Note that the values in the FREE column of vmstat -z output are >> quite large. The cached items are reclaimed only when the

Re: 11.2-STABLE kernel wired memory leak

2019-02-15 Thread Lev Serebryakov
Hello Eugene, Tuesday, February 12, 2019, 10:18:09 PM, you wrote: > Do you have/had some memory pressure here? Growth of swap usage? After several days, ARC is even smaller, but Wired is the same: Mem: 88M Active, 904M Inact, 29G Wired, 1121M Free ARC: 13G Total, 5288M MFU, 6937M MRU, 2880K

Re: 11.2-STABLE kernel wired memory leak

2019-02-14 Thread Stefan Esser
Am 13.02.19 um 10:59 schrieb Andriy Gapon: > On 12/02/2019 20:17, Eugene Grosbein wrote: >> 13.02.2019 1:14, Eugene Grosbein wrote: >> >>> Use following command to see how much memory is wasted in your case: >>> >>> vmstat -z | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort >>> -k1,1

Re: 11.2-STABLE kernel wired memory leak

2019-02-13 Thread Eugene Grosbein
On 13.02.2019 19:04, Eugene Grosbein wrote: > 12.02.2019 23:34, Mark Johnston wrote: > >> On Tue, Feb 12, 2019 at 11:14:31PM +0700, Eugene Grosbein wrote: >>> Hi! >>> >>> Long story short: 11.2-STABLE/amd64 r335757 leaked over 4600MB kernel wired >>> memory over 81 days uptime >>> out of 8GB

Re: 11.2-STABLE kernel wired memory leak

2019-02-13 Thread Karl Denninger
On 2/13/2019 03:59, Andriy Gapon wrote: > On 12/02/2019 20:17, Eugene Grosbein wrote: >> 13.02.2019 1:14, Eugene Grosbein wrote: >> >>> Use following command to see how much memory is wasted in your case: >>> >>> vmstat -z | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort >>> -k1,1

Re: 11.2-STABLE kernel wired memory leak

2019-02-13 Thread Eugene Grosbein
12.02.2019 23:34, Mark Johnston wrote: > On Tue, Feb 12, 2019 at 11:14:31PM +0700, Eugene Grosbein wrote: >> Hi! >> >> Long story short: 11.2-STABLE/amd64 r335757 leaked over 4600MB kernel wired >> memory over 81 days uptime >> out of 8GB total RAM. >> >> Details follow. >> >> I have a

Re: 11.2-STABLE kernel wired memory leak

2019-02-13 Thread Andriy Gapon
On 12/02/2019 20:17, Eugene Grosbein wrote: > 13.02.2019 1:14, Eugene Grosbein wrote: > >> Use following command to see how much memory is wasted in your case: >> >> vmstat -z | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 >> -rn | head > > Oops, small correction: > >

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Lev Serebryakov
Hello Eugene, Tuesday, February 12, 2019, 10:18:09 PM, you wrote: >> I'm have same problem. >> >> According to top(1) I have 29G Wired, but only 17G Total ARC (12G >> difference! System has 32G of RAM), and this statistic shows: >> >> 5487.5 zio_data_buf_524288 >>920.125

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Mike Tancsa
On 2/12/2019 2:38 PM, Eugene Grosbein wrote: > > mbuf* numbers represent memory being wasted IMHO. > > In contrast to ZFS memory that can contain useful cached data, > contents of freed mbufs cannot be re-used, right? > > Some amount of "free" mbufs in the zone's just fine to serve bursts of

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Eugene Grosbein
13.02.2019 2:18, Mike Tancsa wrote: > On an nfs server, serving a few large files, my 32G box is showing > > vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n", > $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head >11014.3 abd_chunk > 2090.5 zio_data_buf_131072 >1142.67

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Mike Tancsa
On 2/12/2019 1:03 PM, Eugene Grosbein wrote: > It seems page daemon is broken somehow as it did not reclaim several > gigs of wired memory > despite of long period of vm thrashing: > > $ sed 's/:/,/' vmstat-z.txt | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, > $1}' | sort -k1,1 -rn | head >

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Eugene Grosbein
13.02.2019 1:50, Lev Serebryakov wrote: > I'm have same problem. > > According to top(1) I have 29G Wired, but only 17G Total ARC (12G > difference! System has 32G of RAM), and this statistic shows: > > 5487.5 zio_data_buf_524288 >920.125 zio_data_buf_131072 >626

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Eugene Grosbein
13.02.2019 1:18, Mark Johnston wrote: > Depending on the system's workload, it is possible for the caches to > grow quite quickly after a reclaim. If you are able to run kgdb on the > kernel, you can find the time of the last reclaim by comparing the > values of lowmem_uptime and time_uptime.

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Lev Serebryakov
On 12.02.2019 21:48, Eugene Grosbein wrote: > I will reach the console next day only. Is it wise to use kgdb over ssh for > running remote system? :-) It works for me :-) BTW, my is: (kgdb) p time_uptime $1 = 81369 (kgdb) p lowmem_uptime $2 = 0 (yes, this system have been rebooted less than

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Mark Johnston
On Wed, Feb 13, 2019 at 01:48:21AM +0700, Eugene Grosbein wrote: > 13.02.2019 1:42, Mark Johnston wrote: > > >> Yes, I have debugger compiled into running kernel and have console access. > >> What commands should I use? > > > > I meant kgdb(1). If you can run that, try: > > > > (kgdb) p

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Eugene Grosbein
13.02.2019 1:42, Mark Johnston wrote: >> Yes, I have debugger compiled into running kernel and have console access. >> What commands should I use? > > I meant kgdb(1). If you can run that, try: > > (kgdb) p time_uptime > (kgdb) p lowmem_uptime > > If you are willing to drop the system into

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Lev Serebryakov
On 12.02.2019 21:37, Eugene Grosbein wrote: >> vfs.zfs.arc_max=1216348160 > > Each line shows how many megabytes is allocated but currently unused by > corresponding > UMA zone (and unavailable for other consumers). Your numbers are pretty low, > you have nothing to worry about IMHO. I have

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Mark Johnston
On Wed, Feb 13, 2019 at 01:40:06AM +0700, Eugene Grosbein wrote: > 13.02.2019 1:18, Mark Johnston wrote: > > > On Wed, Feb 13, 2019 at 01:03:37AM +0700, Eugene Grosbein wrote: > >> 12.02.2019 23:34, Mark Johnston wrote: > >> > >>> I suspect that the "leaked" memory is simply being used to cache

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Eugene Grosbein
13.02.2019 1:18, Mark Johnston wrote: > On Wed, Feb 13, 2019 at 01:03:37AM +0700, Eugene Grosbein wrote: >> 12.02.2019 23:34, Mark Johnston wrote: >> >>> I suspect that the "leaked" memory is simply being used to cache UMA >>> items. Note that the values in the FREE column of vmstat -z output

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Eugene Grosbein
13.02.2019 1:29, Kurt Jaeger wrote: > On a 8 GB 11.2p8 box doing mostly routing: > > 42.3047 abd_chunk > 40 zio_buf_131072 > 31.75 zio_data_buf_131072 >19.8901 swblk >12.9224 RADIX NODE >11.7344 zio_buf_16384 >10.0664 zio_data_buf_12288 >9.84375

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Kurt Jaeger
Hi! > > Use following command to see how much memory is wasted in your case: > > > > vmstat -z | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort > > -k1,1 -rn | head > > Oops, small correction: > > vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, > $1}' |

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Eugene Grosbein
13.02.2019 1:14, Eugene Grosbein wrote: > Use following command to see how much memory is wasted in your case: > > vmstat -z | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 > -rn | head Oops, small correction: vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n",

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Mark Johnston
On Wed, Feb 13, 2019 at 01:03:37AM +0700, Eugene Grosbein wrote: > 12.02.2019 23:34, Mark Johnston wrote: > > > I suspect that the "leaked" memory is simply being used to cache UMA > > items. Note that the values in the FREE column of vmstat -z output are > > quite large. The cached items are

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Eugene Grosbein
13.02.2019 0:57, Garrett Wollman wrote: > In article > eu...@grosbein.net writes: > >> Long story short: 11.2-STABLE/amd64 r335757 leaked over 4600MB kernel >> wired memory over 81 days uptime >> out of 8GB total RAM. > > Not a whole lot of evidence yet, but anecdotally I'm seeing the same >

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Eugene Grosbein
12.02.2019 23:34, Mark Johnston wrote: > I suspect that the "leaked" memory is simply being used to cache UMA > items. Note that the values in the FREE column of vmstat -z output are > quite large. The cached items are reclaimed only when the page daemon > wakes up to reclaim memory; if there

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Garrett Wollman
In article eu...@grosbein.net writes: >Long story short: 11.2-STABLE/amd64 r335757 leaked over 4600MB kernel >wired memory over 81 days uptime >out of 8GB total RAM. Not a whole lot of evidence yet, but anecdotally I'm seeing the same thing on some huge-memory NFS servers running releng/11.2.

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Karl Denninger
On 2/12/2019 10:49, Eugene Grosbein wrote: > 12.02.2019 23:34, Mark Johnston wrote: > >> I suspect that the "leaked" memory is simply being used to cache UMA >> items. Note that the values in the FREE column of vmstat -z output are >> quite large. The cached items are reclaimed only when the

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Eugene Grosbein
12.02.2019 23:49, Eugene Grosbein wrote: > 12.02.2019 23:34, Mark Johnston wrote: > >> I suspect that the "leaked" memory is simply being used to cache UMA >> items. Note that the values in the FREE column of vmstat -z output are >> quite large. The cached items are reclaimed only when the page

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Eugene Grosbein
12.02.2019 23:34, Mark Johnston wrote: > I suspect that the "leaked" memory is simply being used to cache UMA > items. Note that the values in the FREE column of vmstat -z output are > quite large. The cached items are reclaimed only when the page daemon > wakes up to reclaim memory; if there

Re: 11.2-STABLE kernel wired memory leak

2019-02-12 Thread Mark Johnston
On Tue, Feb 12, 2019 at 11:14:31PM +0700, Eugene Grosbein wrote: > Hi! > > Long story short: 11.2-STABLE/amd64 r335757 leaked over 4600MB kernel wired > memory over 81 days uptime > out of 8GB total RAM. > > Details follow. > > I have a workstation running Xorg, Firefox, Thunderbird,

11.2-STABLE kernel wired memory leak

2019-02-12 Thread Eugene Grosbein
Hi! Long story short: 11.2-STABLE/amd64 r335757 leaked over 4600MB kernel wired memory over 81 days uptime out of 8GB total RAM. Details follow. I have a workstation running Xorg, Firefox, Thunderbird, LibreOffice and occasionally VirtualBox for single VM. It has two identical 320GB HDDs