On 3/11/2018 1:43 PM, Jeff Roberson wrote:
> Hi Folks,
>
> This could be my fault from recent NUMA and concurrency related work. I
> did touch some of the arc back-pressure mechanisms. First, I would like
> to identify whether the wired memory is in the buffer cache. Can those
> of you that have a repro look at sysctl vfs.bufspace and tell me if that
> accounts for the bulk of your wired memory usage? I'm wondering if a
> job ran that pulled in all of the bufs from your root disk and filled up
> the buffer cache which doesn't have a back-pressure mechanism. Then arc
> didn't respond appropriately to lower its usage.
>
> Also, if you could try going back to r328953 or r326346 and let me know
> if the problem exists in either. That would be very helpful. If anyone
> is willing to debug this with me contact me directly and I will send
> some test patches or debugging info after you have done the above steps.
>
> Thank you for the reports.
>
> Jeff
It's like there is no ZFS ARC backpressure at all anymore...
Mem: 8783M Active, 2851M Inact, 3716M Laundry, 53G Wired, 2449M Free
ARC: 40G Total, 27G MFU, 11G MRU, 13M Anon, 631M Header, 955M Other
37G Compressed, 53G Uncompressed, 1.43:1 Ratio
Swap: 116G Total, 18G Used, 98G Free, 15% Inuse, 4K In
vfs.bufspace: 0
I ran out of swap space around 16GB and had to add a larger device on to
keep a package test build going.
Was there ever backpressure? Having to limit the arc size has been a
constant workaround for package building.
I have some large tmpfs at the moment totaling around ~28GB.
> # df -ah|egrep '(Used|tmpfs)'
> Filesystem Size
> Used Avail Capacity Mounted on
> tmpfs 99G
> 520K 99G 0% /tmp
> tmpfs 10G
> 8.0K 10G 0% /poudriere/data/.m/exp-11amd64-commit-test
> tmpfs 10G
> 373M 9.6G 4% /poudriere/data/.m/exp-11amd64-commit-test/ref
> tmpfs 1.0G
> 18M 1.0G 2% /poudriere/data/.m/exp-11amd64-commit-test/ref/.p
> tmpfs 10G
> 176K 10G 0%
> /poudriere/data/.m/exp-11amd64-commit-test/ref/var/db/ports
> tmpfs 10G
> 405M 9.6G 4% /poudriere/data/.m/exp-11amd64-commit-test/01
> tmpfs 10G
> 1.1G 8.9G 11% /poudriere/data/.m/exp-11amd64-commit-test/05
> tmpfs 10G
> 2.4G 7.6G 24% /poudriere/data/.m/exp-11amd64-commit-test/04
> tmpfs 10G
> 578M 9.4G 6% /poudriere/data/.m/exp-11amd64-commit-test/03
> tmpfs 10G
> 1.0G 9.0G 10% /poudriere/data/.m/exp-11amd64-commit-test/02
> tmpfs 10G
> 936M 9.1G 9% /poudriere/data/.m/exp-11amd64-commit-test/06
> tmpfs 10G
> 2.7G 7.3G 27% /poudriere/data/.m/exp-11amd64-commit-test/07
> tmpfs 10G
> 2.7G 7.3G 27% /poudriere/data/.m/exp-11amd64-commit-test/09
> tmpfs 10G
> 875M 9.1G 9% /poudriere/data/.m/exp-11amd64-commit-test/08
> tmpfs 10G
> 2.7G 7.3G 27% /poudriere/data/.m/exp-11amd64-commit-test/10
> tmpfs 10G
> 2.7G 7.3G 27% /poudriere/data/.m/exp-11amd64-commit-test/12
> tmpfs 10G
> 390M 9.6G 4% /poudriere/data/.m/exp-11amd64-commit-test/11
> tmpfs 10G
> 992M 9.0G 10% /poudriere/data/.m/exp-11amd64-commit-test/14
> tmpfs 10G
> 2.9G 7.1G 29% /poudriere/data/.m/exp-11amd64-commit-test/13
> tmpfs 1.0G
> 16K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/01/.p
> tmpfs 1.0G
> 540K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/12/.p
> tmpfs 1.0G
> 1.3M 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/05/.p
> tmpfs 1.0G
> 1.3M 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/06/.p
> tmpfs 1.0G
> 396K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/03/.p
> tmpfs 1.0G
> 540K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/09/.p
> tmpfs 1.0G
> 540K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/10/.p
> tmpfs 1.0G
> 732K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/07/.p
> tmpfs 1.0G
> 28K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/08/.p
> tmpfs 1.0G
> 12K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/11/.p
> tmpfs 1.0G
> 1.3M 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/02/.p
> tmpfs 1.0G
> 852K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/14/.p
> tmpfs 1.0G
> 540K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/13/.p
> tmpfs 1.0G
> 688K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/04/.p
> tmpfs 10G
> 125M 9.9G 1%
> /poudriere/data/.m/exp-11amd64-commit-test/03/usr/local
> tmpfs 10G
> 270M 9.7G 3%
> /poudriere/data/.m/exp-11amd64-commit-test/04/usr/local
> tmpfs 10G
> 414M 9.6G 4%
> /poudriere/data/.m/exp-11amd64-commit-test/07/usr/local
> tmpfs 10G
> 332M 9.7G 3%
> /poudriere/data/.m/exp-11amd64-commit-test/09/usr/local
> tmpfs 10G
> 332M 9.7G 3%
> /poudriere/data/.m/exp-11amd64-commit-test/10/usr/local
> tmpfs 10G
> 332M 9.7G 3%
> /poudriere/data/.m/exp-11amd64-commit-test/12/usr/local
> tmpfs 10G
> 332M 9.7G 3%
> /poudriere/data/.m/exp-11amd64-commit-test/13/usr/local
> tmpfs 10G
> 575M 9.4G 6%
> /poudriere/data/.m/exp-11amd64-commit-test/05/usr/local
> tmpfs 10G
> 575M 9.4G 6%
> /poudriere/data/.m/exp-11amd64-commit-test/02/usr/local
> tmpfs 10G
> 306M 9.7G 3%
> /poudriere/data/.m/exp-11amd64-commit-test/14/usr/local
> tmpfs 10G
> 570M 9.4G 6%
> /poudriere/data/.m/exp-11amd64-commit-test/06/usr/local
> tmpfs 10G
> 36M 10G 0%
> /poudriere/data/.m/exp-11amd64-commit-test/01/usr/local
> tmpfs 10G
> 283M 9.7G 3%
> /poudriere/data/.m/exp-11amd64-commit-test/11/usr/local
> tmpfs 10G
> 301M 9.7G 3%
> /poudriere/data/.m/exp-11amd64-commit-test/08/usr/local
And some large processes.
> # ps aux|sort -rnk 6|head -n 20
> root 2481 100.0 4.3 3334388 3209728 15 RJ 14:40 119:54.18
> /usr/local/bin/python2.7 ../../rpython/bin/rpython --source -Ojit
> targetpypystandalone.py
> root 46466 97.7 0.8 620692 581304 15 RJ 16:44 0:36.07
> /wrkdirs/usr/ports/lang/ghc/work/ghc-8.0.2/inplace/lib/bin/ghc-stage1
> -B/wrkdirs/usr/ports/lang/ghc/work/ghc-8.0.2/inplace/lib -hisuf hi -osuf o
> -hcsuf hc -static -H32m -O -I/usr/include -L/usr/lib -I/usr/loc
> root 44667 100.0 0.2 187664 161444 15 RJ 16:43 1:04.16
> /wrkdirs/usr/ports/lang/gcc6/work/.build/./prev-gcc/cc1plus -quiet
> -nostdinc++ -I . -I c-family -I
> /wrkdirs/usr/ports/lang/gcc6/work/gcc-6.4.0/gcc -I
> /wrkdirs/usr/ports/lang/gcc6/work/gcc-6.4.0/gcc/c-family -
> root 47917 76.5 0.2 173572 140580 15 RJ 16:44 0:12.45
> /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free
> -disable-llvm-verifier -discard-value-names -main-file-name
> MemorySanitizer.cpp -mrelocation-model pic -pic-level 2 -mthread-model
> root 47918 75.9 0.2 164784 136976 15 RJ 16:44 0:12.50
> /wrkdirs/usr/ports/lang/gcc7-devel/work/.build/./prev-gcc/cc1plus -quiet
> -nostdinc++ -I . -I c -I
> /wrkdirs/usr/ports/lang/gcc7-devel/work/gcc-7-20180308/gcc -I
> /wrkdirs/usr/ports/lang/gcc7-devel/work/gcc-7-20
> root 47938 71.5 0.2 159952 134260 15 RJ 16:44 0:11.17
> /wrkdirs/usr/ports/lang/gcc6-devel/work/.build/./prev-gcc/cc1plus -quiet
> -nostdinc++ -I . -I c-family -I
> /wrkdirs/usr/ports/lang/gcc6-devel/work/gcc-6-20180307/gcc -I
> /wrkdirs/usr/ports/lang/gcc6-devel/work/g
> root 48090 48.1 0.2 159420 125748 15 RJ 16:44 0:06.02
> /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free
> -disable-llvm-verifier -discard-value-names -main-file-name
> AlignmentFromAssumptions.cpp -mrelocation-model pic -pic-level 2 -mthre
> root 48110 38.4 0.2 153988 116220 15 RJ 16:44 0:04.35
> /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free
> -disable-llvm-verifier -discard-value-names -main-file-name property-type.cxx
> -mrelocation-model static -mthread-model posix -mdisa
> root 48139 35.4 0.1 131636 101608 15 RJ 16:44 0:03.70
> /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free
> -disable-llvm-verifier -discard-value-names -main-file-name
> InstrInfoEmitter.cpp -mrelocation-model pic -pic-level 2 -mthread-model
> root 48119 34.2 0.1 126396 93768 15 RJ 16:44 0:03.79
> /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free
> -disable-llvm-verifier -discard-value-names -main-file-name
> NaryReassociate.cpp -mrelocation-model pic -pic-level 2 -mthread-model
--
Regards,
Bryan Drewery
_______________________________________________
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
