Re: Memory Leaks or Mem. being misreported
Hi. On Sun, Mar 29, 2020 at 07:48:38PM +0200, n...@dismail.de wrote: > On Sun, Mar 29, 2020 at 07:04:36PM +0300, Reco wrote: > > > afirefox: After using an affected Firefox profile > > > > Ok, let's breakdown it (left is aboot, right is afirefox). > > Here we see about 2Gb of memory consumed: > > > > MemFree:32124660 kB / MemFree:30144704 kB > > > > Such discrepancy can be attributed, first, […] > > > > Summing it up, it gives us 668382 kB of discrepancy. > > […] > > So the big question is - where 650M of memory gone? My money is on > > kernel modules. Maybe it's Intel's VideoRAM steal (don't know if AMD do > > it the same way). Maybe it's some proprietary kernel module. > > > > I know no way of accounting memory usage of kernel module, but I can > > spot some possible candidates from "lsmod" output. > > Thanks. > Afaik there are no proprietary kernel modules in the default buster install > and > I did not install proprietary modules either. BUT I did install proprietary > firmware for the devices. That's an important distinction here. Firmware does not eat RAM, modules do. > Here's the lsmod output > (while some programs are running in case that's relevant): Ok, from the most possible offender to the least one: > amdgpu 5373952 26 > snd_usb_audio 290816 0 > ledtrig_audio 16384 1 snd_hda_codec_generic > cec49152 1 drm_kms_helper > kvm_amd 114688 0 > r8169 90112 0 I'm not saying that you should poke'em with "modprobe -r" - unloading these would be difficult if possible at all - I'm saying that a bugreport is in order. If possible, I'd test it headless - i.e. reboot with amdgpu blacklisted, run firefox in TightVNC, watch the results. But that raises the question whenever your rig will boot at all without the operational videocard, so I'd put it into the "desperate measures" category. Reco
Re: Memory Leaks or Mem. being misreported
On Sun, Mar 29, 2020 at 07:04:36PM +0300, Reco wrote: > > afirefox: After using an affected Firefox profile > > Ok, let's breakdown it (left is aboot, right is afirefox). > Here we see about 2Gb of memory consumed: > > MemFree:32124660 kB / MemFree:30144704 kB > > Such discrepancy can be attributed, first, […] > > Summing it up, it gives us 668382 kB of discrepancy. > […] > So the big question is - where 650M of memory gone? My money is on > kernel modules. Maybe it's Intel's VideoRAM steal (don't know if AMD do > it the same way). Maybe it's some proprietary kernel module. > > I know no way of accounting memory usage of kernel module, but I can > spot some possible candidates from "lsmod" output. Thanks. Afaik there are no proprietary kernel modules in the default buster install and I did not install proprietary modules either. BUT I did install proprietary firmware for the devices. First I only installed firmware-amd-graphics as it is necessary for XServer to work. But the same GPU was already present in my previous desktop system. After I noticed the issue with memory I also installed in that order ·firmware-realtek for the ethernet controller (The controller does work without it, possibly not at full speed, but it works fast enough for my connection) ·amd64-microcode for the CPU But this does not seem to have made any difference. Beside these 3 package the only other installed package from non-free is a font-package. Here's the lsmod output (while some programs are running in case that's relevant): Module Size Used by binfmt_misc24576 1 amdgpu 5373952 26 nls_ascii 16384 1 nls_cp437 20480 1 vfat 20480 1 fat86016 1 vfat mfd_core 20480 1 amdgpu gpu_sched 40960 1 amdgpu i2c_algo_bit 16384 1 amdgpu ttm 122880 1 amdgpu snd_usb_audio 290816 0 drm_kms_helper245760 1 amdgpu snd_hda_codec_realtek 126976 1 snd_hda_codec_generic98304 1 snd_hda_codec_realtek ledtrig_audio 16384 1 snd_hda_codec_generic snd_hda_codec_hdmi 73728 1 snd_hda_intel 53248 10 snd_intel_dspcfg 20480 1 snd_hda_intel snd_hda_codec 163840 4 snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec_realtek snd_hda_core 102400 5 snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec,snd_hda_codec_realtek snd_usbmidi_lib40960 1 snd_usb_audio snd_rawmidi45056 1 snd_usbmidi_lib snd_hwdep 16384 2 snd_usb_audio,snd_hda_codec snd_pcm 131072 6 snd_hda_codec_hdmi,snd_hda_intel,snd_usb_audio,snd_hda_codec,snd_hda_core snd_timer 45056 2 snd_pcm cec49152 1 drm_kms_helper snd 102400 29 snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hwdep,snd_hda_intel,snd_usb_audio,snd_usbmidi_lib,snd_hda_codec,snd_hda_codec_realtek,snd_timer,snd_pcm,snd_rawmidi drm 581632 17 gpu_sched,drm_kms_helper,amdgpu,ttm mc 57344 1 snd_usb_audio kvm_amd 114688 0 kvm 806912 1 kvm_amd eeepc_wmi 16384 0 irqbypass 16384 1 kvm asus_wmi 40960 1 eeepc_wmi battery20480 1 asus_wmi pcspkr 16384 0 sparse_keymap 16384 1 asus_wmi video 53248 1 asus_wmi crct10dif_pclmul 16384 1 crc32_pclmul 16384 0 soundcore 16384 1 snd rfkill 28672 2 asus_wmi ghash_clmulni_intel16384 0 aesni_intel 368640 0 ccp 106496 1 kvm_amd sp5100_tco 20480 0 glue_helper16384 1 aesni_intel crypto_simd16384 1 aesni_intel cryptd 24576 2 crypto_simd,ghash_clmulni_intel evdev 28672 15 k10temp16384 0 rng_core 16384 1 ccp watchdog 28672 1 sp5100_tco wmi_bmof 16384 0 sg 36864 0 efi_pstore 16384 0 efivars20480 1 efi_pstore wmi36864 2 asus_wmi,wmi_bmof acpi_cpufreq 28672 0 button 24576 0 parport_pc 32768 0 ppdev 24576 0 lp 20480 0 parport61440 3 parport_pc,lp,ppdev efivarfs 16384 1 ip_tables 32768 0 x_tables 53248 1 ip_tables autofs453248 2 ext4 770048 3 crc32c_generic 16384 0 crc16 16384 1 ext4 mbcache16384 1 ext4 jbd2 139264 1 ext4 sd_mod 57344 5 hid_generic16384 0 usbhid 65536 0 hid 139264 2 usbhid,hid_generic sr_mod 28672 0 cdrom
Re: Memory Leaks or Mem. being misreported
Hi. On Sun, Mar 29, 2020 at 05:06:01PM +0200, n...@dismail.de wrote: > On Sat, Mar 28, 2020 at 05:10:07PM +0300, Reco wrote: > > /proc/meminfo (please *do not soft* it), and the output of slabtop. > > If you're need to understand where all that memory gone - you're in need > > of proper tools. > > I have saved the output of slabtop and /proc/meminfo for: > aboot: Directly after booting and logging into a graphical i3-session Nothing to see here, but a good baseline. > afirefox: After using an affected Firefox profile Ok, let's breakdown it (left is aboot, right is afirefox). Here we see about 2Gb of memory consumed: MemFree:32124660 kB / MemFree:30144704 kB Such discrepancy can be attributed, first, to the difference in Active and Inactive memory pages: Active: 216940 kB / 766788 kB Inactive: 178292 kB / 676072 kB With the most contribution is given by file-backed memory pages (i.e. filesystem cache): Active(file): 94220 kB / 589536 kB Inactive(file): 167528 kB / 665180 kB Which gives us 992988 kB difference of 1979956 kB. And we see some miniscule tmpfs usage (54532 kB): Active(anon): 122720 kB / Active(anon): 177252 kB Inactive(anon):10892 kB / Inactive(anon):10764 kB Second, assorted kernel structures (KReclaim - WritebackTmp per meminfo) give us 155694 kB. Third, assorted processes (AnonPages + Mapped + AnonHugePages) give us another 108360 kB. > In all cases only some daemons, X-Server, i3 and one or two terminals were > running. > htop and free were reporting memory usage as around 450MiB for aboot and > afill-* > and around 1,3GiB for afirefox. Summing it up, it gives us 668382 kB of discrepancy. Note that both htop and free blatantly lied about actual memory usage, sweetening the deal for about 350M. > afill-alloc: After afirefox and running a simple C program that allocates > almost all of the total physical memory > afill-alloc+swaponoff: After afill-alloc and swapoff and swapon. Too lazy to write it up, I leave it as an exercise to another reader. So the big question is - where 650M of memory gone? My money is on kernel modules. Maybe it's Intel's VideoRAM steal (don't know if AMD do it the same way). Maybe it's some proprietary kernel module. I know no way of accounting memory usage of kernel module, but I can spot some possible candidates from "lsmod" output. Reco
Re: Memory Leaks or Mem. being misreported
On Sat, Mar 28, 2020 at 05:10:07PM +0300, Reco wrote: > /proc/meminfo (please *do not soft* it), and the output of slabtop. > If you're need to understand where all that memory gone - you're in need > of proper tools. I have saved the output of slabtop and /proc/meminfo for: aboot: Directly after booting and logging into a graphical i3-session afirefox: After using an affected Firefox profile afill-alloc: After afirefox and running a simple C program that allocates almost all of the total physical memory afill-alloc+swaponoff: After afill-alloc and swapoff and swapon. In all cases only some daemons, X-Server, i3 and one or two terminals were running. htop and free were reporting memory usage as around 450MiB for aboot and afill-* and around 1,3GiB for afirefox. I've read the part about meminfo here https://www.kernel.org/doc/Documentation/filesystems/proc.txt but I'm not experienced enough to tell if anything/what in the meminfo output is unusual. For reference: htop calculates the memory usage from /proc/meminfo like this: Cached memory = Cached + SReclaimable - Shmem Non cache/buffer memory = MemTotal - MemFree - (Buffers + 'Cached memory') Swap= SwapTotal - SwapFree As they are rather lengthy and there are 4 of them, I attached the output of the commands to this email. According to the mailing list guidelines large attachments should be avoided, but since they are in total 68K and compressed just 8K this should probably be fine. If the attachment doesn't work I can resend it in the mail body. Nils meminfo+slabtop.tar.gz Description: application/gzip
Re: Memory Leaks or Mem. being misreported
On Sat, Mar 28, 2020 at 10:12:52AM -0400, Dan Ritter wrote: > Did this cause a problem, or are you chasing a number you don't > like? I avoided possible problems until now, as every time I expected to need a lot of memory and the "used memory" was larger than expected I rebooted beforehand. Also not understanding why this happens or possibly having misunderstood how to read (h)top's and free's memory usage is a problem in itself and might also lead to further problems in the future. > What different hardware have you tried? I am mainly comparing these two setups, but have briefly tried some other hardware as well: | Previously (unaff.) | Current (affected) ——+——+——— CPU | AMD Athlon II X3 450 | AMD Ryzen 7 2700 GPU | AMD Radeon RX580 | AMD Radeon RX 580 Mainboard | AsRock 770 Extreme3 | ASUS X570 TUF-Gaming RAM | 3×2GB DDR3-1333 | 2×16GB DDR4-2800 Debian and Manjaro were freshly installed after the hardware change, Ubuntu was kept for a while but is now removed. Ubuntu 18.04 did not show this behaviour with the old hardware and kernels 4.15, 5.0, 5.3 (from Ubuntu repos) and various versions of 5.4.xy (vanilla) but did show this behaviour with the new hardware and kernels 5.3 and 5.0(from repo) and various vanilla 5.4 versions. I did not test Ubuntu with new hardware and kernel 4.15. > This all looks fine. You are barely using your memory. See the > available number at the end? That's what would be available for > programs that want to use it. In all 3 instances there were only some daemons, the X-Server, WM and one terminal running as I closed all other applications beforehand. (including Firefox). Also this is not the biggest memory increase I observed, as I said this "idle usage" can increase to several GiB. - On Sat, Mar 28, 2020 at 05:10:07PM +0300, Reco wrote: > /proc/meminfo (please *do not soft* it), and the output of slabtop. > If you're need to understand where all that memory gone - you're in need > of proper tools. Thanks. I will look into it this later this day. Nils
Re: Memory Leaks or Mem. being misreported
Dan Ritter (12020-03-28): > Is this causing a problem for you? Not understanding something is a problem by itself. Regards, -- Nicolas George signature.asc Description: PGP signature
Re: Memory Leaks or Mem. being misreported
n...@dismail.de wrote: > after upgrading my hardware I started to notice what seemed like memory > leaks. Did this cause a problem, or are you chasing a number you don't like? > 650MiB(with MATE) of RAM are being used and after extended use, when closing > all > gui-programs except a terminal, based on prior experience I would expect the > RAM usage to not be more than +120MiB compared to after booting. I am *not* > counting file cache/buffers in this (or I severely misunderstand how to read > free's and top's output). Why would you think this? Is this causing a problem for you? > Or maybe I'm just stupid and misreading top's and free's output. You are probably reading them correctly but misinterpreting what the system is doing. > Also this seems to be hardware specific. What different hardware have you tried? > $ free -h > totalusedfree shared buff/cache > available > Mem: 31Gi 397Mi30Gi11Mi 314Mi > 30Gi > Swap: 6,0Gi 0B 6,0Gi > > --After using and closing Firefox > --(no processes using that much RAM are listed in htop): > $ free -h > totalusedfree shared buff/cache > available > Mem: 31Gi 1,1Gi 2,4Gi13Mi27Gi > 29Gi > Swap: 6,0Gi 0B 6,0Gi > > > --After Backing up which "filled" RAM with file caches: > $ free -h > totalusedfree shared buff/cache > available > Mem: 31Gi 471Mi 328Mi13Mi30Gi > 30Gi > Swap: 6,0Gi 0B 6,0Gi This all looks fine. You are barely using your memory. See the available number at the end? That's what would be available for programs that want to use it. > After that (h)top and free are disagreeing on how much memory is being used > (see SWAP). > --After Firefox (htop,top and free report more or less the same): > $ free -h > totalusedfree shared buff/cache > available > Mem: 31Gi 1,5Gi27Gi92Mi 2,7Gi > 29Gi > Swap: 6,0Gi 0B 6,0Gi > > -- After allocating 30GiB and then exiting: > $ top -b -n 1 | grep "^MiB" > MiB Mem : 32092,0 total, 27747,1 free,744,3 used, 3600,6 buff/cache > MiB Swap: 6144,0 total, 6143,5 free, 0,5 used. 30458,0 avail Mem > $ free -h > totalusedfree shared buff/cache > available > Mem: 31Gi 743Mi27Gi93Mi 3,5Gi > 29Gi > Swap: 6,0Gi 0,0Ki 6,0Gi That's an insignificant difference caused by rounding methods. -dsr-
Re: Memory Leaks or Mem. being misreported
Hi. On Sat, Mar 28, 2020 at 02:56:37PM +0100, n...@dismail.de wrote: > But now I would often end up with something between +600MiB and +2.5GiB. > When looking at top or htop no process using nearly that much memory is > listed. /proc/meminfo (please *do not soft* it), and the output of slabtop. If you're need to understand where all that memory gone - you're in need of proper tools. > Sorry for this wall of text. It's only a wall if you don't use paragraphs. Reco
