On Thursday, August 16, 2018 at 8:03:52 PM UTC+2, Marcus Linsner wrote: > On Thursday, August 16, 2018 at 7:50:14 PM UTC+2, Marcus Linsner wrote: > > On Thursday, August 16, 2018 at 7:35:26 PM UTC+2, Marcus Linsner wrote: > > > $ cat /proc/meminfo > > > MemTotal: 7454500 kB > > > MemFree: 5635088 kB > > > MemAvailable: 6574676 kB > > > Buffers: 53832 kB > > > Cached: 1094368 kB > > > SwapCached: 0 kB > > > Active: 724832 kB > > > Inactive: 747696 kB > > > Active(anon): 233816 kB > > > Inactive(anon): 95768 kB > > > Active(file): 491016 kB > > > Inactive(file): 651928 kB > > > Unevictable: 73568 kB > > > Mlocked: 73568 kB > > > SwapTotal: 0 kB > > > SwapFree: 0 kB > > > Dirty: 292 kB > > > Writeback: 0 kB > > > AnonPages: 398016 kB > > > Mapped: 54320 kB > > > Shmem: 5256 kB > > > Slab: 134680 kB > > > SReclaimable: 74124 kB > > > SUnreclaim: 60556 kB > > > KernelStack: 4800 kB > > > PageTables: 10524 kB > > > NFS_Unstable: 0 kB > > > Bounce: 0 kB > > > WritebackTmp: 0 kB > > > CommitLimit: 3727248 kB > > > Committed_AS: 1332236 kB > > > VmallocTotal: 34359738367 kB > > > VmallocUsed: 0 kB > > > VmallocChunk: 0 kB > > > HardwareCorrupted: 0 kB > > > AnonHugePages: 0 kB > > > ShmemHugePages: 0 kB > > > ShmemPmdMapped: 0 kB > > > CmaTotal: 0 kB > > > CmaFree: 0 kB > > > HugePages_Total: 0 > > > HugePages_Free: 0 > > > HugePages_Rsvd: 0 > > > HugePages_Surp: 0 > > > Hugepagesize: 2048 kB > > > DirectMap4k: 327644 kB > > > DirectMap2M: 14008320 kB > > > DirectMap1G: 0 kB > > > > I resumed the firefox compilation and noticed that the memory jumped back > > to 14GB again - I was sure it was more than that 7.4GB before: > > > > $ cat /proc/meminfo > > MemTotal: 14003120 kB > > MemFree: 4602448 kB > > MemAvailable: 6622252 kB > > Buffers: 186220 kB > > Cached: 1986192 kB > > SwapCached: 0 kB > > Active: 7482024 kB > > Inactive: 1448656 kB > > Active(anon): 6667828 kB > > Inactive(anon): 95780 kB > > Active(file): 814196 kB > > Inactive(file): 1352876 kB > > Unevictable: 73568 kB > > Mlocked: 73568 kB > > SwapTotal: 0 kB > > SwapFree: 0 kB > > Dirty: 306392 kB > > Writeback: 4684 kB > > AnonPages: 6811888 kB > > Mapped: 199164 kB > > Shmem: 5340 kB > > Slab: 239524 kB > > SReclaimable: 177620 kB > > SUnreclaim: 61904 kB > > KernelStack: 5968 kB > > PageTables: 28612 kB > > NFS_Unstable: 0 kB > > Bounce: 0 kB > > WritebackTmp: 0 kB > > CommitLimit: 7001560 kB > > Committed_AS: 8571548 kB > > VmallocTotal: 34359738367 kB > > VmallocUsed: 0 kB > > VmallocChunk: 0 kB > > HardwareCorrupted: 0 kB > > AnonHugePages: 0 kB > > ShmemHugePages: 0 kB > > ShmemPmdMapped: 0 kB > > CmaTotal: 0 kB > > CmaFree: 0 kB > > HugePages_Total: 0 > > HugePages_Free: 0 > > HugePages_Rsvd: 0 > > HugePages_Surp: 0 > > Hugepagesize: 2048 kB > > DirectMap4k: 327644 kB > > DirectMap2M: 14008320 kB > > DirectMap1G: 0 kB > > > > > > Oh man, I'm hitting that disk thrashing again after just a few minutes: > > 202MiB/sec reading, 0.0 writing. > > > > Paused qube, reading stopped. > > Resumed qube sooner than before and it's still thrashing... > > > > It'a a fedora 28 template-based VM. > > > > I shut down another VM and I thought dom0 crashed because it froze for like > > 10 sec before the notification message told me that that VM stopped. > > Ok, I caught kswapd0 at 14% in a 'top' terminal on the offending qube, before > the disk thrashing begun(which froze all terminals too) and then the only > process at 100% after disk thrashing stopped! and here's the continuation of > the log, btw the thrashing only stopped after OOM killed the rustc > process(which, my guess was triggeding kswapd0 to use 100% cpu or what): > > [ 6871.435899] systemd-coredum: 4 output lines suppressed due to ratelimiting > [ 6871.485869] audit: type=1130 audit(1534438842.909:179): pid=1 uid=0 > auid=4294967295 ses=4294967295 msg='unit=systemd-logind comm="systemd" > exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' > [ 6871.486357] audit: type=1130 audit(1534438842.910:180): pid=1 uid=0 > auid=4294967295 ses=4294967295 msg='unit=systemd-logind comm="systemd" > exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' > [ 7076.504392] kauditd_printk_skb: 5 callbacks suppressed > [ 7076.504393] audit: type=1101 audit(1534439047.928:186): pid=5658 uid=1000 > auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success' > [ 7076.504502] audit: type=1123 audit(1534439047.928:187): pid=5658 uid=1000 > auid=1000 ses=1 msg='cwd="/home/user" cmd=636174202F6574632F67726F757073 > terminal=pts/2 res=success' > [ 7076.505193] audit: type=1110 audit(1534439047.929:188): pid=5658 uid=0 > auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success' > [ 7076.506823] audit: type=1105 audit(1534439047.930:189): pid=5658 uid=0 > auid=1000 ses=1 msg='op=PAM:session_open > grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix > acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 > res=success' > [ 7076.508580] audit: type=1106 audit(1534439047.932:190): pid=5658 uid=0 > auid=1000 ses=1 msg='op=PAM:session_close > grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix > acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 > res=success' > [ 7076.508681] audit: type=1104 audit(1534439047.932:191): pid=5658 uid=0 > auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success' > [ 7078.730535] audit: type=1101 audit(1534439050.154:192): pid=5670 uid=1000 > auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success' > [ 7078.730868] audit: type=1123 audit(1534439050.154:193): pid=5670 uid=1000 > auid=1000 ses=1 msg='cwd="/home/user" cmd=636174202F6574632F67726F7570 > terminal=pts/2 res=success' > [ 7078.730905] audit: type=1110 audit(1534439050.154:194): pid=5670 uid=0 > auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success' > [ 7078.731931] audit: type=1105 audit(1534439050.155:195): pid=5670 uid=0 > auid=1000 ses=1 msg='op=PAM:session_open > grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix > acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 > res=success' > [ 7573.970907] kauditd_printk_skb: 2 callbacks suppressed > [ 7573.970908] audit: type=1101 audit(1534439545.394:198): pid=6724 uid=1000 > auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success' > [ 7573.970967] audit: type=1123 audit(1534439545.394:199): pid=6724 uid=1000 > auid=1000 ses=1 msg='cwd="/home/user" cmd="dmesg" terminal=pts/4 res=success' > [ 7573.971028] audit: type=1110 audit(1534439545.394:200): pid=6724 uid=0 > auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success' > [ 7573.971894] audit: type=1105 audit(1534439545.395:201): pid=6724 uid=0 > auid=1000 ses=1 msg='op=PAM:session_open > grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix > acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 > res=success' > [ 7574.015405] audit: type=1106 audit(1534439545.439:202): pid=6724 uid=0 > auid=1000 ses=1 msg='op=PAM:session_close > grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix > acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 > res=success' > [ 7574.015440] audit: type=1104 audit(1534439545.439:203): pid=6724 uid=0 > auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success' > [ 7770.221883] audit: type=1101 audit(1534439741.645:204): pid=7130 uid=1000 > auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success' > [ 7770.222429] audit: type=1123 audit(1534439741.646:205): pid=7130 uid=1000 > auid=1000 ses=1 msg='cwd="/home/user" cmd="dmesg" terminal=pts/4 res=success' > [ 7770.222529] audit: type=1110 audit(1534439741.646:206): pid=7130 uid=0 > auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success' > [ 7770.227554] audit: type=1105 audit(1534439741.651:207): pid=7130 uid=0 > auid=1000 ses=1 msg='op=PAM:session_open > grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix > acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 > res=success' > [ 7773.848220] audit: type=1106 audit(1534439745.271:208): pid=7130 uid=0 > auid=1000 ses=1 msg='op=PAM:session_close > grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix > acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 > res=success' > [ 7773.848431] audit: type=1104 audit(1534439745.271:209): pid=7130 uid=0 > auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success' > [ 7809.671640] audit: type=1101 audit(1534439781.095:210): pid=7217 uid=1000 > auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success' > [ 7809.672118] audit: type=1123 audit(1534439781.095:211): pid=7217 uid=1000 > auid=1000 ses=1 msg='cwd="/home/user" cmd="dmesg" terminal=pts/4 res=success' > [ 7809.672152] audit: type=1110 audit(1534439781.095:212): pid=7217 uid=0 > auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success' > [ 7809.673088] audit: type=1105 audit(1534439781.097:213): pid=7217 uid=0 > auid=1000 ses=1 msg='op=PAM:session_open > grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix > acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 > res=success' > [ 7810.371225] audit: type=1106 audit(1534439781.794:214): pid=7217 uid=0 > auid=1000 ses=1 msg='op=PAM:session_close > grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix > acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 > res=success' > [ 7810.371397] audit: type=1104 audit(1534439781.794:215): pid=7217 uid=0 > auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success' > [ 7813.718144] audit: type=1101 audit(1534439785.142:216): pid=7234 uid=1000 > auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success' > [ 7813.718470] audit: type=1123 audit(1534439785.142:217): pid=7234 uid=1000 > auid=1000 ses=1 msg='cwd="/home/user" > cmd=646D657367202D2D636F6C6F723D616C77617973 terminal=pts/4 res=success' > [ 7813.718496] audit: type=1110 audit(1534439785.142:218): pid=7234 uid=0 > auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success' > [ 7813.719407] audit: type=1105 audit(1534439785.143:219): pid=7234 uid=0 > auid=1000 ses=1 msg='op=PAM:session_open > grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix > acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 > res=success' > [ 7817.505895] audit: type=1106 audit(1534439788.928:220): pid=7234 uid=0 > auid=1000 ses=1 msg='op=PAM:session_close > grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix > acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 > res=success' > [ 7817.506116] audit: type=1104 audit(1534439788.929:221): pid=7234 uid=0 > auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success' > [ 8251.964372] audit: type=1101 audit(1534440223.388:222): pid=8119 uid=1000 > auid=1000 ses=1 msg='op=PAM:accounting grantors=pam_unix acct="user" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success' > [ 8251.964572] audit: type=1123 audit(1534440223.388:223): pid=8119 uid=1000 > auid=1000 ses=1 msg='cwd="/home/user" cmd="dmesg" terminal=pts/4 res=success' > [ 8251.965245] audit: type=1110 audit(1534440223.388:224): pid=8119 uid=0 > auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success' > [ 8251.969268] audit: type=1105 audit(1534440223.393:225): pid=8119 uid=0 > auid=1000 ses=1 msg='op=PAM:session_open > grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix > acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 > res=success' > [ 8253.023385] audit: type=1106 audit(1534440224.447:226): pid=8119 uid=0 > auid=1000 ses=1 msg='op=PAM:session_close > grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix > acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 > res=success' > [ 8253.023615] audit: type=1104 audit(1534440224.447:227): pid=8119 uid=0 > auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" > exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success' > [ 9739.590829] watchdog: BUG: soft lockup - CPU#11 stuck for 163s! > [kswapd0:102] > [ 9739.590913] Modules linked in: binfmt_misc fuse ip6table_filter ip6_tables > xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat > nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c > xen_netfront intel_rapl crct10dif_pclmul crc32_pclmul crc32c_intel > ghash_clmulni_intel intel_rapl_perf pcspkr u2mfn(O) xen_gntdev xen_gntalloc > xenfs xen_privcmd xen_blkback xen_evtchn xen_blkfront > [ 9739.591307] CPU: 11 PID: 102 Comm: kswapd0 Tainted: G O > 4.14.57-1.pvops.qubes.x86_64 #1 > [ 9739.591393] task: ffff88035654dc40 task.stack: ffffc90001f08000 > [ 9739.591472] RIP: 0010:__raw_callee_save___pv_queued_spin_unlock+0x1/0x12 > [ 9739.591536] RSP: 0000:ffffc90001f0bcb8 EFLAGS: 00000206 ORIG_RAX: > ffffffffffffff0c > [ 9739.591609] RAX: ffff880034d179e0 RBX: ffff8803591a4d00 RCX: > 0000000000000000 > [ 9739.591691] RDX: 0000000000000000 RSI: 0000000000000000 RDI: > ffff8803591a4d00 > [ 9739.591762] RBP: 0000000000000018 R08: 28f5c28f5c28f5c3 R09: > 00000000000000bc > [ 9739.591836] R10: ffffc90001f0bdd0 R11: 0000000000000000 R12: > ffffc90001f0bd68 > [ 9739.591911] R13: 0000000000000000 R14: 0000000000000000 R15: > ffff88035a9b0458 > [ 9739.591983] FS: 0000000000000000(0000) GS:ffff88035dac0000(0000) > knlGS:0000000000000000 > [ 9739.592056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 9739.592121] CR2: 0000000000eea000 CR3: 000000000220a006 CR4: > 00000000003606e0 > [ 9739.592202] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 9739.592270] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 9739.592337] Call Trace: > [ 9739.592376] list_lru_count_one+0x47/0x70 > [ 9739.592425] super_cache_count+0x64/0xf0 > [ 9739.592465] shrink_slab.part.50+0x12f/0x400 > [ 9739.592522] shrink_node+0x10e/0x320 > [ 9739.592586] kswapd+0x355/0x720 > [ 9739.592633] kthread+0xff/0x130 > [ 9739.592677] ? mem_cgroup_shrink_node+0x170/0x170 > [ 9739.592729] ? kthread_create_on_node+0x70/0x70 > [ 9739.592781] ret_from_fork+0x35/0x40 > [ 9739.592831] Code: 90 90 90 90 90 90 90 90 90 90 51 52 56 57 41 50 41 51 41 > 52 41 53 e8 cf 04 00 00 41 5b 41 5a 41 59 41 58 5f 5e 5a 59 c3 66 90 52 <b8> > 01 00 00 00 31 d2 f0 0f b0 17 3c 01 75 02 5a c3 56 0f b6 f0 > [ 9744.021459] audit: type=1701 audit(1534441715.445:228): auid=4294967295 > uid=0 gid=0 ses=4294967295 pid=326 comm="systemd-udevd" > exe="/usr/lib/systemd/systemd-udevd" sig=6 res=1 > [ 9781.154297] sh invoked oom-killer: > gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), > order=0, oom_score_adj=0 > [ 9781.154323] sh cpuset=/ mems_allowed=0 > [ 9781.154333] CPU: 6 PID: 13108 Comm: sh Tainted: G O L > 4.14.57-1.pvops.qubes.x86_64 #1 > [ 9781.154349] Call Trace: > [ 9781.154358] dump_stack+0x5c/0x85 > [ 9781.154367] dump_header+0x94/0x21c > [ 9781.154376] ? get_page_from_freelist+0x184/0xbd0 > [ 9781.154386] oom_kill_process+0x212/0x410 > [ 9781.154394] out_of_memory+0x131/0x4b0 > [ 9781.154403] __alloc_pages_slowpath+0xda8/0xe90 > [ 9781.154414] __alloc_pages_nodemask+0x270/0x2b0 > [ 9781.154424] filemap_fault+0x54c/0x860 > [ 9781.154433] ? alloc_set_pte+0x343/0x520 > [ 9781.154442] ? filemap_map_pages+0x2d5/0x510 > [ 9781.154454] ext4_filemap_fault+0x2c/0x40 > [ 9781.154463] __do_fault+0x1e/0xb5 > [ 9781.154471] __handle_mm_fault+0xc40/0xda0 > [ 9781.154481] handle_mm_fault+0xc6/0x1b0 > [ 9781.154489] __do_page_fault+0x255/0x4f0 > [ 9781.154499] do_page_fault+0x33/0x120 > [ 9781.154507] ? page_fault+0x2f/0x50 > [ 9781.154515] page_fault+0x45/0x50 > [ 9781.154523] RIP: 4d5aa6d8: (null) > [ 9781.154531] RSP: 0000:000060b33d2792e0 EFLAGS: 60b33d04550f > [ 9781.154537] Mem-Info: > [ 9781.154554] active_anon:3359753 inactive_anon:23957 isolated_anon:0 > [ 9781.154554] active_file:192 inactive_file:352 isolated_file:0 > [ 9781.154554] unevictable:18392 dirty:0 writeback:1 unstable:0 > [ 9781.154554] slab_reclaimable:10075 slab_unreclaimable:15506 > [ 9781.154554] mapped:1227 shmem:1351 pagetables:10544 bounce:0 > [ 9781.154554] free:30606 free_pcp:1856 free_cma:0 > [ 9781.154612] Node 0 active_anon:13439012kB inactive_anon:95828kB > active_file:768kB inactive_file:828kB unevictable:73568kB isolated(anon):0kB > isolated(file):0kB mapped:4648kB dirty:0kB writeback:4kB shmem:5404kB > shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB > unstable:0kB all_unreclaimable? no > [ 9781.154654] Node 0 DMA free:15908kB min:76kB low:92kB high:108kB > active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB > unevictable:0kB writepending:0kB present:15996kB managed:15908kB mlocked:0kB > kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB > free_cma:0kB > [ 9781.154697] lowmem_reserve[]: 0 3956 13655 13655 13655 > [ 9781.154715] Node 0 DMA32 free:58312kB min:19556kB low:24444kB high:29332kB > active_anon:3878024kB inactive_anon:38876kB active_file:224kB > inactive_file:776kB unevictable:32436kB writepending:0kB present:4159452kB > managed:4071632kB mlocked:32436kB kernel_stack:2384kB pagetables:14536kB > bounce:0kB free_pcp:948kB local_pcp:24kB free_cma:0kB > [ 9781.154764] lowmem_reserve[]: 0 0 9699 9699 9699 > [ 9781.154775] Node 0 Normal free:48204kB min:47948kB low:59932kB > high:71916kB active_anon:9560988kB inactive_anon:56952kB active_file:3120kB > inactive_file:2216kB unevictable:41132kB writepending:4kB present:10160128kB > managed:9915580kB mlocked:41132kB kernel_stack:3712kB pagetables:27640kB > bounce:0kB free_pcp:6484kB local_pcp:632kB free_cma:0kB > [ 9781.154829] lowmem_reserve[]: 0 0 0 0 0 > [ 9781.154838] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) > 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = > 15908kB > [ 9781.154869] Node 0 DMA32: 278*4kB (UME) 206*8kB (UME) 230*16kB (UME) > 267*32kB (UME) 131*64kB (UME) 65*128kB (UME) 38*256kB (UME) 19*512kB (UME) > 5*1024kB (UE) 1*2048kB (M) 0*4096kB = 58312kB > [ 9781.154909] Node 0 Normal: 1113*4kB (UMEH) 541*8kB (UMEH) 479*16kB (UMEH) > 437*32kB (UMEH) 177*64kB (UME) 37*128kB (ME) 10*256kB (ME) 1*512kB (M) > 0*1024kB 0*2048kB 0*4096kB = 49564kB > [ 9781.154944] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 > hugepages_size=1048576kB > [ 9781.154959] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 > hugepages_size=2048kB > [ 9781.154975] 1803 total pagecache pages > [ 9781.154982] 0 pages in swap cache > [ 9781.154990] Swap cache stats: add 3510, delete 3510, find 44/106 > [ 9781.155011] Free swap = 0kB > [ 9781.155040] Total swap = 0kB > [ 9781.155048] 3583894 pages RAM > [ 9781.155055] 0 pages HighMem/MovableOnly > [ 9781.155062] 83114 pages reserved > [ 9781.155069] 0 pages cma reserved > [ 9781.155076] 0 pages hwpoisoned > [ 9781.155083] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents > oom_score_adj name > [ 9781.155106] [ 282] 0 282 30805 100 34 3 0 > 0 qubesdb-daemon > [ 9781.155123] [ 326] 0 326 23697 686 46 3 0 > -1000 systemd-udevd > [ 9781.155139] [ 432] 0 432 3042 781 11 3 0 > 0 haveged > [ 9781.155155] [ 433] 81 433 13278 230 30 3 0 > -900 dbus-daemon > [ 9781.155171] [ 438] 0 438 10243 73 23 3 0 > 0 meminfo-writer > [ 9781.155187] [ 450] 0 450 34209 116 38 3 0 > 0 xl > [ 9781.155202] [ 454] 0 454 18997 203 43 3 0 > 0 qubes-gui > [ 9781.155218] [ 455] 0 455 16536 106 36 3 0 > 0 qrexec-agent > [ 9781.155234] [ 503] 0 503 73994 189 50 3 0 > 0 su > [ 9781.355141] [ 519] 0 519 52863 27 10 3 0 > 0 agetty > [ 9781.355161] [ 520] 0 520 52775 29 11 3 0 > 0 agetty > [ 9781.355179] [ 521] 1000 521 21958 338 45 3 0 > 0 systemd > [ 9781.355196] [ 527] 1000 527 34755 598 64 3 0 > 0 (sd-pam) > [ 9781.355216] [ 570] 1000 570 54160 86 14 3 0 > 0 bash > [ 9781.355234] [ 627] 1000 627 3500 30 12 3 0 > 0 xinit > [ 9781.355253] [ 640] 1000 640 323803 26712 180 4 0 > 0 Xorg > [ 9781.355276] [ 663] 1000 663 53597 67 12 4 0 > 0 qubes-session > [ 9781.355296] [ 675] 1000 675 13197 181 31 3 0 > 0 dbus-daemon > [ 9781.355315] [ 691] 1000 691 7242 119 17 3 0 > 0 ssh-agent > [ 9781.355333] [ 709] 1000 709 16562 127 38 3 0 > 0 qrexec-client-v > [ 9781.355352] [ 725] 1000 725 48107 148 31 3 0 > 0 dconf-service > [ 9781.355371] [ 731] 1000 731 428388 2600 194 5 0 > 0 gsd-xsettings > [ 9781.355389] [ 732] 1000 732 62744 1790 29 3 0 > 0 icon-sender > [ 9781.355408] [ 734] 1000 734 122405 245 40 3 0 > 0 gnome-keyring-d > [ 9781.355426] [ 735] 1000 735 120207 121 37 3 0 > 0 agent > [ 9781.355444] [ 748] 1000 748 438372 2843 212 5 0 > 0 nm-applet > [ 9781.355462] [ 755] 1000 755 128956 339 91 4 0 > 0 pulseaudio > [ 9781.355482] [ 757] 172 757 47723 87 29 3 0 > 0 rtkit-daemon > [ 9781.355500] [ 768] 998 768 657135 1588 91 6 1 > 0 polkitd > [ 9781.355522] [ 771] 1000 771 16528 100 34 3 0 > 0 qrexec-fork-ser > [ 9781.355541] [ 774] 1000 774 52238 17 11 3 0 > 0 sleep > [ 9781.355559] [ 848] 1000 848 87396 143 38 4 0 > 0 at-spi-bus-laun > [ 9781.355578] [ 854] 1000 854 13134 120 31 3 0 > 0 dbus-daemon > [ 9781.355597] [ 858] 1000 858 56364 197 46 3 0 > 0 at-spi2-registr > [ 9781.355616] [ 868] 1000 868 123835 221 46 3 0 > 0 gvfsd > [ 9781.355634] [ 902] 1000 902 105683 143 41 4 0 > 0 gvfsd-fuse > [ 9781.355651] [ 921] 1000 921 214263 6889 146 4 0 > 0 gnome-terminal- > [ 9781.355664] [ 925] 1000 925 206356 367 66 4 0 > 0 xdg-desktop-por > [ 9781.355676] [ 930] 1000 930 157249 162 39 4 0 > 0 xdg-document-po > [ 9781.355688] [ 933] 1000 933 117667 117 33 3 0 > 0 xdg-permission- > [ 9781.355700] [ 943] 1000 943 193292 1125 110 3 0 > 0 xdg-desktop-por > [ 9781.355713] [ 951] 1000 951 54193 171 16 3 0 > 0 bash > [ 9781.355725] [ 1012] 0 1012 257186 47659 220 4 0 > 0 packagekitd > [ 9781.355742] [ 2375] 1000 2375 54193 165 14 3 0 > 0 bash > [ 9781.355755] [ 6264] 1000 6264 63294 229 30 3 0 > 0 top > [ 9781.355768] [ 6748] 1000 6748 54193 161 13 3 0 > 0 bash > [ 9781.355780] [10247] 1000 10247 54193 161 13 3 0 > 0 bash > [ 9781.355793] [30948] 1000 30948 54193 160 16 3 0 > 0 bash > [ 9781.355805] [ 664] 1000 664 54193 156 15 4 0 > 0 bash > [ 9781.355818] [ 3687] 1000 3687 54023 213 14 3 0 > 0 watch > [ 9781.355830] [ 5125] 0 5125 26967 181 46 3 0 > 0 systemd-journal > [ 9781.355843] [ 5136] 0 5136 19325 174 40 3 0 > 0 systemd-logind > [ 9781.355855] [ 9347] 1000 9347 65858 760 38 4 0 > 0 rpmbuild > [ 9781.355868] [ 9446] 1000 9446 53597 74 12 3 0 > 0 sh > [ 9781.355878] [ 9476] 1000 9476 67165 4754 68 3 0 > 0 python2.7 > [ 9781.554909] [ 9489] 1000 9489 30476 4870 62 3 0 > 0 python2.7 > [ 9781.554927] [ 9527] 1000 9527 9587 582 23 3 0 > 0 gmake > [ 9781.554940] [ 9530] 1000 9530 9516 504 22 3 0 > 0 gmake > [ 9781.554953] [ 9633] 1000 9633 9521 484 20 3 0 > 0 gmake > [ 9781.554965] [ 9637] 1000 9637 9654 680 22 3 0 > 0 gmake > [ 9781.554978] [ 9677] 1000 9677 9145 139 22 3 0 > 0 gmake > [ 9781.554991] [ 9683] 1000 9683 53977 4945 46 4 0 > 0 cargo > [ 9781.555018] [ 9955] 1000 9955 738373 611629 1350 6 0 > 0 rustc > [ 9781.555033] [12689] 1000 12689 9175 203 19 3 0 > 0 gmake > [ 9781.555045] [12775] 1000 12775 9145 137 21 3 0 > 0 gmake > [ 9781.555063] [12822] 1000 12822 9172 204 22 4 0 > 0 gmake > [ 9781.555076] [12882] 1000 12882 2943 241 11 3 0 > 0 ccache > [ 9781.555088] [12895] 1000 12895 2941 239 11 3 0 > 0 g++ > [ 9781.555101] [12905] 1000 12905 2305 24 8 3 0 > 0 g++ > [ 9781.555113] [12906] 1000 12906 446815 416509 848 4 0 > 0 cc1plus > [ 9781.555126] [12907] 1000 12907 7225 3466 19 3 0 > 0 as > [ 9781.555137] [12911] 1000 12911 9144 171 22 3 0 > 0 gmake > [ 9781.555149] [12918] 1000 12918 2799 254 12 3 0 > 0 ccache > [ 9781.555162] [12925] 1000 12925 2797 196 11 3 0 > 0 ccache > [ 9781.555174] [12933] 1000 12933 2795 194 10 4 0 > 0 g++ > [ 9781.555187] [12936] 1000 12936 2797 253 10 3 0 > 0 g++ > [ 9781.555199] [12944] 1000 12944 2305 23 8 3 0 > 0 g++ > [ 9781.555211] [12945] 1000 12945 295307 277152 575 4 0 > 0 cc1plus > [ 9781.555224] [12946] 1000 12946 18940 15243 42 3 0 > 0 as > [ 9781.555234] [12947] 1000 12947 2305 24 9 3 0 > 0 g++ > [ 9781.555247] [12948] 1000 12948 361103 334097 695 4 0 > 0 cc1plus > [ 9781.555260] [12949] 1000 12949 9074 5344 22 3 0 > 0 as > [ 9781.555271] [12958] 1000 12958 2766 184 10 3 0 > 0 ccache > [ 9781.555286] [12970] 1000 12970 2780 216 10 3 0 > 0 ccache > [ 9781.555298] [12980] 1000 12980 2764 182 10 3 0 > 0 g++ > [ 9781.555310] [12988] 1000 12988 2305 23 9 3 0 > 0 g++ > [ 9781.555322] [12989] 1000 12989 263737 244263 513 4 0 > 0 cc1plus > [ 9781.555334] [12990] 1000 12990 9502 5774 23 3 0 > 0 as > [ 9781.555345] [12991] 1000 12991 2779 214 11 3 0 > 0 g++ > [ 9781.555357] [12998] 1000 12998 2791 188 10 3 0 > 0 ccache > [ 9781.555369] [13001] 1000 13001 2305 24 10 3 0 > 0 g++ > [ 9781.555381] [13002] 1000 13002 318726 292631 610 4 0 > 0 cc1plus > [ 9781.555393] [13003] 1000 13003 4982 1252 13 3 0 > 0 as > [ 9781.555404] [13006] 1000 13006 2789 186 11 4 0 > 0 g++ > [ 9781.555416] [13011] 1000 13011 2305 24 9 3 0 > 0 g++ > [ 9781.555428] [13012] 1000 13012 245657 226072 475 4 0 > 0 cc1plus > [ 9781.555440] [13013] 1000 13013 6862 3134 17 3 0 > 0 as > [ 9781.555453] [13021] 1000 13021 9145 137 22 4 0 > 0 gmake > [ 9781.555465] [13023] 1000 13023 9144 136 21 3 0 > 0 gmake > [ 9781.555477] [13026] 1000 13026 2781 232 11 3 0 > 0 ccache > [ 9781.555489] [13027] 1000 13027 2786 183 9 3 0 > 0 ccache > [ 9781.555502] [13040] 1000 13040 2762 179 11 3 0 > 0 ccache > [ 9781.555515] [13041] 1000 13041 2793 188 11 3 0 > 0 ccache > [ 9781.555527] [13048] 1000 13048 2761 178 10 3 0 > 0 g++ > [ 9781.754559] [13051] 1000 13051 2784 181 10 3 0 > 0 g++ > [ 9781.754574] [13054] 1000 13054 2791 186 11 3 0 > 0 g++ > [ 9781.754590] [13057] 1000 13057 2779 231 10 3 0 > 0 g++ > [ 9781.754607] [13062] 1000 13062 2304 23 9 3 0 > 0 g++ > [ 9781.754624] [13063] 1000 13063 213546 192190 412 4 0 > 0 cc1plus > [ 9781.754641] [13064] 1000 13064 4981 1221 16 3 0 > 0 as > [ 9781.754655] [13065] 1000 13065 2305 22 9 3 0 > 0 g++ > [ 9781.754672] [13066] 1000 13066 209797 189104 403 4 0 > 0 cc1plus > [ 9781.754689] [13067] 1000 13067 4981 1219 15 3 0 > 0 as > [ 9781.754702] [13068] 1000 13068 2304 23 9 3 0 > 0 g++ > [ 9781.754717] [13069] 1000 13069 208141 188217 402 3 0 > 0 cc1plus > [ 9781.754729] [13070] 1000 13070 4981 1223 13 3 0 > 0 as > [ 9781.754740] [13073] 1000 13073 2305 24 9 3 0 > 0 g++ > [ 9781.754754] [13074] 1000 13074 204312 186045 395 3 0 > 0 cc1plus > [ 9781.754770] [13075] 1000 13075 4982 1248 16 3 0 > 0 as > [ 9781.754783] [13080] 1000 13080 2762 176 10 3 0 > 0 ccache > [ 9781.754796] [13089] 1000 13089 2761 174 11 3 0 > 0 g++ > [ 9781.754808] [13094] 1000 13094 2304 22 8 3 0 > 0 g++ > [ 9781.754821] [13095] 1000 13095 104070 81528 194 3 0 > 0 cc1plus > [ 9781.754835] [13096] 1000 13096 4981 1222 16 3 0 > 0 as > [ 9781.754846] [13107] 1000 13107 54023 213 11 3 0 > 0 watch > [ 9781.754859] [13108] 1000 13108 53597 59 14 3 0 > 0 sh > [ 9781.754870] [13109] 0 13109 11239 48 23 3 0 > 0 systemd-coredum > [ 9781.754883] Out of memory: Kill process 9955 (rustc) score 175 or > sacrifice child > [ 9781.754904] Killed process 9955 (rustc) total-vm:2953492kB, > anon-rss:2446516kB, file-rss:0kB, shmem-rss:0kB > [ 9781.838565] oom_reaper: reaped process 9955 (rustc), now anon-rss:0kB, > file-rss:0kB, shmem-rss:0kB > [ 9781.944947] audit: type=1701 audit(1534441753.368:229): auid=4294967295 > uid=0 gid=0 ses=4294967295 pid=5125 comm="systemd-journal" > exe="/usr/lib/systemd/systemd-journald" sig=6 res=1 > [ 9782.129302] systemd[1]: Started Process Core Dump (PID 13109/UID 0). > [ 9782.131966] systemd[1]: systemd-journald.service: Main process exited, > code=dumped, status=6/ABRT > [ 9782.132076] systemd[1]: systemd-journald.service: Failed with result > 'watchdog'. > [ 9782.133213] systemd[1]: systemd-journald.service: Service has no hold-off > time, scheduling restart. > [ 9782.133287] systemd[1]: systemd-journald.service: Scheduled restart job, > restart counter is at 2. > [ 9782.133437] systemd[1]: Stopped Flush Journal to Persistent Storage. > [ 9782.133479] systemd[1]: Stopping Flush Journal to Persistent Storage... > [ 9782.133498] systemd[1]: Stopped Journal Service. > [ 9782.135777] systemd[1]: Starting Journal Service... > [ 9782.136153] audit: type=1130 audit(1534441753.555:230): pid=1 uid=0 > auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@1-13109-0 > comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? > res=success' > [ 9782.136189] audit: type=1131 audit(1534441753.556:231): pid=1 uid=0 > auid=4294967295 ses=4294967295 msg='unit=systemd-journald comm="systemd" > exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' > [ 9782.136220] audit: type=1131 audit(1534441753.557:232): pid=1 uid=0 > auid=4294967295 ses=4294967295 msg='unit=systemd-journal-flush comm="systemd" > exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' > [ 9782.136251] audit: type=1130 audit(1534441753.557:233): pid=1 uid=0 > auid=4294967295 ses=4294967295 msg='unit=systemd-journald comm="systemd" > exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' > [ 9782.136281] audit: type=1131 audit(1534441753.557:234): pid=1 uid=0 > auid=4294967295 ses=4294967295 msg='unit=systemd-journald comm="systemd" > exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' > [ 9782.153162] audit: type=1305 audit(1534441753.576:235): audit_enabled=1 > old=1 auid=4294967295 ses=4294967295 res=1 > [ 9782.168207] systemd-journald[13113]: File > /var/log/journal/12c5e521101c444594b96b53751551a8/system.journal corrupted or > uncleanly shut down, renaming and replacing. > [ 9782.202464] systemd-coredump[13111]: MESSAGE=Process 5125 > (systemd-journal) of user 0 dumped core. > [ 9782.202488] systemd-coredump[13111]: Coredump diverted to > /var/lib/systemd/coredump/core.systemd-journal.0.f4dc34a546c24e2aa7f99cdcd659015f.5125.1534441753000000.lz4 > [ 9782.202518] systemd-coredump[13111]: Stack trace of thread 5125: > [ 9782.202531] systemd-coredump[13111]: #0 0x00007bd86c2981f7 fsync > (libpthread.so.0) > [ 9782.202546] systemd-coredump[13111]: #1 0x00007bd86ca3607b > journal_file_set_online (libsystemd-shared-238.so) > [ 9782.202564] systemd-coredump[13111]: #2 0x00007bd86ca3716a > journal_file_append_object (libsystemd-shared-238.so) > [ 9782.202583] systemd-coredump[13111]: #3 0x00007bd86ca37dc3 > journal_file_append_data (libsystemd-shared-238.so) > [ 9782.202602] systemd-coredump[13111]: #4 0x00007bd86ca3a0d1 > journal_file_append_entry (libsystemd-shared-238.so) > [ 9782.202622] systemd-coredump[13111]: #5 0x0000569e8af706c8 > dispatch_message_real (systemd-journald) > [ 9782.202638] systemd-coredump[13111]: #6 0x0000569e8af6c5e5 > server_read_dev_kmsg (systemd-journald) > [ 9782.206525] systemd-coredum: 6 output lines suppressed due to ratelimiting > [ 9782.219312] systemd[1]: Started Journal Service. > [ 9782.219459] audit: type=1130 audit(1534441753.643:236): pid=1 uid=0 > auid=4294967295 ses=4294967295 msg='unit=systemd-journald comm="systemd" > exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' > [ 9782.233297] audit: type=1130 audit(1534441753.657:237): pid=1 uid=0 > auid=4294967295 ses=4294967295 msg='unit=systemd-journal-flush comm="systemd" > exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' > [ 9782.327795] audit: type=1130 audit(1534441753.751:238): pid=1 uid=0 > auid=4294967295 ses=4294967295 msg='unit=systemd-udevd comm="systemd" > exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
The good news is that I've realized that the OOM triggering was legit: I had firefox set to use 12 cores at once and 14GiB of RAM was clearly not enough! (8 and no ccache was good though - did compile it twice like so) The bad news is that I still don't know why the disk-read thrashing was happening for me, but I will default to blame the OOM (even though no swap was active, ie. I swapoff-ed the swap partition earlier) due to previous experience with OOM triggering on bare-metal hardware: I seem to remember SSD disk activity led being full-on during an impending OOM and everything freezing! -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/1c390347-066f-4386-bbed-fff57b171adb%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
