Hello, I’m running guacamole successfully with more than 40 concurrent RDP users to some 700+ VM’s. Never experienced any kind of performance/resource depletion issues.
Today I configure a simple SSH connection, with user/password authentication. Guacd is being killed by kernel with OOM as soon as I try to connect to the SSH connection. I’m trying this on one of our failover servers which has literally 0 other users on it. Can you please advise what’s wrong? Below is the output of syslog. Regards, Bogdan Nov 6 00:22:36 guacamole-2 guacd[121853]: Creating new client for protocol "ssh" Nov 6 00:22:36 guacamole-2 guacd[121853]: Connection ID is "$5c70fb7d-aad8-4e00-8733-23a04a037155" Nov 6 00:22:37 guacamole-2 guacd[129389]: User "@580d952b-5f78-4c69-8f20-de4fb16e3c7d" joined connection "$5c70fb7d-aad8-4e00-8733-23a04a037155" (1 users now present) Nov 6 00:22:48 guacamole-2 kernel: [19022155.396995] guacd invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397000] CPU: 1 PID: 129393 Comm: guacd Not tainted 5.0.0-1035-azure #37-Ubuntu Nov 6 00:22:48 guacamole-2 kernel: [19022155.397001] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007 06/02/2017 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397002] Call Trace: Nov 6 00:22:48 guacamole-2 kernel: [19022155.397009] dump_stack+0x57/0x75 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397012] dump_header+0x57/0x315 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397015] ? sched_clock_local+0x17/0x90 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397017] oom_kill_process+0x254/0x280 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397019] out_of_memory+0x11b/0x510 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397023] __alloc_pages_slowpath+0xb32/0xe80 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397026] ? blk_flush_plug_list+0xd1/0x100 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397029] __alloc_pages_nodemask+0x2a7/0x2c0 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397033] alloc_pages_current+0x6a/0xe0 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397037] __page_cache_alloc+0x6a/0xa0 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397039] filemap_fault+0x395/0x830 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397041] ? xas_load+0xc/0x80 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397042] ? xas_find+0x15b/0x1a0 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397044] ? filemap_map_pages+0x18b/0x380 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397049] ext4_filemap_fault+0x31/0x44 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397052] __do_fault+0x57/0x115 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397054] __handle_mm_fault+0xe00/0x1340 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397056] handle_mm_fault+0xcd/0x230 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397059] __do_page_fault+0x291/0x4c0 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397061] do_page_fault+0x31/0x110 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397065] ? page_fault+0x8/0x30 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397067] page_fault+0x1e/0x30 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397069] RIP: 0033:0x7fcaf84927e8 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397075] Code: Bad RIP value. Nov 6 00:22:48 guacamole-2 kernel: [19022155.397076] RSP: 002b:00007fcaf96a8c00 EFLAGS: 00010206 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397077] RAX: 00007fc3fd2fd7d0 RBX: 00007fca61a78d90 RCX: 00007fcaec000020 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397078] RDX: 00007fc3fd2fd7d0 RSI: 00007fc3fd2ff7d0 RDI: 00007fcafd2d0c40 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397079] RBP: 00007fca67fff800 R08: 0000000001300000 R09: 0000000000002841 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397080] R10: fffffffffffff000 R11: 00000000012fe000 R12: 00007fcaec0218f0 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397081] R13: 0000000000000a00 R14: 000000000000000e R15: 00007fcaf4003d40 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397083] Mem-Info: Nov 6 00:22:48 guacamole-2 kernel: [19022155.397087] active_anon:3849716 inactive_anon:514 isolated_anon:0 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397087] active_file:188 inactive_file:312 isolated_file:0 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397087] unevictable:0 dirty:0 writeback:19 unstable:0 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397087] slab_reclaimable:44859 slab_unreclaimable:44064 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397087] mapped:508 shmem:635 pagetables:16980 bounce:0 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397087] free:35804 free_pcp:62 free_cma:0 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397091] Node 0 active_anon:15398864kB inactive_anon:2056kB active_file:752kB inactive_file:1248kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:2032kB dirty:0kB writeback:76kB shmem:2540kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 1452032kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no Nov 6 00:22:48 guacamole-2 kernel: [19022155.397092] Node 0 DMA free:15908kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Nov 6 00:22:48 guacamole-2 kernel: [19022155.397095] lowmem_reserve[]: 0 924 15976 15976 15976 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397098] Node 0 DMA32 free:63880kB min:3908kB low:4884kB high:5860kB active_anon:898560kB inactive_anon:0kB active_file:12kB inactive_file:152kB unevictable:0kB writepending:0kB present:1032128kB managed:966592kB mlocked:0kB kernel_stack:0kB pagetables:3508kB bounce:0kB free_pcp:248kB local_pcp:248kB free_cma:0kB Nov 6 00:22:48 guacamole-2 kernel: [19022155.397101] lowmem_reserve[]: 0 0 15051 15051 15051 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397103] Node 0 Normal free:63428kB min:63608kB low:79508kB high:95408kB active_anon:14500044kB inactive_anon:2056kB active_file:1008kB inactive_file:928kB unevictable:0kB writepending:108kB present:15728640kB managed:15419444kB mlocked:0kB kernel_stack:9728kB pagetables:64412kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Nov 6 00:22:48 guacamole-2 kernel: [19022155.397106] lowmem_reserve[]: 0 0 0 0 0 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397108] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB Nov 6 00:22:48 guacamole-2 kernel: [19022155.397116] Node 0 DMA32: 6*4kB (U) 6*8kB (U) 7*16kB (U) 6*32kB (U) 8*64kB (UM) 3*128kB (UM) 3*256kB (UM) 3*512kB (UM) 1*1024kB (M) 1*2048kB (U) 14*4096kB (M) = 63992kB Nov 6 00:22:48 guacamole-2 kernel: [19022155.397124] Node 0 Normal: 703*4kB (UME) 1009*8kB (ME) 709*16kB (ME) 667*32kB (ME) 178*64kB (E) 18*128kB (ME) 3*256kB (UME) 2*512kB (UM) 1*1024kB (M) 0*2048kB 1*4096kB (U) = 64180kB Nov 6 00:22:48 guacamole-2 kernel: [19022155.397133] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Nov 6 00:22:48 guacamole-2 kernel: [19022155.397134] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Nov 6 00:22:48 guacamole-2 kernel: [19022155.397134] 1168 total pagecache pages Nov 6 00:22:48 guacamole-2 kernel: [19022155.397136] 0 pages in swap cache Nov 6 00:22:48 guacamole-2 kernel: [19022155.397137] Swap cache stats: add 0, delete 0, find 0/0 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397137] Free swap = 0kB Nov 6 00:22:48 guacamole-2 kernel: [19022155.397138] Total swap = 0kB Nov 6 00:22:48 guacamole-2 kernel: [19022155.397138] 4194190 pages RAM Nov 6 00:22:48 guacamole-2 kernel: [19022155.397139] 0 pages HighMem/MovableOnly Nov 6 00:22:48 guacamole-2 kernel: [19022155.397140] 93704 pages reserved Nov 6 00:22:48 guacamole-2 kernel: [19022155.397140] 0 pages cma reserved Nov 6 00:22:48 guacamole-2 kernel: [19022155.397141] 0 pages hwpoisoned Nov 6 00:22:48 guacamole-2 kernel: [19022155.397141] Tasks state (memory values in pages): Nov 6 00:22:48 guacamole-2 kernel: [19022155.397142] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name Nov 6 00:22:48 guacamole-2 kernel: [19022155.397147] [ 524] 0 524 26475 45 102400 0 0 lvmetad Nov 6 00:22:48 guacamole-2 kernel: [19022155.397149] [ 1365] 0 1365 17686 219 176128 0 0 systemd-logind Nov 6 00:22:48 guacamole-2 kernel: [19022155.397151] [ 1389] 103 1389 12598 250 147456 0 -900 dbus-daemon Nov 6 00:22:48 guacamole-2 kernel: [19022155.397153] [ 1426] 0 1426 42708 1960 233472 0 0 networkd-dispat Nov 6 00:22:48 guacamole-2 kernel: [19022155.397155] [ 1433] 0 1433 95669 146 106496 0 0 lxcfs Nov 6 00:22:48 guacamole-2 kernel: [19022155.397157] [ 1449] 0 1449 7961 77 110592 0 0 cron Nov 6 00:22:48 guacamole-2 kernel: [19022155.397159] [ 1464] 0 1464 27627 93 114688 0 0 irqbalance Nov 6 00:22:48 guacamole-2 kernel: [19022155.397161] [ 1499] 0 1499 7082 52 102400 0 0 atd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397163] [ 1511] 0 1511 46918 1974 258048 0 0 unattended-upgr Nov 6 00:22:48 guacamole-2 kernel: [19022155.397165] [ 1514] 0 1514 4102 36 77824 0 0 agetty Nov 6 00:22:48 guacamole-2 kernel: [19022155.397166] [ 1528] 0 1528 3721 33 69632 0 0 agetty Nov 6 00:22:48 guacamole-2 kernel: [19022155.397168] [ 1641] 0 1641 72220 230 200704 0 0 polkitd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397170] [ 1665] 0 1665 18074 189 184320 0 -1000 sshd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397171] [ 1840] 0 1840 1126 16 53248 0 0 none Nov 6 00:22:48 guacamole-2 kernel: [19022155.397173] [ 4714] 0 4714 8039 127 102400 0 0 omiserver Nov 6 00:22:48 guacamole-2 kernel: [19022155.397175] [ 4718] 999 4718 8070 147 106496 0 0 omiengine Nov 6 00:22:48 guacamole-2 kernel: [19022155.397177] [ 5017] 998 5017 51718 9941 450560 0 0 omsagent Nov 6 00:22:48 guacamole-2 kernel: [19022155.397179] [ 5171] 0 5171 35464 2748 188416 0 0 python2 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397181] [ 5265] 0 5265 618015 11983 606208 0 0 mdsd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397182] [ 5319] 0 5319 97667 302 184320 0 0 omiagent Nov 6 00:22:48 guacamole-2 kernel: [19022155.397184] [ 14981] 0 14981 48591 388 233472 0 0 nginx Nov 6 00:22:48 guacamole-2 kernel: [19022155.397186] [ 14982] 33 14982 49417 803 274432 0 0 nginx Nov 6 00:22:48 guacamole-2 kernel: [19022155.397188] [ 14983] 33 14983 49635 1218 274432 0 0 nginx Nov 6 00:22:48 guacamole-2 kernel: [19022155.397190] [ 14984] 33 14984 49360 553 266240 0 0 nginx Nov 6 00:22:48 guacamole-2 kernel: [19022155.397191] [ 14985] 33 14985 49353 584 270336 0 0 nginx Nov 6 00:22:48 guacamole-2 kernel: [19022155.397193] [ 117573] 102 117573 66844 1486 172032 0 0 rsyslogd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397195] [ 5622] 0 5622 28679 3652 114688 0 0 mysqld_exporter Nov 6 00:22:48 guacamole-2 kernel: [19022155.397197] [ 108816] 100 108816 20043 184 188416 0 0 systemd-network Nov 6 00:22:48 guacamole-2 kernel: [19022155.397199] [ 108851] 101 108851 17807 294 184320 0 0 systemd-resolve Nov 6 00:22:48 guacamole-2 kernel: [19022155.397201] [ 108895] 62583 108895 35483 150 180224 0 0 systemd-timesyn Nov 6 00:22:48 guacamole-2 kernel: [19022155.397203] [ 108933] 0 108933 80803 3024 651264 0 0 systemd-journal Nov 6 00:22:48 guacamole-2 kernel: [19022155.397204] [ 110156] 0 110156 10780 283 114688 0 -1000 systemd-udevd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397207] [ 110483] 0 110483 19949 3122 200704 0 0 python3 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397209] [ 108720] 114 108720 626580 88768 1138688 0 0 mysqld Nov 6 00:22:48 guacamole-2 kernel: [19022155.397210] [ 14752] 0 14752 60042 4409 245760 0 0 python3 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397212] [ 33218] 106 33218 7148 46 90112 0 0 uuidd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397214] [ 118237] 0 118237 13541 283 143360 0 0 haproxy Nov 6 00:22:48 guacamole-2 kernel: [19022155.397216] [ 118250] 113 118250 13831 543 135168 0 0 haproxy Nov 6 00:22:48 guacamole-2 kernel: [19022155.397218] [ 118841] 115 118841 179471 2445 159744 0 0 prometheus-node Nov 6 00:22:48 guacamole-2 kernel: [19022155.397220] [ 118904] 0 118904 326804 9539 454656 0 0 filebeat Nov 6 00:22:48 guacamole-2 kernel: [19022155.397222] [ 121853] 0 121853 535671 2887 327680 0 0 guacd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397224] [ 26207] 0 26207 91755 451 294912 0 0 packagekitd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397226] [ 10112] 0 10112 29615 163 126976 0 0 bdregd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397228] [ 10120] 0 10120 179554 129 159744 0 0 bdlogd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397230] [ 10137] 0 10137 314988 2169 290816 0 0 epagd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397232] [ 10174] 0 10174 253276 5745 278528 0 0 bdepsecd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397233] [ 10184] 0 10184 47409 131 122880 0 0 bdmond Nov 6 00:22:48 guacamole-2 kernel: [19022155.397235] [ 10244] 0 10244 68044 9237 221184 0 0 bdupdated Nov 6 00:22:48 guacamole-2 kernel: [19022155.397237] [ 10260] 0 10260 857586 211994 2531328 0 0 bdsrvscand.bin Nov 6 00:22:48 guacamole-2 kernel: [19022155.397239] [ 7652] 0 7652 3035 234 69632 0 0 hv_kvp_daemon Nov 6 00:22:48 guacamole-2 kernel: [19022155.397241] [ 14745] 0 14745 72000 221 196608 0 0 accounts-daemon Nov 6 00:22:48 guacamole-2 kernel: [19022155.397244] [ 128855] 0 128855 28035 260 258048 0 0 sshd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397245] [ 128862] 1000 128862 19202 298 188416 0 0 systemd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397248] [ 128866] 1000 128866 65443 735 270336 0 0 (sd-pam) Nov 6 00:22:48 guacamole-2 kernel: [19022155.397250] [ 129012] 1000 129012 28066 268 253952 0 0 sshd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397252] [ 129013] 1000 129013 5803 428 90112 0 0 bash Nov 6 00:22:48 guacamole-2 kernel: [19022155.397253] [ 129027] 0 129027 17599 139 184320 0 0 sudo Nov 6 00:22:48 guacamole-2 kernel: [19022155.397255] [ 129028] 0 129028 16906 121 180224 0 0 su Nov 6 00:22:48 guacamole-2 kernel: [19022155.397257] [ 129033] 0 129033 5810 446 86016 0 0 bash Nov 6 00:22:48 guacamole-2 kernel: [19022155.397259] [ 129151] 111 129151 1972200 99471 1490944 0 0 java Nov 6 00:22:48 guacamole-2 kernel: [19022155.397260] [ 129263] 0 129263 1989 16 61440 0 0 tail Nov 6 00:22:48 guacamole-2 kernel: [19022155.397262] [ 129315] 0 129315 1989 16 57344 0 0 tail Nov 6 00:22:48 guacamole-2 kernel: [19022155.397264] [ 129389] 0 129389 7285214 3363183 54263808 0 0 guacd Nov 6 00:22:48 guacamole-2 kernel: [19022155.397265] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/guacd.service,task=guacd,pid=129389,uid=0 Nov 6 00:22:48 guacamole-2 kernel: [19022155.397272] Out of memory: Kill process 129389 (guacd) score 823 or sacrifice child Nov 6 00:22:48 guacamole-2 kernel: [19022155.417111] Killed process 129389 (guacd) total-vm:29140856kB, anon-rss:13452732kB, file-rss:0kB, shmem-rss:0kB Nov 6 00:22:48 guacamole-2 kernel: [19022155.768328] oom_reaper: reaped process 129389 (guacd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Nov 6 00:22:49 guacamole-2 guacd[121853]: Connection "$5c70fb7d-aad8-4e00-8733-23a04a037155" removed.
