Re: Re: guacd process randomly aborts
Hi Nick, We have doubled the memory on this virtual machine from 2GB to 4GB. It has been nearly a month and so far the random signal 6 aborting of the guacd processes have stopped. However, now we're getting a new error message every couple days (see attached). When this happens I need to log in to the server and kill -9 the guacd processes and then restart the process. We had never received this message before doubling the memory. The network in our virtual environment has never been an issue. Any ideas? Thanks! On Fri, Aug 16, 2024 at 3:05 PM Brian Filipek wrote: > Hi Nick, > > This is very helpful by narrowing this down to a memory allocation > problem. I have the "top" application open so I'll keep an eye on it. We > gave 2GB of memory to this particular virtual machine. We never have more > than 5 users connecting at a time and all they use it for is RDP. Looks > like there's a guacd process of about 150MB resident memory for each > connection but also an extra guacd process. I assume that's a parent > process. I'm also going to keep an eye on Tomcat though. It's currently > referencing 2GB of memory but with 236MB resident. If Tomcat has a memory > leak maybe that ends up preventing new guacd connections. > > Thank you for your help! I'll see how these numbers look on Monday. > > Brian > > On Fri, Aug 16, 2024 at 10:49 AM Nick Couchman wrote: > >> On Fri, Aug 16, 2024 at 10:08 AM Brian Filipek >> wrote: >> >>> Hi Nick, >>> >>> This problem continues in FreeBSD with guacd randomly signal 6 aborting >>> approximately once a week.When this happens I need to log in and restart >>> the guacd service. We had a crash last night and here are the last few >>> lines of /var/log/messages: >>> >>> Aug 15 11:53:14 guacamole guacd[17699]: Client did not terminate in a >>> timely manner. Forcibly terminating client and any child processes. >>> Aug 15 15:35:14 guacamole guacd[17752]: Client did not terminate in a >>> timely manner. Forcibly terminating client and any child processes. >>> Aug 15 15:59:19 guacamole guacd[32874]: User is not responding. >>> Aug 16 00:09:22 guacamole guacd[65139]: User is not responding. >>> Aug 16 03:05:28 guacamole guacd[79034]: User is not responding. >>> Aug 16 03:25:22 guacamole kernel: pid 75166 (guacd), jid 0, uid 899: >>> exited on signal 6 (no core dump - other error) >>> Aug 16 03:25:22 guacamole kernel: pid 1758 (guacd), jid 0, uid 899: >>> exited on signal 6 (no core dump - other error) >>> Aug 16 03:25:22 guacamole guacd[67047]: Unable to shutdown internal >>> socket for connection $277e-d090-441a-b2d5-c337bf55b13c. Corresponding >>> process may remain running but inactive. >>> Aug 16 08:20:40 guacamole login[1783]: ROOT LOGIN (root) ON ttyv0 >>> >>> >> I did a quick Google search on "signal 6" and found that the general >> consensus seems to be that this is a "voluntary" termination by the process >> (guacd), usually triggered by a call to the "abort()" function. A quick >> search through the guacd source code shows that the only times we every >> call this function are in the libguac memory allocation functions ( >> https://github.com/apache/guacamole-server/blob/main/src/libguac/mem.c), >> during memory arithmetic processes (add, subtract, multiple), and during >> the realloc process. So, my _guess_ here is that it is failing to allocate >> required memory for some operation and is aborting. >> >> It's possible there's a memory leak somewhere in guacd that is >> manifesting itself in your particular case on a semi-regular basis, but >> hard to say. The interesting things to try to track this down further would >> be: >> * Any information you have on resources utilization - particularly memory >> - that would show any sort of a trend of memory utilization growing up to >> the point where the process crashes, resetting, then growing again, etc. >> * Any information on resource constraints for the process - that is, are >> you running it in such a way that you've limited the available memory to >> that specific process (a FreeBSD jail, or whatever the FreeBSD equivalent >> might be of the Linux "limits" - any limits on heap space, threads, etc. >> * If you can collect a stack trace during the abort, it would be really >> useful to drill down and figure where in the code this is being triggered - >> if it's in one of the four places in the mem.c file that I mentioned, or if >> something else is causing it. >> >> -Nick >> >>> > > -- > > Regards, > > Brian Filipek > > Technical Support Specialist > > 55740 Currant Road > > Mishawaka, IN 46545 > > (574) 855-5745 > > bfili...@betterworldbooks.com > > betterworldbooks.com > > Better World Books is a for-profit social enterprise that collects and > sells books online with each sale generating funds for literacy initiatives > around the world. > > -- Regards, Brian Filipek Technical Support Specialist 55740 Currant Road Mishawaka, IN 46545 (574) 855-5745 bfili...@betterworldbooks.com betterworldbooks.com Bett
Re: Re: guacd process randomly aborts
Hi Nick, This is very helpful by narrowing this down to a memory allocation problem. I have the "top" application open so I'll keep an eye on it. We gave 2GB of memory to this particular virtual machine. We never have more than 5 users connecting at a time and all they use it for is RDP. Looks like there's a guacd process of about 150MB resident memory for each connection but also an extra guacd process. I assume that's a parent process. I'm also going to keep an eye on Tomcat though. It's currently referencing 2GB of memory but with 236MB resident. If Tomcat has a memory leak maybe that ends up preventing new guacd connections. Thank you for your help! I'll see how these numbers look on Monday. Brian On Fri, Aug 16, 2024 at 10:49 AM Nick Couchman wrote: > On Fri, Aug 16, 2024 at 10:08 AM Brian Filipek > wrote: > >> Hi Nick, >> >> This problem continues in FreeBSD with guacd randomly signal 6 aborting >> approximately once a week.When this happens I need to log in and restart >> the guacd service. We had a crash last night and here are the last few >> lines of /var/log/messages: >> >> Aug 15 11:53:14 guacamole guacd[17699]: Client did not terminate in a >> timely manner. Forcibly terminating client and any child processes. >> Aug 15 15:35:14 guacamole guacd[17752]: Client did not terminate in a >> timely manner. Forcibly terminating client and any child processes. >> Aug 15 15:59:19 guacamole guacd[32874]: User is not responding. >> Aug 16 00:09:22 guacamole guacd[65139]: User is not responding. >> Aug 16 03:05:28 guacamole guacd[79034]: User is not responding. >> Aug 16 03:25:22 guacamole kernel: pid 75166 (guacd), jid 0, uid 899: >> exited on signal 6 (no core dump - other error) >> Aug 16 03:25:22 guacamole kernel: pid 1758 (guacd), jid 0, uid 899: >> exited on signal 6 (no core dump - other error) >> Aug 16 03:25:22 guacamole guacd[67047]: Unable to shutdown internal >> socket for connection $277e-d090-441a-b2d5-c337bf55b13c. Corresponding >> process may remain running but inactive. >> Aug 16 08:20:40 guacamole login[1783]: ROOT LOGIN (root) ON ttyv0 >> >> > I did a quick Google search on "signal 6" and found that the general > consensus seems to be that this is a "voluntary" termination by the process > (guacd), usually triggered by a call to the "abort()" function. A quick > search through the guacd source code shows that the only times we every > call this function are in the libguac memory allocation functions ( > https://github.com/apache/guacamole-server/blob/main/src/libguac/mem.c), > during memory arithmetic processes (add, subtract, multiple), and during > the realloc process. So, my _guess_ here is that it is failing to allocate > required memory for some operation and is aborting. > > It's possible there's a memory leak somewhere in guacd that is manifesting > itself in your particular case on a semi-regular basis, but hard to say. > The interesting things to try to track this down further would be: > * Any information you have on resources utilization - particularly memory > - that would show any sort of a trend of memory utilization growing up to > the point where the process crashes, resetting, then growing again, etc. > * Any information on resource constraints for the process - that is, are > you running it in such a way that you've limited the available memory to > that specific process (a FreeBSD jail, or whatever the FreeBSD equivalent > might be of the Linux "limits" - any limits on heap space, threads, etc. > * If you can collect a stack trace during the abort, it would be really > useful to drill down and figure where in the code this is being triggered - > if it's in one of the four places in the mem.c file that I mentioned, or if > something else is causing it. > > -Nick > >> -- Regards, Brian Filipek Technical Support Specialist 55740 Currant Road Mishawaka, IN 46545 (574) 855-5745 bfili...@betterworldbooks.com betterworldbooks.com Better World Books is a for-profit social enterprise that collects and sells books online with each sale generating funds for literacy initiatives around the world.
Re: Re: guacd process randomly aborts
On Fri, Aug 16, 2024 at 10:08 AM Brian Filipek wrote: > Hi Nick, > > This problem continues in FreeBSD with guacd randomly signal 6 aborting > approximately once a week.When this happens I need to log in and restart > the guacd service. We had a crash last night and here are the last few > lines of /var/log/messages: > > Aug 15 11:53:14 guacamole guacd[17699]: Client did not terminate in a > timely manner. Forcibly terminating client and any child processes. > Aug 15 15:35:14 guacamole guacd[17752]: Client did not terminate in a > timely manner. Forcibly terminating client and any child processes. > Aug 15 15:59:19 guacamole guacd[32874]: User is not responding. > Aug 16 00:09:22 guacamole guacd[65139]: User is not responding. > Aug 16 03:05:28 guacamole guacd[79034]: User is not responding. > Aug 16 03:25:22 guacamole kernel: pid 75166 (guacd), jid 0, uid 899: > exited on signal 6 (no core dump - other error) > Aug 16 03:25:22 guacamole kernel: pid 1758 (guacd), jid 0, uid 899: exited > on signal 6 (no core dump - other error) > Aug 16 03:25:22 guacamole guacd[67047]: Unable to shutdown internal socket > for connection $277e-d090-441a-b2d5-c337bf55b13c. Corresponding process > may remain running but inactive. > Aug 16 08:20:40 guacamole login[1783]: ROOT LOGIN (root) ON ttyv0 > > I did a quick Google search on "signal 6" and found that the general consensus seems to be that this is a "voluntary" termination by the process (guacd), usually triggered by a call to the "abort()" function. A quick search through the guacd source code shows that the only times we every call this function are in the libguac memory allocation functions ( https://github.com/apache/guacamole-server/blob/main/src/libguac/mem.c), during memory arithmetic processes (add, subtract, multiple), and during the realloc process. So, my _guess_ here is that it is failing to allocate required memory for some operation and is aborting. It's possible there's a memory leak somewhere in guacd that is manifesting itself in your particular case on a semi-regular basis, but hard to say. The interesting things to try to track this down further would be: * Any information you have on resources utilization - particularly memory - that would show any sort of a trend of memory utilization growing up to the point where the process crashes, resetting, then growing again, etc. * Any information on resource constraints for the process - that is, are you running it in such a way that you've limited the available memory to that specific process (a FreeBSD jail, or whatever the FreeBSD equivalent might be of the Linux "limits" - any limits on heap space, threads, etc. * If you can collect a stack trace during the abort, it would be really useful to drill down and figure where in the code this is being triggered - if it's in one of the four places in the mem.c file that I mentioned, or if something else is causing it. -Nick >
RE: Re: guacd process randomly aborts
Hi Nick, This problem continues in FreeBSD with guacd randomly signal 6 aborting approximately once a week.When this happens I need to log in and restart the guacd service. We had a crash last night and here are the last few lines of /var/log/messages: Aug 15 11:53:14 guacamole guacd[17699]: Client did not terminate in a timely manner. Forcibly terminating client and any child processes. Aug 15 15:35:14 guacamole guacd[17752]: Client did not terminate in a timely manner. Forcibly terminating client and any child processes. Aug 15 15:59:19 guacamole guacd[32874]: User is not responding. Aug 16 00:09:22 guacamole guacd[65139]: User is not responding. Aug 16 03:05:28 guacamole guacd[79034]: User is not responding. Aug 16 03:25:22 guacamole kernel: pid 75166 (guacd), jid 0, uid 899: exited on signal 6 (no core dump - other error) Aug 16 03:25:22 guacamole kernel: pid 1758 (guacd), jid 0, uid 899: exited on signal 6 (no core dump - other error) Aug 16 03:25:22 guacamole guacd[67047]: Unable to shutdown internal socket for connection $277e-d090-441a-b2d5-c337bf55b13c. Corresponding process may remain running but inactive. Aug 16 08:20:40 guacamole login[1783]: ROOT LOGIN (root) ON ttyv0 You can see it aborted at 3:25 this morning and I restarted the service at 8:20. Also here are some lines from the debug.log if helpful: Aug 16 03:05:28 guacamole guacd[79034]: Requesting termination of client... Aug 16 03:05:28 guacamole guacd[79034]: CLIPRDR (clipboard redirection) channel disconnected. Aug 16 03:05:28 guacamole guacd[79034]: SVC "rdpdr" disconnected. Aug 16 03:05:28 guacamole guacd[79034]: SVC "rdpsnd" disconnected. Aug 16 03:05:28 guacamole guacd[79034]: Client terminated successfully. Aug 16 03:05:28 guacamole guacd[1758]: Unable to request termination of client process: No such process Aug 16 03:05:28 guacamole guacd[1758]: All child processes for connection "$e242b101-dc6a-48c8-9ede-eb256ef2472a" have been terminated. Aug 16 03:25:22 guacamole guacd[75166]: ERRINFO_LOGOFF_BY_USER (0x000C):The disconnection was initiated by the user logging off their session on the server. Aug 16 03:25:22 guacamole guacd[75166]: rdp_set_error_info:freerdp_set_last_error_ex ERRINFO_LOGOFF_BY_USER [0x0001000C] Aug 16 03:25:22 guacamole guacd[75166]: Disconnect reason code: 0xC. Aug 16 03:25:22 guacamole guacd[75166]: CLIPRDR (clipboard redirection) channel disconnected. Aug 16 03:25:22 guacamole guacd[75166]: SVC "rdpdr" disconnected. Aug 16 03:25:22 guacamole guacd[75166]: SVC "rdpsnd" disconnected. Aug 16 03:25:22 guacamole guacd[1758]: Unable to request termination of client process: No such process Aug 16 03:25:22 guacamole guacd[1758]: All child processes for connection "$315622cb-41ce-4726-876d-7e2908161755" have been terminated. Aug 16 03:25:22 guacamole guacd[67047]: Requesting termination of client... Aug 16 03:25:22 guacamole guacd[67047]: CLIPRDR (clipboard redirection) channel disconnected. Aug 16 03:25:22 guacamole guacd[67047]: SVC "rdpdr" disconnected. Aug 16 03:25:22 guacamole guacd[67047]: SVC "rdpsnd" disconnected. Aug 16 03:25:22 guacamole guacd[67047]: Client terminated successfully. Aug 16 08:20:46 guacamole guacd[1740]: Successfully bound AF_INET socket to host 127.0.0.1, port 4822 I can send you more parts of the debug.log if you need. This is running Guacamole server 1.5.4 on FreeBSD 14.1. Let me know if there's anything we can adjust. Thank you! On 2023/07/10 18:10:55 Nick Couchman wrote: > On Mon, Jul 10, 2023 at 2:06 PM Brian Filipek > wrote: > > > Hi Nick, > > > > Yes last week I enabled debug logging so we'll see what it says next time > > this happens. > > > > Sounds good. > > > > > > We used to have Guacamole running on Debian but the experience was awful. > > Anytime a user attempted to RDP to a Windows server in which they had a > > running but disconnected session Guacamole would not allow them to connect > > to it. I would then have to remote into whichever Windows server and log > > the user off. Then they would be able to connect to it with Guacamole. It > > works great in FreeBSD other than this problem which seems to happen every > > few weeks. > > > > > Interesting - I'm not sure why running on Linux would cause this, unless > it's something to do with the FreeRDP version. Personally I run on both > CentOS7 and CentOS Stream 8 and routinely disconnect from sessions and > reconnect with no issue at all. > > > > Memory is definitely not an issue here. We aren't doing anything fancy > > with it. According to top each session only uses a few MB. > > > > > Yeah, sounds like a pretty low-traffic use-case. > > -Nick >
Re: guacd process randomly aborts
On Mon, Jul 10, 2023 at 2:06 PM Brian Filipek wrote: > Hi Nick, > > Yes last week I enabled debug logging so we'll see what it says next time > this happens. > Sounds good. > > We used to have Guacamole running on Debian but the experience was awful. > Anytime a user attempted to RDP to a Windows server in which they had a > running but disconnected session Guacamole would not allow them to connect > to it. I would then have to remote into whichever Windows server and log > the user off. Then they would be able to connect to it with Guacamole. It > works great in FreeBSD other than this problem which seems to happen every > few weeks. > > Interesting - I'm not sure why running on Linux would cause this, unless it's something to do with the FreeRDP version. Personally I run on both CentOS7 and CentOS Stream 8 and routinely disconnect from sessions and reconnect with no issue at all. > Memory is definitely not an issue here. We aren't doing anything fancy > with it. According to top each session only uses a few MB. > > Yeah, sounds like a pretty low-traffic use-case. -Nick
Re: guacd process randomly aborts
Hi Nick, Yes last week I enabled debug logging so we'll see what it says next time this happens. We used to have Guacamole running on Debian but the experience was awful. Anytime a user attempted to RDP to a Windows server in which they had a running but disconnected session Guacamole would not allow them to connect to it. I would then have to remote into whichever Windows server and log the user off. Then they would be able to connect to it with Guacamole. It works great in FreeBSD other than this problem which seems to happen every few weeks. Memory is definitely not an issue here. We aren't doing anything fancy with it. According to top each session only uses a few MB. Thank you for your help! On Mon, Jul 10, 2023 at 1:03 PM Nick Couchman wrote: > On Wed, Jul 5, 2023 at 12:26 PM Brian Filipek > wrote: > > > > Greetings! > > > > Approximately once or twice a month our users will put in a trouble > ticket saying they've logged into Guacamole and then receive this error > when trying to RDP to a server: "An internal error has occurred within the > Guacamole server, and the connection has been terminated. If the problem > persists, please notify your system administrator, or check your system > logs." > > > > When this happens I'll log into the Guacamole server and issue a > "service guacd restart" which fixes it. Looking at the system message log > there are always a lot of guacd exited on signal 11 messages, but when this > particular problem happens the message is guacd exited on signal 6. > > You might want to bump up guacd logging to debug and see if anything > more useful shows up. Or, run it in debug in the foreground, maybe > even with a debugger, and see what happens. > > > > > The server is currently running FreeBSD 13.2, although in the past it > had run 13.1, 13.0, and possibly 12.X. All ports are up to date as well. > > Sorry, I've never run it on FreeBSD, always Linux, so no experience, here. > > > > > We give this VM 2GB of RAM and there are never more than 5 people using > Guacamole at a time. > > Yeah, unless something else is contributing to lack of resources > available for guacd, it doesn't sound like a resource issue. > > -Nick > > - > To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org > For additional commands, e-mail: user-h...@guacamole.apache.org > > -- Regards, Brian Filipek Technical Support Specialist 55740 Currant Road Mishawaka, IN 46545 (574) 855-5745 bfili...@betterworldbooks.com betterworldbooks.com Better World Books is a for-profit social enterprise that collects and sells books online with each sale generating funds for literacy initiatives around the world.
Re: guacd process randomly aborts
On Wed, Jul 5, 2023 at 12:26 PM Brian Filipek wrote: > > Greetings! > > Approximately once or twice a month our users will put in a trouble ticket > saying they've logged into Guacamole and then receive this error when trying > to RDP to a server: "An internal error has occurred within the Guacamole > server, and the connection has been terminated. If the problem persists, > please notify your system administrator, or check your system logs." > > When this happens I'll log into the Guacamole server and issue a "service > guacd restart" which fixes it. Looking at the system message log there are > always a lot of guacd exited on signal 11 messages, but when this particular > problem happens the message is guacd exited on signal 6. You might want to bump up guacd logging to debug and see if anything more useful shows up. Or, run it in debug in the foreground, maybe even with a debugger, and see what happens. > > The server is currently running FreeBSD 13.2, although in the past it had run > 13.1, 13.0, and possibly 12.X. All ports are up to date as well. Sorry, I've never run it on FreeBSD, always Linux, so no experience, here. > > We give this VM 2GB of RAM and there are never more than 5 people using > Guacamole at a time. Yeah, unless something else is contributing to lack of resources available for guacd, it doesn't sound like a resource issue. -Nick - To unsubscribe, e-mail: user-unsubscr...@guacamole.apache.org For additional commands, e-mail: user-h...@guacamole.apache.org
guacd process randomly aborts
Greetings! Approximately once or twice a month our users will put in a trouble ticket saying they've logged into Guacamole and then receive this error when trying to RDP to a server: "An internal error has occurred within the Guacamole server, and the connection has been terminated. If the problem persists, please notify your system administrator, or check your system logs." When this happens I'll log into the Guacamole server and issue a "service guacd restart" which fixes it. Looking at the system message log there are always a lot of guacd exited on signal 11 messages, but when this particular problem happens the message is guacd exited on signal 6. The server is currently running FreeBSD 13.2, although in the past it had run 13.1, 13.0, and possibly 12.X. All ports are up to date as well. We give this VM 2GB of RAM and there are never more than 5 people using Guacamole at a time. Has anybody else experienced this running Guacamole on FreeBSD? Thank you!
