Re: Possible memory leak in Tomcat 8.5.57 Websocket
Okay, I've uploaded the images. Here are the links: https://i.imgur.com/zAohUmm.png https://i.imgur.com/SVp7x6R.png https://i.imgur.com/F97A6b3.png On Tue, 11 Aug 2020 at 20:21, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Alex, > > On 8/11/20 11:47, Alex Maltinsky wrote: > > Hi Folks > > > > We ran into what looks like a memory leak in tomcat 8.5.57 on > > Ubuntu 18.04 running on Openjdk 11.0.5 > > > > Our app maintains permanent websocket connections with multiple > > clients (also written in Java, using the Tyrus websocket client - > > version 1.13.1). Recently, a few clients began misbehaving. They're > > opening multiple websocket sessions in parallel which our app > > doesn't allow. When this happens, the app closes the previous > > websocket session and starts talking to the client over the new one > > session exclusively. This issue happened to two of our clients that > > have a bunch of network connectivity issues which somehow triggered > > this behavior. However, I'm not here to debug the misbehaving > > clients, the client issue is being investigated in parallel. > > However, it appears that this behavior triggered a memory leak in > > tomcat. Slowly but surely our heap keeps filling up with byte > > arrays that seem to have come from the misbehaving clients (we know > > this from the contents of the arrays). Below are the paths to the > > gc roots for a few of these arrays that contain data from one such > > client. The heap dump was taken *over an hour after the client was > > blocked by its IP address using iptables*. So we believe there's no > > reason for these buffers to still exist in memory. Please note that > > the path to the GC roots are all inside tomcat, our app is nowhere > > to be found. > > > > We would appreciate any guidance or suggestions. > > > > Thanks! > > > > image.png > > > > image.png > > > > image.png > > Your image attachments have been removed. Can you post them elsewhere > + links in your message, or transcribe them into text? > > Alternatively, do you have a simple test case that reproduces the proble > m? > > - -chris > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8y04IACgkQHPApP6U8 > pFhoSBAAjLc+roLwg8Qj7HVaHXGWOyucUqAsQOa7jMucQTQkMfs3PiuIfkMxvRp0 > SdRU0uK81E3gIjkwtjeACyeqZIxjExI2RIIe/A4kwtqcZmHkvE/wgayaWM4ncLlI > tIrppKyEOb3G5aRwoRlaJ4TZxcm7EnvDKVLC8XTPZeQUtoKBS7zeVQFUZiW72gkM > UU2RllLITeFsjTZGZfQ6I++XHOgrOMKhvswkxRl0zwSzBZM4Sg44xBUKQPj6b8iQ > d3t6Z9Cy2ynTP1idI1reqEXHeajHLcrlG3VwDwOzQhhgN0TXL+5HEzomAFin9eaI > D59tPPEClBmP1+1C3/Bse8XYOTUdDIzA7EoSEgXsaaQbkMmapUx6XUH1l0CGlzhI > oMUvzkvmMuM4k/MLkD7elsG3YTORLK+ouuwd5gDd6o3xfSy3FXgsAbYxpRMQ/vcs > zkuJ9trREe9uKCsb/MCtLx4cms3ndTo/Q44cEmpdHVMCeQNKnBeQzn9icEzJU7ib > cF6fIHoI6ifLw9v1nzO1Lr/v3LiDm2WoptUZSLOcB0+dhaz+gLCId9gz6A0yw2DQ > VzufR5whkZezS/6Rm40J6r8jLItrXfUALm6eDmulfGDUU6Y9JjQe/oUDdQYArKTE > lVU7JBCc/4wVrZL6DsfnqDecfprRI01wTjXSJ8IP9s6xega2Czc= > =R1Cn > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Possible memory leak in Tomcat 8.5.57 Websocket
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Alex, On 8/11/20 11:47, Alex Maltinsky wrote: > Hi Folks > > We ran into what looks like a memory leak in tomcat 8.5.57 on > Ubuntu 18.04 running on Openjdk 11.0.5 > > Our app maintains permanent websocket connections with multiple > clients (also written in Java, using the Tyrus websocket client - > version 1.13.1). Recently, a few clients began misbehaving. They're > opening multiple websocket sessions in parallel which our app > doesn't allow. When this happens, the app closes the previous > websocket session and starts talking to the client over the new one > session exclusively. This issue happened to two of our clients that > have a bunch of network connectivity issues which somehow triggered > this behavior. However, I'm not here to debug the misbehaving > clients, the client issue is being investigated in parallel. > However, it appears that this behavior triggered a memory leak in > tomcat. Slowly but surely our heap keeps filling up with byte > arrays that seem to have come from the misbehaving clients (we know > this from the contents of the arrays). Below are the paths to the > gc roots for a few of these arrays that contain data from one such > client. The heap dump was taken *over an hour after the client was > blocked by its IP address using iptables*. So we believe there's no > reason for these buffers to still exist in memory. Please note that > the path to the GC roots are all inside tomcat, our app is nowhere > to be found. > > We would appreciate any guidance or suggestions. > > Thanks! > > image.png > > image.png > > image.png Your image attachments have been removed. Can you post them elsewhere + links in your message, or transcribe them into text? Alternatively, do you have a simple test case that reproduces the proble m? - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8y04IACgkQHPApP6U8 pFhoSBAAjLc+roLwg8Qj7HVaHXGWOyucUqAsQOa7jMucQTQkMfs3PiuIfkMxvRp0 SdRU0uK81E3gIjkwtjeACyeqZIxjExI2RIIe/A4kwtqcZmHkvE/wgayaWM4ncLlI tIrppKyEOb3G5aRwoRlaJ4TZxcm7EnvDKVLC8XTPZeQUtoKBS7zeVQFUZiW72gkM UU2RllLITeFsjTZGZfQ6I++XHOgrOMKhvswkxRl0zwSzBZM4Sg44xBUKQPj6b8iQ d3t6Z9Cy2ynTP1idI1reqEXHeajHLcrlG3VwDwOzQhhgN0TXL+5HEzomAFin9eaI D59tPPEClBmP1+1C3/Bse8XYOTUdDIzA7EoSEgXsaaQbkMmapUx6XUH1l0CGlzhI oMUvzkvmMuM4k/MLkD7elsG3YTORLK+ouuwd5gDd6o3xfSy3FXgsAbYxpRMQ/vcs zkuJ9trREe9uKCsb/MCtLx4cms3ndTo/Q44cEmpdHVMCeQNKnBeQzn9icEzJU7ib cF6fIHoI6ifLw9v1nzO1Lr/v3LiDm2WoptUZSLOcB0+dhaz+gLCId9gz6A0yw2DQ VzufR5whkZezS/6Rm40J6r8jLItrXfUALm6eDmulfGDUU6Y9JjQe/oUDdQYArKTE lVU7JBCc/4wVrZL6DsfnqDecfprRI01wTjXSJ8IP9s6xega2Czc= =R1Cn -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Possible memory leak in Tomcat 8.5.57 Websocket
Hi Folks We ran into what looks like a memory leak in tomcat 8.5.57 on Ubuntu 18.04 running on Openjdk 11.0.5 Our app maintains permanent websocket connections with multiple clients (also written in Java, using the Tyrus websocket client - version 1.13.1). Recently, a few clients began misbehaving. They're opening multiple websocket sessions in parallel which our app doesn't allow. When this happens, the app closes the previous websocket session and starts talking to the client over the new one session exclusively. This issue happened to two of our clients that have a bunch of network connectivity issues which somehow triggered this behavior. However, I'm not here to debug the misbehaving clients, the client issue is being investigated in parallel. However, it appears that this behavior triggered a memory leak in tomcat. Slowly but surely our heap keeps filling up with byte arrays that seem to have come from the misbehaving clients (we know this from the contents of the arrays). Below are the paths to the gc roots for a few of these arrays that contain data from one such client. The heap dump was taken *over an hour after the client was blocked by its IP address using iptables*. So we believe there's no reason for these buffers to still exist in memory. Please note that the path to the GC roots are all inside tomcat, our app is nowhere to be found. We would appreciate any guidance or suggestions. Thanks! [image: image.png] [image: image.png] [image: image.png]
Fwd: Possible memory leak in Tomcat 8.5.57 Websocket
Hi Folks We ran into what looks like a memory leak in tomcat 8.5.57 on Ubuntu 18.04 running on Openjdk 11.0.5 Our app maintains permanent websocket connections with multiple clients (also written in Java, using the Tyrus websocket client - version 1.13.1). Recently, a few clients began misbehaving. They're opening multiple websocket sessions in parallel which our app doesn't allow. When this happens, the app closes the previous websocket session and starts talking to the client over the new one session exclusively. This issue happened to two of our clients that have a bunch of network connectivity issues which somehow triggered this behavior. However, I'm not here to debug the misbehaving clients, the client issue is being investigated in parallel. However, it appears that this behavior triggered a memory leak in tomcat. Slowly but surely our heap keeps filling up with byte arrays that seem to have come from the misbehaving clients (we know this from the contents of the arrays). Below are the paths to the gc roots for a few of these arrays that contain data from one such client. The heap dump was taken *over an hour after the client was blocked by its IP address using iptables*. So we believe there's no reason for these buffers to still exist in memory. Please note that the path to the GC roots are all inside tomcat, our app is nowhere to be found. We would appreciate any guidance or suggestions. Thanks! [image: image.png] [image: image.png] [image: image.png]
