Re: Possible memory leak in Tomcat 8.5.57 Websocket

2020-08-11 Thread Alex Maltinsky
Okay, I've uploaded the images. Here are the links:

https://i.imgur.com/zAohUmm.png
https://i.imgur.com/SVp7x6R.png
https://i.imgur.com/F97A6b3.png

On Tue, 11 Aug 2020 at 20:21, Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Alex,
>
> On 8/11/20 11:47, Alex Maltinsky wrote:
> > Hi Folks
> >
> > We ran into what looks like a memory leak in tomcat 8.5.57 on
> > Ubuntu 18.04 running on Openjdk 11.0.5
> >
> > Our app maintains permanent websocket connections with multiple
> > clients (also written in Java, using the Tyrus websocket client -
> > version 1.13.1). Recently, a few clients began misbehaving. They're
> > opening multiple websocket sessions in parallel which our app
> > doesn't allow. When this happens, the app closes the previous
> > websocket session and starts talking to the client over the new one
> > session exclusively. This issue happened to two of our clients that
> > have a bunch of network connectivity issues which somehow triggered
> > this behavior. However, I'm not here to debug the misbehaving
> > clients, the client issue is being investigated in parallel.
> > However, it appears that this behavior triggered a memory leak in
> > tomcat. Slowly but surely our heap keeps filling up with byte
> > arrays that seem to have come from the misbehaving clients (we know
> > this from the contents of the arrays). Below are the paths to the
> > gc roots for a few of these arrays that contain data from one such
> > client. The heap dump was taken *over an hour after the client was
> > blocked by its IP address using iptables*. So we believe there's no
> > reason for these buffers to still exist in memory. Please note that
> > the path to the GC roots are all inside tomcat, our app is nowhere
> > to be found.
> >
> > We would appreciate any guidance or suggestions.
> >
> > Thanks!
> >
> > image.png
> >
> > image.png
> >
> > image.png
>
> Your image attachments have been removed. Can you post them elsewhere
> + links in your message, or transcribe them into text?
>
> Alternatively, do you have a simple test case that reproduces the proble
> m?
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8y04IACgkQHPApP6U8
> pFhoSBAAjLc+roLwg8Qj7HVaHXGWOyucUqAsQOa7jMucQTQkMfs3PiuIfkMxvRp0
> SdRU0uK81E3gIjkwtjeACyeqZIxjExI2RIIe/A4kwtqcZmHkvE/wgayaWM4ncLlI
> tIrppKyEOb3G5aRwoRlaJ4TZxcm7EnvDKVLC8XTPZeQUtoKBS7zeVQFUZiW72gkM
> UU2RllLITeFsjTZGZfQ6I++XHOgrOMKhvswkxRl0zwSzBZM4Sg44xBUKQPj6b8iQ
> d3t6Z9Cy2ynTP1idI1reqEXHeajHLcrlG3VwDwOzQhhgN0TXL+5HEzomAFin9eaI
> D59tPPEClBmP1+1C3/Bse8XYOTUdDIzA7EoSEgXsaaQbkMmapUx6XUH1l0CGlzhI
> oMUvzkvmMuM4k/MLkD7elsG3YTORLK+ouuwd5gDd6o3xfSy3FXgsAbYxpRMQ/vcs
> zkuJ9trREe9uKCsb/MCtLx4cms3ndTo/Q44cEmpdHVMCeQNKnBeQzn9icEzJU7ib
> cF6fIHoI6ifLw9v1nzO1Lr/v3LiDm2WoptUZSLOcB0+dhaz+gLCId9gz6A0yw2DQ
> VzufR5whkZezS/6Rm40J6r8jLItrXfUALm6eDmulfGDUU6Y9JjQe/oUDdQYArKTE
> lVU7JBCc/4wVrZL6DsfnqDecfprRI01wTjXSJ8IP9s6xega2Czc=
> =R1Cn
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


Possible memory leak in Tomcat 8.5.57 Websocket

2020-08-11 Thread Alex Maltinsky
Hi Folks

We ran into what looks like a memory leak in tomcat 8.5.57 on Ubuntu 18.04
running on Openjdk 11.0.5

Our app maintains permanent websocket connections with multiple clients
(also written in Java, using the Tyrus websocket client - version 1.13.1).
Recently, a few clients began misbehaving. They're opening multiple
websocket sessions in parallel which our app doesn't allow. When this
happens, the app closes the previous websocket session and starts talking
to the client over the new one session exclusively. This issue happened to
two of our clients that have a bunch of network connectivity issues which
somehow triggered this behavior. However, I'm not here to debug the
misbehaving clients, the client issue is being investigated in parallel.
However, it appears that this behavior triggered a memory leak in tomcat.
Slowly but surely our heap keeps filling up with byte arrays that seem to
have come from the misbehaving clients (we know this from the contents of
the arrays).
Below are the paths to the gc roots for a few of these arrays that contain
data from one such client. The heap dump was taken *over an hour after the
client was blocked by its IP address using iptables*. So we believe there's
no reason for these buffers to still exist in memory. Please note that the
path to the GC roots are all inside tomcat, our app is nowhere to be found.

We would appreciate any guidance or suggestions.

Thanks!

[image: image.png]

[image: image.png]


[image: image.png]


Fwd: Possible memory leak in Tomcat 8.5.57 Websocket

2020-08-11 Thread Alex Maltinsky
Hi Folks

We ran into what looks like a memory leak in tomcat 8.5.57 on Ubuntu 18.04
running on Openjdk 11.0.5

Our app maintains permanent websocket connections with multiple clients
(also written in Java, using the Tyrus websocket client - version 1.13.1).
Recently, a few clients began misbehaving. They're opening multiple
websocket sessions in parallel which our app doesn't allow. When this
happens, the app closes the previous websocket session and starts talking
to the client over the new one session exclusively. This issue happened to
two of our clients that have a bunch of network connectivity issues which
somehow triggered this behavior. However, I'm not here to debug the
misbehaving clients, the client issue is being investigated in parallel.
However, it appears that this behavior triggered a memory leak in tomcat.
Slowly but surely our heap keeps filling up with byte arrays that seem to
have come from the misbehaving clients (we know this from the contents of
the arrays).
Below are the paths to the gc roots for a few of these arrays that contain
data from one such client. The heap dump was taken *over an hour after the
client was blocked by its IP address using iptables*. So we believe there's
no reason for these buffers to still exist in memory. Please note that the
path to the GC roots are all inside tomcat, our app is nowhere to be found.

We would appreciate any guidance or suggestions.

Thanks!

[image: image.png]

[image: image.png]


[image: image.png]