Re: [dspace-tech] Possible memory leak in dspace.

[Anis]

Hey, thanks for the links, it's just that nothing in our investigation would 
refer to solr leak. Need to go deeper.

We're using dspace 3.2

- Anis

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

Re: [dspace-tech] Possible memory leak in dspace.

[Hardy Pottinger]

Hi, there are a number of issues for you to look into:

https://jira.duraspace.org/browse/DS-2832
https://jira.duraspace.org/browse/DS-2869

(both fixed in DSpace 5.4)

https://jira.duraspace.org/browse/DS-2788

(not yet fixed, but kind of ephemeral)

rambling thoughts on tuning Solr:
http://hardyoyo.thebignow.com/article/49/data-analysis-and-server-optimization-yesterday-was-a-good-day

and a way to tame that aggressive web crawler:
http://hardyoyo.thebignow.com/article/50/manage-all-web-crawler-sessions-like-user-sessions-yes-please

--Hardy

On Fri, Nov 4, 2016 at 10:30 AM, Luiz dos Santos  wrote:

> Hi Anis,
>
>  Sorry, I can't help you, but I am curious about what Hardy have to
> say.
>
> Best regards
> Luiz Claudio Santos
> http://luizclaudiosantos.me/
>
> On Fri, Nov 4, 2016 at 11:17 AM, Hardy Pottinger <
> hardy.pottin...@gmail.com> wrote:
>
>> This is starting to sound familiar. What version of DSpace are you
>> running?
>>
>> --Hardy
>>
>> On Fri, Nov 4, 2016 at 9:29 AM, Anis  wrote:
>>
>>> And anyways, whole bitstreams shouldn't be in DSpace apps memory, no?
>>>
>>> - Anis
>>>
>>>
>>> On Friday, November 4, 2016 at 4:23:05 PM UTC+2, Anis wrote:

 Yes I understand that, but the image above is from a live process, and
 there was no attack happening (this is confirmed), so for some reason there
 is 5 gigs of byte arrays in the memory, tomcats uptime is about 16 hours.

 - Anis

 On Friday, November 4, 2016 at 3:48:13 PM UTC+2, Luiz dos Santos wrote:
>
> Hi Anis,
>
> This seems like you have a lot of bytes "[B" in memory, bitstream
> I guess, maybe this guys stay in the memory all the time because the bot 
> is
> calling them over and over again. I already got have some bots that make 3
> or 5 request per second in the same item, you should try to figure out in
> DSpace log the most frequently ip's. You should see in the DSpace log some
> exception saying that the database is running out of connection as well.
>  If you got the DSpace UI frozen, the restart of tomcat can help.
>
> Best regards
> Luiz
>
> Luiz Claudio Santos
> http://luizclaudiosantos.me/
>
> On Fri, Nov 4, 2016 at 8:16 AM, Anis  wrote:
>
>> We thought so too and we started blocking known violators. The
>> attachment shows the live heap map, and the byte arrays are still
>> dominating, taking 5 out of the 8Gb heap space, these are probably
>> responses to requests, but why are not they GC'd and why are they staying
>> in the memory? Maybe this is normal behaviour, so some clarification 
>> would
>> be nice.
>>
>> - Anis
>>
>>
>> 
>>
>>
>> On Friday, November 4, 2016 at 2:08:34 PM UTC+2, Luiz dos Santos
>> wrote:
>>>
>>> Hi Anis,
>>>
>>>To me it seems more like a bot attack then a leak of memory, do
>>> you have a strategy to deal with malicious bots?
>>>
>>> Best
>>> Luiz
>>>
>>> On Friday, November 4, 2016, Anis  wrote:
>>>
 Hi!


 Our busiest repository was crashing between every 15-30 minutes and
 it seems it was related to out of memory problems and the amount of
 requests DSpace is getting. We captured the heap dump from one of
 our crashes, and as you can see from the image, 78% (3.12GB) of the 
 heap
 space is populated by byte arrays. A lot of these byte arrays are 
 buffered
 1MB arrays that are part of an outputstream from
 HttpServletResponseBufferingWrapper, and to my understanding this
 class should send the 404 response and if you actually check the 
 contents
 of the arrays, they have the 404 HTML code. It also seems that the
 HttpServletResponseBufferingWrappers, that are in the memory are
 never sent, as their response bytesWritten are all 0.

 We resolved the problem by letting Apache handle most of the 404,
 cleaning our theme files from references, that are not found anymore,
 capping out maxdb connections to 50 and maxidle to 30. It is possible, 
 that
 we just had way too many 404 responses to serve, but it would be nice 
 to
 hear if someone else has had or is having this problem and if it 
 happens
 because of the amount of requests, it would be nice to know why.


 - Anis




 


 

Re: [dspace-tech] Possible memory leak in dspace.

[Luiz dos Santos]

Hi Anis,

 Sorry, I can't help you, but I am curious about what Hardy have to say.

Best regards
Luiz Claudio Santos
http://luizclaudiosantos.me/

On Fri, Nov 4, 2016 at 11:17 AM, Hardy Pottinger 
wrote:

> This is starting to sound familiar. What version of DSpace are you running?
>
> --Hardy
>
> On Fri, Nov 4, 2016 at 9:29 AM, Anis  wrote:
>
>> And anyways, whole bitstreams shouldn't be in DSpace apps memory, no?
>>
>> - Anis
>>
>>
>> On Friday, November 4, 2016 at 4:23:05 PM UTC+2, Anis wrote:
>>>
>>> Yes I understand that, but the image above is from a live process, and
>>> there was no attack happening (this is confirmed), so for some reason there
>>> is 5 gigs of byte arrays in the memory, tomcats uptime is about 16 hours.
>>>
>>> - Anis
>>>
>>> On Friday, November 4, 2016 at 3:48:13 PM UTC+2, Luiz dos Santos wrote:

 Hi Anis,

 This seems like you have a lot of bytes "[B" in memory, bitstream I
 guess, maybe this guys stay in the memory all the time because the bot is
 calling them over and over again. I already got have some bots that make 3
 or 5 request per second in the same item, you should try to figure out in
 DSpace log the most frequently ip's. You should see in the DSpace log some
 exception saying that the database is running out of connection as well.
  If you got the DSpace UI frozen, the restart of tomcat can help.

 Best regards
 Luiz

 Luiz Claudio Santos
 http://luizclaudiosantos.me/

 On Fri, Nov 4, 2016 at 8:16 AM, Anis  wrote:

> We thought so too and we started blocking known violators. The
> attachment shows the live heap map, and the byte arrays are still
> dominating, taking 5 out of the 8Gb heap space, these are probably
> responses to requests, but why are not they GC'd and why are they staying
> in the memory? Maybe this is normal behaviour, so some clarification would
> be nice.
>
> - Anis
>
>
> 
>
>
> On Friday, November 4, 2016 at 2:08:34 PM UTC+2, Luiz dos Santos wrote:
>>
>> Hi Anis,
>>
>>To me it seems more like a bot attack then a leak of memory, do
>> you have a strategy to deal with malicious bots?
>>
>> Best
>> Luiz
>>
>> On Friday, November 4, 2016, Anis  wrote:
>>
>>> Hi!
>>>
>>>
>>> Our busiest repository was crashing between every 15-30 minutes and
>>> it seems it was related to out of memory problems and the amount of
>>> requests DSpace is getting. We captured the heap dump from one of
>>> our crashes, and as you can see from the image, 78% (3.12GB) of the heap
>>> space is populated by byte arrays. A lot of these byte arrays are 
>>> buffered
>>> 1MB arrays that are part of an outputstream from
>>> HttpServletResponseBufferingWrapper, and to my understanding this
>>> class should send the 404 response and if you actually check the 
>>> contents
>>> of the arrays, they have the 404 HTML code. It also seems that the
>>> HttpServletResponseBufferingWrappers, that are in the memory are
>>> never sent, as their response bytesWritten are all 0.
>>>
>>> We resolved the problem by letting Apache handle most of the 404,
>>> cleaning our theme files from references, that are not found anymore,
>>> capping out maxdb connections to 50 and maxidle to 30. It is possible, 
>>> that
>>> we just had way too many 404 responses to serve, but it would be nice to
>>> hear if someone else has had or is having this problem and if it happens
>>> because of the amount of requests, it would be nice to know why.
>>>
>>>
>>> - Anis
>>>
>>>
>>>
>>>
>>> 
>>>
>>>
>>> 
>>>
>>>
>>>
>>> 
>>>
>>>
>>> 
>>>
>>>
>>>
>>>
>>
>> --
> You received this message because you are subscribed to the Google
> Groups "DSpace Technical Support" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to dspace-tech...@googlegroups.com.
> To post to this group, send email to dsp

Re: [dspace-tech] Possible memory leak in dspace.

[Hardy Pottinger]

This is starting to sound familiar. What version of DSpace are you running?

--Hardy

On Fri, Nov 4, 2016 at 9:29 AM, Anis  wrote:

> And anyways, whole bitstreams shouldn't be in DSpace apps memory, no?
>
> - Anis
>
>
> On Friday, November 4, 2016 at 4:23:05 PM UTC+2, Anis wrote:
>>
>> Yes I understand that, but the image above is from a live process, and
>> there was no attack happening (this is confirmed), so for some reason there
>> is 5 gigs of byte arrays in the memory, tomcats uptime is about 16 hours.
>>
>> - Anis
>>
>> On Friday, November 4, 2016 at 3:48:13 PM UTC+2, Luiz dos Santos wrote:
>>>
>>> Hi Anis,
>>>
>>> This seems like you have a lot of bytes "[B" in memory, bitstream I
>>> guess, maybe this guys stay in the memory all the time because the bot is
>>> calling them over and over again. I already got have some bots that make 3
>>> or 5 request per second in the same item, you should try to figure out in
>>> DSpace log the most frequently ip's. You should see in the DSpace log some
>>> exception saying that the database is running out of connection as well.
>>>  If you got the DSpace UI frozen, the restart of tomcat can help.
>>>
>>> Best regards
>>> Luiz
>>>
>>> Luiz Claudio Santos
>>> http://luizclaudiosantos.me/
>>>
>>> On Fri, Nov 4, 2016 at 8:16 AM, Anis  wrote:
>>>
 We thought so too and we started blocking known violators. The
 attachment shows the live heap map, and the byte arrays are still
 dominating, taking 5 out of the 8Gb heap space, these are probably
 responses to requests, but why are not they GC'd and why are they staying
 in the memory? Maybe this is normal behaviour, so some clarification would
 be nice.

 - Anis


 


 On Friday, November 4, 2016 at 2:08:34 PM UTC+2, Luiz dos Santos wrote:
>
> Hi Anis,
>
>To me it seems more like a bot attack then a leak of memory, do you
> have a strategy to deal with malicious bots?
>
> Best
> Luiz
>
> On Friday, November 4, 2016, Anis  wrote:
>
>> Hi!
>>
>>
>> Our busiest repository was crashing between every 15-30 minutes and
>> it seems it was related to out of memory problems and the amount of
>> requests DSpace is getting. We captured the heap dump from one of
>> our crashes, and as you can see from the image, 78% (3.12GB) of the heap
>> space is populated by byte arrays. A lot of these byte arrays are 
>> buffered
>> 1MB arrays that are part of an outputstream from
>> HttpServletResponseBufferingWrapper, and to my understanding this
>> class should send the 404 response and if you actually check the contents
>> of the arrays, they have the 404 HTML code. It also seems that the
>> HttpServletResponseBufferingWrappers, that are in the memory are
>> never sent, as their response bytesWritten are all 0.
>>
>> We resolved the problem by letting Apache handle most of the 404,
>> cleaning our theme files from references, that are not found anymore,
>> capping out maxdb connections to 50 and maxidle to 30. It is possible, 
>> that
>> we just had way too many 404 responses to serve, but it would be nice to
>> hear if someone else has had or is having this problem and if it happens
>> because of the amount of requests, it would be nice to know why.
>>
>>
>> - Anis
>>
>>
>>
>>
>> 
>>
>>
>> 
>>
>>
>>
>> 
>>
>>
>> 
>>
>>
>>
>>
>
> --
 You received this message because you are subscribed to the Google
 Groups "DSpace Technical Support" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to dspace-tech...@googlegroups.com.
 To post to this group, send email to dspac...@googlegroups.com.
 Visit this group at https://groups.google.com/group/dspace-tech.
 For more options, visit https://groups.google.com/d/optout.

>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "DSpace Technical Support" group.
> To unsubscribe from this group and stop receivin

Re: [dspace-tech] Possible memory leak in dspace.

[Anis]

And anyways, whole bitstreams shouldn't be in DSpace apps memory, no?

- Anis

On Friday, November 4, 2016 at 4:23:05 PM UTC+2, Anis wrote:
>
> Yes I understand that, but the image above is from a live process, and 
> there was no attack happening (this is confirmed), so for some reason there 
> is 5 gigs of byte arrays in the memory, tomcats uptime is about 16 hours.
>
> - Anis
>
> On Friday, November 4, 2016 at 3:48:13 PM UTC+2, Luiz dos Santos wrote:
>>
>> Hi Anis,
>>
>> This seems like you have a lot of bytes "[B" in memory, bitstream I 
>> guess, maybe this guys stay in the memory all the time because the bot is 
>> calling them over and over again. I already got have some bots that make 3 
>> or 5 request per second in the same item, you should try to figure out in 
>> DSpace log the most frequently ip's. You should see in the DSpace log some 
>> exception saying that the database is running out of connection as well.
>>  If you got the DSpace UI frozen, the restart of tomcat can help.
>>
>> Best regards
>> Luiz 
>>
>> Luiz Claudio Santos
>> http://luizclaudiosantos.me/
>>
>> On Fri, Nov 4, 2016 at 8:16 AM, Anis  wrote:
>>
>>> We thought so too and we started blocking known violators. The 
>>> attachment shows the live heap map, and the byte arrays are still 
>>> dominating, taking 5 out of the 8Gb heap space, these are probably 
>>> responses to requests, but why are not they GC'd and why are they staying 
>>> in the memory? Maybe this is normal behaviour, so some clarification would 
>>> be nice.
>>>
>>> - Anis
>>>
>>>
>>> 
>>>
>>>
>>> On Friday, November 4, 2016 at 2:08:34 PM UTC+2, Luiz dos Santos wrote:

 Hi Anis, 

To me it seems more like a bot attack then a leak of memory, do you 
 have a strategy to deal with malicious bots?

 Best 
 Luiz

 On Friday, November 4, 2016, Anis  wrote:

> Hi!
>
>
> Our busiest repository was crashing between every 15-30 minutes and it 
> seems it was related to out of memory problems and the amount of requests 
> DSpace is getting. We captured the heap dump from one of our crashes, 
> and as you can see from the image, 78% (3.12GB) of the heap space is 
> populated by byte arrays. A lot of these byte arrays are buffered 1MB 
> arrays that are part of an outputstream from 
> HttpServletResponseBufferingWrapper, and to my understanding this class 
> should send the 404 response and if you actually check the contents of 
> the 
> arrays, they have the 404 HTML code. It also seems that the 
> HttpServletResponseBufferingWrappers, that are in the memory are never 
> sent, as their response bytesWritten are all 0.
>
> We resolved the problem by letting Apache handle most of the 404, 
> cleaning our theme files from references, that are not found anymore, 
> capping out maxdb connections to 50 and maxidle to 30. It is possible, 
> that 
> we just had way too many 404 responses to serve, but it would be nice to 
> hear if someone else has had or is having this problem and if it happens 
> because of the amount of requests, it would be nice to know why.
>
>
> - Anis
>
>
>
>
> 
>
>
> 
>
>
>
> 
>
>
> 
>
>
>
>

 -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "DSpace Technical Support" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to dspace-tech...@googlegroups.com.
>>> To post to this group, send email to dspac...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/dspace-tech.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more o

Re: [dspace-tech] Possible memory leak in dspace.

[Anis]

Yes I understand that, but the image above is from a live process, and 
there was no attack happening (this is confirmed), so for some reason there 
is 5 gigs of byte arrays in the memory, tomcats uptime is about 16 hours.

- Anis

On Friday, November 4, 2016 at 3:48:13 PM UTC+2, Luiz dos Santos wrote:
>
> Hi Anis,
>
> This seems like you have a lot of bytes "[B" in memory, bitstream I 
> guess, maybe this guys stay in the memory all the time because the bot is 
> calling them over and over again. I already got have some bots that make 3 
> or 5 request per second in the same item, you should try to figure out in 
> DSpace log the most frequently ip's. You should see in the DSpace log some 
> exception saying that the database is running out of connection as well.
>  If you got the DSpace UI frozen, the restart of tomcat can help.
>
> Best regards
> Luiz 
>
> Luiz Claudio Santos
> http://luizclaudiosantos.me/
>
> On Fri, Nov 4, 2016 at 8:16 AM, Anis > 
> wrote:
>
>> We thought so too and we started blocking known violators. The attachment 
>> shows the live heap map, and the byte arrays are still dominating, taking 5 
>> out of the 8Gb heap space, these are probably responses to requests, but 
>> why are not they GC'd and why are they staying in the memory? Maybe this is 
>> normal behaviour, so some clarification would be nice.
>>
>> - Anis
>>
>>
>> 
>>
>>
>> On Friday, November 4, 2016 at 2:08:34 PM UTC+2, Luiz dos Santos wrote:
>>>
>>> Hi Anis, 
>>>
>>>To me it seems more like a bot attack then a leak of memory, do you 
>>> have a strategy to deal with malicious bots?
>>>
>>> Best 
>>> Luiz
>>>
>>> On Friday, November 4, 2016, Anis  wrote:
>>>
 Hi!


 Our busiest repository was crashing between every 15-30 minutes and it 
 seems it was related to out of memory problems and the amount of requests 
 DSpace is getting. We captured the heap dump from one of our crashes, 
 and as you can see from the image, 78% (3.12GB) of the heap space is 
 populated by byte arrays. A lot of these byte arrays are buffered 1MB 
 arrays that are part of an outputstream from 
 HttpServletResponseBufferingWrapper, and to my understanding this class 
 should send the 404 response and if you actually check the contents of the 
 arrays, they have the 404 HTML code. It also seems that the 
 HttpServletResponseBufferingWrappers, that are in the memory are never 
 sent, as their response bytesWritten are all 0.

 We resolved the problem by letting Apache handle most of the 404, 
 cleaning our theme files from references, that are not found anymore, 
 capping out maxdb connections to 50 and maxidle to 30. It is possible, 
 that 
 we just had way too many 404 responses to serve, but it would be nice to 
 hear if someone else has had or is having this problem and if it happens 
 because of the amount of requests, it would be nice to know why.


 - Anis




 


 



 


 




>>>
>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "DSpace Technical Support" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to dspace-tech...@googlegroups.com .
>> To post to this group, send email to dspac...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/dspace-tech.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

Re: [dspace-tech] Possible memory leak in dspace.

[Luiz dos Santos]

Hi Anis,

This seems like you have a lot of bytes "[B" in memory, bitstream I
guess, maybe this guys stay in the memory all the time because the bot is
calling them over and over again. I already got have some bots that make 3
or 5 request per second in the same item, you should try to figure out in
DSpace log the most frequently ip's. You should see in the DSpace log some
exception saying that the database is running out of connection as well.
 If you got the DSpace UI frozen, the restart of tomcat can help.

Best regards
Luiz

Luiz Claudio Santos
http://luizclaudiosantos.me/

On Fri, Nov 4, 2016 at 8:16 AM, Anis  wrote:

> We thought so too and we started blocking known violators. The attachment
> shows the live heap map, and the byte arrays are still dominating, taking 5
> out of the 8Gb heap space, these are probably responses to requests, but
> why are not they GC'd and why are they staying in the memory? Maybe this is
> normal behaviour, so some clarification would be nice.
>
> - Anis
>
>
> 
>
>
> On Friday, November 4, 2016 at 2:08:34 PM UTC+2, Luiz dos Santos wrote:
>>
>> Hi Anis,
>>
>>To me it seems more like a bot attack then a leak of memory, do you
>> have a strategy to deal with malicious bots?
>>
>> Best
>> Luiz
>>
>> On Friday, November 4, 2016, Anis  wrote:
>>
>>> Hi!
>>>
>>>
>>> Our busiest repository was crashing between every 15-30 minutes and it
>>> seems it was related to out of memory problems and the amount of requests
>>> DSpace is getting. We captured the heap dump from one of our crashes,
>>> and as you can see from the image, 78% (3.12GB) of the heap space is
>>> populated by byte arrays. A lot of these byte arrays are buffered 1MB
>>> arrays that are part of an outputstream from 
>>> HttpServletResponseBufferingWrapper,
>>> and to my understanding this class should send the 404 response and if you
>>> actually check the contents of the arrays, they have the 404 HTML code. It
>>> also seems that the HttpServletResponseBufferingWrappers, that are in
>>> the memory are never sent, as their response bytesWritten are all 0.
>>>
>>> We resolved the problem by letting Apache handle most of the 404,
>>> cleaning our theme files from references, that are not found anymore,
>>> capping out maxdb connections to 50 and maxidle to 30. It is possible, that
>>> we just had way too many 404 responses to serve, but it would be nice to
>>> hear if someone else has had or is having this problem and if it happens
>>> because of the amount of requests, it would be nice to know why.
>>>
>>>
>>> - Anis
>>>
>>>
>>>
>>>
>>> 
>>>
>>>
>>> 
>>>
>>>
>>>
>>> 
>>>
>>>
>>> 
>>>
>>>
>>>
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "DSpace Technical Support" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dspace-tech+unsubscr...@googlegroups.com.
> To post to this group, send email to dspace-tech@googlegroups.com.
> Visit this group at https://groups.google.com/group/dspace-tech.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

Re: [dspace-tech] Possible memory leak in dspace.

[Anis]

We thought so too and we started blocking known violators. The attachment 
shows the live heap map, and the byte arrays are still dominating, taking 5 
out of the 8Gb heap space, these are probably responses to requests, but 
why are not they GC'd and why are they staying in the memory? Maybe this is 
normal behaviour, so some clarification would be nice.

- Anis




On Friday, November 4, 2016 at 2:08:34 PM UTC+2, Luiz dos Santos wrote:
>
> Hi Anis, 
>
>To me it seems more like a bot attack then a leak of memory, do you 
> have a strategy to deal with malicious bots?
>
> Best 
> Luiz
>
> On Friday, November 4, 2016, Anis > 
> wrote:
>
>> Hi!
>>
>>
>> Our busiest repository was crashing between every 15-30 minutes and it 
>> seems it was related to out of memory problems and the amount of requests 
>> DSpace is getting. We captured the heap dump from one of our crashes, 
>> and as you can see from the image, 78% (3.12GB) of the heap space is 
>> populated by byte arrays. A lot of these byte arrays are buffered 1MB 
>> arrays that are part of an outputstream from 
>> HttpServletResponseBufferingWrapper, and to my understanding this class 
>> should send the 404 response and if you actually check the contents of the 
>> arrays, they have the 404 HTML code. It also seems that the 
>> HttpServletResponseBufferingWrappers, that are in the memory are never 
>> sent, as their response bytesWritten are all 0.
>>
>> We resolved the problem by letting Apache handle most of the 404, 
>> cleaning our theme files from references, that are not found anymore, 
>> capping out maxdb connections to 50 and maxidle to 30. It is possible, that 
>> we just had way too many 404 responses to serve, but it would be nice to 
>> hear if someone else has had or is having this problem and if it happens 
>> because of the amount of requests, it would be nice to know why.
>>
>>
>> - Anis
>>
>>
>>
>>
>> 
>>
>>
>> 
>>
>>
>>
>> 
>>
>>
>> 
>>
>>
>>
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

Re: [dspace-tech] Possible memory leak in dspace.

[Luiz dos Santos]

Hi Anis,

   To me it seems more like a bot attack then a leak of memory, do you have
a strategy to deal with malicious bots?

Best
Luiz

On Friday, November 4, 2016, Anis  wrote:

> Hi!
>
>
> Our busiest repository was crashing between every 15-30 minutes and it
> seems it was related to out of memory problems and the amount of requests
> DSpace is getting. We captured the heap dump from one of our crashes, and
> as you can see from the image, 78% (3.12GB) of the heap space is populated
> by byte arrays. A lot of these byte arrays are buffered 1MB arrays that are
> part of an outputstream from HttpServletResponseBufferingWrapper, and to
> my understanding this class should send the 404 response and if you
> actually check the contents of the arrays, they have the 404 HTML code. It
> also seems that the HttpServletResponseBufferingWrappers, that are in the
> memory are never sent, as their response bytesWritten are all 0.
>
> We resolved the problem by letting Apache handle most of the 404, cleaning
> our theme files from references, that are not found anymore, capping out
> maxdb connections to 50 and maxidle to 30. It is possible, that we just had
> way too many 404 responses to serve, but it would be nice to hear if
> someone else has had or is having this problem and if it happens because of
> the amount of requests, it would be nice to know why.
>
>
> - Anis
>
>
>
>
> 
>
>
> 
>
>
>
> 
>
>
> 
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "DSpace Technical Support" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dspace-tech+unsubscr...@googlegroups.com
> 
> .
> To post to this group, send email to dspace-tech@googlegroups.com
> .
> Visit this group at https://groups.google.com/group/dspace-tech.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Luiz Claudio Santos
http://luizclaudiosantos.me/

-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

[dspace-tech] Possible memory leak in dspace.

[Anis]

 

Hi!


Our busiest repository was crashing between every 15-30 minutes and it 
seems it was related to out of memory problems and the amount of requests 
DSpace is getting. We captured the heap dump from one of our crashes, and 
as you can see from the image, 78% (3.12GB) of the heap space is populated 
by byte arrays. A lot of these byte arrays are buffered 1MB arrays that are 
part of an outputstream from HttpServletResponseBufferingWrapper, and to my 
understanding this class should send the 404 response and if you actually 
check the contents of the arrays, they have the 404 HTML code. It also 
seems that the HttpServletResponseBufferingWrappers, that are in the memory 
are never sent, as their response bytesWritten are all 0.

We resolved the problem by letting Apache handle most of the 404, cleaning 
our theme files from references, that are not found anymore, capping out 
maxdb connections to 50 and maxidle to 30. It is possible, that we just had 
way too many 404 responses to serve, but it would be nice to hear if 
someone else has had or is having this problem and if it happens because of 
the amount of requests, it would be nice to know why.


- Anis














-- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to dspace-tech+unsubscr...@googlegroups.com.
To post to this group, send email to dspace-tech@googlegroups.com.
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.