[akka-user] memory issue in akka application

[slowhandblues]

We have an application in Akka and having a problem with increasing heap 
memory that is growing with time and eventually crashing the application. 
On analyzing the heap dump we can see all our log entries are getting 
persisted in the memory. Below is object tree for the same

Akka dispatcher --> Unbounded mail box --> concurrent linked queue --> 
envelop object consisting of all the log information

Our actors are untyped actor and we are using akka default event logger for 
logging

Any idea what could be the problem with this?


Thanks,

Somak

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] memory issue in akka application

[Roland Kuhn]

As Johannes says you are most likely overloading the logging system. 
Unfortunately the default event handlers do not offer protection against this 
but you can easily write your own (start by copying this one 
)
 to use a bounded Mailbox or a bounded internal queue before dispatching to the 
actual Logger (in a non-blocking fashion or in a child Actor).

Regards,

Roland

> 24 jul 2015 kl. 20:13 skrev slowhandblues :
> 
> Hi,
> 
> Thanks for the response. Actually we are not creating any separate actor for 
> log (unless akka creates internally). How do we identify the actor? The way 
> we are trying to write logs is pretty much same as defined in akka 
> documentation (http://doc.akka.io/docs/akka/2.0/java/logging.html) along with 
> logback. 
> 
> On Friday, 24 July 2015 17:22:02 UTC+5:30, slowhandblues wrote:
> We have an application in Akka and having a problem with increasing heap 
> memory that is growing with time and eventually crashing the application. On 
> analyzing the heap dump we can see all our log entries are getting persisted 
> in the memory. Below is object tree for the same
> 
> Akka dispatcher --> Unbounded mail box --> concurrent linked queue --> 
> envelop object consisting of all the log information
> 
> Our actors are untyped actor and we are using akka default event logger for 
> logging
> 
> Any idea what could be the problem with this?
> 
> 
> 
> Thanks,
> 
> Somak
> 
> 
> -- 
> >> Read the docs: http://akka.io/docs/ 
> >> Check the FAQ: 
> >> http://doc.akka.io/docs/akka/current/additional/faq.html 
> >> 
> >> Search the archives: https://groups.google.com/group/akka-user 
> >> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to akka-user+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to akka-user@googlegroups.com 
> .
> Visit this group at http://groups.google.com/group/akka-user 
> .
> For more options, visit https://groups.google.com/d/optout 
> .



Dr. Roland Kuhn
Akka Tech Lead
Typesafe  – Reactive apps on the JVM.
twitter: @rolandkuhn
 

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] memory issue in akka application

[Somak De]

Hi Ronald,

Thanks for your answer. I will definitely explore the approach you have
suggested. In the meantime, as a workaround solution, we have reduced
amount of logging in the code and used standalone org.slf4j.logger instead
of akka.event.logger. That reduced the amount of heap memory growth
drastically. But I have few doubts/question

1. Our application has a very aggressive roadmap this year and as the
application grows, how org.slf4j.logger perform in a highly concurrent
environment? I was assuming that akka logger has been built to handle such
scenarios, so using a stand alone logger will do better?

2. I am curious to understand what exactly you meant by overloading the
logger application. Is it because to many request coming to the logger
mailbox? How using a bounded mailbox(LinkedBlockingQueue) help here, wont
that cause the threads to wait, causing degradation?

To give an overview of amount of login that caused the issue, our
application has pick load of 100 req/min continuously for 3 hours. Average
each req creates 50 log entries. So roughly there is 50 entries in 3
hours. This is in a production application server

Thanks

On Mon, Jul 27, 2015 at 7:16 PM, Roland Kuhn  wrote:

> As Johannes says you are most likely overloading the logging system.
> Unfortunately the default event handlers do not offer protection against
> this but you can easily write your own (start by copying this one
> )
> to use a bounded Mailbox or a bounded internal queue before dispatching to
> the actual Logger (in a non-blocking fashion or in a child Actor).
>
> Regards,
>
> Roland
>
> 24 jul 2015 kl. 20:13 skrev slowhandblues :
>
> Hi,
>
> Thanks for the response. Actually we are not creating any separate actor
> for log (unless akka creates internally). How do we identify the actor? The
> way we are trying to write logs is pretty much same as defined in akka
> documentation (http://doc.akka.io/docs/akka/2.0/java/logging.html) along
> with logback.
>
> On Friday, 24 July 2015 17:22:02 UTC+5:30, slowhandblues wrote:
>>
>> We have an application in Akka and having a problem with increasing heap
>> memory that is growing with time and eventually crashing the application.
>> On analyzing the heap dump we can see all our log entries are getting
>> persisted in the memory. Below is object tree for the same
>>
>> Akka dispatcher --> Unbounded mail box --> concurrent linked queue -->
>> envelop object consisting of all the log information
>>
>> Our actors are untyped actor and we are using akka default event logger
>> for logging
>>
>> Any idea what could be the problem with this?
>>
>>
>> Thanks,
>>
>> Somak
>>
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at http://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> *Dr. Roland Kuhn*
> *Akka Tech Lead*
> Typesafe  – Reactive apps on the JVM.
> twitter: @rolandkuhn
> 
>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] memory issue in akka application

[Roland Kuhn]

No matter which logging solution you choose, if you try to log more messages 
than it can handle then something has to give: either the system’s message 
queues grow out of bounds or you start dropping log messages. If you use a 
bounded mailbox then make sure to set the wait time to 0s because otherwise the 
logging system will slow down your application. If on the other hand your 
“logging” must be done as a matter of correctness (or requirements) then I 
would rather call it auditing; in that case it is entirely reasonable to make 
the business processes wait for its completion.

Regards,

Roland

> 27 jul 2015 kl. 20:42 skrev Somak De :
> 
> Hi Ronald,
> 
> Thanks for your answer. I will definitely explore the approach you have 
> suggested. In the meantime, as a workaround solution, we have reduced amount 
> of logging in the code and used standalone org.slf4j.logger instead of 
> akka.event.logger. That reduced the amount of heap memory growth drastically. 
> But I have few doubts/question 
> 
> 1. Our application has a very aggressive roadmap this year and as the 
> application grows, how org.slf4j.logger perform in a highly concurrent 
> environment? I was assuming that akka logger has been built to handle such 
> scenarios, so using a stand alone logger will do better?
> 
> 2. I am curious to understand what exactly you meant by overloading the 
> logger application. Is it because to many request coming to the logger 
> mailbox? How using a bounded mailbox(LinkedBlockingQueue) help here, wont 
> that cause the threads to wait, causing degradation?
> 
> To give an overview of amount of login that caused the issue, our application 
> has pick load of 100 req/min continuously for 3 hours. Average each req 
> creates 50 log entries. So roughly there is 50 entries in 3 hours. This 
> is in a production application server
> 
> Thanks
> 
> On Mon, Jul 27, 2015 at 7:16 PM, Roland Kuhn  > wrote:
> As Johannes says you are most likely overloading the logging system. 
> Unfortunately the default event handlers do not offer protection against this 
> but you can easily write your own (start by copying this one 
> )
>  to use a bounded Mailbox or a bounded internal queue before dispatching to 
> the actual Logger (in a non-blocking fashion or in a child Actor).
> 
> Regards,
> 
> Roland
> 
>> 24 jul 2015 kl. 20:13 skrev slowhandblues > >:
>> 
>> Hi,
>> 
>> Thanks for the response. Actually we are not creating any separate actor for 
>> log (unless akka creates internally). How do we identify the actor? The way 
>> we are trying to write logs is pretty much same as defined in akka 
>> documentation (http://doc.akka.io/docs/akka/2.0/java/logging.html 
>> ) along with logback. 
>> 
>> On Friday, 24 July 2015 17:22:02 UTC+5:30, slowhandblues wrote:
>> We have an application in Akka and having a problem with increasing heap 
>> memory that is growing with time and eventually crashing the application. On 
>> analyzing the heap dump we can see all our log entries are getting persisted 
>> in the memory. Below is object tree for the same
>> 
>> Akka dispatcher --> Unbounded mail box --> concurrent linked queue --> 
>> envelop object consisting of all the log information
>> 
>> Our actors are untyped actor and we are using akka default event logger for 
>> logging
>> 
>> Any idea what could be the problem with this?
>> 
>> 
>> 
>> Thanks,
>> 
>> Somak
>> 
>> 
>> -- 
>> >> Read the docs: http://akka.io/docs/ 
>> >> Check the FAQ: 
>> >> http://doc.akka.io/docs/akka/current/additional/faq.html 
>> >> 
>> >> Search the archives: https://groups.google.com/group/akka-user 
>> >> 
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to akka-user+unsubscr...@googlegroups.com 
>> .
>> To post to this group, send email to akka-user@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/akka-user 
>> .
>> For more options, visit https://groups.google.com/d/optout 
>> .
> 
> 
> 
> Dr. Roland Kuhn
> Akka Tech Lead
> Typesafe  – Reactive apps on the JVM.
> twitter: @rolandkuhn
>  
> 
> 
> 
> -- 
> >> Read the docs: http://akka.io/docs/ 
> >> Check the FAQ: 
> >> http://doc.akka.io/docs/akka/current/additional/faq.html 
> 

Re: [akka-user] memory issue in akka application

[Jim Hazen]

If you'd like to handle this in logback land and not deal with altering 
akka's logging strategy, you can also configure logback to work in bounded 
space and drop overflows as well.

Take a look at logback's AsyncAppender.

http://logback.qos.ch/manual/appenders.html

You'll configure your logback appender normally and then wrap it in your 
async appender.

For example:


 



-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.