[
https://issues.apache.org/jira/browse/ODE-943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501246#comment-13501246
]
Mateusz Nowakowski commented on ODE-943:
----------------------------------------
After running the test with the workaround provider - there is still a memory
leak.
If the traffic is high, i mean that one of these condition in
SimpleScheduler.doLoadImmediate() are fulfilled :
line 683: if (_outstandingJobs.size() > _todoLimit/2) return true;
or
lines 687-694:
// don't load more than we can chew
final int batch = Math.min((int) (_immediateInterval * _tps /
1000), _todoLimit-_outstandingJobs.size());
// jobs might have been enqueued by #addTodoOnCommit meanwhile
if (batch<=0) {
if (__log.isDebugEnabled()) __log.debug("Max capacity reached:
"+_outstandingJobs.size()+" jobs dispacthed i.e. queued or being executed");
return true;
}
it is likely that line _processedSinceLastLoadTask.clear() is never executed,
so the leak is the same.
If the traffic goes down everything is cleaned - but it depends on the traffic
profile and it is dangerous (potential OOM)
There is another workaround:
Turn off this "defence" mechanism by adding these lines :
ode.scheduler.queueLength=2147483647
ode.scheduler.transactionsPerSecond=1000
to
etc/org.apache.ode.jbi.cfg
But in the end SimpleScheduler needs to rewritten b/c this mechanism is
potential killer.
> NoClassDefFoundError for org.apache.log4j.helpers.AbsoluteTimeDateFormat in
> SimpleScheduler.doLoadImmediate() leads memory leak
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: ODE-943
> URL: https://issues.apache.org/jira/browse/ODE-943
> Project: ODE
> Issue Type: Bug
> Components: BPEL Runtime, JBI Integration
> Affects Versions: 1.3.5, 1.4
> Environment: Ubuntu 11.04 64bit, Oracle JDK 1.6, Servicemix 4.3.0
> Reporter: Abdulkadir Yaman
> Fix For: 1.3.6, 1.4
>
>
> On a fresh installation of Servicemix 4.3.0 does not expose
> org.apache.log4j.helpers by default although Pax logging bundle contains this
> package. At line:707 of Ode_1.3.5 version and line:732 of trunk,
> AbsoluteTimeDateFormat f = new AbsoluteTimeDateFormat(); line leads to
> NoClassDefFoundError for org.apache.log4j.helpers.AbsoluteTimeDateFormat. As
> this error is a subclass of throwable, it is not caught in catch() block.
> This issue leads to huge memory leak under load as
> _processedSinceLastLoadTask.clear(); line can not be reached ever, 1 million
> requests will make your 1gb heap size drained and get your servicemix in a
> fullgc cycles.
> I worked around this issue by wrapping log4j jar into servicemix by karaf@
> osgi:install -s wrap:mvn:log4j/log4j/1.2.13 , and restarting whole system.
> Also log4j dependencies are imported as resolution:optional in MANIFEST-MF in
> ode-jbi bundle.
> To reproduce;
> 1 - reduce heap size in servicemix executable
> 2 - add jvm option : -verbosegc to watch gc and fullgc cycles on karaf
> console or -Xloggc:somefile.out for tailing a file for output
> 3 - set KARAF_DEBUG=true in case you want to debug and see
> _processedSinceLastLoadTask.size() shows increasing number of entries
> 3 - download servicemix 4.3.0 from apache
> 4 - in karaf console, features:install ode
> 5 - deploy a simple bpel flow, features:install examples-ode-ping-pong
> 6 - generate load either via JMeter or Soapui
> 7 - take a heap dump either via jmap or via JConsole finding mbean
> com.sun.management-->HotSpotDiagnostic--> operations -->
> dumpHeap(heapdump.snapshot) , you can find dump file under $SMX_HOME
> 8 - analyse it via yourkit profiler or jprofiler, whatever suits you, you
> will see one object retain most of the memory, which is
> org/apache/ode/scheduler/simple/SimpleScheduler on object explorer window.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
