Hello All,
We are using camel 3.1 for trading over a FIX connection. Every once in
a while, the app stalls on startup.. We are not seeing any
OutOfMemoryExceptions. We dropped a thread dump and I'm not seeing
anything obvious. BTW Dropping the thread dump causes the app to wakeup
and have no further issues for the day.
Here is the Dump. I removed the GangWorker GC and CMS threads from
output for brevity.
Any help would be greatly appreciated. This might just be a quickfixj
issue, and I've posted in their forum.
Thanks,
Geoff
05:00:09,817 [main] DEBUG
org.apache.camel.management.DefaultManagementAgent - Registered MBean
with objectname:
org.apache.camel:context=nyl01a-1103/camel,type=components,name="quickfi
x"
05:00:09,817 [main] INFO
org.apache.camel.component.quickfixj.QuickfixjComponent - QuickFIX/J
component started
05:00:09,817 [main] DEBUG org.apache.camel.impl.DefaultComponent -
Creating endpoint
uri=[quickfix://marketAxessCdsPriceConfig_US?sessionID=FIXT.1.1%3ACSUSCD
SINV-%3EMA], path=[marketAxessCdsPriceConfig_US],
parameters=[{sessionID=FIXT.1.1:CSUSCDSINV->MA}]
05:00:09,817 [main] INFO
org.apache.camel.component.quickfixj.QuickfixjComponent - Creating
QuickFIX/J engine using settings: marketAxessCdsPriceConfig_US
2012-08-27 08:49:25
Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.0-b15 mixed
mode):
"Low Memory Detector" daemon prio=10 tid=0x00002aaaf8d01400 nid=0x79e5
runnable [0x0000000000000000..0x00000000000000
00]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x00002aaaf8a3a400 nid=0x79e4
waiting on condition [0x0000000000000000..0x000000
0041d443e0]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x00002aaaf8a37800 nid=0x79e3
waiting on condition [0x0000000000000000..0x000000
0041c433b0]
java.lang.Thread.State: RUNNABLE
"JDWP Event Helper Thread" daemon prio=10 tid=0x00002aaaf8a22400
nid=0x79e2 runnable [0x0000000000000000..0x000000000
0000000]
java.lang.Thread.State: RUNNABLE
"JDWP Transport Listener: dt_socket" daemon prio=10
tid=0x00002aaaf8a1f000 nid=0x79e1 runnable [0x0000000000000000..0
x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00002aaaf8a11000 nid=0x79e0
waiting on condition [0x0000000000000000..0x0000
000000000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x00002aaaf8a0f400
nid=0x79df waiting on condition [0x000000000000
0000..0x0000000041840fd8]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00002aaaf89f1c00 nid=0x79d6 in
Object.wait() [0x000000004173f000..0x000000004173fca0
]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab65a0018> (a
java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x00002aaab65a0018> (a
java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x00002aaaf89eac00 nid=0x79d5 in
Object.wait() [0x000000004163e000..0x00000000
4163ec20]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab65a3f18> (a
java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00002aaab65a3f18> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x0000000040121800 nid=0x79a2 runnable
[0x0000000040227000..0x000000004022af60]
java.lang.Thread.State: RUNNABLE
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
- locked <0x00002aaab65a2510> (a
sun.misc.Launcher$AppClassLoader)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
- locked <0x00002aaab65a2510> (a
sun.misc.Launcher$AppClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
- locked <0x00002aaab65a2510> (a
sun.misc.Launcher$AppClassLoader)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at
quickfix.DefaultMessageFactory.discoverFactory(DefaultMessageFactory.jav
a:51)
at
quickfix.DefaultMessageFactory.<init>(DefaultMessageFactory.java:44)
at
org.apache.camel.component.quickfixj.QuickfixjEngine.<init>(QuickfixjEng
ine.java:128)
at
org.apache.camel.component.quickfixj.QuickfixjEngine.<init>(QuickfixjEng
ine.java:115)
at
org.apache.camel.component.quickfixj.QuickfixjComponent.createEndpoint(Q
uickfixjComponent.java:69)
- locked <0x00002aaaaf640820> (a java.lang.Object)
at
org.apache.camel.impl.DefaultComponent.createEndpoint(DefaultComponent.j
ava:75)
at
org.apache.camel.impl.DefaultCamelContext.getEndpoint(DefaultCamelContex
t.java:425)
at
org.apache.camel.util.CamelContextHelper.getMandatoryEndpoint(CamelConte
xtHelper.java:48)
at
org.apache.camel.model.RouteDefinition.resolveEndpoint(RouteDefinition.j
ava:180)
at
org.apache.camel.impl.DefaultRouteContext.resolveEndpoint(DefaultRouteCo
ntext.java:110)
at
org.apache.camel.impl.DefaultRouteContext.resolveEndpoint(DefaultRouteCo
ntext.java:116)
at
org.apache.camel.model.SendDefinition.resolveEndpoint(SendDefinition.jav
a:61)
at
org.apache.camel.model.SendDefinition.createProcessor(SendDefinition.jav
a:55)
at
org.apache.camel.model.ProcessorDefinition.createOutputsProcessor(Proces
sorDefinition.java:395)
at
org.apache.camel.model.ProcessorDefinition.createOutputsProcessor(Proces
sorDefinition.java:134)
at
org.apache.camel.impl.DefaultRouteContext.createProcessor(DefaultRouteCo
ntext.java:106)
at
org.apache.camel.model.ProcessorDefinition.createChildProcessor(Processo
rDefinition.java:153)
at
org.apache.camel.model.FilterDefinition.createFilterProcessor(FilterDefi
nition.java:68)
at
org.apache.camel.model.FilterDefinition.createProcessor(FilterDefinition
.java:62)
at
org.apache.camel.model.FilterDefinition.createProcessor(FilterDefinition
.java:34)
at
org.apache.camel.model.ProcessorDefinition.makeProcessor(ProcessorDefini
tion.java:437)
at
org.apache.camel.model.ProcessorDefinition.addRoutes(ProcessorDefinition
.java:183)
at
org.apache.camel.model.RouteDefinition.addRoutes(RouteDefinition.java:79
1)
at
org.apache.camel.model.RouteDefinition.addRoutes(RouteDefinition.java:16
5)
at
org.apache.camel.impl.DefaultCamelContext.startRoute(DefaultCamelContext
.java:697)
at
org.apache.camel.impl.DefaultCamelContext.startRouteDefinitions(DefaultC
amelContext.java:1651)
at
org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelConte
xt.java:1440)
at
org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.ja
va:1338)
- locked <0x00002aaab71e11c8> (a
org.apache.camel.spring.SpringCamelContext)
at
org.apache.camel.spring.SpringCamelContext.doStart(SpringCamelContext.ja
va:167)
at
org.apache.camel.impl.ServiceSupport.start(ServiceSupport.java:67)
at
org.apache.camel.impl.ServiceSupport.start(ServiceSupport.java:54)
at
org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java
:1316)
at
org.apache.camel.spring.SpringCamelContext.maybeStart(SpringCamelContext
.java:211)
at
org.apache.camel.spring.SpringCamelContext.onApplicationEvent(SpringCame
lContext.java:106)
at
org.apache.camel.spring.CamelContextFactoryBean.onApplicationEvent(Camel
ContextFactoryBean.java:240)
at
org.springframework.context.event.SimpleApplicationEventMulticaster.mult
icastEvent(SimpleApplicationEventM
ulticaster.java:97)
at
org.springframework.context.support.AbstractApplicationContext.publishEv
ent(AbstractApplicationContext.jav
a:303)
at
org.springframework.context.support.AbstractApplicationContext.finishRef
resh(AbstractApplicationContext.ja
va:911)
at
org.springframework.context.support.AbstractApplicationContext.refresh(A
bstractApplicationContext.java:428
)
- locked <0x00002aaab65a0060> (a java.lang.Object)
at
org.springframework.context.support.ClassPathXmlApplicationContext.<init
>(ClassPathXmlApplicationContext.j
ava:139)
at
org.springframework.context.support.ClassPathXmlApplicationContext.<init
>(ClassPathXmlApplicationContext.j
ava:93)
at
org.apache.camel.spring.Main.createDefaultApplicationContext(Main.java:1
85)
at org.apache.camel.spring.Main.doStart(Main.java:139)
at
org.apache.camel.impl.ServiceSupport.start(ServiceSupport.java:67)
at
org.apache.camel.impl.ServiceSupport.start(ServiceSupport.java:54)
at org.apache.camel.impl.MainSupport.run(MainSupport.java:136)
at com.csfb.fid.credit.cds.MainSpring.boot(MainSpring.java:28)
at com.csfb.fid.credit.cds.MainSpring.main(MainSpring.java:13)
"VM Thread" prio=10 tid=0x00002aaaf89e5400 nid=0x79d4 runnable
"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00002aaaf6c92c00
nid=0x79b7 waiting on condition
"VM Periodic Task Thread" prio=10 tid=0x00002aaaf8d03c00 nid=0x79e6
waiting on condition
JNI global references: 4877
Heap
par new generation total 19136K, used 2663K [0x00002aaaae5a0000,
0x00002aaaafa60000, 0x00002aaab65a0000)
eden space 17024K, 8% used [0x00002aaaae5a0000, 0x00002aaaae700928,
0x00002aaaaf640000)
from space 2112K, 59% used [0x00002aaaaf640000, 0x00002aaaaf779438,
0x00002aaaaf850000)
to space 2112K, 0% used [0x00002aaaaf850000, 0x00002aaaaf850000,
0x00002aaaafa60000)
concurrent mark-sweep generation total 109824K, used 10286K
[0x00002aaab65a0000, 0x00002aaabd0e0000, 0x00002aaaee5a0
000)
concurrent-mark-sweep perm gen total 47556K, used 47535K
[0x00002aaaee5a0000, 0x00002aaaf1411000, 0x00002aaaf65a0000
)
Geoffrey A Gershaw
CREDIT SUISSE
Information Technology | Credit eTrading Development, KFVB 525
7033 Louis Stephens Drive | 27560 Research Triangle Park | United States
Phone +1 919 994 6412
geoffrey.gers...@credit-suisse.com | www.credit-suisse.com
<http://www.credit-suisse.com/>
===============================================================================
Please access the attached hyperlink for an important electronic communications
disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================
