[ https://issues.apache.org/jira/browse/DRILL-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Deneche A. Hakim reassigned DRILL-3842: --------------------------------------- Assignee: Deneche A. Hakim > JVM dies if drill attempts to read too many files in the directory that blow > up heap > ------------------------------------------------------------------------------------- > > Key: DRILL-3842 > URL: https://issues.apache.org/jira/browse/DRILL-3842 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow > Affects Versions: 1.1.0, 1.2.0 > Reporter: Victoria Markman > Assignee: Deneche A. Hakim > Priority: Critical > > Run {{select count(*) from t1}} where t1 directory consists of 1.9 million > little parquet files. The outcome: drillbit is dead and out of working set. > 1. Client never got response back from the server > 2. drillbit.log > {code} > 2015-09-25 17:56:55,935 [29fa756f-894d-0340-3661-b925bff0fe11:foreman] INFO > o.a.d.exec.store.parquet.Metadata - Took 47999 ms to get file statuses > 2015-09-25 18:43:19,871 [BitServer-4] INFO > o.a.d.exec.rpc.control.ControlServer - RPC connection /10.10.88.135:31011 > <--> /10.10.88.135:51675 (control server) timed out. Timeout was set to 300 > seconds. Closing connection. > 2015-09-25 18:50:06,026 [BitServer-3] INFO > o.a.d.exec.rpc.control.ControlClient - Channel closed /10.10.88.135:51675 > <--> /10.10.88.135:31011. > 2015-09-25 18:50:06,032 [UserServer-1] ERROR > o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication. > Connection: /10.10.88.135:31010 <--> /10.10.88.133:51612 (user client). > Closing connection. > java.lang.OutOfMemoryError: Java heap space > {code} > drillbit.out > {code} > Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread "main-SendThread(atsqa4-133.qa.lab:5181)" > Exception in thread "WorkManager.StatusThread" java.lang.OutOfMemoryError: > Java heap space > 2015-09-25 18:53:52 > Full thread dump OpenJDK 64-Bit Server VM (24.65-b04 mixed mode): > {code} > jstack > {code} > [Fri Sep 25 18:53:29 ] # jstack 63205 > 63205: Unable to open socket file: target process not responding or HotSpot > VM not loaded > The -F option can be used when the target process is not responding > {code} > jstack -F > {code} > Attaching to process ID 63205, please wait... > Debugger attached successfully. > Server compiler detected. > JVM version is 24.65-b04 > java.lang.RuntimeException: Unable to deduce type of thread from address > 0x0000000004093800 (expected type JavaThread, CompilerThread, ServiceThread, > JvmtiAgentThread, or SurrogateLockerThread) > at > sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162) > at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150) > at > sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149) > at > sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56) > at > sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39) > at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52) > at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at sun.tools.jstack.JStack.runJStackTool(JStack.java:136) > at sun.tools.jstack.JStack.main(JStack.java:102) > Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for > type of address 0x0000000004093800 > at > sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62) > at > sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80) > at > sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158) > ... 15 more > Exception in thread "main" java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at sun.tools.jstack.JStack.runJStackTool(JStack.java:136) > at sun.tools.jstack.JStack.main(JStack.java:102) > Caused by: java.lang.RuntimeException: Unable to deduce type of thread from > address 0x0000000004093800 (expected type JavaThread, CompilerThread, > ServiceThread, JvmtiAgentThread, or SurrogateLockerThread) > at > sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162) > at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150) > at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:65) > at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) > at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) > at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) > at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) > ... 6 more > Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for > type of address 0x0000000004093800 > at > sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62) > at > sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80) > at > sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158) > ... 12 more > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)