[ https://issues.apache.org/jira/browse/ATLAS-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ayub Pathan updated ATLAS-2199: ------------------------------- Priority: Blocker (was: Critical) > DeadLock in atlas while producing metadata with hive view > 300 cols. > --------------------------------------------------------------------- > > Key: ATLAS-2199 > URL: https://issues.apache.org/jira/browse/ATLAS-2199 > Project: Atlas > Issue Type: Bug > Components: atlas-core > Affects Versions: 0.8.2 > Reporter: Ayub Pathan > Priority: Blocker > Fix For: 0.8.2 > > Attachments: application.log, atlas.20171009-192118.err, > atlas.20171009-192118.out, thread-dump.txt > > > Steps to reproduce: > 1. Produce messages using hive hook - create view with 300 columns. Example, > command below > {noformat} > create table t1_300(col0 string, col1 string, col2 string, col3 string, col4 > string, col5 string, col6 string, col7 string, col8 string, col9 string, > col10 string, col11 string, col12 string, col13 string, col14 string, col15 > string, col16 string, col17 string, col18 string, col19 string, col20 string, > col21 string, col22 string, col23 string, col24 string, col25 string, col26 > string, col27 string, col28 string, col29 string, col30 string, col31 string, > col32 string, col33 string, col34 string, col35 string, col36 string, col37 > string, col38 string, col39 string, col40 string, col41 string, col42 string, > col43 string, col44 string, col45 string, col46 string, col47 string, col48 > string, col49 string, col50 string, col51 string, col52 string, col53 string, > col54 string, col55 string, col56 string, col57 string, col58 string, col59 > string, col60 string, col61 string, col62 string, col63 string, col64 string, > col65 string, col66 string, col67 string, col68 string, col69 string, col70 > string, col71 string, col72 string, col73 string, col74 string, col75 string, > col76 string, col77 string, col78 string, col79 string, col80 string, col81 > string, col82 string, col83 string, col84 string, col85 string, col86 string, > col87 string, col88 string, col89 string, col90 string, col91 string, col92 > string, col93 string, col94 string, col95 string, col96 string, col97 string, > col98 string, col99 string, col100 string, col101 string, col102 string, > col103 string, col104 string, col105 string, col106 string, col107 string, > col108 string, col109 string, col110 string, col111 string, col112 string, > col113 string, col114 string, col115 string, col116 string, col117 string, > col118 string, col119 string, col120 string, col121 string, col122 string, > col123 string, col124 string, col125 string, col126 string, col127 string, > col128 string, col129 string, col130 string, col131 string, col132 string, > col133 string, col134 string, col135 string, col136 string, col137 string, > col138 string, col139 string, col140 string, col141 string, col142 string, > col143 string, col144 string, col145 string, col146 string, col147 string, > col148 string, col149 string, col150 string, col151 string, col152 string, > col153 string, col154 string, col155 string, col156 string, col157 string, > col158 string, col159 string, col160 string, col161 string, col162 string, > col163 string, col164 string, col165 string, col166 string, col167 string, > col168 string, col169 string, col170 string, col171 string, col172 string, > col173 string, col174 string, col175 string, col176 string, col177 string, > col178 string, col179 string, col180 string, col181 string, col182 string, > col183 string, col184 string, col185 string, col186 string, col187 string, > col188 string, col189 string, col190 string, col191 string, col192 string, > col193 string, col194 string, col195 string, col196 string, col197 string, > col198 string, col199 string, col200 string, col201 string, col202 string, > col203 string, col204 string, col205 string, col206 string, col207 string, > col208 string, col209 string, col210 string, col211 string, col212 string, > col213 string, col214 string, col215 string, col216 string, col217 string, > col218 string, col219 string, col220 string, col221 string, col222 string, > col223 string, col224 string, col225 string, col226 string, col227 string, > col228 string, col229 string, col230 string, col231 string, col232 string, > col233 string, col234 string, col235 string, col236 string, col237 string, > col238 string, col239 string, col240 string, col241 string, col242 string, > col243 string, col244 string, col245 string, col246 string, col247 string, > col248 string, col249 string, col250 string, col251 string, col252 string, > col253 string, col254 string, col255 string, col256 string, col257 string, > col258 string, col259 string, col260 string, col261 string, col262 string, > col263 string, col264 string, col265 string, col266 string, col267 string, > col268 string, col269 string, col270 string, col271 string, col272 string, > col273 string, col274 string, col275 string, col276 string, col277 string, > col278 string, col279 string, col280 string, col281 string, col282 string, > col283 string, col284 string, col285 string, col286 string, col287 string, > col288 string, col289 string, col290 string, col291 string, col292 string, > col293 string, col294 string, col295 string, col296 string, col297 string, > col298 string, col299 string); > create view v_t1_300 as select * from t1_300; > {noformat} > 2. With this Atlas throws out of memory exception. Below error log confirming > the same. > {noformat} > [root@ctr-e134-1499953498516-209460-01-000004 atlas]# cat > atlas.20171009-192118.err > Exception in thread "pool-2-thread-6" Exception in thread "pool-2-thread-2" > java.lang.OutOfMemoryError: Java heap space > at java.util.HashMap$KeySet.iterator(HashMap.java:916) > at java.util.HashSet.iterator(HashSet.java:172) > at sun.nio.ch.Util$3.iterator(Util.java:324) > at > org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600) > at > org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Exception in thread "Thread-22" Exception in thread "pool-2-thread-3" > Exception in thread "HashSessionScavenger-0" java.lang.OutOfMemoryError: Java > heap space > at java.util.HashMap$KeySet.iterator(HashMap.java:916) > at java.util.HashSet.iterator(HashSet.java:172) > at sun.nio.ch.Util$3.iterator(Util.java:324) > at > org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600) > at > org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > java.lang.OutOfMemoryError: Java heap space > at > java.util.concurrent.ConcurrentHashMap$ValuesView.iterator(ConcurrentHashMap.java:4683) > at > org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:314) > at > org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:285) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > java.lang.OutOfMemoryError: Java heap space > at java.util.HashMap$KeySet.iterator(HashMap.java:916) > at java.util.HashSet.iterator(HashSet.java:172) > at sun.nio.ch.Util$3.iterator(Util.java:324) > at > org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600) > at > org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > java.lang.OutOfMemoryError: Java heap space > Exception in thread "pool-2-thread-7" java.lang.OutOfMemoryError: Java heap > space > Exception in thread "pool-2-thread-1" java.lang.OutOfMemoryError: Java heap > space > [root@ctr-e134-1499953498516-209460-01-000004 atlas]# > {noformat} > 3. After this step, Atlas became unresponsive, no requests are honoured. All > requests are timedout. > 4. JSTACK output confirming the deadlock. > {noformat} > [root@ctr-e134-1499953498516-209460-01-000004 atlas]# ps axu | grep Atlas > atlas 131887 3.6 0.9 20609684 2612572 ? Sl Oct09 31:03 > /usr/lib/jvm/java-openjdk/bin/java -Datlas.log.dir=/grid/0/log/atlas > -Datlas.log.file=application.log -Datlas.home=/usr/hdp/.3.0-175/atlas > -Datlas.conf=/usr/hdp/current/atlas-server/conf -Xms2048m -Xmx2048m > -XX:MaxNewSize=600m -XX:MetaspaceSize=100m -XX:MaxMetaspaceSize=512m -server > -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled > -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled > -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError > -XX:HeapDumpPath=/grid/0/log/atlas/atlas_server.hprof > -Xloggc:/grid/0/log/atlas/gc-worker.log -verbose:gc -XX:+UseGCLogFileRotation > -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails > -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps > -Dlog4j.configuration=atlas-log4j.xml > -Djava..auth.login.config=/usr/hdp/current/atlas-server/conf/atlas_jaas.conf > -classpath > /usr/hdp/current/atlas-server/conf:/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/classes:/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/lib/*:/usr/hdp/.3.0-175/atlas/libext/*:/etc/hbase/conf > org.apache.atlas.Atlas -app /usr/hdp/current/atlas-server/server/webapp/atlas > root 1666487 0.0 0.0 10632 968 pts/1 S+ 09:34 0:00 grep > --color=auto Atlas > [root@ctr-e134-1499953498516-209460-01-000004 atlas]# > {noformat} > {noformat} > [root@ctr-e134-1499953498516-209460-01-000004 atlas]# > /usr/lib/jvm/java-openjdk/bin/jstack -F -l 131887 > Attaching to process ID 131887, please wait... > Debugger attached successfully. > Server compiler detected. > JVM version is 25.144-b01 > Deadlock Detection: > java.lang.RuntimeException: Unable to deduce type of thread from address > 0x00007f8838013800 (expected type JavaThread, CompilerThread, ServiceThread, > JvmtiAgentThread, or SurrogateLockerThread) > at > sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:169) > at sun.jvm.hotspot.runtime.Threads.first(Threads.java:153) > at > sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149) > at > sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56) > at > sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39) > at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:62) > at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) > at sun.jvm.hotspot.tools.JStack.run(JStack.java:66) > at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260) > at sun.jvm.hotspot.tools.Tool.start(Tool.java:223) > at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118) > at sun.jvm.hotspot.tools.JStack.main(JStack.java:92) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.tools.jstack.JStack.runJStackTool(JStack.java:140) > at sun.tools.jstack.JStack.main(JStack.java:106) > Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for > type of address 0x00007f8838013800 > at > sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62) > at > sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80) > at > sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:165) > ... 17 more > Can't print deadlocks:Unable to deduce type of thread from address > 0x00007f8838013800 (expected type JavaThread, CompilerThread, ServiceThread, > JvmtiAgentThread, or SurrogateLockerThread) > Exception in thread "main" java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.tools.jstack.JStack.runJStackTool(JStack.java:140) > at sun.tools.jstack.JStack.main(JStack.java:106) > Caused by: java.lang.InternalError: Metadata does not appear to be polymorphic > at > sun.jvm.hotspot.types.basic.BasicTypeDataBase.findDynamicTypeForAddress(BasicTypeDataBase.java:278) > at > sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:102) > at sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:68) > at sun.jvm.hotspot.memory.DictionaryEntry.klass(DictionaryEntry.java:71) > at > sun.jvm.hotspot.memory.DictionaryEntry.equals(DictionaryEntry.java:79) > at sun.jvm.hotspot.memory.Dictionary.getEntry(Dictionary.java:99) > at sun.jvm.hotspot.memory.Dictionary.find(Dictionary.java:87) > at > sun.jvm.hotspot.memory.SystemDictionary.find(SystemDictionary.java:159) > at > sun.jvm.hotspot.memory.SystemDictionary.find(SystemDictionary.java:151) > at > sun.jvm.hotspot.memory.SystemDictionary.getAbstractOwnableSynchronizerKlass(SystemDictionary.java:135) > at > sun.jvm.hotspot.runtime.ConcurrentLocksPrinter.fillLocks(ConcurrentLocksPrinter.java:66) > at > sun.jvm.hotspot.runtime.ConcurrentLocksPrinter.<init>(ConcurrentLocksPrinter.java:36) > at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:71) > at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) > at sun.jvm.hotspot.tools.JStack.run(JStack.java:66) > at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260) > at sun.jvm.hotspot.tools.Tool.start(Tool.java:223) > at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118) > at sun.jvm.hotspot.tools.JStack.main(JStack.java:92) > ... 6 more > [root@ctr-e134-1499953498516-209460-01-000004 atlas]# > {noformat} > Attaching the thread dump output as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)