[jira] [Commented] (SPARK-27021) Leaking Netty event loop group for shuffle chunk fetch requests
[ https://issues.apache.org/jira/browse/SPARK-27021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997288#comment-16997288 ] roncenzhao commented on SPARK-27021: [~attilapiros] Thanks. The issue is the same problem we have encountered. > Leaking Netty event loop group for shuffle chunk fetch requests > --- > > Key: SPARK-27021 > URL: https://issues.apache.org/jira/browse/SPARK-27021 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.4.0, 2.4.1, 3.0.0 >Reporter: Attila Zsolt Piros >Assignee: Attila Zsolt Piros >Priority: Major > Fix For: 3.0.0 > > Attachments: image-2019-12-14-23-23-50-384.png > > > The extra event loop group created for handling shuffle chunk fetch requests > are never closed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27021) Leaking Netty event loop group for shuffle chunk fetch requests
[ https://issues.apache.org/jira/browse/SPARK-27021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996445#comment-16996445 ] roncenzhao commented on SPARK-27021: [~attilapiros] Thank you. We have one problem about the memory leak of `StreamState` in `OneForOneStreamManager` which cause the `NodeManager` OOM. Most of the memory in NM is used by `StreamState`, like this: !image-2019-12-14-23-23-50-384.png! This may be caused by the shuffle service because we find the `StreamState` include some application which were already finished. Would you have any idea about this problem? Thanks~ > Leaking Netty event loop group for shuffle chunk fetch requests > --- > > Key: SPARK-27021 > URL: https://issues.apache.org/jira/browse/SPARK-27021 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.4.0, 2.4.1, 3.0.0 >Reporter: Attila Zsolt Piros >Assignee: Attila Zsolt Piros >Priority: Major > Fix For: 3.0.0 > > Attachments: image-2019-12-14-23-23-50-384.png > > > The extra event loop group created for handling shuffle chunk fetch requests > are never closed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27021) Leaking Netty event loop group for shuffle chunk fetch requests
[ https://issues.apache.org/jira/browse/SPARK-27021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-27021: --- Attachment: image-2019-12-14-23-23-50-384.png > Leaking Netty event loop group for shuffle chunk fetch requests > --- > > Key: SPARK-27021 > URL: https://issues.apache.org/jira/browse/SPARK-27021 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.4.0, 2.4.1, 3.0.0 >Reporter: Attila Zsolt Piros >Assignee: Attila Zsolt Piros >Priority: Major > Fix For: 3.0.0 > > Attachments: image-2019-12-14-23-23-50-384.png > > > The extra event loop group created for handling shuffle chunk fetch requests > are never closed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27021) Leaking Netty event loop group for shuffle chunk fetch requests
[ https://issues.apache.org/jira/browse/SPARK-27021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16995333#comment-16995333 ] roncenzhao commented on SPARK-27021: [~attilapiros] [~vanzin] I hava two problems: # whether this bug also affect spark2.4.3 or not ? # whether this bug cause the NodeManager OOM? Look forward to your reply~ > Leaking Netty event loop group for shuffle chunk fetch requests > --- > > Key: SPARK-27021 > URL: https://issues.apache.org/jira/browse/SPARK-27021 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.4.0, 2.4.1, 3.0.0 >Reporter: Attila Zsolt Piros >Assignee: Attila Zsolt Piros >Priority: Major > Fix For: 3.0.0 > > > The extra event loop group created for handling shuffle chunk fetch requests > are never closed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26736) if filter condition `And` has non-determined sub function it does not do partition prunning
[ https://issues.apache.org/jira/browse/SPARK-26736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-26736: --- Summary: if filter condition `And` has non-determined sub function it does not do partition prunning (was: if filter condition has rand() function it does not do partition prunning) > if filter condition `And` has non-determined sub function it does not do > partition prunning > --- > > Key: SPARK-26736 > URL: https://issues.apache.org/jira/browse/SPARK-26736 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.4.0 >Reporter: roncenzhao >Priority: Minor > > Example: > A partitioned table definition: > _create table test(id int) partitioned by (dt string);_ > The following sql does not do partition prunning: > _select * from test where dt='20190101' and rand() < 0.5;_ > > I think it should do partition prunning in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-26736) if filter condition has rand() function it does not do partition prunning
roncenzhao created SPARK-26736: -- Summary: if filter condition has rand() function it does not do partition prunning Key: SPARK-26736 URL: https://issues.apache.org/jira/browse/SPARK-26736 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 2.4.0 Reporter: roncenzhao Example: A partitioned table definition: _create table test(id int) partitioned by (dt string);_ The following sql does not do partition prunning: _select * from test where dt='20190101' and rand() < 0.5;_ I think it should do partition prunning in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug
[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16214220#comment-16214220 ] roncenzhao commented on SPARK-22272: I think the simple way is to set 'spark.file.transferTo' false. I do it in our production env and the problem is never seen again. > killing task may cause the executor progress hang because of the JVM bug > > > Key: SPARK-22272 > URL: https://issues.apache.org/jira/browse/SPARK-22272 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.2 > Environment: java version "1.7.0_75" > hadoop version 2.5.0 >Reporter: roncenzhao > Attachments: 26883.jstack, screenshot-1.png, screenshot-2.png > > > JVM bug: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8132693 > We kill the task using 'Thread.interrupt()' and the ShuffleMapTask use nio to > merge all partitions files when 'spark.file.transferTo' is true(default), so > it may cause the jvm bug. > When the driver send one task to this bad executor, the task will never run > and as a result the job will hang forever without handling. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug
[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16205372#comment-16205372 ] roncenzhao commented on SPARK-22272: The spark 2.1.x still supports jdk1.7 so IMHO I think spark should avoid this problem if you have known the bug in jdk1.7. Thanks~ > killing task may cause the executor progress hang because of the JVM bug > > > Key: SPARK-22272 > URL: https://issues.apache.org/jira/browse/SPARK-22272 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.1 > Environment: java version "1.7.0_75" > hadoop version 2.5.0 >Reporter: roncenzhao > Attachments: 26883.jstack, screenshot-1.png, screenshot-2.png > > > JVM bug: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8132693 > We kill the task using 'Thread.interrupt()' and the ShuffleMapTask use nio to > merge all partitions files when 'spark.file.transferTo' is true(default), so > it may cause the jvm bug. > When the driver send one task to this bad executor, the task will never run > and as a result the job will hang forever without handling. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug
[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203264#comment-16203264 ] roncenzhao commented on SPARK-22272: The jdk bug is not resolved yet. If spark does not work around it, I think spark should change the default value of 'spark.file.transferTo' to false in case of this problem. > killing task may cause the executor progress hang because of the JVM bug > > > Key: SPARK-22272 > URL: https://issues.apache.org/jira/browse/SPARK-22272 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.1 > Environment: java version "1.7.0_75" > hadoop version 2.5.0 >Reporter: roncenzhao > Attachments: 26883.jstack, screenshot-1.png, screenshot-2.png > > > JVM bug: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8132693 > We kill the task using 'Thread.interrupt()' and the ShuffleMapTask use nio to > merge all partitions files when 'spark.file.transferTo' is true(default), so > it may cause the jvm bug. > When the driver send one task to this bad executor, the task will never run > and as a result the job will hang forever without handling. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug
[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-22272: --- Description: JVM bug: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8132693 We kill the task using 'Thread.interrupt()' and the ShuffleMapTask use nio to merge all partitions files when 'spark.file.transferTo' is true(default), so it may cause the jvm bug. When the driver send one task to this bad executor, the task will never run and as a result the job will hang forever without handling. was: JVM bug: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8132693 We kill the task using 'Thread.interrupt()' and the ShuffleMapTask use nio to merge all partitions files when 'spark.file.transferTo' is true(default), so it may cause the jvm bug. > killing task may cause the executor progress hang because of the JVM bug > > > Key: SPARK-22272 > URL: https://issues.apache.org/jira/browse/SPARK-22272 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.1 > Environment: java version "1.7.0_75" > hadoop version 2.5.0 >Reporter: roncenzhao > Attachments: 26883.jstack, screenshot-1.png, screenshot-2.png > > > JVM bug: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8132693 > We kill the task using 'Thread.interrupt()' and the ShuffleMapTask use nio to > merge all partitions files when 'spark.file.transferTo' is true(default), so > it may cause the jvm bug. > When the driver send one task to this bad executor, the task will never run > and as a result the job will hang forever without handling. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug
[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-22272: --- Attachment: 26883.jstack > killing task may cause the executor progress hang because of the JVM bug > > > Key: SPARK-22272 > URL: https://issues.apache.org/jira/browse/SPARK-22272 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.1 > Environment: java version "1.7.0_75" > hadoop version 2.5.0 >Reporter: roncenzhao > Attachments: 26883.jstack, screenshot-1.png, screenshot-2.png > > > JVM bug: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8132693 > We kill the task using 'Thread.interrupt()' and the ShuffleMapTask use nio to > merge all partitions files when 'spark.file.transferTo' is true(default), so > it may cause the jvm bug. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug
[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-22272: --- Attachment: screenshot-1.png > killing task may cause the executor progress hang because of the JVM bug > > > Key: SPARK-22272 > URL: https://issues.apache.org/jira/browse/SPARK-22272 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.1 > Environment: java version "1.7.0_75" > hadoop version 2.5.0 >Reporter: roncenzhao > Attachments: screenshot-1.png, screenshot-2.png > > > JVM bug: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8132693 > We kill the task using 'Thread.interrupt()' and the ShuffleMapTask use nio to > merge all partitions files when 'spark.file.transferTo' is true(default), so > it may cause the jvm bug. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug
[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-22272: --- Attachment: screenshot-2.png > killing task may cause the executor progress hang because of the JVM bug > > > Key: SPARK-22272 > URL: https://issues.apache.org/jira/browse/SPARK-22272 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.1 > Environment: java version "1.7.0_75" > hadoop version 2.5.0 >Reporter: roncenzhao > Attachments: screenshot-1.png, screenshot-2.png > > > JVM bug: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8132693 > We kill the task using 'Thread.interrupt()' and the ShuffleMapTask use nio to > merge all partitions files when 'spark.file.transferTo' is true(default), so > it may cause the jvm bug. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug
roncenzhao created SPARK-22272: -- Summary: killing task may cause the executor progress hang because of the JVM bug Key: SPARK-22272 URL: https://issues.apache.org/jira/browse/SPARK-22272 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.1.1 Environment: java version "1.7.0_75" hadoop version 2.5.0 Reporter: roncenzhao JVM bug: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8132693 We kill the task using 'Thread.interrupt()' and the ShuffleMapTask use nio to merge all partitions files when 'spark.file.transferTo' is true(default), so it may cause the jvm bug. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-21794) exception about reading task serial data(broadcast) value when the storage memory is not enough to unroll
[ https://issues.apache.org/jira/browse/SPARK-21794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao closed SPARK-21794. -- Resolution: Duplicate > exception about reading task serial data(broadcast) value when the storage > memory is not enough to unroll > - > > Key: SPARK-21794 > URL: https://issues.apache.org/jira/browse/SPARK-21794 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.1, 2.1.0 >Reporter: roncenzhao > Attachments: error stack.png > > > ``` > 17/08/09 19:27:43 ERROR Utils: Exception encountered > java.util.NoSuchElementException > at > org.apache.spark.util.collection.PrimitiveVector$$anon$1.next(PrimitiveVector.scala:58) > at > org.apache.spark.storage.memory.PartiallyUnrolledIterator.next(MemoryStore.scala:697) > at > org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) > at scala.Option.map(Option.scala:146) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178) > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1276) > at > org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:174) > at > org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:89) > at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:72) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 17/08/09 19:27:43 INFO UnifiedMemoryManager: Will not store broadcast_5 as > the required space (1048576 bytes) exceeds our memory limit (878230 bytes) > 17/08/09 19:27:43 WARN MemoryStore: Failed to reserve initial memory > threshold of 1024.0 KB for computing block broadcast_5 in memory. > 17/08/09 19:27:43 WARN MemoryStore: Not enough space to cache broadcast_5 in > memory! (computed 384.0 B so far) > 17/08/09 19:27:43 INFO MemoryStore: Memory use = 857.6 KB (blocks) + 0.0 B > (scratch space shared across 0 tasks(s)) = 857.6 KB. Storage limit = 857.6 KB. > 17/08/09 19:27:43 ERROR Utils: Exception encountered > java.util.NoSuchElementException > at > org.apache.spark.util.collection.PrimitiveVector$$anon$1.next(PrimitiveVector.scala:58) > at > org.apache.spark.storage.memory.PartiallyUnrolledIterator.next(MemoryStore.scala:697) > at > org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) > at scala.Option.map(Option.scala:146) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178) > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1276) > at > org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:174) > at > org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:89) > at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:72) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec
[jira] [Commented] (SPARK-21794) exception about reading task serial data(broadcast) value when the storage memory is not enough to unroll
[ https://issues.apache.org/jira/browse/SPARK-21794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136237#comment-16136237 ] roncenzhao commented on SPARK-21794: [~yuming] Thanks, this is resolved by your PR. I close this issue now. > exception about reading task serial data(broadcast) value when the storage > memory is not enough to unroll > - > > Key: SPARK-21794 > URL: https://issues.apache.org/jira/browse/SPARK-21794 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.1, 2.1.0 >Reporter: roncenzhao > Attachments: error stack.png > > > ``` > 17/08/09 19:27:43 ERROR Utils: Exception encountered > java.util.NoSuchElementException > at > org.apache.spark.util.collection.PrimitiveVector$$anon$1.next(PrimitiveVector.scala:58) > at > org.apache.spark.storage.memory.PartiallyUnrolledIterator.next(MemoryStore.scala:697) > at > org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) > at scala.Option.map(Option.scala:146) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178) > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1276) > at > org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:174) > at > org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:89) > at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:72) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 17/08/09 19:27:43 INFO UnifiedMemoryManager: Will not store broadcast_5 as > the required space (1048576 bytes) exceeds our memory limit (878230 bytes) > 17/08/09 19:27:43 WARN MemoryStore: Failed to reserve initial memory > threshold of 1024.0 KB for computing block broadcast_5 in memory. > 17/08/09 19:27:43 WARN MemoryStore: Not enough space to cache broadcast_5 in > memory! (computed 384.0 B so far) > 17/08/09 19:27:43 INFO MemoryStore: Memory use = 857.6 KB (blocks) + 0.0 B > (scratch space shared across 0 tasks(s)) = 857.6 KB. Storage limit = 857.6 KB. > 17/08/09 19:27:43 ERROR Utils: Exception encountered > java.util.NoSuchElementException > at > org.apache.spark.util.collection.PrimitiveVector$$anon$1.next(PrimitiveVector.scala:58) > at > org.apache.spark.storage.memory.PartiallyUnrolledIterator.next(MemoryStore.scala:697) > at > org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) > at scala.Option.map(Option.scala:146) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178) > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1276) > at > org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:174) > at > org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:89) > at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:72) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at org.apache.spark.executor.Executor$TaskRunn
[jira] [Updated] (SPARK-21794) exception about reading task serial data(broadcast) value when the storage memory is not enough to unroll
[ https://issues.apache.org/jira/browse/SPARK-21794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-21794: --- Affects Version/s: (was: 2.1.1) 2.1.0 > exception about reading task serial data(broadcast) value when the storage > memory is not enough to unroll > - > > Key: SPARK-21794 > URL: https://issues.apache.org/jira/browse/SPARK-21794 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.1, 2.1.0 >Reporter: roncenzhao > Attachments: error stack.png > > > ``` > 17/08/09 19:27:43 ERROR Utils: Exception encountered > java.util.NoSuchElementException > at > org.apache.spark.util.collection.PrimitiveVector$$anon$1.next(PrimitiveVector.scala:58) > at > org.apache.spark.storage.memory.PartiallyUnrolledIterator.next(MemoryStore.scala:697) > at > org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) > at scala.Option.map(Option.scala:146) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178) > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1276) > at > org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:174) > at > org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:89) > at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:72) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 17/08/09 19:27:43 INFO UnifiedMemoryManager: Will not store broadcast_5 as > the required space (1048576 bytes) exceeds our memory limit (878230 bytes) > 17/08/09 19:27:43 WARN MemoryStore: Failed to reserve initial memory > threshold of 1024.0 KB for computing block broadcast_5 in memory. > 17/08/09 19:27:43 WARN MemoryStore: Not enough space to cache broadcast_5 in > memory! (computed 384.0 B so far) > 17/08/09 19:27:43 INFO MemoryStore: Memory use = 857.6 KB (blocks) + 0.0 B > (scratch space shared across 0 tasks(s)) = 857.6 KB. Storage limit = 857.6 KB. > 17/08/09 19:27:43 ERROR Utils: Exception encountered > java.util.NoSuchElementException > at > org.apache.spark.util.collection.PrimitiveVector$$anon$1.next(PrimitiveVector.scala:58) > at > org.apache.spark.storage.memory.PartiallyUnrolledIterator.next(MemoryStore.scala:697) > at > org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) > at scala.Option.map(Option.scala:146) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178) > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1276) > at > org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:174) > at > org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:89) > at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:72) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurren
[jira] [Updated] (SPARK-21794) exception about reading task serial data(broadcast) value when the storage memory is not enough to unroll
[ https://issues.apache.org/jira/browse/SPARK-21794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-21794: --- Attachment: error stack.png > exception about reading task serial data(broadcast) value when the storage > memory is not enough to unroll > - > > Key: SPARK-21794 > URL: https://issues.apache.org/jira/browse/SPARK-21794 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.1, 2.1.1 >Reporter: roncenzhao > Attachments: error stack.png > > > ``` > 17/08/09 19:27:43 ERROR Utils: Exception encountered > java.util.NoSuchElementException > at > org.apache.spark.util.collection.PrimitiveVector$$anon$1.next(PrimitiveVector.scala:58) > at > org.apache.spark.storage.memory.PartiallyUnrolledIterator.next(MemoryStore.scala:697) > at > org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) > at scala.Option.map(Option.scala:146) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178) > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1276) > at > org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:174) > at > org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:89) > at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:72) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 17/08/09 19:27:43 INFO UnifiedMemoryManager: Will not store broadcast_5 as > the required space (1048576 bytes) exceeds our memory limit (878230 bytes) > 17/08/09 19:27:43 WARN MemoryStore: Failed to reserve initial memory > threshold of 1024.0 KB for computing block broadcast_5 in memory. > 17/08/09 19:27:43 WARN MemoryStore: Not enough space to cache broadcast_5 in > memory! (computed 384.0 B so far) > 17/08/09 19:27:43 INFO MemoryStore: Memory use = 857.6 KB (blocks) + 0.0 B > (scratch space shared across 0 tasks(s)) = 857.6 KB. Storage limit = 857.6 KB. > 17/08/09 19:27:43 ERROR Utils: Exception encountered > java.util.NoSuchElementException > at > org.apache.spark.util.collection.PrimitiveVector$$anon$1.next(PrimitiveVector.scala:58) > at > org.apache.spark.storage.memory.PartiallyUnrolledIterator.next(MemoryStore.scala:697) > at > org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) > at scala.Option.map(Option.scala:146) > at > org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178) > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1276) > at > org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:174) > at > org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:89) > at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:72) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(Thread
[jira] [Created] (SPARK-21794) exception about reading task serial data(broadcast) value when the storage memory is not enough to unroll
roncenzhao created SPARK-21794: -- Summary: exception about reading task serial data(broadcast) value when the storage memory is not enough to unroll Key: SPARK-21794 URL: https://issues.apache.org/jira/browse/SPARK-21794 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.1.1, 2.0.1 Reporter: roncenzhao ``` 17/08/09 19:27:43 ERROR Utils: Exception encountered java.util.NoSuchElementException at org.apache.spark.util.collection.PrimitiveVector$$anon$1.next(PrimitiveVector.scala:58) at org.apache.spark.storage.memory.PartiallyUnrolledIterator.next(MemoryStore.scala:697) at org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30) at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) at scala.Option.map(Option.scala:146) at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178) at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1276) at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:174) at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:65) at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:65) at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:89) at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:72) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) at org.apache.spark.scheduler.Task.run(Task.scala:86) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 17/08/09 19:27:43 INFO UnifiedMemoryManager: Will not store broadcast_5 as the required space (1048576 bytes) exceeds our memory limit (878230 bytes) 17/08/09 19:27:43 WARN MemoryStore: Failed to reserve initial memory threshold of 1024.0 KB for computing block broadcast_5 in memory. 17/08/09 19:27:43 WARN MemoryStore: Not enough space to cache broadcast_5 in memory! (computed 384.0 B so far) 17/08/09 19:27:43 INFO MemoryStore: Memory use = 857.6 KB (blocks) + 0.0 B (scratch space shared across 0 tasks(s)) = 857.6 KB. Storage limit = 857.6 KB. 17/08/09 19:27:43 ERROR Utils: Exception encountered java.util.NoSuchElementException at org.apache.spark.util.collection.PrimitiveVector$$anon$1.next(PrimitiveVector.scala:58) at org.apache.spark.storage.memory.PartiallyUnrolledIterator.next(MemoryStore.scala:697) at org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30) at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1$$anonfun$2.apply(TorrentBroadcast.scala:178) at scala.Option.map(Option.scala:146) at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178) at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1276) at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:174) at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:65) at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:65) at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:89) at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:72) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) at org.apache.spark.scheduler.Task.run(Task.scala:86) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ``` This exception is caused in `MemoryStore.putIteratorAsValues()`. When `keepUnrolling` is false in the first time, the `vector: SizeTrackingVector` is not null and is empty. So when call the iterator method
[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs
[
https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16104373#comment-16104373
]
roncenzhao commented on SPARK-17321:
Hi, I encounter this problem too. Any process about this bug? Thanks~
> YARN shuffle service should use good disk from yarn.nodemanager.local-dirs
> --
>
> Key: SPARK-17321
> URL: https://issues.apache.org/jira/browse/SPARK-17321
> Project: Spark
> Issue Type: Bug
> Components: YARN
>Affects Versions: 1.6.2, 2.0.0
>Reporter: yunjiong zhao
>
> We run spark on yarn, after enabled spark dynamic allocation, we notice some
> spark application failed randomly due to YarnShuffleService.
> From log I found
> {quote}
> 2016-08-29 11:33:03,450 ERROR org.apache.spark.network.TransportContext:
> Error while initializing Netty pipeline
> java.lang.NullPointerException
> at
> org.apache.spark.network.server.TransportRequestHandler.(TransportRequestHandler.java:77)
> at
> org.apache.spark.network.TransportContext.createChannelHandler(TransportContext.java:159)
> at
> org.apache.spark.network.TransportContext.initializePipeline(TransportContext.java:135)
> at
> org.apache.spark.network.server.TransportServer$1.initChannel(TransportServer.java:123)
> at
> org.apache.spark.network.server.TransportServer$1.initChannel(TransportServer.java:116)
> at
> io.netty.channel.ChannelInitializer.channelRegistered(ChannelInitializer.java:69)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRegistered(AbstractChannelHandlerContext.java:133)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRegistered(AbstractChannelHandlerContext.java:119)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRegistered(DefaultChannelPipeline.java:733)
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:450)
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.access$100(AbstractChannel.java:378)
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:424)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
> Which caused by the first disk in yarn.nodemanager.local-dirs was broken.
> If we enabled spark.yarn.shuffle.stopOnFailure(SPARK-16505) we might lost
> hundred nodes which is unacceptable.
> We have 12 disks in yarn.nodemanager.local-dirs, so why not use other good
> disks if the first one is broken?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20019) spark can not load alluxio fileSystem after adding jar
[ https://issues.apache.org/jira/browse/SPARK-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15936541#comment-15936541 ] roncenzhao commented on SPARK-20019: [~srowen] Should I create a PR for this problem? > spark can not load alluxio fileSystem after adding jar > -- > > Key: SPARK-20019 > URL: https://issues.apache.org/jira/browse/SPARK-20019 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: roncenzhao > Attachments: exception_stack.png > > > The follwing sql cannot load alluxioSystem and it throws > `ClassNotFoundException`. > ``` > add jar /xxx/xxx/alluxioxxx.jar; > set fs.alluxio.impl=alluxio.hadoop.FileSystem; > select * from alluxionTbl; > ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-20019) spark can not load alluxio fileSystem after adding jar
[ https://issues.apache.org/jira/browse/SPARK-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15932621#comment-15932621 ] roncenzhao edited comment on SPARK-20019 at 3/20/17 1:25 PM: - [~srowen] I change the classLoader of `sparkContext.hadoopConfiguration` to the new one and it works. Actually I don't know what the 'load the driver at this stage' mean. Could you give me some more specific description? Thanks~ was (Author: roncenzhao): I change the classLoader of `sparkContext.hadoopConfiguration` to the new one and it works. Actually I don't know what the 'load the driver at this stage' mean. Could you give me some more specific description? Thanks~ > spark can not load alluxio fileSystem after adding jar > -- > > Key: SPARK-20019 > URL: https://issues.apache.org/jira/browse/SPARK-20019 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: roncenzhao > Attachments: exception_stack.png > > > The follwing sql cannot load alluxioSystem and it throws > `ClassNotFoundException`. > ``` > add jar /xxx/xxx/alluxioxxx.jar; > set fs.alluxio.impl=alluxio.hadoop.FileSystem; > select * from alluxionTbl; > ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20019) spark can not load alluxio fileSystem after adding jar
[ https://issues.apache.org/jira/browse/SPARK-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15932621#comment-15932621 ] roncenzhao commented on SPARK-20019: I change the classLoader of `sparkContext.hadoopConfiguration` to the new one and it works. Actually I don't know what the 'load the driver at this stage' mean. Could you give me some more specific description? Thanks~ > spark can not load alluxio fileSystem after adding jar > -- > > Key: SPARK-20019 > URL: https://issues.apache.org/jira/browse/SPARK-20019 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: roncenzhao > Attachments: exception_stack.png > > > The follwing sql cannot load alluxioSystem and it throws > `ClassNotFoundException`. > ``` > add jar /xxx/xxx/alluxioxxx.jar; > set fs.alluxio.impl=alluxio.hadoop.FileSystem; > select * from alluxionTbl; > ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20017) Functions "str_to_map" and "explode" throws NPE exceptioin
[
https://issues.apache.org/jira/browse/SPARK-20017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15931776#comment-15931776
]
roncenzhao commented on SPARK-20017:
[~maropu] Please check it, Thanks~
> Functions "str_to_map" and "explode" throws NPE exceptioin
> --
>
> Key: SPARK-20017
> URL: https://issues.apache.org/jira/browse/SPARK-20017
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: roncenzhao
> Attachments: screenshot-1.png
>
>
> ```
> val sqlDf = spark.sql("select k,v from (select str_to_map('') as map_col from
> range(2)) tbl lateral view explode(map_col) as k,v")
> sqlDf.show
> ```
> The sql throws NPE exception.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20019) spark can not load alluxio fileSystem after adding jar
[ https://issues.apache.org/jira/browse/SPARK-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15931702#comment-15931702 ] roncenzhao commented on SPARK-20019: The root cause of this exception is: The `SessionState.addJar()` only set the new jarClassLoader into current thread contextClassLoader and the classLoader of `sparkContext.hadoopConfiguration` is the old one which does not include the new jar path. In the function `InsertIntoHiveTable.getStagingDir()`, it throws the `classNotFoundException` when `inputPath.getFileSystem(hadoopConf)` is called. Because it loads the FileSystem using the `classLoader` of `hadoopConf`. > spark can not load alluxio fileSystem after adding jar > -- > > Key: SPARK-20019 > URL: https://issues.apache.org/jira/browse/SPARK-20019 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: roncenzhao > Attachments: exception_stack.png > > > The follwing sql cannot load alluxioSystem and it throws > `ClassNotFoundException`. > ``` > add jar /xxx/xxx/alluxioxxx.jar; > set fs.alluxio.impl=alluxio.hadoop.FileSystem; > select * from alluxionTbl; > ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-20019) spark can not load alluxio fileSystem after adding jar
[ https://issues.apache.org/jira/browse/SPARK-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-20019: --- Attachment: exception_stack.png > spark can not load alluxio fileSystem after adding jar > -- > > Key: SPARK-20019 > URL: https://issues.apache.org/jira/browse/SPARK-20019 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: roncenzhao > Attachments: exception_stack.png > > > The follwing sql cannot load alluxioSystem and it throws > `ClassNotFoundException`. > ``` > add jar /xxx/xxx/alluxioxxx.jar; > set fs.alluxio.impl=alluxio.hadoop.FileSystem; > select * from alluxionTbl; > ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-20019) spark can not load alluxio fileSystem after adding jar
roncenzhao created SPARK-20019: -- Summary: spark can not load alluxio fileSystem after adding jar Key: SPARK-20019 URL: https://issues.apache.org/jira/browse/SPARK-20019 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.1.0 Reporter: roncenzhao The follwing sql cannot load alluxioSystem and it throws `ClassNotFoundException`. ``` add jar /xxx/xxx/alluxioxxx.jar; set fs.alluxio.impl=alluxio.hadoop.FileSystem; select * from alluxionTbl; ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20017) Functions "str_to_map" and "explode" throws NPE exceptioin
[
https://issues.apache.org/jira/browse/SPARK-20017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15931691#comment-15931691
]
roncenzhao commented on SPARK-20017:
The root cause of this exception is:
The `StringToMap`'s `dataType` is `MapType(StringType, StringType,
valueContainsNull = false)` and the result of `str_to_map('')` is `"" -> null`,
so in the function `explode`, it will throw NPE exception.
I think we should change the `dataType` of `StringToMap` to
`MapType(StringType, StringType)`, the default value of `valueContainsNull` is
`true`.
> Functions "str_to_map" and "explode" throws NPE exceptioin
> --
>
> Key: SPARK-20017
> URL: https://issues.apache.org/jira/browse/SPARK-20017
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: roncenzhao
> Attachments: screenshot-1.png
>
>
> ```
> val sqlDf = spark.sql("select k,v from (select str_to_map('') as map_col from
> range(2)) tbl lateral view explode(map_col) as k,v")
> sqlDf.show
> ```
> The sql throws NPE exception.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-20017) Functions "str_to_map" and "explode" throws NPE exceptioin
[
https://issues.apache.org/jira/browse/SPARK-20017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
roncenzhao updated SPARK-20017:
---
Attachment: screenshot-1.png
> Functions "str_to_map" and "explode" throws NPE exceptioin
> --
>
> Key: SPARK-20017
> URL: https://issues.apache.org/jira/browse/SPARK-20017
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: roncenzhao
> Attachments: screenshot-1.png
>
>
> ```
> val sqlDf = spark.sql("select k,v from (select str_to_map('') as map_col from
> range(2)) tbl lateral view explode(map_col) as k,v")
> sqlDf.show
> ```
> The sql throws NPE exception.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-20017) Functions "str_to_map" and "explode" throws NPE exceptioin
roncenzhao created SPARK-20017:
--
Summary: Functions "str_to_map" and "explode" throws NPE exceptioin
Key: SPARK-20017
URL: https://issues.apache.org/jira/browse/SPARK-20017
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.1.0
Reporter: roncenzhao
```
val sqlDf = spark.sql("select k,v from (select str_to_map('') as map_col from
range(2)) tbl lateral view explode(map_col) as k,v")
sqlDf.show
```
The sql throws NPE exception.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19328) using spark thrift server gets memory leak problem in `ExecutorsListener`
[ https://issues.apache.org/jira/browse/SPARK-19328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833802#comment-15833802 ] roncenzhao commented on SPARK-19328: [~srowen] In the heap dump file, it shows that the `executorToLogUrls` and `executorIdToData` hold the majority of memory. So I think we should add some strategies to trim the stale data. > using spark thrift server gets memory leak problem in `ExecutorsListener` > - > > Key: SPARK-19328 > URL: https://issues.apache.org/jira/browse/SPARK-19328 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.2 > Environment: spark2.0.2 >Reporter: roncenzhao > Attachments: screenshot-1.png > > > When I use spark-thrift-server, the memory usage will gradually increase. > I found that in `ExecutorsListener` all the data about executors is never > trimmed. > So do we need some operations to trim the data ? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19328) using spark thrift server gets memory leak problem in `ExecutorsListener`
[ https://issues.apache.org/jira/browse/SPARK-19328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-19328: --- Attachment: screenshot-1.png > using spark thrift server gets memory leak problem in `ExecutorsListener` > - > > Key: SPARK-19328 > URL: https://issues.apache.org/jira/browse/SPARK-19328 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.2 > Environment: spark2.0.2 >Reporter: roncenzhao > Attachments: screenshot-1.png > > > When I use spark-thrift-server, the memory usage will gradually increase. > I found that in `ExecutorsListener` all the data about executors is never > trimmed. > So do we need some operations to trim the data ? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19328) using spark thrift server gets memory leak problem in `ExecutorsListener`
roncenzhao created SPARK-19328: -- Summary: using spark thrift server gets memory leak problem in `ExecutorsListener` Key: SPARK-19328 URL: https://issues.apache.org/jira/browse/SPARK-19328 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.0.2 Environment: spark2.0.2 Reporter: roncenzhao When I use spark-thrift-server, the memory usage will gradually increase. I found that in `ExecutorsListener` all the data about executors is never trimmed. So do we need some operations to trim the data ? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19187) querying from parquet partitioned table throws FileNotFoundException when some partitions' hdfs locations do not exist
[ https://issues.apache.org/jira/browse/SPARK-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15821269#comment-15821269 ] roncenzhao commented on SPARK-19187: I think this problem has been resolved in SPARK-17599. I will close this issue. Thanks~ > querying from parquet partitioned table throws FileNotFoundException when > some partitions' hdfs locations do not exist > -- > > Key: SPARK-19187 > URL: https://issues.apache.org/jira/browse/SPARK-19187 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.2 >Reporter: roncenzhao > > Hi, all. > When the parquet partitioned table's some partition's hdfs paths do not > exist, querying from it throws FileNotFoundException . > The error stack is : > ` > TaskSetManager: Lost task 522.0 in stage 1.0 (TID 523, > sd-hadoop-datanode-50-135.i > dc.vip.com): java.io.FileNotFoundException: File does not exist: > hdfs://bipcluster/bip/external_table/vip > dw/dw_log_app_pageinfo_clean_spark_parquet/dt=20161223/hm=1730 > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1128) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120) > at > org.apache.spark.sql.execution.datasources.HadoopFsRelation$$anonfun$7$$anonfun$apply$3.apply( > fileSourceInterfaces.scala:465) > at > org.apache.spark.sql.execution.datasources.HadoopFsRelation$$anonfun$7$$anonfun$apply$3.apply( > fileSourceInterfaces.scala:462) > at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) > at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) > at scala.collection.Iterator$class.foreach(Iterator.scala:893) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) > at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) > at > scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104) > at > scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48) > at > scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310) > at scala.collection.AbstractIterator.to(Iterator.scala:1336) > at > scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302) > at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336) > at > scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289) > at scala.collection.AbstractIterator.toArray(Iterator.scala:1336) > at > org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:912) > at > org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:912) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19187) querying from parquet partitioned table throws FileNotFoundException when some partitions' hdfs locations do not exist
[ https://issues.apache.org/jira/browse/SPARK-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15821196#comment-15821196 ] roncenzhao commented on SPARK-19187: In the method `HadoopTableReader.makeRDDForPartitionedTable()` we will verify the partition paths if `spark.sql.hive.verifyPartitionPath` is true. So can we add a configuration item to control whether throw exception or not in this case? > querying from parquet partitioned table throws FileNotFoundException when > some partitions' hdfs locations do not exist > -- > > Key: SPARK-19187 > URL: https://issues.apache.org/jira/browse/SPARK-19187 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.2 >Reporter: roncenzhao > > Hi, all. > When the parquet partitioned table's some partition's hdfs paths do not > exist, querying from it throws FileNotFoundException . > The error stack is : > ` > TaskSetManager: Lost task 522.0 in stage 1.0 (TID 523, > sd-hadoop-datanode-50-135.i > dc.vip.com): java.io.FileNotFoundException: File does not exist: > hdfs://bipcluster/bip/external_table/vip > dw/dw_log_app_pageinfo_clean_spark_parquet/dt=20161223/hm=1730 > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1128) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120) > at > org.apache.spark.sql.execution.datasources.HadoopFsRelation$$anonfun$7$$anonfun$apply$3.apply( > fileSourceInterfaces.scala:465) > at > org.apache.spark.sql.execution.datasources.HadoopFsRelation$$anonfun$7$$anonfun$apply$3.apply( > fileSourceInterfaces.scala:462) > at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) > at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) > at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) > at scala.collection.Iterator$class.foreach(Iterator.scala:893) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) > at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) > at > scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104) > at > scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48) > at > scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310) > at scala.collection.AbstractIterator.to(Iterator.scala:1336) > at > scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302) > at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336) > at > scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289) > at scala.collection.AbstractIterator.toArray(Iterator.scala:1336) > at > org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:912) > at > org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:912) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19187) querying from parquet partitioned table throws FileNotFoundException when some partitions' hdfs locations do not exist
roncenzhao created SPARK-19187: -- Summary: querying from parquet partitioned table throws FileNotFoundException when some partitions' hdfs locations do not exist Key: SPARK-19187 URL: https://issues.apache.org/jira/browse/SPARK-19187 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.0.2 Reporter: roncenzhao Hi, all. When the parquet partitioned table's some partition's hdfs paths do not exist, querying from it throws FileNotFoundException . The error stack is : ` TaskSetManager: Lost task 522.0 in stage 1.0 (TID 523, sd-hadoop-datanode-50-135.i dc.vip.com): java.io.FileNotFoundException: File does not exist: hdfs://bipcluster/bip/external_table/vip dw/dw_log_app_pageinfo_clean_spark_parquet/dt=20161223/hm=1730 at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1128) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120) at org.apache.spark.sql.execution.datasources.HadoopFsRelation$$anonfun$7$$anonfun$apply$3.apply( fileSourceInterfaces.scala:465) at org.apache.spark.sql.execution.datasources.HadoopFsRelation$$anonfun$7$$anonfun$apply$3.apply( fileSourceInterfaces.scala:462) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48) at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310) at scala.collection.AbstractIterator.to(Iterator.scala:1336) at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336) at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289) at scala.collection.AbstractIterator.toArray(Iterator.scala:1336) at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:912) at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:912) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) at org.apache.spark.scheduler.Task.run(Task.scala:86) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19169) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files
[ https://issues.apache.org/jira/browse/SPARK-19169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15818635#comment-15818635 ] roncenzhao commented on SPARK-19169: I have the two doubts: 1. In the method `HiveTableScanExec.addColumnMetadataToConf(conf)`, we set the `serdeConstants.LIST_COLUMN_TYPES` and `serdeConstants.LIST_COLUMNS` into `hadoopConf`. 2. In the method `HadoopTableReader.initializeLocalJobConfFunc(path, tableDesc)`, we set the table's properties which include `serdeConstants.LIST_COLUMN_TYPES` and `serdeConstants.LIST_COLUMNS` into jobConf. I think it's the two points that cause this problem. When I remove this two methods, the sql will run successfully. I don't know why we must set the `serdeConstants.LIST_COLUMN_TYPES` and `serdeConstants.LIST_COLUMNS` into jobConf. Thanks~ > columns changed orc table encouter 'IndexOutOfBoundsException' when read the > old schema files > - > > Key: SPARK-19169 > URL: https://issues.apache.org/jira/browse/SPARK-19169 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.2 >Reporter: roncenzhao > > We hava an orc table called orc_test_tbl and hava inserted some data into it. > After that, we change the table schema by droping some columns. > When reading the old schema file, we get the follow exception. > ``` > java.lang.IndexOutOfBoundsException: toIndex = 65 > at java.util.ArrayList.subListRangeCheck(ArrayList.java:962) > at java.util.ArrayList.subList(ArrayList.java:954) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113) > at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ``` -- T
[jira] [Commented] (SPARK-19169) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files
[ https://issues.apache.org/jira/browse/SPARK-19169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15818552#comment-15818552 ] roncenzhao commented on SPARK-19169: I do not think this is a misusage of ORC. If we do not set the `serdeConstants.LIST_COLUMN_TYPES` and `serdeConstants.LIST_COLUMNS` of table schema into `hadoopConf` and let the orc reader get the schema info by reading orc file, we can run the sql successfully. > columns changed orc table encouter 'IndexOutOfBoundsException' when read the > old schema files > - > > Key: SPARK-19169 > URL: https://issues.apache.org/jira/browse/SPARK-19169 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.2 >Reporter: roncenzhao > > We hava an orc table called orc_test_tbl and hava inserted some data into it. > After that, we change the table schema by droping some columns. > When reading the old schema file, we get the follow exception. > ``` > java.lang.IndexOutOfBoundsException: toIndex = 65 > at java.util.ArrayList.subListRangeCheck(ArrayList.java:962) > at java.util.ArrayList.subList(ArrayList.java:954) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113) > at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19175) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files
[ https://issues.apache.org/jira/browse/SPARK-19175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15817989#comment-15817989 ] roncenzhao commented on SPARK-19175: [~srowen] I am very sorry for my mistake and I will be more careful next time. Let's discuss this problem in this jira. Thanks~ > columns changed orc table encouter 'IndexOutOfBoundsException' when read the > old schema files > - > > Key: SPARK-19175 > URL: https://issues.apache.org/jira/browse/SPARK-19175 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.2, 2.1.0 > Environment: spark2.0.2 >Reporter: roncenzhao > > We hava an orc table called orc_test_tbl and hava inserted some data into it. > After that, we change the table schema by droping some columns. > When reading the old schema file, we get the follow exception. > But hive can read it successfully. > ``` > java.lang.IndexOutOfBoundsException: toIndex = 65 > at java.util.ArrayList.subListRangeCheck(ArrayList.java:962) > at java.util.ArrayList.subList(ArrayList.java:954) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113) > at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19175) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files
[ https://issues.apache.org/jira/browse/SPARK-19175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15817970#comment-15817970 ] roncenzhao commented on SPARK-19175: Sorry for that. I will be more careful next time. I think this bug is a little different from SPARK-16628. The root cause of this bug is that we read the old schema file with the new schema of the table. If we read the old schema file with the schema in the orcFile, it will not throw exception. > columns changed orc table encouter 'IndexOutOfBoundsException' when read the > old schema files > - > > Key: SPARK-19175 > URL: https://issues.apache.org/jira/browse/SPARK-19175 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.2, 2.1.0 > Environment: spark2.0.2 >Reporter: roncenzhao > > We hava an orc table called orc_test_tbl and hava inserted some data into it. > After that, we change the table schema by droping some columns. > When reading the old schema file, we get the follow exception. > But hive can read it successfully. > ``` > java.lang.IndexOutOfBoundsException: toIndex = 65 > at java.util.ArrayList.subListRangeCheck(ArrayList.java:962) > at java.util.ArrayList.subList(ArrayList.java:954) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113) > at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19170) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files
[ https://issues.apache.org/jira/browse/SPARK-19170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15817957#comment-15817957 ] roncenzhao commented on SPARK-19170: Sorry for that!!! > columns changed orc table encouter 'IndexOutOfBoundsException' when read the > old schema files > - > > Key: SPARK-19170 > URL: https://issues.apache.org/jira/browse/SPARK-19170 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.2 >Reporter: roncenzhao > > We hava an orc table called orc_test_tbl and hava inserted some data into it. > After that, we change the table schema by droping some columns. > When reading the old schema file, we get the follow exception. > ``` > java.lang.IndexOutOfBoundsException: toIndex = 65 > at java.util.ArrayList.subListRangeCheck(ArrayList.java:962) > at java.util.ArrayList.subList(ArrayList.java:954) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113) > at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-19173) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files
[ https://issues.apache.org/jira/browse/SPARK-19173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao closed SPARK-19173. -- > columns changed orc table encouter 'IndexOutOfBoundsException' when read the > old schema files > - > > Key: SPARK-19173 > URL: https://issues.apache.org/jira/browse/SPARK-19173 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.2 >Reporter: roncenzhao > > We hava an orc table called orc_test_tbl and hava inserted some data into it. > After that, we change the table schema by droping some columns. > When reading the old schema file, we get the follow exception. > ``` > java.lang.IndexOutOfBoundsException: toIndex = 65 > at java.util.ArrayList.subListRangeCheck(ArrayList.java:962) > at java.util.ArrayList.subList(ArrayList.java:954) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113) > at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-19171) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files
[ https://issues.apache.org/jira/browse/SPARK-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao closed SPARK-19171. -- duplicate > columns changed orc table encouter 'IndexOutOfBoundsException' when read the > old schema files > - > > Key: SPARK-19171 > URL: https://issues.apache.org/jira/browse/SPARK-19171 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.2 >Reporter: roncenzhao > > We hava an orc table called orc_test_tbl and hava inserted some data into it. > After that, we change the table schema by droping some columns. > When reading the old schema file, we get the follow exception. > ``` > java.lang.IndexOutOfBoundsException: toIndex = 65 > at java.util.ArrayList.subListRangeCheck(ArrayList.java:962) > at java.util.ArrayList.subList(ArrayList.java:954) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113) > at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-19175) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files
[ https://issues.apache.org/jira/browse/SPARK-19175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15817717#comment-15817717 ] roncenzhao edited comment on SPARK-19175 at 1/11/17 8:56 AM: - I have the two doubts: 1. In the method `HiveTableScanExec.addColumnMetadataToConf(conf)`, we set the `serdeConstants.LIST_COLUMN_TYPES` and `serdeConstants.LIST_COLUMNS` into `hadoopConf`. 2. In the method `HadoopTableReader.initializeLocalJobConfFunc(path, tableDesc)`, we set the table's properties which include `serdeConstants.LIST_COLUMN_TYPES` and `serdeConstants.LIST_COLUMNS` into jobConf. I think it's the two points that cause this problem. When I remove this two methods, the sql will run successfully. I don't know why we must set the `serdeConstants.LIST_COLUMN_TYPES` and `serdeConstants.LIST_COLUMNS` into jobConf. Would anyone please tell me the reason? Thanks ~~~ was (Author: roncenzhao): I have the two doubts: 1. In the method `HiveTableScanExec.addColumnMetadataToConf(conf)`, we set the `serdeConstants.LIST_COLUMN_TYPES` and `serdeConstants.LIST_COLUMNS` into `hadoopConf`. 2. In the method `HadoopTableReader.initializeLocalJobConfFunc(path, tableDesc)`, we set the table's properties which include `serdeConstants.LIST_COLUMN_TYPES` and `serdeConstants.LIST_COLUMNS` into jobConf. I think it's the two points that cause this problem. When I remove this two method, the sql will run successfully. I don't know why we must set the `serdeConstants.LIST_COLUMN_TYPES` and `serdeConstants.LIST_COLUMNS` into jobConf. Would anyone please tell me the reason? Thanks ~~~ > columns changed orc table encouter 'IndexOutOfBoundsException' when read the > old schema files > - > > Key: SPARK-19175 > URL: https://issues.apache.org/jira/browse/SPARK-19175 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.2, 2.1.0 > Environment: spark2.0.2 >Reporter: roncenzhao > > We hava an orc table called orc_test_tbl and hava inserted some data into it. > After that, we change the table schema by droping some columns. > When reading the old schema file, we get the follow exception. > But hive can read it successfully. > ``` > java.lang.IndexOutOfBoundsException: toIndex = 65 > at java.util.ArrayList.subListRangeCheck(ArrayList.java:962) > at java.util.ArrayList.subList(ArrayList.java:954) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113) > at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.
[jira] [Commented] (SPARK-19175) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files
[ https://issues.apache.org/jira/browse/SPARK-19175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15817717#comment-15817717 ] roncenzhao commented on SPARK-19175: I have the two doubts: 1. In the method `HiveTableScanExec.addColumnMetadataToConf(conf)`, we set the `serdeConstants.LIST_COLUMN_TYPES` and `serdeConstants.LIST_COLUMNS` into `hadoopConf`. 2. In the method `HadoopTableReader.initializeLocalJobConfFunc(path, tableDesc)`, we set the table's properties which include `serdeConstants.LIST_COLUMN_TYPES` and `serdeConstants.LIST_COLUMNS` into jobConf. I think it's the two points that cause this problem. When I remove this two method, the sql will run successfully. I don't know why we must set the `serdeConstants.LIST_COLUMN_TYPES` and `serdeConstants.LIST_COLUMNS` into jobConf. Would anyone please tell me the reason? Thanks ~~~ > columns changed orc table encouter 'IndexOutOfBoundsException' when read the > old schema files > - > > Key: SPARK-19175 > URL: https://issues.apache.org/jira/browse/SPARK-19175 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.2, 2.1.0 > Environment: spark2.0.2 >Reporter: roncenzhao > > We hava an orc table called orc_test_tbl and hava inserted some data into it. > After that, we change the table schema by droping some columns. > When reading the old schema file, we get the follow exception. > But hive can read it successfully. > ``` > java.lang.IndexOutOfBoundsException: toIndex = 65 > at java.util.ArrayList.subListRangeCheck(ArrayList.java:962) > at java.util.ArrayList.subList(ArrayList.java:954) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226) > at > org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113) > at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) > at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) > at org.apache.spark.scheduler.Task.run(Task.scala:86) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.Thr
[jira] [Created] (SPARK-19175) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files
roncenzhao created SPARK-19175: -- Summary: columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files Key: SPARK-19175 URL: https://issues.apache.org/jira/browse/SPARK-19175 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.1.0, 2.0.2 Environment: spark2.0.2 Reporter: roncenzhao We hava an orc table called orc_test_tbl and hava inserted some data into it. After that, we change the table schema by droping some columns. When reading the old schema file, we get the follow exception. But hive can read it successfully. ``` java.lang.IndexOutOfBoundsException: toIndex = 65 at java.util.ArrayList.subListRangeCheck(ArrayList.java:962) at java.util.ArrayList.subList(ArrayList.java:954) at org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161) at org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113) at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) at org.apache.spark.scheduler.Task.run(Task.scala:86) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19174) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files
roncenzhao created SPARK-19174: -- Summary: columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files Key: SPARK-19174 URL: https://issues.apache.org/jira/browse/SPARK-19174 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.0.2 Reporter: roncenzhao We hava an orc table called orc_test_tbl and hava inserted some data into it. After that, we change the table schema by droping some columns. When reading the old schema file, we get the follow exception. ``` java.lang.IndexOutOfBoundsException: toIndex = 65 at java.util.ArrayList.subListRangeCheck(ArrayList.java:962) at java.util.ArrayList.subList(ArrayList.java:954) at org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161) at org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113) at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) at org.apache.spark.scheduler.Task.run(Task.scala:86) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19173) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files
roncenzhao created SPARK-19173: -- Summary: columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files Key: SPARK-19173 URL: https://issues.apache.org/jira/browse/SPARK-19173 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.0.2 Reporter: roncenzhao We hava an orc table called orc_test_tbl and hava inserted some data into it. After that, we change the table schema by droping some columns. When reading the old schema file, we get the follow exception. ``` java.lang.IndexOutOfBoundsException: toIndex = 65 at java.util.ArrayList.subListRangeCheck(ArrayList.java:962) at java.util.ArrayList.subList(ArrayList.java:954) at org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161) at org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113) at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) at org.apache.spark.scheduler.Task.run(Task.scala:86) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19172) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files
roncenzhao created SPARK-19172: -- Summary: columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files Key: SPARK-19172 URL: https://issues.apache.org/jira/browse/SPARK-19172 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.0.2 Reporter: roncenzhao We hava an orc table called orc_test_tbl and hava inserted some data into it. After that, we change the table schema by droping some columns. When reading the old schema file, we get the follow exception. ``` java.lang.IndexOutOfBoundsException: toIndex = 65 at java.util.ArrayList.subListRangeCheck(ArrayList.java:962) at java.util.ArrayList.subList(ArrayList.java:954) at org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161) at org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113) at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) at org.apache.spark.scheduler.Task.run(Task.scala:86) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19171) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files
roncenzhao created SPARK-19171: -- Summary: columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files Key: SPARK-19171 URL: https://issues.apache.org/jira/browse/SPARK-19171 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.0.2 Reporter: roncenzhao We hava an orc table called orc_test_tbl and hava inserted some data into it. After that, we change the table schema by droping some columns. When reading the old schema file, we get the follow exception. ``` java.lang.IndexOutOfBoundsException: toIndex = 65 at java.util.ArrayList.subListRangeCheck(ArrayList.java:962) at java.util.ArrayList.subList(ArrayList.java:954) at org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161) at org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113) at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) at org.apache.spark.scheduler.Task.run(Task.scala:86) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19169) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files
roncenzhao created SPARK-19169: -- Summary: columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files Key: SPARK-19169 URL: https://issues.apache.org/jira/browse/SPARK-19169 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.0.2 Reporter: roncenzhao We hava an orc table called orc_test_tbl and hava inserted some data into it. After that, we change the table schema by droping some columns. When reading the old schema file, we get the follow exception. ``` java.lang.IndexOutOfBoundsException: toIndex = 65 at java.util.ArrayList.subListRangeCheck(ArrayList.java:962) at java.util.ArrayList.subList(ArrayList.java:954) at org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161) at org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113) at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) at org.apache.spark.scheduler.Task.run(Task.scala:86) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19170) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files
roncenzhao created SPARK-19170: -- Summary: columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files Key: SPARK-19170 URL: https://issues.apache.org/jira/browse/SPARK-19170 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.0.2 Reporter: roncenzhao We hava an orc table called orc_test_tbl and hava inserted some data into it. After that, we change the table schema by droping some columns. When reading the old schema file, we get the follow exception. ``` java.lang.IndexOutOfBoundsException: toIndex = 65 at java.util.ArrayList.subListRangeCheck(ArrayList.java:962) at java.util.ArrayList.subList(ArrayList.java:954) at org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161) at org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:539) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1215) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113) at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:208) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) at org.apache.spark.scheduler.Task.run(Task.scala:86) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18981) The last job hung when speculation is on
[ https://issues.apache.org/jira/browse/SPARK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-18981: --- Description: CONF: spark.speculation true spark.dynamicAllocation.minExecutors0 spark.executor.cores 2 When I run the follow app, the bug will trigger. ``` sc.runJob(job1) sleep(100s) sc.runJob(job2) // the job2 will hang and never be scheduled ``` The triggering condition is described as follows: condition1: During the sleeping time, the executors will be released and the # of the executor will be zero some seconds later. The #numExecutorsTarget in 'ExecutorAllocationManager' will be 0. condition2: In 'ExecutorAllocationListener.onTaskEnd()', the numRunningTasks will be negative during the ending of job1's tasks. condition3: The job2 only hava one task. result: In the method 'ExecutorAllocationManager.updateAndSyncNumExecutorsTarget()', we will calculate #maxNeeded in 'maxNumExecutorsNeeded()'. Obviously, #numRunningOrPendingTasks will be negative and the #maxNeeded will be 0 or negative. So the 'ExecutorAllocationManager' will not request container from yarn. The app will hang. In the attachments, submitting the app by 'run_scala.sh' will lead to the 'hang' problem as the 'job_hang.png' shows. was: CONF: spark.speculation true spark.dynamicAllocation.minExecutors0 spark.executor.cores 4 When I run the follow app, the bug will trigger. ``` sc.runJob(job1) sleep(100s) sc.runJob(job2) // the job2 will hang and never be scheduled ``` The triggering condition is described as follows: condition1: During the sleeping time, the executors will be released and the # of the executor will be zero some seconds later. The #numExecutorsTarget in 'ExecutorAllocationManager' will be 0. condition2: In 'ExecutorAllocationListener.onTaskEnd()', the numRunningTasks will be negative during the ending of job1's tasks. condition3: The job2 only hava one task. result: In the method 'ExecutorAllocationManager.updateAndSyncNumExecutorsTarget()', we will calculate #maxNeeded in 'maxNumExecutorsNeeded()'. Obviously, #numRunningOrPendingTasks will be negative and the #maxNeeded will be 0 or negative. So the 'ExecutorAllocationManager' will not request container from yarn. The app will hang. In the attachments, submitting the app by 'run_scala.sh' will lead to the 'hang' problem as the 'job_hang.png' shows. > The last job hung when speculation is on > > > Key: SPARK-18981 > URL: https://issues.apache.org/jira/browse/SPARK-18981 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.2 > Environment: spark2.0.2 > hadoop2.5.0 >Reporter: roncenzhao >Priority: Critical > Attachments: Test.scala, job_hang.png, run_scala.sh > > > CONF: > spark.speculation true > spark.dynamicAllocation.minExecutors0 > spark.executor.cores 2 > When I run the follow app, the bug will trigger. > ``` > sc.runJob(job1) > sleep(100s) > sc.runJob(job2) // the job2 will hang and never be scheduled > ``` > The triggering condition is described as follows: > condition1: During the sleeping time, the executors will be released and the > # of the executor will be zero some seconds later. The #numExecutorsTarget in > 'ExecutorAllocationManager' will be 0. > condition2: In 'ExecutorAllocationListener.onTaskEnd()', the numRunningTasks > will be negative during the ending of job1's tasks. > condition3: The job2 only hava one task. > result: > In the method 'ExecutorAllocationManager.updateAndSyncNumExecutorsTarget()', > we will calculate #maxNeeded in 'maxNumExecutorsNeeded()'. Obviously, > #numRunningOrPendingTasks will be negative and the #maxNeeded will be 0 or > negative. So the 'ExecutorAllocationManager' will not request container from > yarn. The app will hang. > In the attachments, submitting the app by 'run_scala.sh' will lead to the > 'hang' problem as the 'job_hang.png' shows. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18981) The last job hung when speculation is on
[ https://issues.apache.org/jira/browse/SPARK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-18981: --- Description: CONF: spark.speculation true spark.dynamicAllocation.minExecutors0 spark.executor.cores 4 When I run the follow app, the bug will trigger. ``` sc.runJob(job1) sleep(100s) sc.runJob(job2) // the job2 will hang and never be scheduled ``` The triggering condition is described as follows: condition1: During the sleeping time, the executors will be released and the # of the executor will be zero some seconds later. The #numExecutorsTarget in 'ExecutorAllocationManager' will be 0. condition2: In 'ExecutorAllocationListener.onTaskEnd()', the numRunningTasks will be negative during the ending of job1's tasks. condition3: The job2 only hava one task. result: In the method 'ExecutorAllocationManager.updateAndSyncNumExecutorsTarget()', we will calculate #maxNeeded in 'maxNumExecutorsNeeded()'. Obviously, #numRunningOrPendingTasks will be negative and the #maxNeeded will be 0 or negative. So the 'ExecutorAllocationManager' will not request container from yarn. The app will hang. In the attachments, submitting the app by 'run_scala.sh' will lead to the 'hang' problem as the 'job_hang.png' shows. was: CONF: spark.speculation true spark.dynamicAllocation.minExecutors0 spark.executor.cores 4 When I run the follow app, the bug will trigger. ``` sc.runJob(job1) sleep(100s) sc.runJob(job2) // the job2 will hang and never be scheduled ``` The triggering condition is described as follows: condition1: During the sleeping time, the executors will be released and the # of the executor will be zero some seconds later. The #numExecutorsTarget in 'ExecutorAllocationManager' will be 0. condition2: In 'ExecutorAllocationListener.onTaskEnd()', the numRunningTasks will be negative during the ending of job1's tasks. condition3: The job2 only hava one task. result: In the method 'ExecutorAllocationManager.updateAndSyncNumExecutorsTarget()', we will calculate #maxNeeded in 'maxNumExecutorsNeeded()'. Obviously, #numRunningOrPendingTasks will be negative and the #maxNeeded will be 0 or negative. So the 'ExecutorAllocationManager' will not request container from yarn. The app will hang. > The last job hung when speculation is on > > > Key: SPARK-18981 > URL: https://issues.apache.org/jira/browse/SPARK-18981 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.2 > Environment: spark2.0.2 > hadoop2.5.0 >Reporter: roncenzhao >Priority: Critical > Attachments: Test.scala, job_hang.png, run_scala.sh > > > CONF: > spark.speculation true > spark.dynamicAllocation.minExecutors0 > spark.executor.cores 4 > When I run the follow app, the bug will trigger. > ``` > sc.runJob(job1) > sleep(100s) > sc.runJob(job2) // the job2 will hang and never be scheduled > ``` > The triggering condition is described as follows: > condition1: During the sleeping time, the executors will be released and the > # of the executor will be zero some seconds later. The #numExecutorsTarget in > 'ExecutorAllocationManager' will be 0. > condition2: In 'ExecutorAllocationListener.onTaskEnd()', the numRunningTasks > will be negative during the ending of job1's tasks. > condition3: The job2 only hava one task. > result: > In the method 'ExecutorAllocationManager.updateAndSyncNumExecutorsTarget()', > we will calculate #maxNeeded in 'maxNumExecutorsNeeded()'. Obviously, > #numRunningOrPendingTasks will be negative and the #maxNeeded will be 0 or > negative. So the 'ExecutorAllocationManager' will not request container from > yarn. The app will hang. > In the attachments, submitting the app by 'run_scala.sh' will lead to the > 'hang' problem as the 'job_hang.png' shows. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18981) The last job hung when speculation is on
[ https://issues.apache.org/jira/browse/SPARK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-18981: --- Attachment: run_scala.sh > The last job hung when speculation is on > > > Key: SPARK-18981 > URL: https://issues.apache.org/jira/browse/SPARK-18981 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.2 > Environment: spark2.0.2 > hadoop2.5.0 >Reporter: roncenzhao >Priority: Critical > Attachments: Test.scala, job_hang.png, run_scala.sh > > > CONF: > spark.speculation true > spark.dynamicAllocation.minExecutors0 > spark.executor.cores 4 > When I run the follow app, the bug will trigger. > ``` > sc.runJob(job1) > sleep(100s) > sc.runJob(job2) // the job2 will hang and never be scheduled > ``` > The triggering condition is described as follows: > condition1: During the sleeping time, the executors will be released and the > # of the executor will be zero some seconds later. The #numExecutorsTarget in > 'ExecutorAllocationManager' will be 0. > condition2: In 'ExecutorAllocationListener.onTaskEnd()', the numRunningTasks > will be negative during the ending of job1's tasks. > condition3: The job2 only hava one task. > result: > In the method 'ExecutorAllocationManager.updateAndSyncNumExecutorsTarget()', > we will calculate #maxNeeded in 'maxNumExecutorsNeeded()'. Obviously, > #numRunningOrPendingTasks will be negative and the #maxNeeded will be 0 or > negative. So the 'ExecutorAllocationManager' will not request container from > yarn. The app will hang. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18981) The last job hung when speculation is on
[ https://issues.apache.org/jira/browse/SPARK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-18981: --- Attachment: Test.scala > The last job hung when speculation is on > > > Key: SPARK-18981 > URL: https://issues.apache.org/jira/browse/SPARK-18981 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.2 > Environment: spark2.0.2 > hadoop2.5.0 >Reporter: roncenzhao >Priority: Critical > Attachments: Test.scala, job_hang.png > > > CONF: > spark.speculation true > spark.dynamicAllocation.minExecutors0 > spark.executor.cores 4 > When I run the follow app, the bug will trigger. > ``` > sc.runJob(job1) > sleep(100s) > sc.runJob(job2) // the job2 will hang and never be scheduled > ``` > The triggering condition is described as follows: > condition1: During the sleeping time, the executors will be released and the > # of the executor will be zero some seconds later. The #numExecutorsTarget in > 'ExecutorAllocationManager' will be 0. > condition2: In 'ExecutorAllocationListener.onTaskEnd()', the numRunningTasks > will be negative during the ending of job1's tasks. > condition3: The job2 only hava one task. > result: > In the method 'ExecutorAllocationManager.updateAndSyncNumExecutorsTarget()', > we will calculate #maxNeeded in 'maxNumExecutorsNeeded()'. Obviously, > #numRunningOrPendingTasks will be negative and the #maxNeeded will be 0 or > negative. So the 'ExecutorAllocationManager' will not request container from > yarn. The app will hang. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18981) The last job hung when speculation is on
[ https://issues.apache.org/jira/browse/SPARK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-18981: --- Attachment: job_hang.png > The last job hung when speculation is on > > > Key: SPARK-18981 > URL: https://issues.apache.org/jira/browse/SPARK-18981 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.2 > Environment: spark2.0.2 > hadoop2.5.0 >Reporter: roncenzhao >Priority: Critical > Attachments: job_hang.png > > > CONF: > spark.speculation true > spark.dynamicAllocation.minExecutors0 > spark.executor.cores 4 > When I run the follow app, the bug will trigger. > ``` > sc.runJob(job1) > sleep(100s) > sc.runJob(job2) // the job2 will hang and never be scheduled > ``` > The triggering condition is described as follows: > condition1: During the sleeping time, the executors will be released and the > # of the executor will be zero some seconds later. The #numExecutorsTarget in > 'ExecutorAllocationManager' will be 0. > condition2: In 'ExecutorAllocationListener.onTaskEnd()', the numRunningTasks > will be negative during the ending of job1's tasks. > condition3: The job2 only hava one task. > result: > In the method 'ExecutorAllocationManager.updateAndSyncNumExecutorsTarget()', > we will calculate #maxNeeded in 'maxNumExecutorsNeeded()'. Obviously, > #numRunningOrPendingTasks will be negative and the #maxNeeded will be 0 or > negative. So the 'ExecutorAllocationManager' will not request container from > yarn. The app will hang. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18981) The last job hung when speculation is on
[ https://issues.apache.org/jira/browse/SPARK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-18981: --- Description: CONF: spark.speculation true spark.dynamicAllocation.minExecutors0 spark.executor.cores 4 When I run the follow app, the bug will trigger. ``` sc.runJob(job1) sleep(100s) sc.runJob(job2) // the job2 will hang and never be scheduled ``` The triggering condition is described as follows: condition1: During the sleeping time, the executors will be released and the # of the executor will be zero some seconds later. The #numExecutorsTarget in 'ExecutorAllocationManager' will be 0. condition2: In 'ExecutorAllocationListener.onTaskEnd()', the numRunningTasks will be negative during the ending of job1's tasks. condition3: The job2 only hava one task. result: In the method 'ExecutorAllocationManager.updateAndSyncNumExecutorsTarget()', we will calculate #maxNeeded in 'maxNumExecutorsNeeded()'. Obviously, #numRunningOrPendingTasks will be negative and the #maxNeeded will be 0 or negative. So the 'ExecutorAllocationManager' will not request container from yarn. The app will hang. was: related settings: spark.speculation true spark.dynamicAllocation.minExecutors0 spark.executor.cores 4 When I run the follow app, the bug will trigger. ``` sc.runJob(job1) sleep(100s) sc.runJob(job2) // the job2 will hang and never be scheduled ``` The triggering condition is described as follows: condition1: During the sleeping time, the executors will be released and the # of the executor will be zero some seconds later. The #numExecutorsTarget in 'ExecutorAllocationManager' will be 0. condition2: In 'ExecutorAllocationListener.onTaskEnd()', the numRunningTasks will be negative during the ending of job1's tasks. condition3: The job2 only hava one task. result: In the method 'ExecutorAllocationManager.updateAndSyncNumExecutorsTarget()', we will calculate #maxNeeded in 'maxNumExecutorsNeeded()'. Obviously, #numRunningOrPendingTasks will be negative and the #maxNeeded will be 0 or negative. So the 'ExecutorAllocationManager' will not request container from yarn. The app will hang. > The last job hung when speculation is on > > > Key: SPARK-18981 > URL: https://issues.apache.org/jira/browse/SPARK-18981 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.2 > Environment: spark2.0.2 > hadoop2.5.0 >Reporter: roncenzhao >Priority: Critical > > CONF: > spark.speculation true > spark.dynamicAllocation.minExecutors0 > spark.executor.cores 4 > When I run the follow app, the bug will trigger. > ``` > sc.runJob(job1) > sleep(100s) > sc.runJob(job2) // the job2 will hang and never be scheduled > ``` > The triggering condition is described as follows: > condition1: During the sleeping time, the executors will be released and the > # of the executor will be zero some seconds later. The #numExecutorsTarget in > 'ExecutorAllocationManager' will be 0. > condition2: In 'ExecutorAllocationListener.onTaskEnd()', the numRunningTasks > will be negative during the ending of job1's tasks. > condition3: The job2 only hava one task. > result: > In the method 'ExecutorAllocationManager.updateAndSyncNumExecutorsTarget()', > we will calculate #maxNeeded in 'maxNumExecutorsNeeded()'. Obviously, > #numRunningOrPendingTasks will be negative and the #maxNeeded will be 0 or > negative. So the 'ExecutorAllocationManager' will not request container from > yarn. The app will hang. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18981) The last job hung when speculation is on
[ https://issues.apache.org/jira/browse/SPARK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-18981: --- Description: related settings: spark.speculation true spark.dynamicAllocation.minExecutors0 spark.executor.cores 4 When I run the follow app, the bug will trigger. ``` sc.runJob(job1) sleep(100s) sc.runJob(job2) // the job2 will hang and never be scheduled ``` The triggering condition is described as follows: condition1: During the sleeping time, the executors will be released and the # of the executor will be zero some seconds later. The #numExecutorsTarget in 'ExecutorAllocationManager' will be 0. condition2: In 'ExecutorAllocationListener.onTaskEnd()', the numRunningTasks will be negative during the ending of job1's tasks. condition3: The job2 only hava one task. result: In the method 'ExecutorAllocationManager.updateAndSyncNumExecutorsTarget()', we will calculate #maxNeeded in 'maxNumExecutorsNeeded()'. Obviously, #numRunningOrPendingTasks will be negative and the #maxNeeded will be 0 or negative. So the 'ExecutorAllocationManager' will not request container from yarn. The app will hang. was: related settings: spark.speculation true spark.dynamicAllocation.minExecutors0 spark.executor.cores 4 When I run the follow app, the bug will trigger. ``` sc.runJob(job1) sleep(100s) sc.runJob(job2) // the job2 will hang and never be scheduled ``` The triggering condition is described as follows: condition1: During the sleeping time, the executors will be released and the # of the executor will be zero some seconds later. The #numExecutorsTarget in 'ExecutorAllocationManager' will be 0. condition2: In 'ExecutorAllocationListener.onTaskEnd()', the numRunningTasks will be negative during the ending of job1's tasks. condition3: The job2 only hava one task. result: In the method 'ExecutorAllocationManager.updateAndSyncNumExecutorsTarget()', we will calculate #maxNeeded in 'maxNumExecutorsNeeded()'. Obviously, #numRunningOrPendingTasks will be negative and the #maxNeeded will be 0 or negative. So the 'ExecutorAllocationManager' will not request container from yarn. The app will be hung. > The last job hung when speculation is on > > > Key: SPARK-18981 > URL: https://issues.apache.org/jira/browse/SPARK-18981 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.2 > Environment: spark2.0.2 > hadoop2.5.0 >Reporter: roncenzhao >Priority: Critical > > related settings: > spark.speculation true > spark.dynamicAllocation.minExecutors0 > spark.executor.cores 4 > When I run the follow app, the bug will trigger. > ``` > sc.runJob(job1) > sleep(100s) > sc.runJob(job2) // the job2 will hang and never be scheduled > ``` > The triggering condition is described as follows: > condition1: During the sleeping time, the executors will be released and the > # of the executor will be zero some seconds later. The #numExecutorsTarget in > 'ExecutorAllocationManager' will be 0. > condition2: In 'ExecutorAllocationListener.onTaskEnd()', the numRunningTasks > will be negative during the ending of job1's tasks. > condition3: The job2 only hava one task. > result: > In the method 'ExecutorAllocationManager.updateAndSyncNumExecutorsTarget()', > we will calculate #maxNeeded in 'maxNumExecutorsNeeded()'. Obviously, > #numRunningOrPendingTasks will be negative and the #maxNeeded will be 0 or > negative. So the 'ExecutorAllocationManager' will not request container from > yarn. The app will hang. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18981) The last job hung when speculation is on
roncenzhao created SPARK-18981: -- Summary: The last job hung when speculation is on Key: SPARK-18981 URL: https://issues.apache.org/jira/browse/SPARK-18981 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.0.2 Environment: spark2.0.2 hadoop2.5.0 Reporter: roncenzhao Priority: Critical related settings: spark.speculation true spark.dynamicAllocation.minExecutors0 spark.executor.cores 4 When I run the follow app, the bug will trigger. ``` sc.runJob(job1) sleep(100s) sc.runJob(job2) // the job2 will hang and never be scheduled ``` The triggering condition is described as follows: condition1: During the sleeping time, the executors will be released and the # of the executor will be zero some seconds later. The #numExecutorsTarget in 'ExecutorAllocationManager' will be 0. condition2: In 'ExecutorAllocationListener.onTaskEnd()', the numRunningTasks will be negative during the ending of job1's tasks. condition3: The job2 only hava one task. result: In the method 'ExecutorAllocationManager.updateAndSyncNumExecutorsTarget()', we will calculate #maxNeeded in 'maxNumExecutorsNeeded()'. Obviously, #numRunningOrPendingTasks will be negative and the #maxNeeded will be 0 or negative. So the 'ExecutorAllocationManager' will not request container from yarn. The app will be hung. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18074) UDFs don't work on non-local environment
[ https://issues.apache.org/jira/browse/SPARK-18074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15673145#comment-15673145 ] roncenzhao commented on SPARK-18074: Hi, if I remove the conf `spark.serializer org.apache.spark.serializer.KryoSerializer` in spark-defaults.conf, it will run successfully. > UDFs don't work on non-local environment > > > Key: SPARK-18074 > URL: https://issues.apache.org/jira/browse/SPARK-18074 > Project: Spark > Issue Type: Bug >Reporter: Alberto Andreotti > > It seems that UDFs fail to deserialize when they are sent to the remote > cluster. This is an app that can help reproduce the problem, > https://github.com/albertoandreottiATgmail/spark_udf > and this is the stack trace with the exception, > org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in > stage 4.0 failed 4 times, most recent failure: Lost task 0.3 in stage 4.0 > (TID 77, 192.168.1.194): java.lang.ClassCastException: cannot assign instance > of scala.collection.immutable.List$SerializationProxy to field > org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type > scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD > at > java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2083) > at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1261) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1996) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > scala.collection.immutable.List$SerializationProxy.readObject(List.scala:477) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > scala.collection.immutable.List$SerializationProxy.readObject(List.scala:477) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > scala.collection.immutable.List$SerializationProxy.readObject(List.scala:477) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.
[jira] [Commented] (SPARK-18074) UDFs don't work on non-local environment
[ https://issues.apache.org/jira/browse/SPARK-18074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15672950#comment-15672950 ] roncenzhao commented on SPARK-18074: I have encountered this problem, too. If any one can give some method to solve this problem? > UDFs don't work on non-local environment > > > Key: SPARK-18074 > URL: https://issues.apache.org/jira/browse/SPARK-18074 > Project: Spark > Issue Type: Bug >Reporter: Alberto Andreotti > > It seems that UDFs fail to deserialize when they are sent to the remote > cluster. This is an app that can help reproduce the problem, > https://github.com/albertoandreottiATgmail/spark_udf > and this is the stack trace with the exception, > org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in > stage 4.0 failed 4 times, most recent failure: Lost task 0.3 in stage 4.0 > (TID 77, 192.168.1.194): java.lang.ClassCastException: cannot assign instance > of scala.collection.immutable.List$SerializationProxy to field > org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type > scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD > at > java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2083) > at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1261) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1996) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > scala.collection.immutable.List$SerializationProxy.readObject(List.scala:477) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > scala.collection.immutable.List$SerializationProxy.readObject(List.scala:477) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > scala.collection.immutable.List$SerializationProxy.readObject(List.scala:477) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >
[jira] [Closed] (SPARK-16564) DeadLock happens when ‘StaticMemoryManager‘ release in-memory block
[ https://issues.apache.org/jira/browse/SPARK-16564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao closed SPARK-16564. -- Resolution: Resolved In master branch, the release logical has been refactoring, the dead lock problem is solved. > DeadLock happens when ‘StaticMemoryManager‘ release in-memory block > --- > > Key: SPARK-16564 > URL: https://issues.apache.org/jira/browse/SPARK-16564 > Project: Spark > Issue Type: Bug >Affects Versions: 1.6.1 > Environment: spark.memory.useLegacyMode=true >Reporter: roncenzhao > > the condition causes dead lock is: > Thread 1. 'BlockManagerSlaveEndpoint' receives 'RemoveBroadcast' or > 'RemoveBlock' or .., executor begins to remove block named 'blockId1' > Thread 2. the other rdd begins to run and must 'evictBlocksToFreeSpace()' for > more memory, unfortunately the chosen block to evict is 'blockId1' > As follows, > thread 1 is holding blockId1's blockInfo lock(0x00053ab39f58) and > waitting for StaticMemoryManager's lock(0x00039f04fea8). > thread 2 is holding StaticMemoryManager's lock(0x00039f04fea8) and > waitting for blockId1's blockInfo lock(0x00053ab39f58). > This condition causes dead lock. > stackTrace: > Found one Java-level deadlock: > = > "block-manager-slave-async-thread-pool-24": > waiting to lock monitor 0x7f0b004ec278 (object 0x00039f04fea8, a > org.apache.spark.memory.StaticMemoryManager), > which is held by "Executor task launch worker-11" > "Executor task launch worker-11": > waiting to lock monitor 0x7f0b01354018 (object 0x00053ab39f58, a > org.apache.spark.storage.BlockInfo), > which is held by "block-manager-slave-async-thread-pool-22" > "block-manager-slave-async-thread-pool-22": > waiting to lock monitor 0x7f0b004ec278 (object 0x00039f04fea8, a > org.apache.spark.memory.StaticMemoryManager), > which is held by "Executor task launch worker-11" > Java stack information for the threads listed above: > === > "Executor task launch worker-11": > at > org.apache.spark.storage.BlockManager.dropFromMemory(BlockManager.scala:1032) > - waiting to lock <0x00053ab39f58> (a > org.apache.spark.storage.BlockInfo) > at > org.apache.spark.storage.BlockManager.dropFromMemory(BlockManager.scala:1009) > at > org.apache.spark.storage.MemoryStore$$anonfun$evictBlocksToFreeSpace$2.apply(MemoryStore.scala:460) > at > org.apache.spark.storage.MemoryStore$$anonfun$evictBlocksToFreeSpace$2.apply(MemoryStore.scala:449) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at > org.apache.spark.storage.MemoryStore.evictBlocksToFreeSpace(MemoryStore.scala:449) > - locked <0x00039f04fea8> (a > org.apache.spark.memory.StaticMemoryManager) > at > org.apache.spark.memory.StorageMemoryPool.acquireMemory(StorageMemoryPool.scala:89) > - locked <0x00039f04fea8> (a > org.apache.spark.memory.StaticMemoryManager) > at > org.apache.spark.memory.StaticMemoryManager.acquireUnrollMemory(StaticMemoryManager.scala:83) > - locked <0x00039f04fea8> (a > org.apache.spark.memory.StaticMemoryManager) > at > org.apache.spark.storage.MemoryStore.reserveUnrollMemoryForThisTask(MemoryStore.scala:493) > - locked <0x00039f04fea8> (a > org.apache.spark.memory.StaticMemoryManager) > at > org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:291) > at > org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:178) > at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:85) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:268) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:89) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > "block-manager-slave-async-thread-pool-22": > at org.apache.spark.storage.MemoryStore.remove(MemoryStore.scala:216) > - waiting to lock <0x00039f04fea8> (a > org.apache.spark.memory.StaticMemoryManager) > at > org.apache.spark.storage.BlockManager.removeBlock(Block
[jira] [Created] (SPARK-16564) DeadLock happens when ‘StaticMemoryManager‘ release in-memory block
roncenzhao created SPARK-16564: -- Summary: DeadLock happens when ‘StaticMemoryManager‘ release in-memory block Key: SPARK-16564 URL: https://issues.apache.org/jira/browse/SPARK-16564 Project: Spark Issue Type: Bug Affects Versions: 1.6.1 Environment: spark.memory.useLegacyMode=true Reporter: roncenzhao the condition causes dead lock is: Thread 1. 'BlockManagerSlaveEndpoint' receives 'RemoveBroadcast' or 'RemoveBlock' or .., executor begins to remove block named 'blockId1' Thread 2. the other rdd begins to run and must 'evictBlocksToFreeSpace()' for more memory, unfortunately the chosen block to evict is 'blockId1' As follows, thread 1 is holding blockId1's blockInfo lock(0x00053ab39f58) and waitting for StaticMemoryManager's lock(0x00039f04fea8). thread 2 is holding StaticMemoryManager's lock(0x00039f04fea8) and waitting for blockId1's blockInfo lock(0x00053ab39f58). This condition causes dead lock. stackTrace: Found one Java-level deadlock: = "block-manager-slave-async-thread-pool-24": waiting to lock monitor 0x7f0b004ec278 (object 0x00039f04fea8, a org.apache.spark.memory.StaticMemoryManager), which is held by "Executor task launch worker-11" "Executor task launch worker-11": waiting to lock monitor 0x7f0b01354018 (object 0x00053ab39f58, a org.apache.spark.storage.BlockInfo), which is held by "block-manager-slave-async-thread-pool-22" "block-manager-slave-async-thread-pool-22": waiting to lock monitor 0x7f0b004ec278 (object 0x00039f04fea8, a org.apache.spark.memory.StaticMemoryManager), which is held by "Executor task launch worker-11" Java stack information for the threads listed above: === "Executor task launch worker-11": at org.apache.spark.storage.BlockManager.dropFromMemory(BlockManager.scala:1032) - waiting to lock <0x00053ab39f58> (a org.apache.spark.storage.BlockInfo) at org.apache.spark.storage.BlockManager.dropFromMemory(BlockManager.scala:1009) at org.apache.spark.storage.MemoryStore$$anonfun$evictBlocksToFreeSpace$2.apply(MemoryStore.scala:460) at org.apache.spark.storage.MemoryStore$$anonfun$evictBlocksToFreeSpace$2.apply(MemoryStore.scala:449) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.storage.MemoryStore.evictBlocksToFreeSpace(MemoryStore.scala:449) - locked <0x00039f04fea8> (a org.apache.spark.memory.StaticMemoryManager) at org.apache.spark.memory.StorageMemoryPool.acquireMemory(StorageMemoryPool.scala:89) - locked <0x00039f04fea8> (a org.apache.spark.memory.StaticMemoryManager) at org.apache.spark.memory.StaticMemoryManager.acquireUnrollMemory(StaticMemoryManager.scala:83) - locked <0x00039f04fea8> (a org.apache.spark.memory.StaticMemoryManager) at org.apache.spark.storage.MemoryStore.reserveUnrollMemoryForThisTask(MemoryStore.scala:493) - locked <0x00039f04fea8> (a org.apache.spark.memory.StaticMemoryManager) at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:291) at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:178) at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:85) at org.apache.spark.rdd.RDD.iterator(RDD.scala:268) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) "block-manager-slave-async-thread-pool-22": at org.apache.spark.storage.MemoryStore.remove(MemoryStore.scala:216) - waiting to lock <0x00039f04fea8> (a org.apache.spark.memory.StaticMemoryManager) at org.apache.spark.storage.BlockManager.removeBlock(BlockManager.scala:1114) - locked <0x00053ab39f58> (a org.apache.spark.storage.BlockInfo) at org.apache.spark.storage.BlockManager$$anonfun$removeBroadcast$2.apply(BlockManager.scala:1101) at org.apache.spark.storage.BlockManager$$anonfun$removeBroadcast$2.apply(BlockManager.scala:1101) at scala.collection.immutable.Set$Set2.foreach(Set.scala:94) at org.a
