[
https://issues.apache.org/jira/browse/SPARK-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14742018#comment-14742018
]
Maciej Bryński edited comment on SPARK-10538 at 9/12/15 12:24 PM:
------------------------------------------------------------------
OK.
I managed to isolate the problem.
I have two dataframes:
1) Data dataframe
2) Dictionary dataframe
Counts of data group by foreign key to dictionary are following:
key, count
1, 5398567
2, 59912
3, 3678
4, 74461
5, 12845
When I did a join - result is partitioned by join key, so one of the partitions
is too big to process.
As a workaround:
Is there any possibility to force broadcast join from pyspark (or spark sql)?
I found this, but it's only for Scala.
https://github.com/apache/spark/pull/6751/files
was (Author: maver1ck):
OK.
I managed to isolate the problem.
I have two dataframes:
1) Data dataframe
2) Dictionary dataframe
Counts of data group by foreign key to dictionary are following:
key, count
1, 5398567
2, 59912
3, 3678
4, 74461
5, 12845
When I did a join - result is partitioned by join key, so one of the partitions
is too big to process.
Is there any possibility to force broadcast join from pyspark (or spark sql)?
I found this, but it's only for Scala.
https://github.com/apache/spark/pull/6751/files
> java.lang.NegativeArraySizeException during join
> ------------------------------------------------
>
> Key: SPARK-10538
> URL: https://issues.apache.org/jira/browse/SPARK-10538
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0
> Reporter: Maciej Bryński
> Attachments: screenshot-1.png
>
>
> Hi,
> I've got a problem during joining tables in PySpark. (in my example 20 of
> them)
> I can observe that during calculation of first partition (on one of
> consecutive joins) there is a big shuffle read size (294.7 MB / 146 records)
> vs on others partitions (approx. 272.5 KB / 113 record)
> I can also observe that just before the crash python process going up to few
> gb of RAM.
> After some time there is an exception:
> {code}
> java.lang.NegativeArraySizeException
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
> Source)
> at
> org.apache.spark.sql.execution.TungstenProject$$anonfun$3$$anonfun$apply$3.apply(basicOperators.scala:90)
> at
> org.apache.spark.sql.execution.TungstenProject$$anonfun$3$$anonfun$apply$3.apply(basicOperators.scala:88)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.insertAll(BypassMergeSortShuffleWriter.java:119)
> at
> org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:73)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> at org.apache.spark.scheduler.Task.run(Task.scala:88)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> I'm running this on 2 nodes cluster (12 cores, 64 GB RAM)
> Config:
> {code}
> spark.driver.memory 10g
> spark.executor.extraJavaOptions -XX:-UseGCOverheadLimit -XX:+UseParallelGC
> -Dfile.encoding=UTF8
> spark.executor.memory 60g
> spark.storage.memoryFraction 0.05
> spark.shuffle.memoryFraction 0.75
> spark.driver.maxResultSize 10g
> spark.cores.max 24
> spark.kryoserializer.buffer.max 1g
> spark.default.parallelism 200
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
