[
https://issues.apache.org/jira/browse/SPARK-21172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16138361#comment-16138361
]
Lasantha Fernando commented on SPARK-21172:
-------------------------------------------
I've encountered the same issue with Spark 2.1.1 as well. Am hitting this
continuously for a particular job without being able to finish the job at all.
I can get results if I split the job to smaller jobs and run them separately.
But need to run the whole job together. I've tried the following parameter
modifications without any luck.
* Change the {{spark.sql.shuffle.partitions}}
* Change {{spark.files.fetchTimeout}}
* Increase {{spark.shuffle.file.buffer}}
* Increase {{spark.unsafe.sorter.spill.reader.buffer.size}}
* Change {{spark.io.compression.codec}}
* Enable/disable {{spark.shuffle.compress}}
* Enable/disable {{spark.shuffle.spill.compress}}
* Enable/disable {{spark.file.transferTo}}
* Change the garbage collector to use parallel GC
{{spark.executor.extraJavaOptions -XX:ParallelGCThreads=4 -XX:+UseParallelGC}}
* Enable/disable off heap memory {{spark.memory.offHeap.enabled}}
Also being trying some of the suggestions from the slide deck
[here|https://www.slideshare.net/databricks/tuning-apache-spark-for-largescale-workloads-gaoxiang-liu-and-sital-kedia]
and some suggestions from SPARK-4105. Was not able to resolve the issue so far
with any of these suggestions.
> EOFException reached end of stream in UnsafeRowSerializer
> ---------------------------------------------------------
>
> Key: SPARK-21172
> URL: https://issues.apache.org/jira/browse/SPARK-21172
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
> Affects Versions: 2.0.1
> Reporter: liupengcheng
> Labels: shuffle
>
> Spark sql job failed because of the following Exception. Seems like a bug in
> shuffle stage.
> Shuffle read size for single task is tens of GB
> {code}
> org.apache.spark.SparkException: Task failed while writing rows
> at
> org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:264)
> at
> org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(InsertIntoHadoopFsRelationCommand.scala:143)
> at
> org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(InsertIntoHadoopFsRelationCommand.scala:143)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
> at org.apache.spark.scheduler.Task.run(Task.scala:86)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.EOFException: reached end of stream after reading 9034374
> bytes; 1684891936 bytes expected
> at
> org.spark_project.guava.io.ByteStreams.readFully(ByteStreams.java:735)
> at
> org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$3$$anon$1.next(UnsafeRowSerializer.scala:127)
> at
> org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$3$$anon$1.next(UnsafeRowSerializer.scala:110)
> at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
> at
> org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:30)
> at
> org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:43)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
> at
> org.apache.spark.sql.execution.datasources.DefaultWriterContainer$$anonfun$writeRows$1.apply$mcV$sp(WriterContainer.scala:255)
> at
> org.apache.spark.sql.execution.datasources.DefaultWriterContainer$$anonfun$writeRows$1.apply(WriterContainer.scala:253)
> at
> org.apache.spark.sql.execution.datasources.DefaultWriterContainer$$anonfun$writeRows$1.apply(WriterContainer.scala:253)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1345)
> at
> org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:259)
> ... 8 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
