[jira] [Commented] (SPARK-2984) FileNotFoundException on _temporary directory
[
https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15960906#comment-15960906
]
Hemang Nagar commented on SPARK-2984:
-
Is there any work going on this issue, or anything related to this, as it seems
nobody has been able to resolve this, and a lot of people including me have
this issue?
> FileNotFoundException on _temporary directory
> -
>
> Key: SPARK-2984
> URL: https://issues.apache.org/jira/browse/SPARK-2984
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.1.0
>Reporter: Andrew Ash
>Assignee: Josh Rosen
>Priority: Critical
> Fix For: 1.3.0
>
>
> We've seen several stacktraces and threads on the user mailing list where
> people are having issues with a {{FileNotFoundException}} stemming from an
> HDFS path containing {{_temporary}}.
> I ([~aash]) think this may be related to {{spark.speculation}}. I think the
> error condition might manifest in this circumstance:
> 1) task T starts on a executor E1
> 2) it takes a long time, so task T' is started on another executor E2
> 3) T finishes in E1 so moves its data from {{_temporary}} to the final
> destination and deletes the {{_temporary}} directory during cleanup
> 4) T' finishes in E2 and attempts to move its data from {{_temporary}}, but
> those files no longer exist! exception
> Some samples:
> {noformat}
> 14/08/11 08:05:08 ERROR JobScheduler: Error running job streaming job
> 140774430 ms.0
> java.io.FileNotFoundException: File
> hdfs://hadoopc/user/csong/output/human_bot/-140774430.out/_temporary/0/task_201408110805__m_07
> does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:654)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:708)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:360)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
> at
> org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
> at
> org.apache.spark.SparkHadoopWriter.commitJob(SparkHadoopWriter.scala:126)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:841)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:724)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:643)
> at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1068)
> at
> org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:773)
> at
> org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:771)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:41)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
> at scala.util.Try$.apply(Try.scala:161)
> at org.apache.spark.streaming.scheduler.Job.run(Job.scala:32)
> at
> org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:172)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> -- Chen Song at
> http://apache-spark-user-list.1001560.n3.nabble.com/saveAsTextFiles-file-not-found-exception-td10686.html
> {noformat}
> I am running a Spark Streaming job that uses saveAsTextFiles to save results
> into hdfs files. However, it has an exception after 20 batches
> result-140631234/_temporary/0/task_201407251119__m_03 does not
> exist.
> {noformat}
> and
> {noformat}
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
> No lease on /apps/data/vddil/real-time/checkpoint/temp: File does not exist.
> Holder DFSClient_NONMAPREDUCE_327993456_13 does not have any open files.
> at
> org.apache.hadoop.hdfs.server.
[jira] [Commented] (SPARK-12917) Add DML support to Spark SQL for HIVE
[ https://issues.apache.org/jira/browse/SPARK-12917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15145057#comment-15145057 ] Hemang Nagar commented on SPARK-12917: -- Yes it is a transaction table feature, and since Hive supports transactions now, can Spark also provide support for the same ? > Add DML support to Spark SQL for HIVE > - > > Key: SPARK-12917 > URL: https://issues.apache.org/jira/browse/SPARK-12917 > Project: Spark > Issue Type: New Feature > Components: SQL >Affects Versions: 1.6.0 >Reporter: Hemang Nagar >Priority: Blocker > > Spark SQL should be updated to support the DML operations that are being > supported by Hive since 0.14 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12917) Add DML support to Spark SQL for HIVE
[ https://issues.apache.org/jira/browse/SPARK-12917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15114616#comment-15114616 ] Hemang Nagar commented on SPARK-12917: -- Update and Delete operations are supported in Hive 0.14 and after that, we need Spark to support it. Also, need Insert by values operations to be supported. For example, insert into table values(1, "john doe"), this gives an unsupported operation exception in Spark. > Add DML support to Spark SQL for HIVE > - > > Key: SPARK-12917 > URL: https://issues.apache.org/jira/browse/SPARK-12917 > Project: Spark > Issue Type: New Feature > Components: SQL >Affects Versions: 1.6.0 >Reporter: Hemang Nagar >Priority: Blocker > > Spark SQL should be updated to support the DML operations that are being > supported by Hive since 0.14 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-12917) Add DML support to Spark SQL for HIVE
Hemang Nagar created SPARK-12917: Summary: Add DML support to Spark SQL for HIVE Key: SPARK-12917 URL: https://issues.apache.org/jira/browse/SPARK-12917 Project: Spark Issue Type: New Feature Components: SQL Affects Versions: 1.6.0 Reporter: Hemang Nagar Priority: Blocker Spark SQL should be updated to support the DML operations that are being supported by Hive since 0.14 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
