GitHub user jeanlyn opened a pull request:
https://github.com/apache/spark/pull/6833
[SPARK-8379][SQL]avoid speculative tasks write to the same file
The issue link
[SPARK-8379](https://issues.apache.org/jira/browse/SPARK-8379)
Currently,when we insert data to the dynamic partition with speculative
tasks we will get the Exception
```
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
Lease mismatch on
/tmp/hive-jeanlyn/hive_2015-06-15_15-20-44_734_8801220787219172413-1/-ext-10000/ds=2015-06-15/type=2/part-00301.lzo
owned by DFSClient_attempt_201506031520_0011_m_000189_0_-1513487243_53
but is accessed by
DFSClient_attempt_201506031520_0011_m_000042_0_-1275047721_57
```
This pr try to write the data to temporary dir when using dynamic parition
avoid the speculative tasks writing the same file
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jeanlyn/spark speculation
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6833.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6833
----
commit e19a3bd77b6b9f44479e51659e244e9809b2963d
Author: jeanlyn <jeanly...@gmail.com>
Date: 2015-06-15T16:38:16Z
avoid speculative tasks write same file
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
