<user@spark.apache.org<mailto:user@spark.apache.org>>
Subject: Re: Missing output partition file in S3
On 15 Sep 2016, at 19:37, Chen, Kevin
<kevin.c...@neustar.biz<mailto:kevin.c...@neustar.biz>> wrote:
Hi,
Has any one encountered an issue of missing output partition file in S3
> Begin forwarded message:
>
> From: "Chen, Kevin" <kevin.c...@neustar.biz>
> Subject: Re: Missing output partition file in S3
> Date: September 19, 2016 at 10:54:44 AM PDT
> To: Steve Loughran <ste...@hortonworks.com>
> Cc: "user@spark.a
uot;
<user@spark.apache.org<mailto:user@spark.apache.org>>
Subject: Re: Missing output partition file in S3
On 15 Sep 2016, at 19:37, Chen, Kevin
<kevin.c...@neustar.biz<mailto:kevin.c...@neustar.biz>> wrote:
Hi,
Has any one encountered an issue of missing output par
Sent from my iPhone
> On Sep 15, 2016, at 1:37 PM, Chen, Kevin <kevin.c...@neustar.biz> wrote:
>
> Hi,
>
> Has any one encountered an issue of missing output partition file in S3 ? My
> spark job writes output to a S3 location. Occasionally, I noticed one
>
are you using speculation?
On 15 September 2016 at 21:37, Chen, Kevin <kevin.c...@neustar.biz> wrote:
> Hi,
>
> Has any one encountered an issue of missing output partition file in S3 ?
> My spark job writes output to a S3 location. Occasionally, I noticed one
> part
On 15 Sep 2016, at 19:37, Chen, Kevin
<kevin.c...@neustar.biz<mailto:kevin.c...@neustar.biz>> wrote:
Hi,
Has any one encountered an issue of missing output partition file in S3 ? My
spark job writes output to a S3 location. Occasionally, I noticed one partition
file is missing.
Hi,
Has any one encountered an issue of missing output partition file in S3 ? My
spark job writes output to a S3 location. Occasionally, I noticed one partition
file is missing. As a result, one chunk of data was lost. If I rerun the same
job, the problem usually goes away. This has been
runtime, and then stop our Spark
job completely ASAP if it happens?
Thanks,
Nicolas
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Missing-output-partition-file-in-S3-tp21326.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
