Re: Handling Empty RDD

[Yogesh Vyas]

Hi,
I finally got it working.
I was using the updateStateByKey() function to maintain the previous
value of the state, and I found that the event list was empty. Hence
handling the empty event list by using event.isEmtpy() sort out the
problem.

On Sun, May 22, 2016 at 7:59 PM, Ted Yu  wrote:
> You mean when rdd.isEmpty() returned false, saveAsTextFile still produced
> empty file ?
>
> Can you show code snippet that demonstrates this ?
>
> Cheers
>
> On Sun, May 22, 2016 at 5:17 AM, Yogesh Vyas  wrote:
>>
>> Hi,
>> I am reading files using textFileStream, performing some action onto
>> it and then saving it to HDFS using saveAsTextFile.
>> But whenever there is no file to read, Spark will write and empty RDD(
>> [] ) to HDFS.
>> So, how to handle the empty RDD.
>>
>> I checked rdd.isEmpty() and rdd.count>0, but both of them does not works.
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Handling Empty RDD

[Ted Yu]

You mean when rdd.isEmpty() returned false, saveAsTextFile still produced
empty file ?

Can you show code snippet that demonstrates this ?

Cheers

On Sun, May 22, 2016 at 5:17 AM, Yogesh Vyas  wrote:

> Hi,
> I am reading files using textFileStream, performing some action onto
> it and then saving it to HDFS using saveAsTextFile.
> But whenever there is no file to read, Spark will write and empty RDD(
> [] ) to HDFS.
> So, how to handle the empty RDD.
>
> I checked rdd.isEmpty() and rdd.count>0, but both of them does not works.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

Handling Empty RDD

[Yogesh Vyas]

Hi,
I am reading files using textFileStream, performing some action onto
it and then saving it to HDFS using saveAsTextFile.
But whenever there is no file to read, Spark will write and empty RDD(
[] ) to HDFS.
So, how to handle the empty RDD.

I checked rdd.isEmpty() and rdd.count>0, but both of them does not works.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org