I think you could use `repartition` to make sure there would be no empty 
partitions.

You could also try `coalesce` to combine partitions , but it can't make sure 
there are no more empty partitions.

Best Regards,

Yi Tian
[email protected]




On Oct 18, 2014, at 20:30, [email protected] wrote:

> Hi,
> 
> I am developing program using Spark where I am using filter such as:
>  
> cleanedData = distData.map(json_extractor.extract_json).filter(lambda x: x != 
> None and x != '')
> cleanedData.saveAsTextFile(sys.argv[3])
>  
>  
> It happens to me that there is saved lot of empty files (probably from those 
> partitions that should have been filtered out). Is there some way, how to 
> prevent Spark from saving these empty files?
>  
> Thank you in advance for any help.
>  
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to