If the file is not splittable(can I assume the log file is splittable,
though) can you advise on how spark handles such case…? If Spark can't what
is the widely used practice?
On 3 Sep 2016 7:29 pm, "Raghavendra Pandey" <raghavendra.pan...@gmail.com>
wrote:
If your file format is splittable say TSV, CSV etc, it will be distributed
across all executors.
On Sat, Sep 3, 2016 at 3:38 PM, Somasundaram Sekar <somasundar.sekar@
tigeranalytics.com> wrote:
> Hi All,
>
>
>
> Would like to gain some understanding on the questions listed below,
>
>
>
> 1. When processing a large file with Apache Spark, with, say,
> sc.textFile("somefile.xml"), does it split it for parallel processing
> across executors or, will it be processed as a single chunk in a single
> executor?
>
> 2. When using dataframes, with implicit XMLContext from Databricks
> is there any optimization prebuilt for such large file processing?
>
>
>
> Please help!!!
>
>
>
> http://stackoverflow.com/questions/39305310/does-spark-proce
> ss-large-file-in-the-single-worker
>
>
>
> Regards,
>
> Somasundaram S
>
