Hi All,
Would like to gain some understanding on the questions listed below,
1. When processing a large file with Apache Spark, with, say,
sc.textFile("somefile.xml"), does it split it for parallel processing
across executors or, will it be processed as a single chunk in a single
executor?
2. When using dataframes, with implicit XMLContext from Databricks is
there any optimization prebuilt for such large file processing?
Please help!!!
http://stackoverflow.com/questions/39305310/does-spark-process-large-file-in-the-single-worker
Regards,
Somasundaram S
