Hi All, I am reading entire directory of gz XML files with wholeTextFiles.
I understand as it is gz and with wholeTextFiles the individual files are not splittable but why the entire directory is read by one executor, single task. I have provided number of executors as number of files in that directory. Is the only option here is to repartition after the xmls are read and parsed with JaxB. Regards, Pradeep --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
