Hi, I am wondering if anyone has successfully enabled "mapreduce.input.fileinputformat.list-status.num-threads" in Spark jobs. I usually set this property to 25 to speed up file listing in MR jobs (Hive and Pig). But for some reason, this property does not take effect in Spark HadoopRDD resulting in serious delay in file listing.
I verified that the property is indeed set in HadoopRDD by logging the value of the property in the getPartitions() function. I also tried to attach VisualVM to Spark and Pig clients, which look as follows- In Pig, I can see 25 threads running in parallel for file listing- [image: Inline image 1] In Spark, I only see 2 threads running in parallel for file listing- [image: Inline image 2] What's strange is that the # of concurrent threads in Spark is throttled no matter how high I set "mapreduce.input.fileinputformat.list-status.num-threads". Is anyone using Spark with this property enabled? If so, can you please share how you do it? Thanks! Cheolsoo
