sparkContext.textFile does not honour the minPartitions argument

I am trying to read a file into a single partition but it seems like sparkContext.textFile ignores the passed minPartitions value. I know I can repartition the RDD but I was curious to know if this is expected or if this is a bug that needs to be further investigated?

Re: sparkContext.textFile does not honour the minPartitions argument

Hi Ankit, Optional number of partitions value is to increase number of partitions not reduce it from default value. On Thu, Jan 1, 2015 at 10:43 AM, Aniket Bhatnagar < aniket.bhatna...@gmail.com> wrote: > I am trying to read a file into a single partition but it seems like > sparkContext.textFil

Re: sparkContext.textFile does not honour the minPartitions argument

Thanks everyone. I studied the source code and realized minPartitions is passed over to Hadoop's InputFormat and its upto the InputFormat implementation to use the parameter as a hint. Thanks, Aniket On Fri, Jan 2, 2015, 7:13 AM Rishi Yadav wrote: > Hi Ankit, > > Optional number of partitions v