Are you using hadoop-1.0? Hadoop doesn't support splittable bz2 files before 1.2 (or a later version). But due to a bug (https://issues.apache.org/jira/browse/HADOOP-10614), you should try hadoop-2.5.0. -Xiangrui
On Wed, Aug 27, 2014 at 2:49 PM, jerryye <jerr...@gmail.com> wrote: > Hi, > I'm running on the master branch and I noticed that textFile ignores > minPartition for bz2 files. Is anyone else seeing the same thing? I tried > varying minPartitions for a bz2 file and rdd.partitions.size was always 1 > whereas doing it for a non-bz2 file worked. > > Not sure if this matters or not but I'm reading from s3. > > - jerry > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/minPartitions-ignored-for-bz2-tp12948.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
