Before doing saveAsParquetFile, you can call the repartition and provide a decent number which will result in the total number of output files generated.
Thanks Best Regards On Mon, Nov 3, 2014 at 1:12 PM, ag007 <agre...@mac.com> wrote: > Hi there, > > I have a pySpark job that is simply taking a tab separated CSV outputting > it > to a Parquet file. The code is based on the SQL write parquet example. > (Using a different inferred schema, only 35 columns). The input files range > from 100MB to 12 Gb. > > I have tried different different block sizes from 10MB through to 1 Gb, I > have tried different parallelism. The total part files total about 1:5 > compression. > > I am trying to get large parquet files. Having this many small files will > cause problems to my name node. I have over 500,000 of these files. > > Your assistance would be greatly appreciated. > > cheers, > Ag > > PS Another solution may be if there is a parquet concat tool around. I > couldn't see one. I understand that this tool would have to adjust the > footer. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Parquet-files-are-only-6-20MB-in-size-tp17935.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
