Re: How does Spark determine in-memory partition count when reading Parquet ~files?

2016-10-22 Thread shea.parkes
o tune that parameter appropriately for our work flow. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-does-Spark-determine-in-memory-partition-count-when-reading-Parquet-files-tp27921p27943.html Sent from the Apache Spark User List mailing list archive at Nabb

Re: How does Spark determine in-memory partition count when reading Parquet ~files?

2016-10-19 Thread Michael Armbrust
1001560.n3.nabble.com/How-does-Spark-determine-in- > memory-partition-count-when-reading-Parquet-files-tp27921.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >

How does Spark determine in-memory partition count when reading Parquet ~files?

2016-10-18 Thread shea.parkes
ase to understand how many partitions will result from reading a parquet ~file? Thanks, Shea -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-does-Spark-determine-in-memory-partition-count-when-reading-Parquet-files-tp27921.html Sent from the Apache Spark U