I need to save the dataframe to parquet format and need some input on choosing the appropriate block size to help efficiently parallelize/localize the data to the executors. Should I be using parquet block size or hdfs block size and what is the optimal block size to use on a 100 node cluster?
Thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Dataframe-to-parquet-using-hdfs-or-parquet-block-size-tp26693.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
