I had a similar requirement and I come up with a small algorithem to
determine number of partitions based on cluster size and input data.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Question-regarding-spark-data-partition-and-coalesce-Need-info-on-my
in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Question-regarding-spark-data-partition-and-coalesce-Need-info-on-my-use-case-tp12214p12311.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
in that
direction. Any help is much appreciated.
I am using spark 1.0 on yarn.
Thanks,
AG
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Question-regarding-spark-data-partition-and-coalesce-Need-info-on-my-use-case-tp12214.html
Sent from the Apache Spark User
-spark-data-partition-and-coalesce-Need-info-on-my-use-case-tp12214.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e
