Hi Saurabh,
You can do the following to print the number of entries in each partition.
You may need to grep executor logs for the counts.
val rdd = sc.parallelize(1 to 100, 4)
rdd.foreachPartition(it => println("Record count in partition" + it.size))
Hope this is what you are looking for.
Thanks
Srinivas
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/How-to-check-for-No-of-Records-per-partition-in-Dataframe-tp18764p18805.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
