Hello, Nipun In my opinion, the „converting the dataframe to an RDD“ wouldn’t be a costly operation since Dataframe (Dataset) operations are under the hood operated always as RDDs. I don’t know which version of Spark you operate, but I suppose you utilize the 2.0. I would, therefore go for:
dataFrame.rdd.partitions That returns Array of partitions (writen in SCALA). Regards, Jan From: Nipun Parasrampuria [mailto:paras...@umn.edu] Sent: Thursday, October 27, 2016 12:01 AM To: user@spark.apache.org Subject: No of partitions in a Dataframe How do I find the number of partitions in a dataframe without converting the dataframe to an RDD(I'm assuming that it's a costly operation). If there's no way to do so, I wonder why the API doesn't include a method like that(an explanation for why such a method would be useless, perhaps) Thanks! Nipun
