> I do not think that SPARK will automatically determine the partitions. > Actually > it does not automatically determine the partitions. In case a table has a few > million records, it all goes through the driver.
Hi Gourav Actualy spark jdbc driver is able to deal direclty with partitions. Sparks creates a jdbc connection for each partition. All details explained in this post : http://www.gatorsmile.io/numpartitionsinjdbc/ Also an example with greenplum database: http://engineering.pivotal.io/post/getting-started-with-greenplum-spark/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
