Re: SparkSession for RDBMS

2016-08-03 Thread Takeshi Yamamuro
Hi, If these bounaries are not given, spark tries to read all the data as a single parition. See: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala#L56 // maropu On Wed, Aug 3, 2016 at 11:19 PM, Selvam Raman

SparkSession for RDBMS

2016-08-03 Thread Selvam Raman
Hi All, I would like to read the data from RDBMS to spark (2.0) using sparksession. How can i decide upper boundary, lower boundary and partitions. is there any specific approach available. How Sqoop2 decides number of partitions, upper and lower boundary if we are not specifying anything. --