Re: Question on using pseudo columns in spark jdbc options

2017-12-07 Thread रविशंकर नायर
It works perfectly. You can use pseudo columns like ROWNUM in Oracle and RRN in DB2. To avoid skewing you can apply the great coalesce function...Spark is sparkling.. Best, On Thu, Dec 7, 2017 at 2:20 PM, Tomasz Dudek wrote: > Hey Ravion, > > yes, you can obviously specify other column than a

Re: Question on using pseudo columns in spark jdbc options

2017-12-07 Thread Tomasz Dudek
Hey Ravion, yes, you can obviously specify other column than a primary key. Be aware though, that if the key range is not spread evenly (for example in your code, if there's a "gap" in primary keys and no row has id between 0 and 17220) some of the executors may not assist in loading data (because

Question on using pseudo columns in spark jdbc options

2017-12-02 Thread रविशंकर नायर
Hi all, I am using a query to fetch data from MYSQL as follows: var df = spark.read. format("jdbc"). option("url", "jdbc:mysql://10.0.0.192:3306/retail_db"). option("driver" ,"com.mysql.jdbc.Driver"). option("user", "retail_dba"). option("password", "cloudera"). option("dbtable", "orders"). optio