Re: Hive to Oracle using Spark - Type(Date) conversion issue

Use unix time and write the unix time to oracle as number column type ,create virtual column in oracle database for the unix time like “oracle_time generated always as (to_date('1970010108','MMDDHH24')+(1/24/60/60)*unixtime ) > On Mar 20, 2018, at 11:08 PM, Gurusamy Thirupathy wrote: >

Re: Hive to Oracle using Spark - Type(Date) conversion issue

HI Jorn, Thanks for your sharing different options, yes we are trying to build a generic tool for Hive to Spark export. FYI, currently we are using sqoop, we are trying to migrate from sqoop to spark. Thanks -G On Tue, Mar 20, 2018 at 2:17 AM, Jörn Franke wrote: > Write

Re: Hive to Oracle using Spark - Type(Date) conversion issue

Write your own Spark UDF. Apply it to all varchar columns. Within this udf you can use the SimpleDateFormat parse method. If this method returns null you return the content as varchar if not you return a date. If the content is null you return null. Alternatively you can define an insert

Re: Hive to Oracle using Spark - Type(Date) conversion issue

Hi guha, Thanks for your quick response, option a and b are in our table already. For option b, again the same problem, we don't know which column is date. Thanks, -G On Sun, Mar 18, 2018 at 9:36 PM, Deepak Sharma wrote: > The other approach would to write to temp

Re: Hive to Oracle using Spark - Type(Date) conversion issue

Hi The is not with spark in this case, it is with Oracle. If you do not know which columns to apply date-related conversion rule, then you have a problem. You should try either a) Define some config file where you can define table name, date column name and date-format @ source so that you can

Re: Hive to Oracle using Spark - Type(Date) conversion issue

The other approach would to write to temp table and then merge the data. But this may be expensive solution. Thanks Deepak On Mon, Mar 19, 2018, 08:04 Gurusamy Thirupathy wrote: > Hi, > > I am trying to read data from Hive as DataFrame, then trying to write the > DF into

Hive to Oracle using Spark - Type(Date) conversion issue

Hi, I am trying to read data from Hive as DataFrame, then trying to write the DF into the Oracle data base. In this case, the date field/column in hive is with Type Varchar(20) but the corresponding column type in Oracle is Date. While reading from hive , the hive table names are dynamically