Use unix time and write the unix time to oracle as number column type ,create
virtual column in oracle database for the unix time like “oracle_time
generated always as (to_date('1970010108','MMDDHH24')+(1/24/60/60)*unixtime
)
> On Mar 20, 2018, at 11:08 PM, Gurusamy Thirupathy wrote:
>
>
HI Jorn,
Thanks for your sharing different options, yes we are trying to build a
generic tool for Hive to Spark export.
FYI, currently we are using sqoop, we are trying to migrate from sqoop to
spark.
Thanks
-G
On Tue, Mar 20, 2018 at 2:17 AM, Jörn Franke wrote:
> Write your own Spark UDF. App
Write your own Spark UDF. Apply it to all varchar columns.
Within this udf you can use the SimpleDateFormat parse method. If this method
returns null you return the content as varchar if not you return a date. If the
content is null you return null.
Alternatively you can define an insert functi
Hi guha,
Thanks for your quick response, option a and b are in our table already.
For option b, again the same problem, we don't know which column is date.
Thanks,
-G
On Sun, Mar 18, 2018 at 9:36 PM, Deepak Sharma
wrote:
> The other approach would to write to temp table and then merge the dat
Hi
The is not with spark in this case, it is with Oracle. If you do not know
which columns to apply date-related conversion rule, then you have a
problem.
You should try either
a) Define some config file where you can define table name, date column
name and date-format @ source so that you can
The other approach would to write to temp table and then merge the data.
But this may be expensive solution.
Thanks
Deepak
On Mon, Mar 19, 2018, 08:04 Gurusamy Thirupathy wrote:
> Hi,
>
> I am trying to read data from Hive as DataFrame, then trying to write the
> DF into the Oracle data base. I
Hi,
I am trying to read data from Hive as DataFrame, then trying to write the
DF into the Oracle data base. In this case, the date field/column in hive
is with Type Varchar(20)
but the corresponding column type in Oracle is Date. While reading from
hive , the hive table names are dynamically decid
