Re: Converting a string of format of 'dd/MM/yyyy' in Spark sql

Hi Bijay. At the moment it is only POC getting CSV data for invoices on a daily basis, importing into HDFS and store it in ORC table (non transactional as Spark cannot read from it) in Hive database. I have written both Hive version and Spark version. The Hive version is pretty stable as below.

Re: Converting a string of format of 'dd/MM/yyyy' in Spark sql

Hi, I have written the UDF for doing same in pyspark DataFrame since some of my dates are before unix standard time epoch of 1/1/1970. I have more than 250 columns and applying custom date_format UDF to more than 50 columns. I am getting OOM error and poor performance because of UDF. What's your

Re: Converting a string of format of 'dd/MM/yyyy' in Spark sql

Minor correction UK date is dd/MM/ scala> sql("select paymentdate, TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP(paymentdate,'dd/MM/'),'-MM-dd')) AS newdate from tmp").first res47: org.apache.spark.sql.Row = [10/02/2014,2014-02-10] Dr Mich Talebzadeh LinkedIn *

Re: Converting a string of format of 'dd/MM/yyyy' in Spark sql

Can you try this one? spark-sql> select paymentdate, TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP(paymentdate,'MM/dd/'),'-MM-dd')) from tmp; 10/02/2014 2014-10-02 spark-sql> From: Tamas Szuromi > Date: Thursday, March

Re: Converting a string of format of 'dd/MM/yyyy' in Spark sql

Thanks guys. Unfortunately neither is working sql("select paymentdate, unix_timestamp(paymentdate) from tmp").first res28: org.apache.spark.sql.Row = [10/02/2014,null] Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Converting a string of format of 'dd/MM/yyyy' in Spark sql

Actually, you should run sql("select paymentdate, unix_timestamp(paymentdate, "dd/MM/") from tmp").first But keep in mind you will get a unix timestamp! On 24 March 2016 at 17:29, Mich Talebzadeh wrote: > Thanks guys. > > Unfortunately neither is working > >

Re: Converting a string of format of 'dd/MM/yyyy' in Spark sql

Mich, Can you try the value for paymentdata to this format paymentdata='2015-01-01 23:59:59' , to_date(paymentdate) and see if it helps. On Thursday, March 24, 2016, Tamas Szuromi wrote: > Hi Mich, > > Take a look >

Re: Converting a string of format of 'dd/MM/yyyy' in Spark sql

Hi Mich, Take a look https://spark.apache.org/docs/1.6.1/api/java/org/apache/spark/sql/functions.html#unix_timestamp(org.apache.spark.sql.Column,%20java.lang.String) cheers, Tamas On 24 March 2016 at 14:29, Mich Talebzadeh wrote: > > Hi, > > I am trying to convert