Thanks.
Tried this
scala> val a = df.filter(col("Total") > "").map(p =>
Invoices(p(0).toString,
p(1).toString.TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP(p(1),"dd/MM/yyyy"),"yyyy-MM-dd")),
p(2).toString.substring(1).replace(",", "").toDouble,
p(3).toString.substring(1).replace(",", "").toDouble,
p(4).toString.substring(1).replace(",", "").toDouble))
<console>:23: error: value TO_DATE is not a member of String
val a = df.filter(col("Total") > "").map(p =>
Invoices(p(0).toString,
p(1).toString.TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP(p(1),"dd/MM/yyyy"),"yyyy-MM-dd")),
p(2).toString.substring(1).replace(",", "").toDouble,
p(3).toString.substring(1).replace(",", "").toDouble,
p(4).toString.substring(1).replace(",", "").toDouble))
^
^
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 24 March 2016 at 22:00, Alexander Krasnukhin <the.malk...@gmail.com>
wrote:
> You can invoke exactly the same functions on scala side as well i.e.
> http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.functions$
>
> Have you tried them?
>
> On Thu, Mar 24, 2016 at 10:29 PM, Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>>
>> Hi,
>>
>> Read a CSV in with the following schema
>>
>> scala> df.printSchema
>> root
>> |-- Invoice Number: string (nullable = true)
>> |-- Payment date: string (nullable = true)
>> |-- Net: string (nullable = true)
>> |-- VAT: string (nullable = true)
>> |-- Total: string (nullable = true)
>>
>> I use mapping as below
>>
>> case class Invoices(Invoicenumber: String, Paymentdate: String, Net:
>> Double, VAT: Double, Total: Double)
>>
>> val a = df.filter(col("Total") > "").map(p => Invoices(p(0).toString,
>> p(1).toString, p(2).toString.substring(1).replace(",", "").toDouble,
>> p(3).toString.substring(1).replace(",", "").toDouble,
>> p(4).toString.substring(1).replace(",", "").toDouble))
>>
>>
>> I want to convert p(1).toString to datetime like below when I used in sql
>>
>> TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP(paymentdate,'dd/MM/yyyy'),'yyyy-MM-dd'))
>> AS paymentdate
>>
>>
>> Thanks
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn *
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>
>
>
> --
> Regards,
> Alexander
>
