[jira] [Commented] (SPARK-26994) Enhance StructField to accept number format or date format

2019-03-04 Thread Murali Aakula (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783425#comment-16783425
 ] 

Murali Aakula commented on SPARK-26994:
---

We could have defined that format conversion in the setter methods. There are 
multiple alternatives to solve our problem. I'm sure that everyone may have 
implemented in one or the other alternatives. Also, there will be performance 
issues while converting form Raw to User defined object (mentioned in the spark 
documentation). 

So I thought that, it will actually make sense, if the parsers/readers (Json, 
CSV, ..) can handle this conversion. StructField just holds the 
values(patterns) just like column name and the column datatype. The conversion 
logic will be in the parsers/readers. Hope this makesense.

> Enhance StructField to accept number format or date format
> --
>
> Key: SPARK-26994
> URL: https://issues.apache.org/jira/browse/SPARK-26994
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Murali Aakula
>Priority: Major
>
> Enhance StructField to accept number format or date format and Enahcne 
> reader/steramreader and writer/streamwriter to use these formats.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26994) Enhance StructField to accept number format or date format

2019-02-28 Thread Hyukjin Kwon (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781314#comment-16781314
 ] 

Hyukjin Kwon commented on SPARK-26994:
--

Can't you just define one function for this conversion? I don't think we should 
add automatic conversion logic into {{StructField}}. Those conversions should 
happen via SQL expression and APIs.

> Enhance StructField to accept number format or date format
> --
>
> Key: SPARK-26994
> URL: https://issues.apache.org/jira/browse/SPARK-26994
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core, SQL
>Affects Versions: 2.4.0
>Reporter: Murali Aakula
>Priority: Major
>
> Enhance StructField to accept number format or date format and Enahcne 
> reader/steramreader and writer/streamwriter to use these formats.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26994) Enhance StructField to accept number format or date format

2019-02-28 Thread Murali Aakula (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780572#comment-16780572
 ] 

Murali Aakula commented on SPARK-26994:
---

Sample Input:

id,cpt,schedule_date,dstamp,weight,miles
1,MON-14:30,2018-11-30 10:00,29/11/2018 21:07:47,"10,110.12","1,010.12"
2,MON-14:30,2018-11-30 11:00,29/11/2018 21:17:47,"1,110.12","10.10"

Currently, we are reading few columns as string(schedule_date, dstamp, weight, 
miles) then converting them using spark sql functions as needed(date, date, 
double, double). Kind of Two step process. We can set one date and one 
timestamp format on spark config, but not multiple.

> Enhance StructField to accept number format or date format
> --
>
> Key: SPARK-26994
> URL: https://issues.apache.org/jira/browse/SPARK-26994
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core, SQL
>Affects Versions: 2.4.0
>Reporter: Murali Aakula
>Priority: Major
>
> Enhance StructField to accept number format or date format and Enahcne 
> reader/steramreader and writer/streamwriter to use these formats.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26994) Enhance StructField to accept number format or date format

2019-02-27 Thread Hyukjin Kwon (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780117#comment-16780117
 ] 

Hyukjin Kwon commented on SPARK-26994:
--

Can you show expected input and output in the JIRA?

> Enhance StructField to accept number format or date format
> --
>
> Key: SPARK-26994
> URL: https://issues.apache.org/jira/browse/SPARK-26994
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core, SQL
>Affects Versions: 2.4.0
>Reporter: Murali Aakula
>Priority: Major
>
> Enhance StructField to accept number format or date format and Enahcne 
> reader/steramreader and writer/streamwriter to use these formats.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org