Re: [I] [Bug] 无法将datetime类型数据转换成日期格式 [doris-spark-connector]

2024-12-03 Thread via GitHub


weilai201 commented on issue #200:
URL: 
https://github.com/apache/doris-spark-connector/issues/200#issuecomment-2516050404

   > > ### Search before asking
   > > 
   > > * [x]   I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   > > 
   > > ### Version
   > > 1.3.1
   > > ### What's Wrong?
   > > doris-spark-connector无法将doris库中datetime类型数据在spark中转换为java 
Date或timestamp类型。目前统一转换为String类型,对于使用非常不方便!
   > > ### What You Expected?
   > > doris中的datetime类型转换为java中的Date或Timestamp类型
   > > ### How to Reproduce?
   > > _No response_
   > > ### Anything Else?
   > > _No response_
   > > ### Are you willing to submit PR?
   > > 
   > > * [ ]   Yes I am willing to submit a PR!
   > > 
   > > ### Code of Conduct
   > > 
   > > * [x]   I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   > 
   > 我也遇到同样的问题了,目前我临时的解决办法是调整了部分代码,rowBatch处理的时候把datetimev2转为timestamp, 
https://github.com/dingyufei615/doris-spark-connector/tree/support_datetimev2_to_timastamp
 新增了一个参数`doris.read.datetimev2.as.timestamp.enabled` ,设置为`true` 
即可,我当前使用的是Spark3.3 Scala2.12,目前测试使用没遇到问题。社区后面也说了会重构这部分,如果不着急的话可以等后续的版本。
   
   👍!我目前也是通过修改源码做了支持!期望你的代码能够合并进去


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [I] [Bug] 无法将datetime类型数据转换成日期格式 [doris-spark-connector]

2024-12-03 Thread via GitHub


weilai201 commented on issue #200:
URL: 
https://github.com/apache/doris-spark-connector/issues/200#issuecomment-2516049114

   > s中的datetime类型转换为java中的Date或Timesta
   
   👍!我目前也是通过修改源码做了支持


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [I] [Bug] 无法将datetime类型数据转换成日期格式 [doris-spark-connector]

2024-12-01 Thread via GitHub


dingyufei615 commented on issue #200:
URL: 
https://github.com/apache/doris-spark-connector/issues/200#issuecomment-2510628719

   > ### Search before asking
   > * [x]  I had searched in the 
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and 
found no similar issues.
   > 
   > ### Version
   > 1.3.1
   > 
   > ### What's Wrong?
   > doris-spark-connector无法将doris库中datetime类型数据在spark中转换为java 
Date或timestamp类型。目前统一转换为String类型,对于使用非常不方便!
   > 
   > ### What You Expected?
   > doris中的datetime类型转换为java中的Date或Timestamp类型
   > 
   > ### How to Reproduce?
   > _No response_
   > 
   > ### Anything Else?
   > _No response_
   > 
   > ### Are you willing to submit PR?
   > * [ ]  Yes I am willing to submit a PR!
   > 
   > ### Code of Conduct
   > * [x]  I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   
   我也遇到同样的问题了,目前我临时的解决办法是调整了部分代码,rowBatch处理的时候把datetimev2转为timestamp,
   
https://github.com/dingyufei615/doris-spark-connector/tree/support_datetimev2_to_timastamp
   新增了一个参数`doris.read.datetimev2.as.timestamp.enabled` ,设置为`true` 
即可,我当前使用的是Spark3.3 Scala2.12,目前测试使用没遇到问题。社区后面也说了会重构这部分,如果不着急的话可以等后续的版本。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [I] [Bug] 无法将datetime类型数据转换成日期格式 [doris-spark-connector]

2024-12-01 Thread via GitHub


dingyufei615 commented on issue #200:
URL: 
https://github.com/apache/doris-spark-connector/issues/200#issuecomment-2510471938

   > > Would it be possible to add a parameter that allows users to choose 
whether to convert datetime to timestamp? In Spark, using string-type dates can 
sometimes cause confusion and require additional conversion steps for users.
   > 
   > For the current code structure, all functions are in one module. Since 
different spark versions map the date and timestamp types in catalyst to 
different java classes, it is difficult to be compatible with different 
versions of the API. But the good news is that an important refactoring is 
underway and will be released in the next version. So after this, different 
data deserialization can be performed for different spark versions, and this 
problem will not be difficult to solve. This optimization can be planned to be 
released in the version after next, and you are also welcome to contribute this 
optimization to the community.
   
   Nice! Has the refactored code been merged already? I would be happy to 
contribute code for this optimization.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [I] [Bug] 无法将datetime类型数据转换成日期格式 [doris-spark-connector]

2024-12-01 Thread via GitHub


gnehil commented on issue #200:
URL: 
https://github.com/apache/doris-spark-connector/issues/200#issuecomment-2510420169

   > Would it be possible to add a parameter that allows users to choose 
whether to convert datetime to timestamp? In Spark, using string-type dates can 
sometimes cause confusion and require additional conversion steps for users.
   
   For the current code structure, all functions are
   in one module. Since different spark versions map the date and timestamp
   types in catalyst to different java classes, it is difficult to be
   compatible with different versions of the API.
   But the good news is that an important refactoring is underway and will be
   released in the next version.
   So after this, different data deserialization can be performed for
   different spark versions, and this problem will not be difficult to solve.
   This optimization can be planned to be released in the version after next,
   and you are also welcome to contribute this optimization to the community.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [I] [Bug] 无法将datetime类型数据转换成日期格式 [doris-spark-connector]

2024-12-01 Thread via GitHub


gnehil commented on issue #200:
URL: 
https://github.com/apache/doris-spark-connector/issues/200#issuecomment-2510418441

   Hi, Yufei
   
   Thanks for your interest in the community.
   This is a good suggestion.For the current code structure, all functions are
   in one module. Since different spark versions map the date and timestamp
   types in catalyst to different java classes, it is difficult to be
   compatible with different versions of the API.
   But the good news is that an important refactoring is underway and will be
   released in the next version.
   So after this, different data deserialization can be performed for
   different spark versions, and this problem will not be difficult to solve.
   This optimization can be planned to be released in the version after next,
   and you are also welcome to contribute this optimization to the community.
   
   Best regards
   
   Li Heng
   
   Yufei Ding ***@***.***> 于2024年11月29日周五 20:17写道:
   
   > Would it be possible to add a parameter that allows users to choose
   > whether to convert datetime to timestamp? In Spark, using string-type dates
   > can sometimes cause confusion and require additional conversion steps for
   > users.
   >
   > —
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   > You are receiving this because you commented.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [I] [Bug] 无法将datetime类型数据转换成日期格式 [doris-spark-connector]

2024-11-29 Thread via GitHub


dingyufei615 commented on issue #200:
URL: 
https://github.com/apache/doris-spark-connector/issues/200#issuecomment-2507703611

   Would it be possible to add a parameter that allows users to choose whether 
to convert datetime to timestamp? In Spark, using string-type dates can 
sometimes cause confusion and require additional conversion steps for users.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [I] [Bug] 无法将datetime类型数据转换成日期格式 [doris-spark-connector]

2024-07-04 Thread via GitHub


gnehil commented on issue #200:
URL: 
https://github.com/apache/doris-spark-connector/issues/200#issuecomment-2208625068

   Thanks for your suggestion, there are some incompatibilities between spark2 
and spark3, we are working on fixing this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org