Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
PHILO-HE commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2151468471 > > ubase64: #4482 > > I see you've map the from_base64 to unbase64, and respectively I find the base64 is almost the same as to_base64, so it's just a missing or is there any other consideration? @Donvi, seems there are a few semantic differences between Spark's `unbase64` & Velox's `from_base64`. So the simple mapping has not been accepted by the community. See discussion: https://github.com/apache/incubator-gluten/pull/5242#discussion_r1548887962. I guess similarly `to_base64` cannot be mapped due to some unknown differences. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
Donvi commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2151449323 > ubase64: #4482 I see you've map the from_base64 to unbase64, and respectively I find the base64 is almost the same as to_base64, so it's just a missing or is there any other consideration? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
boneanxs commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2141753499 @PHILO-HE I can try to support `array_sort` if no one picked, we internally need this function :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
PHILO-HE commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2139430047 > And there is a test for `collect_set` which used array_sort > > https://github.com/apache/incubator-gluten/blob/d35d1dc5e4450fdf58b8092ea26a0c928de29a48/backends-velox/src/test/scala/org/apache/gluten/execution/VeloxAggregateFunctionsSuite.scala#L846 @xumingming, this test only confirms aggregate is offloaded. In my local test, `array_sort` is not offloaded actually. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
xumingming commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2138963209 > > `shuffle`, `array_sort` are already supported, can be marked as complete. > > @xumingming, seems `sort_array` is supported, but `array_sort` is not. Please spare some time to confirm. Thanks! @PHILO-HE array_sort is marked as supported in the doc: https://github.com/apache/incubator-gluten/blob/e5dcbe3884d5215cc652246476b1ec980c859d4c/docs/velox-backend-support-progress.md?plain=1#L273 And there is a test for `collect_set` which used array_sort https://github.com/apache/incubator-gluten/blob/d35d1dc5e4450fdf58b8092ea26a0c928de29a48/backends-velox/src/test/scala/org/apache/gluten/execution/VeloxAggregateFunctionsSuite.scala#L846 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
Donvi commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2132826236 As I see only `rand` exists and no `randn`, I'm taking randn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
PHILO-HE commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2128849458 > `shuffle`, `array_sort` are already supported, can be marked as complete. Seems `sort_array` is supported, but `array_sort` is not. Please spare some time to confirm. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
PHILO-HE commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2128844727 > > > @PHILO-HE Thanks for your feedback. So i'd like to take date_part. Is to_date supported in gluten now? It doesn't shows in the list. I also would like to pick it. > > > > > > @NEUpanning, this list only maintains working-in-progress functions. I think `to_date` has been supported. See https://github.com/apache/incubator-gluten/blob/main/docs/velox-backend-support-progress.md. > > `date_part` may be supported also. I note the below test in Gluten. You can confirm whether all date patterns have been supported. > > https://github.com/apache/incubator-gluten/blob/d74fc97cf941759c79f440b0df5c5071655b984e/backends-velox/src/test/scala/org/apache/gluten/execution/ScalarFunctionsValidateSuite.scala#L808 > > I can't find any implementation of date_part and to_date function in Velox. Would you like to help me find it? Thanks. @NEUpanning, not a direct replacement. date_part is covered [here](https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/scala/org/apache/gluten/expression/ExpressionConverter.scala#L466). to_date is converted to Cast + GetTimestamp by Spark. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
xumingming commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2124608290 I will take a look at `bround` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
xumingming commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2124030163 `shuffle`, `array_sort` are already supported, can be marked as complete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
NEUpanning commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2121980171 > > @PHILO-HE Thanks for your feedback. So i'd like to take date_part. Is to_date supported in gluten now? It doesn't shows in the list. I also would like to pick it. > > @NEUpanning, this list only maintains working-in-progress functions. I think `to_date` has been supported. See https://github.com/apache/incubator-gluten/blob/main/docs/velox-backend-support-progress.md. > > `date_part` may be supported also. I note the below test in Gluten. You can confirm whether all date patterns have been supported. > > https://github.com/apache/incubator-gluten/blob/d74fc97cf941759c79f440b0df5c5071655b984e/backends-velox/src/test/scala/org/apache/gluten/execution/ScalarFunctionsValidateSuite.scala#L808 I can't find any implementation of date_part and to_date function in Velox. Would you like to help me find it? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
PHILO-HE commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2121466319 > @PHILO-HE Thanks for your feedback. So i'd like to take date_part. Is to_date supported in gluten now? It doesn't shows in the list. I also would like to pick it. @NEUpanning, this list only maintains working-in-progress functions. I think `to_date` has been supported. See https://github.com/apache/incubator-gluten/blob/main/docs/velox-backend-support-progress.md. `date_part` may be supported also. I note the below test in Gluten. You can confirm whether all date patterns have been supported. https://github.com/apache/incubator-gluten/blob/d74fc97cf941759c79f440b0df5c5071655b984e/backends-velox/src/test/scala/org/apache/gluten/execution/ScalarFunctionsValidateSuite.scala#L808 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
NEUpanning commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2119607052 @PHILO-HE Thanks for your feedback. So i'd like to take date_part. Is to_date supported in gluten now? It doesn't shows in the list. I also would like to pick it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
PHILO-HE commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2117417987 > I'd like to take unix_date, thanks. We have supported it in both Gluten & Velox. Just changed its state in the list. Thanks! https://github.com/apache/incubator-gluten/pull/5287 https://github.com/facebookincubator/velox/pull/8725 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
NEUpanning commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2117366460 I'd like to take unix_date, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
xumingming commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2109422932 @marin-ma Thanks for the advice, I will take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
marin-ma commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2109187441 > I'd like to take a look at `stack` function, it seems like a `Generator`, meaning one row of input might return multiple rows of output, does Velox has this generator ability? @xumingming Currently, 4 generator functions are supported : `explode`, `pos_explode`, 'inline` and `json_tuple`. The approach is creating a `ProjectNode` + `UnnestNode` + `ProjectNode` pattern in Velox pipeline. But seems like the `stack` function cannot use this pattern. Perhaps we can build another pipeline by leveraging the `ExpandNode` in Velox (Not sure if this approach really works). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
xumingming commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2109163864 I'd like to take a look at `stack` function, it seems like a `Generator`, meaning one row of input might return multiple rows of output, does Velox has this generator ability? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
ivoson commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2102428935 I'd like to implement `array_append` and `array_insert` for spark 3.4+ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
gaoyangxiaozhu commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2098240491 PR for width_bucket support, https://github.com/apache/incubator-gluten/pull/5634 looks still need velox side change for bucket_number <=0 case will send PR in velox repository to fix -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
Donvi commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2084419656 I'd like and am working in the math function expm1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
honeyhexin commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2084334638 I'd like to take cot, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
zhli1142015 commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2078523538 I'd like to take `soundex` and `levenshtein`, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
PHILO-HE commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2074589292 > hey @zhouyuan could you help add `format_string` and `format_number` in the list and I would take `format_string` and `format_number` later @gaoyangxiaozhu, just added them into the list. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
gaoyangxiaozhu commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2074550645 hey @zhouyuan could you help add `format_string` and `format_number` in the list and I would take `format_string` and `format_number` later -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
lyy-pineapple commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2063280619 > @PHILO-HE hello, I'd like take forall function. and exists(array) also support -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
acvictor commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2060476882 I'd like to try array_size. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
ivoson commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2060456474 I'd like to take flatten function. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
lyy-pineapple commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2058588022 @PHILO-HE hello, I'd like take forall function. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
Yohahaha commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2055471331 I'd like take transform function. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
Yohahaha commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2051313485 I'd like take get function, as known as GetArrayItem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
PHILO-HE commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2050928067 > Hey, @PHILO-HE , what's the plan with concat_ws ? It says "PR Ready" and I see that you have committed an implementation to this branch: [oap-project/velox@c5eec03](https://github.com/oap-project/velox/commit/c5eec030464970b83389c598354c0da4c8fb25ef). Is that the PR being referenced? Is it planned to be merged into velox main? Hi @supermem613, that commit is not used by Gluten main branch. We have another implementation for upstream velox: https://github.com/facebookincubator/velox/pull/8854. It is still under review. I will try to push the progress. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
supermem613 commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2049850094 Hey, @PHILO-HE , what's the plan with concat_ws ? It says "PR Ready" and I see that you have committed an implementation to this branch: https://github.com/oap-project/velox/commit/c5eec030464970b83389c598354c0da4c8fb25ef. Is that the PR being referenced? Is it planned to be merged into velox main? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
Yohahaha commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-204919 I'd like take percentile agg function. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
ivoson commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2042062087 Hi @PHILO-HE I'd like to take filter (array filter), thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
PHILO-HE commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2034062336 Just removed the below supported functions from the above list. Thanks for the contribution! `last_day, unhex, lead, lag, minute, second, may_keys` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
PHILO-HE commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2034042510 > @PHILO-HE , I'd like to pick up base64 and unbase64, please. > > (FYI, looks like there was a PR above for unbase64, but it seems to have been closed without committing ~45-55 days ago, so hopefully I am not conflicting with any work). Hi @supermem613, sorry for the late reply. I note Gluten PR https://github.com/apache/incubator-gluten/pull/5242 is trying to re-use Velox's existing `from_base64` function (proposed for prestosql) for `unbase64`. Not sure whether we can map `base64` to some other function. If there is no semantic difference, we can just re-use the existing Velox functions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
supermem613 commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2020700016 I'd like to pick up base64 and unbase64, please. (FYI, looks like there was a PR above for unbase64, but it seems to have been closed without committing ~45-55 days ago, so hopefully I am not conflicting with any work). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
yma11 commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2014391482 @PHILO-HE I would like to take `map_filter`. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
Re: [I] [VL] Unsupported spark function list [please leave a comment if you plan to pick some] [incubator-gluten]
mskapilks commented on issue #4039: URL: https://github.com/apache/incubator-gluten/issues/4039#issuecomment-2014344838 > I would like to work on `bool_and`, `bool_or` These are already supported it seems. All `bool_and`, `bool_or`, `every`, `some` get converted to min, max of bool column -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org
