andygrove opened a new issue, #7245:
URL: https://github.com/apache/datafusion-comet/issues/7245
### Describe the bug
`lpad` and `rpad` fail natively when their string argument references a
column but evaluates to a scalar for the whole batch. `CometStringLPad` and
`CometStringRPad` only keep a `Literal` string on Spark, so `IF(id > 0, (SELECT
max(s) FROM strs), s)` is serialized. When every row in the batch takes the
subquery branch, the conditional returns a scalar, and the native
`read_side_padding` kernel only accepts an array as its first argument:
```
org.apache.comet.CometNativeException: Unsupported arguments
[Scalar(Utf8("cd")), Scalar(Int32(5)), Scalar(Utf8("0"))] for function
rpad/lpad/read_side_padding.
```
Spark returns the padded value. Default configs are enough to hit this.
### Steps to reproduce
```scala
Seq((1, "ab"), (2, "cd")).toDF("id",
"s").repartition(1).write.parquet("/tmp/strs")
spark.read.parquet("/tmp/strs").createOrReplaceTempView("strs")
sql("SELECT id, lpad(IF(id > 0, (SELECT max(s) FROM strs), s), 5, '0') FROM
strs").collect()
```
`rpad`, and `CASE WHEN id > 0 THEN (SELECT max(s) FROM strs) ELSE s END` as
the string argument, fail the same way.
Reproduced on `main` at c4a0b1f2bc with the default Spark 4.1 profile.
### Expected behavior
Spark's rows: `000cd` for both rows with `lpad`, and `cd000` with `rpad`.
### Additional context
This is the padding half of the scalar-argument problem in #3336. #3336 says
`hour`, `minute`, `second` and `unix_timestamp` need `ConstantFolding` disabled
to fail, but a plain scalar subquery reaches them with default configs too.
`SELECT hour((SELECT max(ts) FROM t)) FROM t` fails with `hour(scalar) should
be fold in Spark JVM side.`, and `unix_timestamp` fails the same way. #5889
carries fixes for both halves that don't depend on its struct support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
