malinjawi commented on issue #11985:
URL: https://github.com/apache/gluten/issues/11985#issuecomment-6659773232
I think this is the same bug I ran into this week, and the cause is on the
Velox side.
When a struct is null at some row, Velox leaves the values of its children
at that row undefined. Velox's Arrow export doesn't skip those rows, so for a
timestamp field it calls `toMicros()` on whatever bytes are left there and
throws. The numbers in your error (`Timestamp(140514369963968,
140514371047552)`) look like leftover pointers, and the three nested
`exportToArrowImpl` frames in your stack fit a timestamp inside a struct.
It reproduces with a native Parquet write, no UDF needed:
```python
spark.range(0, 2000000, 1, 4).selectExpr(
"id", "timestamp_seconds(id * 37) as ts",
"concat(repeat('q', cast(id % 50 as int)), cast(id as string)) as junk"
).write.mode("overwrite").parquet("/tmp/ts_struct/input")
spark.read.parquet("/tmp/ts_struct/input").selectExpr(
"id", "concat(repeat('q', cast(id % 50 as int)), cast(rand() as string))
as junk",
"case when id % 3 = 0 then null else named_struct('t', ts, 'v', id) end
as st"
).write.mode("overwrite").parquet("/tmp/ts_struct/out.parquet")
```
A left join that brings a struct with a timestamp in from the right side
fails the same way.
Fix is up in Velox: facebookincubator/velox#20126. I built Gluten against it
locally, and the CASE WHEN, left-join and nested-struct writes all go through
and read back correctly. Once it lands it should reach us with the next Velox
update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
