WangGuangxin opened a new pull request, #12925:
URL: https://github.com/apache/gluten/pull/12925
## What changes are proposed in this pull request?
This PR reduces driver-side memory usage when converting large `InSet`
expressions to Substrait.
Before this change, `InSetTransformer` eagerly materialized one Substrait
`LiteralNode` for each element in the IN set and stored all of them in
`SingularOrListNode`. For large IN lists, this creates a large
number of intermediate objects and increases retained heap usage before
protobuf serialization.
This PR changes the flow as follows:
- `InSetTransformer` keeps the raw IN-set values together with the child
data type, instead of eagerly building one `LiteralNode` per value.
- `SingularOrListNode` constructs the corresponding Substrait literal
protobuf entries lazily in `toProtobuf()`.
- During protobuf serialization, options are appended to the builder
directly, instead of first collecting them into an extra temporary
`List<Expression>`.
- The existing deterministic ordering is preserved by sorting IN-set
values with the same `Literal(_, valueType).toString()` logic as before.
In short, this patch reduces the number of long-lived intermediate objects
created for large `InSet` expressions and also avoids an extra temporary
protobuf list during serialization.
## Why are the changes needed?
Queries with very large `IN (...)` predicates may be optimized by Spark
into `InSet`. In Gluten's Substrait conversion path, eagerly creating and
retaining one `LiteralNode` per IN-set value can cause high memory
pressure on the driver.
The issue is especially visible for large IN lists, where the conversion
step itself may consume much more memory than necessary even though the final
serialized representation is only needed at protobuf
generation time.
By deferring literal node construction until `toProtobuf()`, we reduce
retained heap usage in the expression tree and lower peak allocation pressure
in the conversion path.
## How was this patch tested?
Manually.
## Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
