heyihong commented on code in PR #51473:
URL: https://github.com/apache/spark/pull/51473#discussion_r2213381273
##########
sql/connect/common/src/main/protobuf/spark/connect/expressions.proto:
##########
@@ -215,13 +215,19 @@ message Expression {
}
message Array {
- DataType element_type = 1;
+ // (Optional) The element type of the array. Only need to set this when
the elements are
+ // empty, since spark 4.1+ supports inferring the element type from the
elements.
+ optional DataType element_type = 1;
repeated Literal elements = 2;
Review Comment:
@zhengruifeng This change optimizes out both arrays and maps, and also
applies to non-primitive types. Also, [the reduction of
function_lit_array.json](https://github.com/apache/spark/pull/51473/files#diff-608710d9b20c4ddde4354bfaee5407da1f4bc4ebdbc33daa1ef5c8a7c8f9b7f3)
in size is obvious.
##########
sql/connect/common/src/main/protobuf/spark/connect/expressions.proto:
##########
@@ -215,13 +215,19 @@ message Expression {
}
message Array {
- DataType element_type = 1;
+ // (Optional) The element type of the array. Only need to set this when
the elements are
+ // empty, since spark 4.1+ supports inferring the element type from the
elements.
+ optional DataType element_type = 1;
repeated Literal elements = 2;
Review Comment:
@zhengruifeng This change optimizes out both arrays and maps, and also
applies to non-primitive types. Also, [the reduction of
function_lit_array.json](https://github.com/apache/spark/pull/51473/files#diff-608710d9b20c4ddde4354bfaee5407da1f4bc4ebdbc33daa1ef5c8a7c8f9b7f3)
in size seems obvious.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
