L. C. Hsieh created SPARK-51931:
-----------------------------------
Summary: Add maxBytesPerOutputBatch to limit the number of bytes
of Arrow output batch
Key: SPARK-51931
URL: https://issues.apache.org/jira/browse/SPARK-51931
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: L. C. Hsieh
While implementing columnar-based operator for Spark, if the operator takes
input from Arrow-based evaluation operator in Spark, the number of bytes of
output batch is unlimited for now. For such columnar-based operator, sometimes
we want to limit the maximum bytes of input batch. If we need to limit the
batch size in bytes, it seems there is no existing way we can do.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
