[
https://issues.apache.org/jira/browse/SPARK-54688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-54688:
-----------------------------------
Labels: pull-request-available (was: )
> Let ArrowBatchIterator return ArrowBatch instead of Array[Byte]
> ---------------------------------------------------------------
>
> Key: SPARK-54688
> URL: https://issues.apache.org/jira/browse/SPARK-54688
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.2.0
> Reporter: Yicong Huang
> Priority: Major
> Labels: pull-request-available
>
> Currently {{ArrowBatchIterator}} returns {{Iterator[Array[Byte]]}}, which
> only provides the raw serialized batch bytes. This limits the ability to
> expose batch-level metadata (e.g., row count, byte sizes) to callers without
> re-parsing the batch.
> {code:scala}
> // Current implementation
> private[sql] class ArrowBatchIterator(...)
> extends Iterator[Array[Byte]] with AutoCloseable {
>
> override def next(): Array[Byte] = {
> // ...
> bytes
> }
> }
> {code}
> Propose to let it return ArrowBatch instead.
> {code:scala}
> private[sql] class ArrowBatchIterator(...)
> extends Iterator[ArrowBatch] with AutoCloseable {
>
> override def next(): ArrowBatch = {
> // ...
> ArrowBatch(rowCount, bytes.length, bytes)
> }
> }
> {code}
> This enables downstream consumers to access batch metadata without
> deserializing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
