GitHub user rdblue opened a pull request:
https://github.com/apache/spark/pull/21118
SPARK-23325: Use InternalRow when reading with DataSourceV2.
## What changes were proposed in this pull request?
This updates the DataSourceV2 API to use InternalRow instead of Row for the
default case with no scan mix-ins. Because the API is changing significantly in
the same places, this also renames ReaderFactory back to ReadTask.
Support for readers that produce Row is added through
SupportsDeprecatedScanRow, which matches the previous API. Readers that used
Row now implement this class and should be migrated to InternalRow.
Readers that previously implemented SupportsScanUnsafeRow have been
migrated to use no SupportsScan mix-ins and produce InternalRow.
## How was this patch tested?
This uses existing tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rdblue/spark
SPARK-23325-datasource-v2-internal-row
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21118.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21118
----
commit eddd049ed78970dda0796a37462e52f53bfeacc4
Author: Ryan Blue <blue@...>
Date: 2018-04-20T20:15:58Z
SPARK-23325: Use InternalRow when reading with DataSourceV2.
This updates the DataSourceV2 API to use InternalRow instead of Row for
the default case with no scan mix-ins. Because the API is changing
significantly in the same places, this also renames ReaderFactory back
to ReadTask.
Support for readers that produce Row is added through
SupportsDeprecatedScanRow, which matches the previous API. Readers that
used Row now implement this class and should be migrated to InternalRow.
Readers that previously implemented SupportsScanUnsafeRow have been
migrated to use no SupportsScan mix-ins and produce InternalRow.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
