Gengliang Wang created SPARK-61820:
--------------------------------------
Summary: Clarify that native data source offsets are progress
boundaries, and that adjacent ranges must not repeat or skip records
Key: SPARK-61820
URL: https://issues.apache.org/jira/browse/SPARK-61820
Project: Spark
Issue Type: Documentation
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
Version 1 of the native data source ABI ({{spark_datasource.h}}) describes the
offsets of a streaming read in terms of the offset that a micro-batch starts
after, and a micro-batch as the data after its start up to and including its
end. This describes them as progress boundaries instead: the library defines
how offsets map to records, equal boundaries describe no data, and adjacent
ranges must neither repeat nor skip records. Committing an offset acknowledges
all the data before its progress boundary.
The rule on adjacent ranges is a requirement on libraries that exactly-once
reads imply but that the header did not state. It applies to every
implementation of the ABI, such as the Rust SDK (SPARK-61282) and the C++ SDK
(https://github.com/apache/spark/pull/59726).
The change updates the comments of {{spark_datasource.h}}, the Native Data
Sources guide ({{docs/sql-native-data-sources.md}}), and the doc comments of
{{NativeLibrary.latestOffset}}, {{ColumnarStreamReader}} and
{{ColumnarMicroBatchStream}}. It does not change the behavior of Spark.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]