Hi Team,
When `spark.sql.catalog.spark_catalog.scan-planning-mode=server`, by default,
RESTTableScan's PlanTableScanRequest has no snapshotId set and is up to the
catalog server to pick up the latest snapshotId. In theory, between Spark’s
datasource schema binding on client side and remote table scan on the server
side, table schema can change and new data can be ingested into the table. The
server can return files written with new schema while the client tries to read
them with old schema. Is there a potential issue regarding schema mismatch as
the result of different snapshotIds are used by the client and the server?
The source of the behavior:
Compared with the local mode, DataTableScan.doPlanFiles() invokes the public
version of SnapshotScan.snapshot(), which picks snapshotId via
table().currentSnapshot() if snapshotId is not provided in the context, so
schema binding and table scanning are based on the same snapshotId in the Table
object.
public Snapshot snapshot() {
return snapshotId() != null ? table().snapshot(snapshotId()) :
table().currentSnapshot();
}
In contrast, RESTTableScan.planFiles() invokes the protected version of
SnapshotScan.snapshot(), which returns null if not provided in the context.
protected Long snapshotId() {
return context().snapshotId();
}
I wonder if the diverged behavior is intentional? If yes, for what reason?
Thanks,
Limin
