Peter Toth created SPARK-60930:
----------------------------------
Summary: InMemoryFileIndex equality ignores the listed files, so
reuse can merge stale and fresh scans
Key: SPARK-60930
URL: https://issues.apache.org/jira/browse/SPARK-60930
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
InMemoryFileIndex.equals compares only the root paths. Two scans of one path
whose file listings differ therefore canonicalize equal, and exchange reuse can
serve one from the other. The typical case is a Dataset analyzed before files
were added, used together with a fresh read after.
Repro, with AQE on or off and broadcast off:
{code:scala}
spark.range(0, 10).write.parquet(path)
sql(s"CREATE OR REPLACE TEMP VIEW pv AS SELECT id FROM parquet.`$path`")
val held = spark.table("pv")
held.collect()
spark.range(100, 105).write.mode("append").parquet(path)
held.join(spark.table("pv"), Seq("id"), "full_outer").count() // 10, expected
15
{code}
{{held.union(spark.table("pv")).count()}} returns the expected 25, because
there is no exchange to reuse.
Other consumers of the same equality:
* apache/spark#58229 (SPARK-58950) groups view occurrences by canonical
equality. With its config on, the union above returns 20.
* CacheManager finds cached data with sameResult. After files are added outside
Spark, a fresh read of a cached path keeps hitting the stale cache until it is
refreshed.
A prototype that also compares the listed files (path, length, modification
time) in equals:
* returns 15 for the join and 25 for the union;
* keeps exchange reuse for two fresh reads of an unchanged path;
* passes FileIndexSuite, ReuseExchangeAndSubquerySuite, ExchangeSuite,
CachedTableSuite, DatasetCacheSuite and FileSourceStrategySuite.
It changes the cache behavior above: the fresh read misses the stale cache and
reads the new files. A real fix also needs a fingerprint computed once per
listing, instead of comparing full listings on every equals, and a decision on
the cache semantics.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
