joelrobin18 opened a new pull request, #58907: URL: https://github.com/apache/spark/pull/58907
### What changes were proposed in this pull request? This PR adds `FileIndex.isTimeTravel`, defaulting to `false`, so V1 file data sources can identify immutable time-travel snapshots. `CacheManager` honors this signal during write-driven cache invalidation by table name and path. `InsertIntoHadoopFsRelationCommand` marks its path-based recache as write-driven. Existing explicit refresh operations continue to include time-travel snapshots. ### Why are the changes needed? [SPARK-53732](https://issues.apache.org/jira/browse/SPARK-53732) changed write-driven cache invalidation from plan matching to table-name matching with `includeTimeTravel = false`. V2 relations honor that flag through `DataSourceV2Relation.timeTravelSpec`, but V1 `LogicalRelationWithTable` was matched only by table name. For providers that lower a pinned read to `LogicalRelation(HadoopFsRelation(FileIndex))`, the time-travel state survives only in the `FileIndex`. A write to the live table therefore cleared and rebuilt the pinned cache. The DataFrame remained registered as cached, but the next action re-executed its plan, including expensive or non-idempotent UDFs. The new signal closes this V1 gap without coupling Spark to a specific data source. JIRA: [SPARK-59631](https://issues.apache.org/jira/browse/SPARK-59631) ### Does this PR introduce _any_ user-facing change? Yes. For V1 data sources that identify an immutable snapshot, appending to or overwriting the live table no longer recomputes an already materialized cache for that snapshot. Live-table caches are still refreshed after writes, and explicit refresh or uncache operations still include pinned snapshots. Before: ```text is_cached=true udf_calls=10->20 tokens_same=false ``` After the coordinated Spark and provider changes: ```text is_cached=true udf_calls=10->10 tokens_same=true ``` ### How was this patch tested? Added `V1TimeTravelCacheSuite` with six cases covering name- and path-based invalidation for pinned and live V1 relations, including automatic write invalidation and explicit refresh. ```bash build/sbt "sql/testOnly org.apache.spark.sql.V1TimeTravelCacheSuite" ``` All six tests passed. The existing V2 time-travel cache test, SQL MiMa, and main/test Scalastyle checks also passed. End-to-end testing reproduced `10->20` UDF executions with unpatched Spark and preserved `10->10` with the coordinated Spark/provider changes. The provider integration matrix passed all four tests in both `AUTO` and `NONE` modes. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (gpt-5.6-sol) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
