deepakpanda93 commented on issue #15128:
URL: https://github.com/apache/hudi/issues/15128#issuecomment-4892043601
This issue was reviewed as part of the JIRA-migrated backlog triage.
This is expected Spark behavior, not a Hudi bug. In a long-lived Spark SQL /
Thrift (JDBC) session, a table's file listing is resolved and cached in that
session's relation/FileIndex on first access, so later queries in the same
connection don't automatically see commits written afterward (by Flink or any
external writer). A new connection is a new session with a fresh listing -
which is why your step 5 (reconnect) already showed correct data.
Reproduced and confirmed on Hudi 1.2.0 / Spark 3.5 with a Hive-metastore
table and two sessions:
-- session A (reader): seed + query -> age = 20 (relation cached)
-- session B (writer): UPDATE cache_repro SET age = 27 WHERE id = 1;
-- session A: SELECT ... -> age = 20 (stale, cached)
REFRESH TABLE cache_repro;
-- session A: SELECT ... -> age = 27 (fresh)
To pick up new data within an existing connection, refresh the cached
metadata:
REFRESH TABLE t1; -- or spark.catalog.refreshTable("t1")
then re-run the query. This applies to all file-based sources in the Spark
Thrift Server, not just Hudi.
Closing as resolved (works as designed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
