[ https://issues.apache.org/jira/browse/HUDI-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ethan Guo updated HUDI-5092: ---------------------------- Attachment: image (1).png > Querying Hudi table throws NoSuchMethodError in Databricks runtime > ------------------------------------------------------------------- > > Key: HUDI-5092 > URL: https://issues.apache.org/jira/browse/HUDI-5092 > Project: Apache Hudi > Issue Type: Bug > Components: spark > Affects Versions: 0.12.0 > Reporter: Ethan Guo > Assignee: Ethan Guo > Priority: Blocker > Fix For: 0.13.1 > > Attachments: image (1).png, image.png > > > Originally reported by the user: > [https://github.com/apache/hudi/issues/6137] > > Crux of the issue is that Databricks's DBR runtime diverges from OSS Spark, > and in that case `FileStatusCache` API is very clearly divergent b/w the two. > There are a few approaches we can take: > # Avoid reliance on Spark's FIleStatusCache implementation altogether and > rely on our own one > # Apply more staggered approach where we first try to use Spark's > FileStatusCache and if it doesn't match expected API, we fallback to our own > impl > > Approach # 1 would actually mean that we're not sharing cache implementation > w/ Spark, which in turn would entail that in some cases we might be keeping 2 > instances of the same cache. Approach # 2 remediates that and allows us to > only fallback in case API is not compatible. -- This message was sent by Atlassian Jira (v8.20.10#820010)