steveloughran commented on PR #10233: URL: https://github.com/apache/iceberg/pull/10233#issuecomment-3139465000
@danielcweeks so here we are 1. no reflection 1. forced update of hadoop version on spark 3.4 and 3.5. That forced update isn't great, but if iceberg compiles with hadoop 3.4.1 then it's already possibly broken somewhere else when run there. We put a lot of effort into backward compatibility of APIs, but its really hard to do forward compatibility (openFile(), createFile() and other builders are the best we have here. A real risk is some overloaded method compiling down differently, different exception signatures etc. That's independ of this PR, which simply finds the problem faster. Ignoring that fundamental issue, I could tweak this one to look for the presence of the new API classes via a loadResource call, and don't attempt bulk delete if not found. Provided all uses of the bulk delete is isolated to guarded method, this wouldn't exacerbate the existing issue. How to test if that worked? remove the forced dependency updates from the spark versions. Thoughts? It's easily done, and isn't the full reflection game, just a probe and a guard -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
