JoshRosen commented on issue #24668: [SPARK-27676][SQL] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles URL: https://github.com/apache/spark/pull/24668#issuecomment-494595508 It looks like this change is breaking the ability to drop a catalog table whose underlying files don't exist / have been deleted. In `DropTableCommand` we have ```scala catalog.refreshTable(tableName) catalog.dropTable(tableName, ifExists, purge) ``` Here, `refresh()` is both clearing old caches (file listing, cached tables, views) _and_ is repopulating some of them (re-listing) and that's triggering the error. To avoid this, I think I can more narrowly-scope this patch's changes to only propagate `FileNotFoundException` if it occurs for non-root-path listings.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org