whua3 opened a new pull request, #11304: URL: https://github.com/apache/gravitino/pull/11304
### What changes were proposed in this pull request? This PR fixes a long-standing race in `BaseGVFSOperations.internalFileSystemCache` where a cached `FileSystem` could be `close()`d while a caller still held it through a long-lived stream, causing `RejectedExecutionException` on the next async IO. See #11303 for details. The fix aligns GVFS with Hadoop's own `FileSystem.CACHE` semantics: **a cached FileSystem is never proactively closed by the cache.** - The `removalListener` no longer calls `fs.close()` by default. Eviction (size or time based) only detaches the entry from the cache map; the underlying FS remains usable by any caller still holding a reference. - A new `Set<FileSystem> allCreatedFileSystems` tracks every FS the operations instance ever created. `BaseGVFSOperations#close()` closes all of them — including those evicted from the cache — at the unambiguous lifecycle end of the GVFS instance. - A new optional config `fs.gravitino.fileset.cache.closeOnEviction` (default `false`) is provided as an emergency escape hatch to fall back to the legacy behaviour. In that mode the evicted FS is also removed from the tracking set to prevent double-close at shutdown. ### Why are the changes needed? Closes #11303. Without this fix, any GVFS-backed Spark/Flink/Hive job whose per-task duration exceeds `evictionMillsAfterAccess` (default 1h) fails with `RejectedExecutionException` on the underlying object-store FS thread pool. This is a re-occurrence of the bug originally tracked in #3928 (2024) and only partially mitigated by #3932 (default raised from 5min to 1h). Increasing the timeout reduces incidence but does not fix the underlying race; long-running batch tasks writing > 1h still hit this. This PR removes the race itself by binding FS lifecycle to GVFS lifecycle (the same pattern Hadoop uses in `FileSystem.CACHE`). ### Does this PR introduce any user-facing change? - No default value changes; cache hit/miss semantics preserved. - One observable change: an evicted entry's underlying `FileSystem` is closed at GVFS shutdown instead of at eviction time. - One new optional config `fs.gravitino.fileset.cache.closeOnEviction` (default `false`) for legacy fallback. ### How was this patch tested? New unit tests in `TestGvfsBase`: - `testFileSystemNotClosedOnEviction` — invalidate the cache and assert the previously-cached FS is still usable. - `testEvictedFileSystemClosedOnGvfsShutdown` — assert that an evicted-but-not-yet-closed FS is closed exactly once when GVFS itself is closed. - `testCacheHitMissSemanticsUnchanged` — regression: same key returns the same cached FS as before. - `testCloseOnEvictionLegacyBehaviour` — verify the legacy escape-hatch path still closes on eviction and avoids double-close. Existing `clients:filesystem-hadoop3` test suite passes locally: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
