Jackie-Jiang opened a new pull request, #19580:
URL: https://github.com/apache/pinot/pull/19580
## Summary
`FileUtils.sizeOfDirectory` in commons-io 2.12+ walks with NIO's
`FileTreeWalker`, which lists a directory and then stats each entry. If an
entry disappears in between, the whole walk aborts with an
`UncheckedIOException`. Pinot moved from commons-io 2.11 to 2.22 in #18308, so
this now bites every site that sizes a directory another thread is writing to.
Two production sites do:
- **`REALTIME_CONSUMER_DIR_USAGE` gauge** in `BaseServerStarter`: walks each
realtime table's consumer directory on every metrics scrape. The mutable Lucene
text index in there flushes many times per second, creating and deleting
`*.tmp` files, so scrapes intermittently log this WARN and the gauge reports
`-1`:
```
WARN [BaseServerStarter] Failed to gather size info for consumer
directories
java.io.UncheckedIOException: java.nio.file.NoSuchFileException:
.../consumers/<segment>/<column>.lucene.v912.index/_zh6_Lucene90FieldsIndexfile_pointers_1yyd.tmp
```
- **`SegmentLocalFSDirectory.getDiskSizeBytes`**: walks the loaded segment's
directory, which a concurrent reload rewrites in place (index handlers delete
old index files, write `.inprogress` markers and temp files, and force-delete
the text index directory). The method only caught `IllegalArgumentException`,
the pre-2.12 failure type, so the `UncheckedIOException` escapes and fails the
whole `/tables/{table}/size` call. The controller's `TableSizeReader` then
treats every segment on that server as missing for that round.
This PR adds
`org.apache.pinot.common.utils.FileUtils.sizeOfDirectory(File)`: a
`listFiles()` / `length()` recursion whose primitives report a concurrently
deleted entry as `null` / `0` instead of throwing. It is the implementation
commons-io itself used through 2.11, plus a symlink skip. Both sites switch to
it, and the now-dead `IllegalArgumentException` catch in `getDiskSizeBytes` is
removed.
The util's Javadoc states when to use it: a directory another thread may be
writing to. The remaining commons-io `sizeOfDirectory` / `sizeOf` callers all
size directories the same thread has just finished producing (segment builds,
minion tasks, untarred uploads), where a genuine I/O error should surface, so
they intentionally stay on commons-io.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
