RiccardoSale opened a new pull request, #20493:
URL: https://github.com/apache/druid/pull/20493
Fixes #20492.
### Description
`FileUtils.writeAtomically` writes to a temporary file named
`.<fileName>.<uuid>` before moving it to the target. This adds 38 characters,
so a file name between 218 and 255 bytes is valid but its temporary name is
not, and the write fails with `File name too long`.
`WorkerTaskManager` persists assigned and completed tasks this way, using
the task ID as the file name. Tasks with such IDs can never start on a
MiddleManager or Indexer, and the Overlord waits
`druid.indexer.runner.taskAssignmentTimeout` before failing each one.
Kill tasks hit this first, because their IDs are much longer than ingestion
task IDs for the same datasource:
| Task | ID format | Characters added to the datasource name |
|---|---|---|
| Kafka ingestion | `index_kafka_<dataSource>_<hash>_<random>` | 37 |
| Batch ingestion | `index_parallel_<dataSource>_<random>_<createdTime>` |
49 |
| Coordinator kill |
`coordinator-issued_kill_<dataSource>_<random>_<intervalStart>_<intervalEnd>_<createdTime>`
| 108 |
So a datasource can ingest normally but never be cleaned up. With the
temporary name's 38 characters, kill tasks fail for datasource names longer
than 109 characters, while ingestion keeps working up to 168 (batch) or 180
(Kafka).
#### Changes
- Name the temporary file `.<uuid>`, without the target file name, so its
length no longer depends on the target. The UUID keeps it unique, and the
target file keeps its full name after the move. This raises the limit for every
task ID to the filesystem limit of 255 bytes.
- Add tests for a 255-character file name and a 255-character task ID
assigned through `WorkerTaskManager`.
#### Proposed follow-up: kill task IDs without the interval
After this fix, kill tasks still support shorter datasource names than
ingestion: up to 147 characters, against 206 for batch and 218 for Kafka.
Leaving the interval out of kill task IDs would make them 50 characters shorter
and raise the kill limit to 197, close to batch ingestion. The interval is
already stored in the task payload. This changes the kill task ID format, so
I'd rather do it in a separate PR if maintainers agree.
#### Release note
Fixed: Tasks with IDs longer than 217 characters no longer fail to start on
MiddleManagers and Indexers with `File name too long`.
<hr>
##### Key changed/added classes in this PR
* `FileUtils`
* `FileUtilsTest`
* `WorkerTaskManagerTest`
<hr>
This PR has:
- [x] been self-reviewed.
- [x] a release note entry in the PR description.
- [x] added or updated unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
