jihoonson opened a new pull request #10376: URL: https://github.com/apache/druid/pull/10376
Part of https://github.com/apache/druid/issues/10352. ### Description This PR implements the live reporting system for parallel ingestion described in #10352. The live reporting system is used to help checking status of each subtask. During ingestion, subtasks can now periodically send a report to the supervisor task including its state and metrics (`LiveMetricsReporter`). When the supervisor task receives a report from a running subtask, it assumes that the task is still running and pauses status checking with Overlord for a while. As a result, it is expected to talk less to the Overlord compared to before which could be a bottleneck in scaling ingestion system. When the supervisor task receives a report from a finished subtask, it still checks the final status of the task with Overlord so that the final status is consistent across the whole cluster. If some reports are missing, the supervisor has a timer per subtask which triggers status checking with the Overlord. As a result, missing reports doesn't affect to the progress of the supervisor task. These are implemented in `TaskMonitor` and `ParallelIndexPhaseRunner`. A new interface `IngestionMetrics` was added in this PR, but it has only no-op implementation yet. I'm planning to add actual implementations as follow-ups. For reviewers, even though this PR adds more than 2000 of new lines, it should be not that hard to review as the new codes are mostly about new POJO classes for different report types. <hr> This PR has: - [x] been self-reviewed. - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.) - [ ] added documentation for new or modified features or behaviors. - [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml) - [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met. - [ ] added integration tests. - [ ] been tested in a test Druid cluster. <!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist above are strictly necessary, but it would be very helpful if you at least self-review the PR. --> <hr> ##### Key changed/added classes in this PR * `LiveMetricsReporter` * `ParallelIndexPhaseRunner` * `TaskMonitor` * `IngestionMetrics` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
