rbankar7 opened a new pull request, #19942:
URL: https://github.com/apache/druid/pull/19942
### Description
When Druid is configured with `druid.emitter=composing`, the composing
emitter was re-resolving all of its child emitters from the Guice injector on
**every** emitted event. Because Druid emits a metric per segment scanned, on
busy nodes this executes tens of thousands of times per second, and each
resolution takes a monitor lock. Lock profiling on affected nodes attributed
the large majority of lock-contention samples to `LifecycleScope$1.get`.
#### Root cause
`ComposingEmitterModule#getEmitter` built its child-emitter list with
`Lists.transform(...)`, which returns a **lazy** Guava view rather than a
materialized list:
```java
List<Emitter> emitters = Lists.transform(
config.getEmitters(),
s -> injector.getInstance(Key.get(Emitter.class, Names.named(s)))
);
return new ComposingEmitter(emitters);
```
`ComposingEmitter.emit(Event)` iterates this list once per event:
```java
public void emit(Event event) {
for (Emitter e : emitters) {
e.emit(event);
}
}
```
Since `emitters` is a lazy view, each iteration re-ran the transform
function, which re-resolved every child emitter from the injector. For
lifecycle-scoped bindings, that resolution goes through `LifecycleScope#get()`,
which is `synchronized` and so takes a monitor lock on every call — even
though, after startup, it does nothing but return an already-cached instance.
The child emitters were therefore being looked up (and locked) on the hottest
path in the process rather than resolved once at startup.
#### Fixed the lock contention in the composing emitter
The child emitters are now resolved **eagerly, exactly once**, into an
`ImmutableList` at construction time, so `ComposingEmitter.emit()` iterates a
plain materialized list with no injector lookups and no locking:
```java
List<Emitter> emitters = config.getEmitters()
.stream()
.map(s ->
injector.getInstance(Key.get(Emitter.class, Names.named(s))))
.collect(ImmutableList.toImmutableList());
return new ComposingEmitter(emitters);
```
This is purely an internal change — the same set of child emitters is
composed in the same order. The only difference is *when* they are resolved:
once during provisioning instead of on every emitted event. This also aligns
`ComposingEmitterModule` with its sibling `SwitchingEmitterModule`, which
already resolves its child emitters eagerly into materialized lists.
#### Release note
Fixed lock contention on the metrics emit path when using the composing
emitter (`druid.emitter=composing`). Child emitters are now resolved once at
startup instead of on every emitted event, which previously caused significant
monitor contention on nodes emitting metrics at high volume.
<hr>
##### Key changed/added classes in this PR
* `ComposingEmitterModule`
<hr>
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency
checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md)
(Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in
[licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
