There is a race between aggregators and ingestion updates. Actually, many aggregators are vulnerable now. See this issue: https://github.com/apache/incubator-druid/pull/3956 and a conversation starting from this message: https://github.com/apache/incubator-druid/pull/5148#discussion_r170906998.
However, you could replace a simple synchronized with ReadWriteLock or Striped<ReadWriteLock> (see ArrayOfDoublesSketchMergeBufferAggregator for example), that would be a useful contribution to Druid. On Thu, 19 Jul 2018 at 13:21, Will Lauer <wla...@oath.com.invalid> wrote: > A colleague recently pointed out to me that all the sketch operations that > take place in SketchAggregator (in the datasketches module) use a > SychronizedUnion class that basically wraps a normal sketch Union and > synchronizes all operations. From what I can tell with other aggregators in > the Druid code base, there doesn't appear to be a need to synchronize. It > looks like Aggregators are always processed from within a single thread. Is > it reasonable to remove all the syncrhonizations from the SketchAggregator > and avoid the performance hit that they impose at runtime? > > Will > > Will Lauer > Senior Principal Architect > > Progress requires pain > > m: 508.561.6427 > > o: 217.255.4262 >
