Hi Himanshu,
We did not experiment with the union operation, but these are toy-example
throughput numbers we got for a *single* thread writing to a theta sketch:
basic (sketches-core) 167M,
lock-based 32M,
our concurrent 115M (223M with 2 threads, and scaling linearly).The
experiments were done in a single sketch environment, and the results might be
different with multiple sketches, however it captures the potential gain of
removing the lock.
Thanks,Eshcar
On Wednesday, July 11, 2018, 2:26:53 AM GMT+3, Himanshu
<[email protected]> wrote:
Yes, SynchronizedUnion was added on seeing concurrency error on the
processes doing realtime ingestion. On other processes, it is not called
concurrently and hope is that synchronization does not cause overhead when
that piece of code is only called in one thread because it gets optimized.
On Tue, Jul 10, 2018 at 10:13 AM, Gian Merlino <[email protected]> wrote:
> Hi Eshcar,
>
> To my knowledge, in the Druid Aggregator and BufferAggregator interfaces,
> the main place where concurrency happens is that "aggregate" and "get" may
> be called simultaneously during realtime ingestion. So if there would be a
> benefit from improving concurrency it would probably end up in that area.
>
> On Tue, Jul 10, 2018 at 2:10 AM Eshcar Hillel <[email protected]>
> wrote:
>
> > Hi All,
> > My name is Eshcar Hillel from Oath research. I'm currently working with
> > Lee Rhodes on committing a new concurrent implementation of the theta
> > sketch to the sketches-core library.I was wondering whether this
> > implementation can help boost the union operation that is applied to
> > multiple sketches at query time in druid.From what I see in the code the
> > sketch aggregator uses the SynchronizedUnion implementation, which
> > basically uses a lock at every single access (update/read) of the union
> > operation. We believe a thread-safe implementation of the union operation
> > can help decrease the inherent overhead of the lock.
> > I will be happy to join the meeting today and briefly discuss this
> option.
> > Thanks,Eshcar
> >
> >
> >
>
