liran-funaro commented on pull request #10001:
URL: https://github.com/apache/druid/pull/10001#issuecomment-687738552
We continue to evaluate Oak incremental-index on our (Verizon Media)
production workload and seeing interesting results.
For example, we run Druid on two identical VMs that consumed events from the
same Kafka feed for an entire day; one with Oak and one with the "vanilla"
Druid (on-heap incremental index).
### Oak vs. on-heap ("vanilla") incremental-index comparison:
* Oak required 35% less flush operations for the entire duration
* Which produced 50% larger partitions
* As a result, Oak spent 50 **minutes** less time in flush operations; i.e.,
almost 40% reduction compared to the on-heap implementation
* This also reduced the time it took to merge and push the segments by
almost a minute.
In the latest Druid Summit, we've seen that these kinds of optimizations are
vital.
Companies invest resources in post-compaction, thus, creating larger
partitions during ingestion can reduce these efforts cost.
We appreciate any effort by the community for moving this forward, starting
by reviewing our design proposal (#10321).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
