[GitHub] [druid] liran-funaro commented on pull request #10001: Optimizing incremental-index ingestion using off-heap key/value map (OakMap)

liran-funaro commented on pull request #10001: URL: https://github.com/apache/druid/pull/10001#issuecomment-660959076 ### Updates We are working with our production teams at Verizon Media toward testing our incremental-index implementation on actual production data. As part of this effo

[GitHub] [druid] liran-funaro commented on pull request #10001: Optimizing incremental-index ingestion using off-heap key/value map (OakMap)

liran-funaro commented on pull request #10001: URL: https://github.com/apache/druid/pull/10001#issuecomment-687738552 We continue to evaluate Oak incremental-index on our (Verizon Media) production workload and seeing interesting results. For example, we run Druid on two identical VMs th

[GitHub] [druid] liran-funaro commented on pull request #10001: Optimizing incremental-index ingestion using off-heap key/value map (OakMap)

liran-funaro commented on pull request #10001: URL: https://github.com/apache/druid/pull/10001#issuecomment-641602050 Hi @jihoonson, @leventov, @gianm, @ebortnik, @b-slim, and @jon-wei. Since you had an interest in our previous discussions regarding `OakIncrementalIndex`, we wanted to up

[GitHub] [druid] liran-funaro commented on pull request #10001: Optimizing incremental-index ingestion using off-heap key/value map (OakMap)

liran-funaro commented on pull request #10001: URL: https://github.com/apache/druid/pull/10001#issuecomment-643264856 @jihoonson Thank you. We are eager to hear your valuable feedback. This is an automated message from the Ap

[GitHub] [druid] liran-funaro commented on pull request #10001: Optimizing incremental-index ingestion using off-heap key/value map (OakMap)

liran-funaro commented on pull request #10001: URL: https://github.com/apache/druid/pull/10001#issuecomment-643274113 @clintropolis We noticed your recent commits modified the same benchmarks as we did in this PR ([commit #a607c95](https://github.com/apache/druid/pull/10001/commits/a607c95

[GitHub] [druid] liran-funaro commented on pull request #10001: Optimizing incremental-index ingestion using off-heap key/value map (OakMap)

liran-funaro commented on pull request #10001: URL: https://github.com/apache/druid/pull/10001#issuecomment-644182922 We noticed a lot of Druid users run their workload on Amazon EC2. We want to point out that this PR will not only improve performance but will also reduce operational costs

[GitHub] [druid] liran-funaro commented on pull request #10001: Optimizing incremental-index ingestion using off-heap key/value map (OakMap)

liran-funaro commented on pull request #10001: URL: https://github.com/apache/druid/pull/10001#issuecomment-646589680 Thanks to @yuanlihan for helping us find bugs on the realtime query side. Our system experiments focused on batch ingestion, so his contribution is highly appreciated. --

[GitHub] [druid] liran-funaro commented on pull request #10001: Optimizing incremental-index ingestion using off-heap key/value map (OakMap)

liran-funaro commented on pull request #10001: URL: https://github.com/apache/druid/pull/10001#issuecomment-649515567 Hi @jihoonson, have you had a chance to check out our issue/PR? We will be happy to answer any questions you might have. --