[GitHub] surekhasaharan commented on issue #6901: Introduce published segment cache in broker
surekhasaharan commented on issue #6901: Introduce published segment cache in broker URL: https://github.com/apache/incubator-druid/pull/6901#issuecomment-459567332 > I wonder if the feature is turned off by default? There is a possibility it could behave unpredictably on a huge cluster. It was not, but it is now, added a config to `PlannerConfig` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on issue #6901: Introduce published segment cache in broker
surekhasaharan commented on issue #6901: Introduce published segment cache in broker URL: https://github.com/apache/incubator-druid/pull/6901#issuecomment-459219973 > @surekhasaharan thanks for checking the memory usage. Would you please add how many segments were in the cluster? At that time, there would be roughly around 67K segments. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on issue #6901: Introduce published segment cache in broker
surekhasaharan commented on issue #6901: Introduce published segment cache in broker URL: https://github.com/apache/incubator-druid/pull/6901#issuecomment-459146133 > Can you also please record the impact on heap size in the broker? We already have pretty massive broker heaps and putting more pressure on them would be problematic. This is used memory comparison on our test broker over a day, grey line is without this patch and blue line is with this patch. There is slightly higher memory usage with current published segments cache on broker. https://user-images.githubusercontent.com/13008746/52018434-3e971700-249f-11e9-9924-6da8200d0982.png";> This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on issue #6901: Introduce published segment cache in broker
surekhasaharan commented on issue #6901: Introduce published segment cache in broker URL: https://github.com/apache/incubator-druid/pull/6901#issuecomment-458245549 > > Heads up, I very briefly experimented with interning data segments back in #3286 but had to back it out because real-time nodes publish with a size of 0, which makes the segment size calculations all break. > > Thanks @drcrallen This issue can indeed happen, I looked at the trail of issues surrounding #3286, I am thinking of addressing #3287 before this PR to avoid incorrect segment sizes. The size issue is resolved with my latest commit. Changed the `DataSegmentInterner` to not rely on the `DataSegment#equals`, it instead uses a combination of `DataSegment#trueEquals` and `size` attribute to find the better segment to intern. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on issue #6901: Introduce published segment cache in broker
surekhasaharan commented on issue #6901: Introduce published segment cache in broker URL: https://github.com/apache/incubator-druid/pull/6901#issuecomment-457027997 > Heads up, I very briefly experimented with interning data segments back in #3286 but had to back it out because real-time nodes publish with a size of 0, which makes the segment size calculations all break. > Thanks @drcrallen This issue can indeed happen, I looked at the trail of issues surrounding #3286, I am thinking of addressing #3287 before this PR to avoid incorrect segment sizes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
