Re: Pull #9224 - Druid Coordinator Pause Feature

2020-01-21 Thread Lucas Capistrant
Maytas, Appreciate the notes. Unfortunately we've considered both those options and they will not work for us. The first approach is a bit too casual for our liking. We need a guarantee that coordination won't run regardless of hour long our HDFS outage is. Plus we would have to do multiple rolli

Re: Pull #9224 - Druid Coordinator Pause Feature

2020-01-20 Thread Maytas Monsereenusorn
I'm still pretty new to Druid and might be wrong but I notice the following points in the documentation for the Coordinator ( https://druid.apache.org/docs/latest/design/coordinator.html): *"The Druid Coordinator runs periodically and the time between each run is a configurable parameter. Each tim

Re: Pull #9224 - Druid Coordinator Pause Feature

2020-01-20 Thread Will Lauer
I have no idea about the implementation, but the concept is certainly one we have been looking for for quite a while in the several clusters I manage. I'm excited to see this capability added to the system. Will On Mon, Jan 20, 2020, 1:55 PM Lucas Capistrant wrote: > Hi all, > > Looking for som

Pull #9224 - Druid Coordinator Pause Feature

2020-01-20 Thread Lucas Capistrant
Hi all, Looking for some feedback on the idea of creating a new dynamic config for the coordinator that allows cluster admins to pause coordination by setting the new config to true (default is false). By pause coordination, I mean to skip running any coordinator helpers every time the coordinator