Hey Samarth, I wrote a comment in the issue - thanks for looking at this, it is a valuable issue IMO.
On Mon, Feb 18, 2019 at 4:26 PM Samarth Jain <samarth.j...@gmail.com> wrote: > I have created an issue > https://github.com/apache/incubator-druid/issues/7068 that outlines the > limitations in current approach which prevents us from parallelizing the > segment load/drop workload. Also have raised a PR > https://github.com/apache/incubator-druid/pull/7088 to help address it. > > On Wed, Jan 30, 2019 at 4:40 PM Gian Merlino <g...@apache.org> wrote: > > > I believe today, if you use the (experimental) HTTP-based load queues, > they > > will parallelize segment downloads. Adding similar functionality for the > > ZK-based load queues would definitely be useful though, since at this > time > > nobody seems to be actively driving a migration to HTTP-based load queues > > being enabled by default. > > > > On Wed, Jan 30, 2019 at 7:20 PM Samarth Jain <sama...@apache.org> wrote: > > > > > We noticed that it takes a long time for the historicals to download > > > segments from deep storage (in our case S3). Looking closer at the code > > in > > > ZKCoordinator, I noticed that the segment download is happening in a > > single > > > threaded fashion. This download happens in the SingleThreadedExecutor > > > service used by the PathChildrenCache. Looking at the commentary on > > > https://github.com/apache/incubator-druid/issues/4421 and > > > https://github.com/apache/incubator-druid/issues/3202, the executor > > > service > > > used in PathChildrenCache can only be single threaded. > > > > > > My proposal is to use a multi threaded ExecutorService that will be > used > > to > > > take action on the events to perform the download. The role of single > > > threaded ExecutorService in PathChildrenCache will be simply to > delegate > > > the download task to this new executor service. > > > > > > Does that sound feasible? IMO, if this happens to be functionally > > correct, > > > it should help significantly boost up the time it is taking historicals > > to > > > download all the assigned segments. > > > > > > I would be more than happy to contribute this enhancement to the > > community. > > > > > > Thanks, > > > Samarth > > > > > >
