I have created an issue https://github.com/apache/incubator-druid/issues/7068 that outlines the limitations in current approach which prevents us from parallelizing the segment load/drop workload. Also have raised a PR https://github.com/apache/incubator-druid/pull/7088 to help address it.
On Wed, Jan 30, 2019 at 4:40 PM Gian Merlino <g...@apache.org> wrote: > I believe today, if you use the (experimental) HTTP-based load queues, they > will parallelize segment downloads. Adding similar functionality for the > ZK-based load queues would definitely be useful though, since at this time > nobody seems to be actively driving a migration to HTTP-based load queues > being enabled by default. > > On Wed, Jan 30, 2019 at 7:20 PM Samarth Jain <sama...@apache.org> wrote: > > > We noticed that it takes a long time for the historicals to download > > segments from deep storage (in our case S3). Looking closer at the code > in > > ZKCoordinator, I noticed that the segment download is happening in a > single > > threaded fashion. This download happens in the SingleThreadedExecutor > > service used by the PathChildrenCache. Looking at the commentary on > > https://github.com/apache/incubator-druid/issues/4421 and > > https://github.com/apache/incubator-druid/issues/3202, the executor > > service > > used in PathChildrenCache can only be single threaded. > > > > My proposal is to use a multi threaded ExecutorService that will be used > to > > take action on the events to perform the download. The role of single > > threaded ExecutorService in PathChildrenCache will be simply to delegate > > the download task to this new executor service. > > > > Does that sound feasible? IMO, if this happens to be functionally > correct, > > it should help significantly boost up the time it is taking historicals > to > > download all the assigned segments. > > > > I would be more than happy to contribute this enhancement to the > community. > > > > Thanks, > > Samarth > > >
