We don't have usage data, but my sense is that the search query is not commonly used, and among people that use the search query, it's not common to rely on "druid.query.search.searchStrategy: auto". So I think it would be ok to remove the feature and have "auto" be an alias for "useIndexes", especially if as part of that, the "useIndexes" approach gets smarter.
On Wed, Nov 15, 2023 at 8:52 PM Clint Wylie <cwy...@apache.org> wrote: > Hi all, just wanted to start a thread to discuss removing the 'auto' > strategy from the native search query, which is the only thing that > uses the 'estimateSelectivity' method of > > https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/index/BitmapColumnIndex.java#L33 > , > which is what I would actually like to remove to make implementing > index suppliers a bit easier and also tidy some stuff up for some > additional changes I would like to make. > > The 'auto' strategy uses these selectivity estimation methods to try > to determine if it should actually use the indexes or not when > performing a search query, however I've been making some improvements > on making index usage itself automatically determine if it should use > indexes or value matchers, which I think is the replacement. The first > part is the changes in https://github.com/apache/druid/pull/13977, > which currently only applies to 'auto' and 'json' columns, but the > strategy could easily be applied to traditional string columns as > well, and the future work I'd like to do would allow cursor creation > to adaptively skip computing the remaining bitmaps of an AND filter > once the intersection is selective enough to make further operations > not worth the cost. > > Once these changes are in place, the 'useIndexes' strategy effectively > becomes the 'auto' strategy because all index evaluation is automatic, > and also much cheaper than the 'auto' strategy which currently has to > repeat some work if it actually does decide to use indexes. > > I suppose the part worth discussing is if this 'auto' strategy is used > enough for search queries that we need the complete replacement in > place before we can remove it, or if I can just go ahead and remove it > now, or if I should wire up the partial improvements used today by > 'auto' and 'json' columns to regular string columns so at least a > partial improvement is available. I think it would make it a bit > easier for me to finish the remaining refactor for the improvements i > have in mind if i can get this method out of my way entirely, but I > can also probably work around it if necessary (e.g. if anyone is > depending on it), so removing it probably isn't a blocker for > finishing this work. > > Thoughts? > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > For additional commands, e-mail: dev-h...@druid.apache.org > >
