Hi all, just wanted to start a thread to discuss removing the 'auto' strategy from the native search query, which is the only thing that uses the 'estimateSelectivity' method of https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/index/BitmapColumnIndex.java#L33, which is what I would actually like to remove to make implementing index suppliers a bit easier and also tidy some stuff up for some additional changes I would like to make.
The 'auto' strategy uses these selectivity estimation methods to try to determine if it should actually use the indexes or not when performing a search query, however I've been making some improvements on making index usage itself automatically determine if it should use indexes or value matchers, which I think is the replacement. The first part is the changes in https://github.com/apache/druid/pull/13977, which currently only applies to 'auto' and 'json' columns, but the strategy could easily be applied to traditional string columns as well, and the future work I'd like to do would allow cursor creation to adaptively skip computing the remaining bitmaps of an AND filter once the intersection is selective enough to make further operations not worth the cost. Once these changes are in place, the 'useIndexes' strategy effectively becomes the 'auto' strategy because all index evaluation is automatic, and also much cheaper than the 'auto' strategy which currently has to repeat some work if it actually does decide to use indexes. I suppose the part worth discussing is if this 'auto' strategy is used enough for search queries that we need the complete replacement in place before we can remove it, or if I can just go ahead and remove it now, or if I should wire up the partial improvements used today by 'auto' and 'json' columns to regular string columns so at least a partial improvement is available. I think it would make it a bit easier for me to finish the remaining refactor for the improvements i have in mind if i can get this method out of my way entirely, but I can also probably work around it if necessary (e.g. if anyone is depending on it), so removing it probably isn't a blocker for finishing this work. Thoughts? --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
