Hi Carolina, The discussion, as I recall, was on the public Drill Slack channel which you are welcome to join. Would also be great if the participants of that discussion could record the info in a Jira ticket.
As I recall, the folks found that there are complications with Drill's Calcite-based query planner when computing costs after the filter pushdown. There was detailed discussion of specific code changes, but I don't know if that fixed the problem or if the discussion petered out. Thanks, - Paul On Wed, Sep 23, 2020 at 11:57 AM Carolina Gomes <ca...@afterdata.ai> wrote: > Hi Paul, > > > That would be great even if you can just copy the discussion here. Being > able to do that would greatly optimize the performance of our product. > > On Mon, Sep 21, 2020 at 5:57 PM Paul Rogers <par0...@gmail.com> wrote: > > > Hi Carolina, > > > > This issue came up recently in one of the Drill Slack channels. I wonder, > > can anyone here summarize the findings from that Slack discussion? > > > > Thanks, > > > > - Paul > > > > > > On Mon, Sep 21, 2020 at 7:35 AM Carolina Gomes <ca...@afterdata.ai> > wrote: > > > > > Also if it helps, I’m using Drill 1.16 in single-node mode. > > > > > > On Mon, Sep 21, 2020 at 10:32 AM Carolina Gomes <ca...@afterdata.ai> > > > wrote: > > > > > > > Hi all, > > > > > > > > I have a question about push down of limit and offset clauses on > Drill. > > > > For my use case, I’d always like for limit and offset clauses to be > > > pushed > > > > down to the data sources, which are always RDBMS databases like SQL > > > Server, > > > > Oracle etc. > > > > > > > > > > > > However, I have noticed the decision to push down seems to happen > > > > depending on the size of the limit clause, and on the number of > columns > > > > being projected. > > > > > > > > > > > > As an example, I have a table of about 250 columns with about 50 > > million > > > > rows. If I do: > > > > > > > > > > > > select * from table limit 1000 —-> limit push down does not happen, > > query > > > > takes 30s while if I change the physical plan to push down the limit > > > > clause, it takes less than 1s. > > > > > > > > select * from table limit 1000000 —-> limit push down does happen, > > query > > > > takes roughly same time as if I queried directly on the source DB. > > > > > > > > Is there a way of easily telling Drill to always pushdown? > > > > -- > > > > [Carolina Gomes] > > > > CEO, AfterData.ai <https://www.afterdata.ai/> > > > > +1 (416) 931 4774 > > > > > > > > > > > > > > > > -- > > > [Carolina Gomes] > > > CEO, AfterData.ai <https://www.afterdata.ai/> > > > +1 (416) 931 4774 > > > > > > -- > [Carolina Gomes] > CEO, AfterData.ai <https://www.afterdata.ai/> > +1 (416) 931 4774 >
