Besides the different optimization rule sets and cost model (Calcite is very extensive in that sense) used by Hive and Drill, I see the following difference of the way how Calcite is used in these two systems.
1. Drill supports query against schema-less storage (text, JSON etc). For Hive or other traditional database systems, each column's data type is known to query planner during planning time. For Drill, that's not the case; the column type could be known only during execution time, when the data is scanned from storage plugin. That's why Drill uses SQL ANY type (= ALL type in Ted's response) in Calcite for such use case. At execution, each column's type would be determined dynamically, and run-time code would be generated based on type resolution. This concept is sometime called as late binding. 2. Select * query support : in Hive or DB system, query planner would expand * into a list of regular columns. For Drill, since the table metadata is not available, Calcite could not expand *, which would be deferred to Drill's execution time. We have to extend Calcite such that the query planner would keep the semantics of *, and make the query semantics checking pass through Calcite for select * query. For instance, Q1: Select * from T1; Q2: Select T1.*, T2.COL_A from T1, T2 where T1.COL1 = T2.COL2; Q3: Select * from ( SELECT * FROM T1) ORDER BY COL1; In all the cases, Drill has to make sure Calcite would do name resolution for * column correctly in planning time, and then expand the * column properly in execution time. On Wed, May 27, 2015 at 8:32 PM, Andrew Brust < andrew.br...@bluebadgeinsights.com> wrote: > That makes sense. Just having trouble mapping that back on Ted's > comment. But I tend to think that's me and my ignorance. > > -----Original Message----- > From: Hanifi Gunes [mailto:hgu...@maprtech.com] > Sent: Wednesday, May 27, 2015 4:48 PM > To: user > Subject: Re: what's the differenct between drill and optiq > > Calcite does parsing & planning of queries. Drill executes in a very > flexible distributed columnar fashion with late binding. > > On Wed, May 27, 2015 at 8:34 AM, Ted Dunning <ted.dunn...@gmail.com> > wrote: > > > Andrew, > > > > What Hive does not have is the extensions that Drill has that allow > > SQL to be type flexible. The ALL type and all of the implications > > both in terms of implementation and user impact it has are a really big > deal. > > > > > > > > On Wed, May 27, 2015 at 6:08 AM, Andrew Brust < > > andrew.br...@bluebadgeinsights.com> wrote: > > > > > Thanks! > > > > > > Sent from my phone > > > <insert witty apology for typos here> > > > > > > ----- Reply message ----- > > > From: "PHANI KUMAR YADAVILLI" <phanikumaryadavi...@gmail.com> > > > To: "user@drill.apache.org" <user@drill.apache.org> > > > Subject: what's the differenct between drill and optiq > > > Date: Wed, May 27, 2015 8:33 AM > > > > > > Yes hive uses calcite. You can refer hive documentation. > > > On May 27, 2015 6:01 PM, "Andrew Brust" < > > > andrew.br...@bluebadgeinsights.com> > > > wrote: > > > > > > > Folks at Hortonworks told me that Hive now uses Calcite as well. > > > > Can anyone here confirm or deny that? > > > > > > > > -----Original Message----- > > > > From: Rajkumar Singh [mailto:rsi...@maprtech.com] > > > > Sent: Wednesday, May 27, 2015 6:52 AM > > > > To: user@drill.apache.org > > > > Subject: Re: what's the differenct between drill and optiq > > > > > > > > Optiq(now known as calcite) is an api for query parser,planner and > > > > optimization, drill uses it for the SQL parsing,validation and > > > > optimization.Drill query planner applies its own custom planner > > > > rules > > to > > > > build the query logical plan. > > > > > > > > Rajkumar Singh > > > > > > > > > > > > > > > > > On May 27, 2015, at 12:04 PM, 陈礼剑 <chenlij...@togeek.cn> wrote: > > > > > > > > > > Hi: > > > > > > > > > > I just want to know the difference between drill and optiq. > > > > > > > > > > > > > > > Is drill just 'extend' optiq to support many other > > > > > 'stores'(hadoop, > > > > mongodb, ...)? > > > > > > > > > > > > > > > ---from davy > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
