Questions on architecture / plans

[Miguel Branco]

Hi,

I'm trying to understand some links in the architecture, and thinking 
whether some of the work we've done can be contributed in any way (it's 
on fast "scan" operators: you're actually referencing to it under "NoDB" 
in your wiki on research/academic papers). But first we'd need to 
understand how the various Drill pieces fit together.

I understand this is an open project, and given stable APIs, we can have 
many implementations of any given part. But let's assume I'd like to 
implement a storage engine plus some scan/filter operator. There's the 
reference interpreter, which does the runtime execution and has its own 
reference operators. The goal there is not performance but just to 
demonstrate/prove functionality. But how do you see that moving forward? 
I've read about references early on about code generation & other 
runtime executors; is there any one already working on those already, or 
is it still far too early? Would that be done by effectively 
"replicating" all the functionality currently being built in the 
reference interpreter, or are there plans to take parts of that out & 
reuse? i.e. are there plans for some common plan interpreter stuff?

Regarding the linking with Optiq: I understand that's work in progress, 
but it's not quite clear to me how certain capabilities are advertised 
between Drill & Optiq: for instance, a storage engine plus scan 
operators have to be known to both systems. Will there be a need to 
advertise those separately to both systems, or will Optiq be embedded in 
Drill in some seamless way? Similarly, it's not quite clear to me how 
one implements a runtime for Drill, considering there's Optiq doing the 
query optimization, and both systems have to know what operators exist 
to optimize them/execute them. Or, to put it differently, how tightly 
coupled will Optiq and Drill be? [And, is there any other plans for 
query optimizations besides Optiq? Not that there's any issue with Optiq 
- I'm learning about it and am very very impressed with it - but if 
there alternative optimizers in the pipeline, then these issues have 
probably been addressed or will be so, soon.]

Any hints / description on your current thoughts would be most welcome. 
In fact, considering the project is now a few months old, it would be 
great to see where it's going *now*. It would allow us to understand 
where our work might fit in, to understand to which components (e.g. 
drill, optiq) it has to link to, and where we can help!

many thanks,
Miguel