Enumerable is flawed because it uses the Volcano execution model, which
requires blocking threads.
Interpreter might be a better execution model. Take a look at
org.apache.calcite.interpreter.Node. The ‘run()’ method returns void but we
could make it return boolean, where ‘false’ means ‘I’m not yet done, call me in
a while’. Then you could have many more queries running than you have threads.
Julian
> On Mar 4, 2020, at 9:48 AM, Scott Reynolds wrote:
>
> List,
>
> Our team's current application runs a thread pool for handling queries.
> Each time a query comes it it is placed in the run queue and executed
> outside of the HTTP RPC thread. This allows us to handle N concurrent
> queries where N is the size of the thread pool. I would like to change this
> to a N:M system allowing our service to handle more concurrent queries then
> the thread pool size. Our TableScan uses Netty RPCs and Queues to fetch
> results from the remote database giving use the opportunity to park the
> current query while awaiting RPC responses.
>
> Has anyone been able to achieve this? Is the best way to own Convention?
>
> Our current service doesn't use JDBC and therefore have direct access to
> the compiled Enumerable. One solution I haven't been able to get working is
> using InterruptedException (or something similar -- Kafka created a
> WakeUpException), which would allow a sub-enumerable the ability to signal
> above that it needs to save it's state and yield.
>
> Is this something that could be added to EnumerableDefaults? If not would
> it be a InterruptableEnumerable convention instead?