[
https://issues.apache.org/jira/browse/CALCITE-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15009453#comment-15009453
]
Julian Hyde commented on CALCITE-849:
-------------------------------------
I just had an idea about how to get co-operative scheduling working without a
big impact. How about building on Interpreter? The current contract is that the
{{Node.run()}} method is called once, and when it returns, it is finished; but
we could change the contract so that it would be called repeatedly until it
calls {{sink.end}}. A node will no longer be occupying a dedicated thread, so
we could have many more nodes than worker threads.
Your Publisher and Subscriber APIs look great for clients of the stream.
However I believe that we need a different API for the workers inside the
engine. I think Node, Source, Sink are what works there.
Regarding your dashboard example (I have called it the "pie chart" use case,
see e.g. http://calcite.apache.org/docs/stream.html). You don't build pie
charts on streams. You build pie charts on relations, albeit standing relations
(aka materialized views) continuously maintained in memory based on the stream.
When you remove that use case I think you are left with the "topic" and "queue"
paradigms, and "topic" is more common.
> Streams/Slow iterators dont close on statement close
> ----------------------------------------------------
>
> Key: CALCITE-849
> URL: https://issues.apache.org/jira/browse/CALCITE-849
> Project: Calcite
> Issue Type: Bug
> Reporter: Jesse Yates
> Assignee: Julian Hyde
> Fix For: 1.5.0
>
> Attachments: calcite-849-bug.patch
>
>
> This is easily seen when querying an infinite stream with a clause that
> cannot be matched
> {code}
> select stream PRODUCT from orders where PRODUCT LIKE 'noMatch';
> select stream * from orders where PRODUCT LIKE 'noMatch';
> {code}
> The issue arises when accessing the results in a multi-threaded context. Yes,
> its not a good idea (and things will break, like here). However, this case
> feels like it ought to be an exception.
> Suppose you are accessing a stream and have a query that doesn't match
> anything on the stream for a long time. Because of the way a ResultSet is
> built, the call to executeQuery() will hang until the first matching result
> is received. In that case, you might want to cancel the query because its
> taking so long. You also want the thing that's accessing the stream (the
> StreamTable implementation) to cancel the querying/collection - via a call to
> close on the passed iterator/enumerable.
> Since the first result was never generated, the ResultSet was never returned
> to the caller. You can get around this by using a second thread and keeping a
> handle to the creating statement. When you go to close that statement though,
> you end up not closing the cursor (and the underlying iterables/enumberables)
> because it never finished getting created.
> It gets even more problematic if you are use select * as the iterable doesn't
> finish getting created in the AvaticaResultSet.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
