ahmedabu98 opened a new pull request, #41315:
URL: https://github.com/apache/beam/pull/41315
Addresses #41313
A SELECT on a streaming (unbounded) source never shows any rows in the Beam
SQL shell. The shell collects the rows in memory and only hands them over once
the pipeline finishes. A streaming pipeline never finishes.
This PR hands the rows over while the pipeline runs. For an unbounded
SELECT, `ResultSet.next()` returns each row as soon as the pipeline produces
it. The shell already prints JDBC rows one at a time, so a query like this now
prints rows as messages arrive on the topic:
```sql
CREATE EXTERNAL TABLE orders (id BIGINT, item VARCHAR, amount DOUBLE)
TYPE kafka
LOCATION 'localhost:9092/orders'
TBLPROPERTIES '{"format": "json"}';
SELECT * FROM orders WHERE amount > 100;
```
Canceling the statement (Ctrl+C in the shell) or closing the result set
cancels the pipeline. If the pipeline fails, the query fails with the
pipeline's error.
Each row also keeps its change kind (`ValueKind`). A follow-up PR adds a
full-screen live view to the shell that shows it as `+I`, `-U`, `+U` or `-D`.
The view plugs in through `LiveResult.setViewer(...)`.
Bounded queries, `LIMIT` queries and INSERTs work as before. Like any SELECT
in the shell, this needs the DirectRunner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
