[
https://issues.apache.org/jira/browse/FLINK-14807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054621#comment-17054621
]
Caizhi Weng commented on FLINK-14807:
-------------------------------------
Hi [~sewen]
What I'm worrying about is not the stress on the JM / TM memory but the stress
on JM disks if no external storage system is configured for the blob server. A
never-ending streaming job or a batch job with very large result set will
eventually make the blob server disks full. If we use back-pressuring and limit
the size of results not consumed by the client we can solve this problem.
Another thing is that currently clients / sinks can neither get the address and
port of the blob server nor can they use the {{BlobClient}}. Blob server
currently seems to be available only for JM.
And for users who would like to see the results as soon as possible, as
accumulators are sent with heartbeats, they'll have to wait for the next
heartbeat to get the latest results. Sending the same results in the
accumulators over and over again also doesn't seem right. With the design I
mentioned above they are free to choose between consistency and visibility.
Even though {{GlobalAggregateManager}} seems to be a bad hack, we can refactor
it once {{OperatorCoordinators}} is implemented.
But all these questions are actually pointing towards a core question: What's
the use cases of {{Table#collect}}? That is to ask, what do we want to provide
with the users? Do we need to support never-ending streaming jobs and batch
jobs with large results? Do we need to provide the users with the freedom to
choose between consistency and visibility? What do you think?
> Add Table#collect api for fetching data to client
> -------------------------------------------------
>
> Key: FLINK-14807
> URL: https://issues.apache.org/jira/browse/FLINK-14807
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
> Affects Versions: 1.9.1
> Reporter: Jeff Zhang
> Priority: Major
> Labels: usability
> Fix For: 1.11.0
>
> Attachments: table-collect-draft.patch, table-collect.png
>
>
> Currently, it is very unconvinient for user to fetch data of flink job unless
> specify sink expclitly and then fetch data from this sink via its api (e.g.
> write to hdfs sink, then read data from hdfs). However, most of time user
> just want to get the data and do whatever processing he want. So it is very
> necessary for flink to provide api Table#collect for this purpose.
>
> Other apis such as Table#head, Table#print is also helpful.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
