Im-Manshushu commented on issue #11258:
URL: https://github.com/apache/doris/issues/11258#issuecomment-1198094741
What you said is quite right. My idea is like this. The previous answer may
not be described clearly
------------------ 原始邮件 ------------------
发件人:
"apache/doris"
***@***.***>;
发送时间: 2022年7月28日(星期四) 晚上8:20
***@***.***>;
抄送: "I'm ~ ***@***.******@***.***>;
主题: Re: [apache/doris] [Feature] JSON data is dynamically written to
the Doris table (Issue #11258)
Yes, encapsulate such a data structure, dynamically load by spelling the URL
on the sink side, and add a keyby operator before sink
At present, flink-doris-connector initiates the stream load of the table
when the flink task starts, instead of doing the stream load when the upstream
data is received. How to do this dynamic stream load? Please describe your
design in detail～
Many users put all the canal logs of all tables in the business library into
one topic, which needs to be distributed before they can use
doris-flink-connector. His idea is to edit a task to synchronize the entire
library. Because currently doris-flink-connector uses http inputstream, that
is, a checkpoint opens a stream, and a streamLoad url is strongly bound. In
this case, we can only use the flink side to cache data, and then a table
generates a buffer, and bind the corresponding table-streamload-url, set a
threshold, such as rows number or batch size to submit tasks, just like
doris-datax-writer.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
