Thank you for the detailed explanation.
On Tue, 29 Aug 2023 at 22:45, Ken Krugler
wrote:
> If you need a specific output order, then merge the three streams, key by
> a constant (like 1), and run that into a KeyedProcessFunction.
>
> That function can buffer out-of-order records, and set up a ti
If you need a specific output order, then merge the three streams, key by a
constant (like 1), and run that into a KeyedProcessFunction.
That function can buffer out-of-order records, and set up a timer to fire when
it gets a MAX_WATERMARK (which indicates that all streams are finished) so that
Thank you so much for taking the time to provide me with such a detailed
response. Your assistance has been incredibly helpful in clarifying my
understanding!
Let me provide you with the exact scenario , I think there might be some
misunderstanding. All the streams are bounded and parallelism is s
ps: Forget the link: Hybrid Source[1]
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/hybridsource/
Hang Ruan 于2023年8月11日周五 10:14写道：
> Hi, Muazim.
>
> I think the Hybird Source[1] may be helpful for your case.
>
> Best,
> Hang
>
> Ken Krugler 于2023年8月1
Hi, Muazim.
I think the Hybird Source[1] may be helpful for your case.
Best,
Hang
Ken Krugler 于2023年8月11日周五 04:18写道：
> As (almost) always, the devil is in the details.
>
> You haven’t said, but I’m assuming you’re writing out multiple files, each
> with a different schema, as otherwise you cou
As (almost) always, the devil is in the details.
You haven’t said, but I’m assuming you’re writing out multiple files, each with
a different schema, as otherwise you could just leverage the existing Flink
support for CSV.
So then you could combine the header/footer streams (adding a flag for he
Thanks for the response!
I have a specific use case where I am writing to a TextFile sink. I have a
Bounded stream of header data and need to merge it with another bounded
stream. While writing the data to a text file the header data should be
written before the original data(from another bounded
Hi Muazim,
In Flink, a stream of data (unless bounded) is assumed to never end.
So in your example below, this means stream 2 would NEVER be emitted, because
stream 1 would never end (there is no time at which you know for sure that
stream 1 is done).
And this in turn means stream 2 would be b
