Co-relate two streams

[Abhinav Sharma]

Hi,

How can I co-relate two streams of different types in Flink?
Scenario: In stream1, I have data in pojo with a field user. In stream2, I
have data in a different pojo which also contains the field user. (However,
other than the user field, they have no common field).

Now what I want to do is relate the two streams such that for every event
in stream1, I want to collect events from stream2 where the user is the
same. Both stream1 and stream2 are unbounded.

I tried using
stream1.connect(stream2).process(new CoProcessFunction) {
private String user;

public void processElement1(Type1 inp, CoProcessFunction.Context ctx, Collector out)  {
user = inp.getUser();
}

public void processElement2(Type2 inp, CoProcessFunction.Context ctx, Collector out)  {
if (user.equals(inp.getUser())) {
out.collect(inp);
}
}
});

But this works only and only if both elements occur simultaneously.

How can I collect the cases with history? Is using ListState required?
Is there some better way to this in Flink?


Requesting help,
Abhinav

Re: Configure operator based on key

[Abhinav Sharma]

Hi Yidan,

Thank you for your reply. I was wondering if there is some way that the
process function can kiw which condition fired the trigger.

Eg: If I set trigger to fire when he object associated with key have value
2, 8, 10 (3 conditions for the trigger to fire), then if he process
function, I want to operate differently on them.

On Mon, Feb 22, 2021, 11:23 AM yidan zhao  wrote:

> You can self-define it using keyedStream.window(GlobalWindows.create()
> ).trigger(self-defined-trigger).
>
> Abhinav Sharma  于2021年2月21日周日 下午3:57写道：
>
>> Hi,
>>
>> Is there some way that I can configure an operator based on the key in a
>> stream?
>> Eg: If the key is 'abcd', then create a window of size X counts, if the
>> key is 'bfgh', then create a window of size Y counts.
>>
>> Is this scenario possible in flink
>>
>>

Configure operator based on key

[Abhinav Sharma]

Hi,

Is there some way that I can configure an operator based on the key in a
stream?
Eg: If the key is 'abcd', then create a window of size X counts, if the key
is 'bfgh', then create a window of size Y counts.

Is this scenario possible in flink

Configure classes

[Abhinav Sharma]

Hi
I am evaluating flink with use case where we need to create a basic flink
pipeline, and inject the classes for map, reduce, process, etc via some xml
configuration (or something equivalent).
Eg:

stream.keyBy(value -> value.getKey())
.window(TumblingProcessingWindow.of(Time.miuntes(1)))
.process(new MyInjectedClass());

Is something like this possible, where a developer can just write
MyInjectedClass, and configure it without writing code in flink app? The
developer needs to write just the process class, and specify which step in
pipeline to inject the class.

Event trigger query

[Abhinav Sharma]

Newbie question: How can I set triggers to stream which execute according
to system time? Eg: I want to sum the elements of streams at 1PM everyday.