Hi Thalita,
We should clarify what exactly you want to achieve. Could you just
describe your use case in prose? We can then write the Beam/Flink pipeline.
-Max
On 10.09.18 18:06, Vergilio, Thalita wrote:
Hi Max,
Thank you very much for that. Is there a way that I could group by window in
Flink? It did occur to me that I should be doing that but, as far as I
understand, Flink requires a keyed stream before a windowing function can be
applied. I couldn't think of a way around it.
Many thanks,
Thalita
-----Original Message-----
From: Maximilian Michels <[email protected]>
Sent: 10 September 2018 16:48
To: [email protected]; Vergilio, Thalita <[email protected]>
Cc: Mullier, Duncan <[email protected]>; Ramachandran, Muthu
<[email protected]>
Subject: Re: Comparison between Beam (with Flink runner) and using the Flink
SDK directly
Hi Thalita,
Your code examples for Beam/Flink are not equivalent.
For the Beam version, you load data from Kafka, then you window, produce a
KV<Window, KafkaRecord>, group by the window, and finally process the groupings.
For Flink, you load from Kafka, parse the Kafka Json, then window and keyby the
id from Kafka, followed by the processing.
The jobs are semantically very different because in Beam you group by an
interval window, but in Flink you group by a key extracted from Kafka.
Best,
Max
On 07.09.18 10:41, Vergilio, Thalita wrote:
Hi,
My name is Thalita and I am a PhD student researching big data
processing using a multi-cloud architecture. I am currently evaluating
the advantages of using an intermediate framework such as Beam in
order to achieve framework agnosticism for processing code (write the
code once, use any framework to run it).
While running some experiments to assess the "cost" of using an
additional framework such as Beam, as opposed to using the runner's
SDK directly, I have come across some bizarre findings. I produced
equivalent processing pipelines using Beam and the Flink SDK. I then
ran them on my Flink cluster (same infrastructure, same settings), and
found that the Beam pipeline performed significantly better than the
Flink pipeline.
There are more details of this experiment here:
https://stackoverflow.com/questions/52182558/how-flink-and-beam-sdks-h
andle-windowing-which-is-more-efficient
I should be grateful for any insight from you guys on:
1.Whether I am comparing like for like (are the code samples equivalent?).
2.Any possible explanations for what I am observing.
3.Your own experiences using Beam vs. using the runner's SDK directly.
Are they different from mine?
Thank you very much for your help. Any advice you can give me is much
much appreciated J
Best wishes,
Thalita
To view the terms under which this email is distributed, please go
to:- http://disclaimer.leedsbeckett.ac.uk/disclaimer/disclaimer.html
To view the terms under which this email is distributed, please go to:-
http://disclaimer.leedsbeckett.ac.uk/disclaimer/disclaimer.html
