Github user rtudoran commented on the issue:
https://github.com/apache/flink/pull/3590
@fhueske @sunjincheng121 @hongyuhong @stefanobortoli
I have run a test to compare the 3 approaches:
-windows based #3550
-processfunction based with events managed in ValueState[Queue] - this PR
-processfunction based with events managed in MapState[Long,JList] #3607
The simple benchmark that I run generates events 1 ms apart (a 5 tuple like
the one we used in the tests). There are 2 scenarios that I run a simple
counting over the window contents
Scenario 1)
2 second window (~2000 events in a window) - 100K events in total generated
Window based solution: 113839 ms
Process based (with Queue): 111792 ms
Process based on MapState: 110533 ms
10 second window (~10000 events in a window) - 200K events in total
generated
Window based solution: 218399ms
Process based (with Queue): 217343ms
Process based on MapState: 217657ms
I would say that the approaches are similar in performance (with some small
advantage for ProcessingFunctions). Regarding the 2 approaches for handing data
in process windows, I would say that the price to pay for
serializing/deserializing the whole list of events is matched by
(serializing/deserializing the timestamp keys + independently deserializing the
events that need to be removed). Considering that the performance are similar
personally I believe that the approach with Queue is preferred because we can
actually gain something (i.e., the order of the events) which will be helpful
in extending the implementation for full SQL
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
