[ https://issues.apache.org/jira/browse/FLINK-7347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aljoscha Krettek updated FLINK-7347: ------------------------------------ Fix Version/s: 1.4.0 > "removeAll" is extremely inefficient in > MessageAcknowledgingSourceBase.notifyCheckpointComplete > ----------------------------------------------------------------------------------------------- > > Key: FLINK-7347 > URL: https://issues.apache.org/jira/browse/FLINK-7347 > Project: Flink > Issue Type: Improvement > Components: DataStream API > Affects Versions: 1.3.1 > Reporter: Yonatan Most > Fix For: 1.4.0 > > > Observe this line in > {{MessageAcknowledgingSourceBase.notifyCheckpointComplete}}: > {code} > idsProcessedButNotAcknowledged.removeAll(checkpoint.f1); > {code} > The implementation of {{removeAll}} is such that if the set is smaller than > the collection to remove, then the set is iterated and every item is checked > for containment in the collection. The type of {{checkpoint.f1}} here is > {{ArrayList}}, so the {{contains}} action is very inefficient, and it is > performed for every item in {{idsProcessedButNotAcknowledged}}. > In our pipeline we had about 10 million events processed, and the checkpoint > was stuck on the {{removeAll}} call for hours. > A simple solution is to make {{idsForCurrentCheckpoint}} a {{HashSet}} > instead of an {{ArrayList}}. The fact that it's a list is not really used > anywhere. -- This message was sent by Atlassian JIRA (v6.4.14#64029)