[
https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308245#comment-15308245
]
Frances Perry commented on BEAM-315:
------------------------------------
The Beam Model (and hence the DataflowRunner) specifies that GoupByKey compares
for key equality using encoded elements.
https://github.com/apache/incubator-beam/blob/638cc20719cb314d9e6b29fa79d45edcf4586dce/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupByKey.java#L50
That's also why GroupByKey.validate() ensures the deterministic coder
requirements -- if the user does a GBK, they must provide a key coder that is
known to produce the same encoding for equal keys.
https://github.com/apache/incubator-beam/blob/638cc20719cb314d9e6b29fa79d45edcf4586dce/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupByKey.java#L188
> GroupByKey/CoGroupByKey doesn't group correctly in batch mode of
> FlinkPipelineRunner
> ------------------------------------------------------------------------------------
>
> Key: BEAM-315
> URL: https://issues.apache.org/jira/browse/BEAM-315
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Affects Versions: 0.1.0-incubating
> Reporter: Pawel Szczur
> Attachments: CoGroupPipelineStringKey.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
