[jira] [Commented] (FLINK-4578) AggregateOperator incorrectly sets ForwardedField with nested composite types
[ https://issues.apache.org/jira/browse/FLINK-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17336853#comment-17336853 ] Flink Jira Bot commented on FLINK-4578: --- This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > AggregateOperator incorrectly sets ForwardedField with nested composite types > - > > Key: FLINK-4578 > URL: https://issues.apache.org/jira/browse/FLINK-4578 > Project: Flink > Issue Type: Bug > Components: API / DataSet >Reporter: Gábor Gévay >Priority: Major > Labels: stale-major > > When an aggregation is called on a grouped DataSet, > {{AggregateOperator.translateToDataFlow}} tries to determine whether the > field that is being aggregated is the same field that the grouping is based > on. If this is not the case, then it adds the ForwardedField property for the > key field. > However, the mechanism that makes this decision breaks when there are nested > composite types involved, because it gets the key positions with > {{getKeys().computeLogicalKeyPositions()}}, which returns the _flat_ > positions, whereas the position of the field to aggregate is counted only on > the outer type. > Example code: https://github.com/ggevay/flink/tree/agg-bad-forwarded-fields > Here, I have changed the WordCount example to have the type > {{Tuple3, String, Integer>}}, and do {{.groupBy(1).sum(2)}} > (which groups by the String field and sums the Integer field). If you set a > breakpoint into {{AggregateOperator.translateToDataFlow}}, you can see that > {{logicalKeyPositions}} contains 2, and {{fields}} also contains 2, which > causes {{keyFieldUsedInAgg}} to be erroneously set to true. The problem is > caused by the Tuple2 being counted as 2 fields in {{logicalKeyPositions}}, > but only 1 field in {{fields}}. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-4578) AggregateOperator incorrectly sets ForwardedField with nested composite types
[ https://issues.apache.org/jira/browse/FLINK-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17328901#comment-17328901 ] Flink Jira Bot commented on FLINK-4578: --- This major issue is unassigned and itself and all of its Sub-Tasks have not been updated for 30 days. So, it has been labeled "stale-major". If this ticket is indeed "major", please either assign yourself or give an update. Afterwards, please remove the label. In 7 days the issue will be deprioritized. > AggregateOperator incorrectly sets ForwardedField with nested composite types > - > > Key: FLINK-4578 > URL: https://issues.apache.org/jira/browse/FLINK-4578 > Project: Flink > Issue Type: Bug > Components: API / DataSet >Reporter: Gábor Gévay >Priority: Major > Labels: stale-major > > When an aggregation is called on a grouped DataSet, > {{AggregateOperator.translateToDataFlow}} tries to determine whether the > field that is being aggregated is the same field that the grouping is based > on. If this is not the case, then it adds the ForwardedField property for the > key field. > However, the mechanism that makes this decision breaks when there are nested > composite types involved, because it gets the key positions with > {{getKeys().computeLogicalKeyPositions()}}, which returns the _flat_ > positions, whereas the position of the field to aggregate is counted only on > the outer type. > Example code: https://github.com/ggevay/flink/tree/agg-bad-forwarded-fields > Here, I have changed the WordCount example to have the type > {{Tuple3, String, Integer>}}, and do {{.groupBy(1).sum(2)}} > (which groups by the String field and sums the Integer field). If you set a > breakpoint into {{AggregateOperator.translateToDataFlow}}, you can see that > {{logicalKeyPositions}} contains 2, and {{fields}} also contains 2, which > causes {{keyFieldUsedInAgg}} to be erroneously set to true. The problem is > caused by the Tuple2 being counted as 2 fields in {{logicalKeyPositions}}, > but only 1 field in {{fields}}. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-4578) AggregateOperator incorrectly sets ForwardedField with nested composite types
[ https://issues.apache.org/jira/browse/FLINK-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15463363#comment-15463363 ] Gabor Gevay commented on FLINK-4578: A hacky solution might be to create some dummy {{Keys}} object from the aggregate field, and call {{computeLogicalKeyPositions}} on it, so that we know the flat position of the key field. > AggregateOperator incorrectly sets ForwardedField with nested composite types > - > > Key: FLINK-4578 > URL: https://issues.apache.org/jira/browse/FLINK-4578 > Project: Flink > Issue Type: Bug > Components: DataSet API >Reporter: Gabor Gevay > > When an aggregation is called on a grouped DataSet, > {{AggregateOperator.translateToDataFlow}} tries to determine whether the > field that is being aggregated is the same field that the grouping is based > on. If this is not the case, then it adds the ForwardedField property for the > key field. > However, the mechanism that makes this decision breaks when there are nested > composite types involved, because it gets the key positions with > {{getKeys().computeLogicalKeyPositions()}}, which returns the _flat_ > positions, whereas the position of the field to aggregate is counted only on > the outer type. > Example code: https://github.com/ggevay/flink/tree/agg-bad-forwarded-fields > Here, I have changed the WordCount example to have the type > {{Tuple3, String, Integer>}}, and do {{.groupBy(1).sum(2)}} > (which groups by the String field and sums the Integer field). If you set a > breakpoint into {{AggregateOperator.translateToDataFlow}}, you can see that > {{logicalKeyPositions}} contains 2, and {{fields}} also contains 2, which > causes {{keyFieldUsedInAgg}} to be erroneously set to true. The problem is > caused by the Tuple2 being counted as 2 fields in {{logicalKeyPositions}}, > but only 1 field in {{fields}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)