[
https://issues.apache.org/jira/browse/FLINK-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612357#comment-14612357
]
ASF GitHub Bot commented on FLINK-1085:
---------------------------------------
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/854#discussion_r33809780
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/operators/CombineTaskTest.java
---
@@ -92,7 +92,35 @@ public void testCombineTask() {
this.outList.clear();
}
-
+
+ @Test
+ public void testOversizedRecordCombineTask() {
+ int keyCnt = 1;
+ int valCnt = 20;
+
+ addInput(new UniformRecordGenerator(keyCnt, valCnt, true));
+ addDriverComparator(this.comparator);
+ addDriverComparator(this.comparator);
+ setOutput(this.outList);
+
+
getTaskConfig().setDriverStrategy(DriverStrategy.SORTED_GROUP_COMBINE);
+ getTaskConfig().setRelativeMemoryDriver(combine_frac);
--- End diff --
I think you cannot reduce the memory further below to enforce records to be
oversized. I think you need to change the data generator to provide you with an
oversized record. I think the tests have a `UnionIterator` where you can mix an
oversized record into the generating iterator.
> Unnecessary failing of GroupReduceCombineDriver
> -----------------------------------------------
>
> Key: FLINK-1085
> URL: https://issues.apache.org/jira/browse/FLINK-1085
> Project: Flink
> Issue Type: Bug
> Components: Local Runtime
> Affects Versions: 0.6.1-incubating, 0.7.0-incubating
> Reporter: Fabian Hueske
> Labels: starter
>
> With a recent update (commit cbbcf7820885a8a9734ffeba637b0182a6637939) the
> GroupReduceCombineDriver was changed to not use an asynchronous partial
> sorter. Instead, the driver fills a sort buffer with records, sorts it,
> combines them, clears the buffer, and continues to fill it again.
> The GroupReduceCombineDriver fails if a record cannot be serialized into an
> empty sort buffer, i.e., if the record is too large for the buffer.
> Alternatively, we should emit a WARN message for the first record that is too
> large and just forward all records which do not fit into the empty sort
> buffer (maybe continue to count how many records were simply forwarded and
> give a second WARN message with this statistic).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
