[
https://issues.apache.org/jira/browse/FLINK-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14595518#comment-14595518
]
ASF GitHub Bot commented on FLINK-1085:
---------------------------------------
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/854#issuecomment-114043020
I think this code actually simply drops oversized records, or am I
overlooking something?
Also, outputting oversized records on WARN debug level seems heavy, it is
not really something that should alert the person that runs the program.
Before merging this, we need to
- Make sure the records are not lost
- Add a test that validates exactly that (not as an ITCase, but a unit
test)
- Reduce the log level for large records to DEBUG
> Unnecessary failing of GroupReduceCombineDriver
> -----------------------------------------------
>
> Key: FLINK-1085
> URL: https://issues.apache.org/jira/browse/FLINK-1085
> Project: Flink
> Issue Type: Bug
> Components: Local Runtime
> Affects Versions: 0.6.1-incubating, 0.7.0-incubating
> Reporter: Fabian Hueske
> Labels: starter
>
> With a recent update (commit cbbcf7820885a8a9734ffeba637b0182a6637939) the
> GroupReduceCombineDriver was changed to not use an asynchronous partial
> sorter. Instead, the driver fills a sort buffer with records, sorts it,
> combines them, clears the buffer, and continues to fill it again.
> The GroupReduceCombineDriver fails if a record cannot be serialized into an
> empty sort buffer, i.e., if the record is too large for the buffer.
> Alternatively, we should emit a WARN message for the first record that is too
> large and just forward all records which do not fit into the empty sort
> buffer (maybe continue to count how many records were simply forwarded and
> give a second WARN message with this statistic).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
