Nikola created FLINK-17192: ------------------------------ Summary: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due to an exception: The record exceeds the maximum size of a sort buffer (current maximum: 72351744 bytes) Key: FLINK-17192 URL: https://issues.apache.org/jira/browse/FLINK-17192 Project: Flink Issue Type: Bug Components: Runtime / Task Affects Versions: 1.8.2 Environment: flink 1.8.2 Reporter: Nikola
We are trying to sort our data and we get to the following exception: {code:java} java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due to an exception: The record exceeds the maximum size of a sort buffer (current maximum: 72351744 bytes). at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:650) at org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1108) at org.apache.flink.runtime.operators.NoOpDriver.run(NoOpDriver.java:82) at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:503) at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: The record exceeds the maximum size of a sort buffer (current maximum: 72351744 bytes). at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:831) Caused by: java.io.IOException: The record exceeds the maximum size of a sort buffer (current maximum: 72351744 bytes). at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ReadingThread.go(UnilateralSortMerger.java:986) at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:827){code} The sort buffer says to be "72351744 bytes" which is about 70MB. Our TM has 30GB memory set for flink. I tried to find the same issue, but nothing really gave me an answer. I tried finding the same issue on JIRA, but I couldn't. [http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Any-way-to-increase-sort-buffer-size-td10152.html] This one is from 2016 so I am not sure if I can trust it anymore since Flink has evolved since Is it possible to increase this sort buffer? -- This message was sent by Atlassian Jira (v8.3.4#803005)