flink1.12.4 写入hdfs报错 java.lang.OutOfMemoryError: Direct buffer memory

2021-11-08 Thread xiao cai
通过flink 1.12.4 streaming file sink 写入hdfs,运行过程中抛出以下异常: 2021-11-08 20:39:05 java.io.IOException: java.lang.OutOfMemoryError: Direct buffer memory at org.apache.hadoop.hdfs.DataStreamer$LastExceptionInStreamer.set(DataStreamer.java:299) at org.apache.hadoop.hdfs.DataStreamer.run(DataStream

Re: flink1.12.4 写入hdfs报错 java.lang.OutOfMemoryError: Direct buffer memory

2021-11-08 Thread Caizhi Weng
Hi! 可以通过配置 taskmanager.memory.task.off-heap.size 指定 direct memory 和 native memory 的大小,详见 [1]。 [1] https://nightlies.apache.org/flink/flink-docs-master/zh/docs/deployment/memory/mem_setup_tm/#%e9%85%8d%e7%bd%ae%e5%a0%86%e5%a4%96%e5%86%85%e5%ad%98%e7%9b%b4%e6%8e%a5%e5%86%85%e5%ad%98%e6%88%96%e6%9c%