Hi: 谢谢答复。 我用的云上的arm虚拟机。 请问下社区有计划支持arm架构下的RocksDB吗?
祝好! Wang Hui -----邮件原件----- 发件人: Yun Tang [mailto:myas...@live.com] 发送时间: 2021年7月16日 14:27 收件人: user-zh@flink.apache.org 主题: Re: flink大窗口性能问题 目前Flink社区版RocksDB尚不支持ARM架构机器。使用RocksDB的话,内存均是堆外管理,与JVM的堆上内存无关。 另外,有个题外话,你们是云上产品还是自建了ARM集群?有点好奇目前国内的ARM集群使用率情况。 祝好 唐云 ________________________________ From: Wanghui (HiCampus) <wanghu...@huawei.com> Sent: Thursday, July 15, 2021 11:33 To: user-zh@flink.apache.org <user-zh@flink.apache.org> Subject: Re: flink大窗口性能问题 我在aarch64 + jre 8的环境下,使用rocksdb state backend时,碰到如下错误: 另外,使用rocksdb可以解决大窗口的oom问题吗,原理是什么? Caused by: java.lang.Exception: Exception while creating StreamOperatorStateContext. at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:222) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:248) at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290) at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$1(StreamTask.java:506) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47) at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:526) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for KeyedProcessOperator_ae33e81d863e4093619373d1e1f77012_(1/1) from any of the 1 provided restore options. at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135) at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:335) at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:148) ... 9 more Caused by: java.io.IOException: Could not load the native RocksDB library at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:948) at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:489) at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:319) at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142) at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121) ... 11 more Caused by: java.lang.UnsatisfiedLinkError: /tmp/rocksdb-lib-bd8659305e92a27fac27481baf57897b/librocksdbjni-linux64.so: /tmp/rocksdb-lib-bd8659305e92a27fac27481baf57897b/librocksdbjni-linux64.so: cannot open shared object file: No such file or directory (Possible cause: can't load AMD 64-bit .so on a AARCH64-bit platform) at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1934) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1817) at java.lang.Runtime.load0(Runtime.java:810) at java.lang.System.load(System.java:1088) at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78) at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56) at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:923) ... 15 more On 2021/07/15 02:47:23, Jingsong Li <j...@gmail.com<mailto:j...@gmail.com>> wrote: > 没用rocksdb吗?> > > > > On Thu, Jul 15, 2021 at 10:46 AM Michael Ran > <gr...@163.com<mailto:gr...@163.com>> wrote:> > > > > > 要么内存增大,或者并行增大,要么窗口改小,同时保留数据时间减少> > > > 在 2021-07-15 10:23:25,"Hui Wang" > > <46...@qq.com.INVALID<mailto:46...@qq.com.INVALID>> 写道:> > > > >flink大窗口缓存数据量过大导致jvm频烦full gc,并且处理速度极低,最终OOM,该如何调优> > > >> > > > > > > -- > > > Best, Jingsong Lee> > >