Re: flink运行一段时间后TaskManager退出,报OutOfMemoryError: Metaspace

2022-06-08 文章 Lijie Wang
看错误是 metaspace OOM 了,可以按照提示,增加 taskmanager.memory.jvm-metaspace.size
的大小,或者增加 TM 总内存大小

Best,
Lijie

weishishuo...@163.com  于2022年6月7日周二 18:37写道:

> 我使用的版本是:
> flink:1.13.2
> flink cdc: flink-connector-jdbc_2.11-1.13.2.jar
> flink-sql-connector-mysql-cdc-2.2.0.jar
> flink-sql-connector-postgres-cdc-2.2.0.jar
>
> 任务比较简单,就是从mysql、pg同步数据到pg,mysql,使用的是sql接口,请问大伙儿有碰到过这个问题吗?
>
> 2022-06-07 18:13:59,393 ERROR
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner  [] - Fatal
> error occurred while executing the TaskManager. Shutting it down...
> java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error
> has occurred. This can mean two things: either the job requires a larger
> size of JVM metaspace to load classes or there is a class loading leak. In
> the first case 'taskmanager.memory.jvm-metaspace.size' configuration option
> should be increased. If the error persists (usually in cluster after
> several job (re-)submissions) then there is probably a class loading leak
> in user code or some of its dependencies which has to be investigated and
> fixed. The task executor has to be shutdown...at
> java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_112]
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> ~[?:1.8.0_112]
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> ~[?:1.8.0_112]
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> ~[?:1.8.0_112]
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> ~[?:1.8.0_112]
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> ~[?:1.8.0_112]
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> ~[?:1.8.0_112]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_112]
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> ~[?:1.8.0_112]
> at
> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
> ~[flink-dist_2.11-1.13.2.jar:1.13.2]
> at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
> [flink-dist_2.11-1.13.2.jar:1.13.2]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) [?:1.8.0_112]
> at io.debezium.relational.Column.editor(Column.java:31)
> [blob_p-343be43b7874de49f6ce1d8bcb6a90a384203530-2e5afb6f8bf4834164c1bb92aaf97a00:2.2.0]
> at
> io.debezium.connector.postgresql.connection.PostgresConnection.readTableColumn(PostgresConnection.java:464)
> [blob_p-343be43b7874de49f6ce1d8bcb6a90a384203530-2e5afb6f8bf4834164c1bb92aaf97a00:2.2.0]
> at
> io.debezium.jdbc.JdbcConnection.getColumnsDetails(JdbcConnection.java:1226)
> [blob_p-343be43b7874de49f6ce1d8bcb6a90a384203530-2e5afb6f8bf4834164c1bb92aaf97a00:2.2.0]
> at
> io.debezium.jdbc.JdbcConnection.readSchema(JdbcConnection.java:1182)
> [blob_p-343be43b7874de49f6ce1d8bcb6a90a384203530-2e5afb6f8bf4834164c1bb92aaf97a00:2.2.0]
> at
> io.debezium.connector.postgresql.PostgresSchema.refresh(PostgresSchema.java:100)
> [blob_p-343be43b7874de49f6ce1d8bcb6a90a384203530-2e5afb6f8bf4834164c1bb92aaf97a00:2.2.0]
> at
> io.debezium.connector.postgresql.PostgresSnapshotChangeEventSource.connectionCreated(PostgresSnapshotChangeEventSource.java:95)
> [blob_p-343be43b7874de49f6ce1d8bcb6a90a384203530-2e5afb6f8bf4834164c1bb92aaf97a00:2.2.0]
> at
> io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:103)
> [blob_p-343be43b7874de49f6ce1d8bcb6a90a384203530-2e5afb6f8bf4834164c1bb92aaf97a00:2.2.0]
>
>
>
>
> weishishuo...@163.com
>


flink运行一段时间后TaskManager退出,报OutOfMemoryError: Metaspace

2022-06-07 文章 weishishuo...@163.com
我使用的版本是:
flink:1.13.2
flink cdc: flink-connector-jdbc_2.11-1.13.2.jar 
flink-sql-connector-mysql-cdc-2.2.0.jar 
flink-sql-connector-postgres-cdc-2.2.0.jar

任务比较简单,就是从mysql、pg同步数据到pg,mysql,使用的是sql接口,请问大伙儿有碰到过这个问题吗?

2022-06-07 18:13:59,393 ERROR 
org.apache.flink.runtime.taskexecutor.TaskManagerRunner  [] - Fatal error 
occurred while executing the TaskManager. Shutting it down...
java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has 
occurred. This can mean two things: either the job requires a larger size of 
JVM metaspace to load classes or there is a class loading leak. In the first 
case 'taskmanager.memory.jvm-metaspace.size' configuration option should be 
increased. If the error persists (usually in cluster after several job 
(re-)submissions) then there is probably a class loading leak in user code or 
some of its dependencies which has to be investigated and fixed. The task 
executor has to be shutdown...at java.lang.ClassLoader.defineClass1(Native 
Method) ~[?:1.8.0_112]
at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[?:1.8.0_112]
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) 
~[?:1.8.0_112]
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) 
~[?:1.8.0_112]
at java.net.URLClassLoader.access$100(URLClassLoader.java:73) ~[?:1.8.0_112]
at java.net.URLClassLoader$1.run(URLClassLoader.java:368) ~[?:1.8.0_112]
at java.net.URLClassLoader$1.run(URLClassLoader.java:362) ~[?:1.8.0_112]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112]
at java.net.URLClassLoader.findClass(URLClassLoader.java:361) ~[?:1.8.0_112]
at 
org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71)
 ~[flink-dist_2.11-1.13.2.jar:1.13.2]
at 
org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
 [flink-dist_2.11-1.13.2.jar:1.13.2]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) [?:1.8.0_112]
at io.debezium.relational.Column.editor(Column.java:31) 
[blob_p-343be43b7874de49f6ce1d8bcb6a90a384203530-2e5afb6f8bf4834164c1bb92aaf97a00:2.2.0]
at 
io.debezium.connector.postgresql.connection.PostgresConnection.readTableColumn(PostgresConnection.java:464)
 
[blob_p-343be43b7874de49f6ce1d8bcb6a90a384203530-2e5afb6f8bf4834164c1bb92aaf97a00:2.2.0]
at 
io.debezium.jdbc.JdbcConnection.getColumnsDetails(JdbcConnection.java:1226) 
[blob_p-343be43b7874de49f6ce1d8bcb6a90a384203530-2e5afb6f8bf4834164c1bb92aaf97a00:2.2.0]
at io.debezium.jdbc.JdbcConnection.readSchema(JdbcConnection.java:1182) 
[blob_p-343be43b7874de49f6ce1d8bcb6a90a384203530-2e5afb6f8bf4834164c1bb92aaf97a00:2.2.0]
at 
io.debezium.connector.postgresql.PostgresSchema.refresh(PostgresSchema.java:100)
 
[blob_p-343be43b7874de49f6ce1d8bcb6a90a384203530-2e5afb6f8bf4834164c1bb92aaf97a00:2.2.0]
at 
io.debezium.connector.postgresql.PostgresSnapshotChangeEventSource.connectionCreated(PostgresSnapshotChangeEventSource.java:95)
 
[blob_p-343be43b7874de49f6ce1d8bcb6a90a384203530-2e5afb6f8bf4834164c1bb92aaf97a00:2.2.0]
at 
io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:103)
 
[blob_p-343be43b7874de49f6ce1d8bcb6a90a384203530-2e5afb6f8bf4834164c1bb92aaf97a00:2.2.0]




weishishuo...@163.com