[
https://issues.apache.org/jira/browse/FLINK-30960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17781713#comment-17781713
]
Bo Shen commented on FLINK-30960:
---------------------------------
This issue still exists in the latest master branch of flink-jdbc-connector.
I would like to contribute a PR for a quick fix if this is OK:
# Introduce a new FlushingException extends RuntimeException, and throw new
type instead in "checkFlushException".
# In JdbcOutputFormat's scheduler thread, add a type check before replacing
existing flushException. If newly caught exception is of "FlushingException",
ignore it.
> OutOfMemory error using jdbc sink
> ---------------------------------
>
> Key: FLINK-30960
> URL: https://issues.apache.org/jira/browse/FLINK-30960
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
> Affects Versions: 1.16.0
> Environment: Using SQL-Client + TableAPI
> Processing Kafka messages to MySQL databases.
> Reporter: Bo Shen
> Priority: Major
> Attachments: heap-memory-metrics.png,
> image-2023-02-08-16-58-27-765.png, taskmanager.log
>
>
> Here in production I have an environment consisting of a kafka source, to
> which third-party providers push a few thousand messages every hour, and I
> use flink jdbc sink to store those messages into a mysql database.
> Recently there are a few taskmanager process crashes. My investigation
> suggests that the cause boils down to the way exceptions are handled in jdbc
> batched mode.
>
> When writing to JDBC failed in batched mode due to some error like
> DataTuncation, the exception is stored in field "flushException" waiting to
> be processed by the task main thread.
> This exception is only checked on the next call to "writeRecords". In my
> case, if the exception happens to occur when processing the last batch of
> records, and there are no further record comming from source for the next
> hour, the flushing thread simply repeatly throws a new RuntimeException
> wrapping the last "flushException" as cause. The new RuntimeException is
> stored as the new "flushException".
> Hence, every time the RuntimeException is thrown, the exception hierachy gets
> bigger, and eventually before the exception is processed on the main thread,
> the jvm runs out of memory, which causes an unexpected process crash.
> !image-2023-02-08-16-58-27-765.png!
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
