gejinzh opened a new issue, #12820:
URL: https://github.com/apache/hudi/issues/12820
**Describe the problem you faced**
The flink task has been running for 17 days; Then an exception occurred
**hudi config**
```
Map<String, String> options= new HashMap<>();
options.put(FlinkOptions.PATH.key(), hudiProperties.getPath());
options.put(FlinkOptions.TABLE_TYPE.key(),
HoodieTableType.MERGE_ON_READ.name());
options.put(FlinkOptions.OPERATION.key(),
WriteOperationType.INSERT.value());
options.put(FlinkOptions.DATABASE_NAME.key(),
hudiProperties.getDatabase());
options.put(FlinkOptions.TABLE_NAME.key(),
hudiProperties.getTableName());
options.put(FlinkOptions.WRITE_TASKS.key(), String.valueOf(1));
options.put(FlinkOptions.HIVE_SYNC_ENABLED.key(),
String.valueOf(true));
options.put(FlinkOptions.HIVE_SYNC_MODE.key(),
HiveSyncMode.HMS.name());
options.put(FlinkOptions.HIVE_SYNC_METASTORE_URIS.key(),
hudiProperties.getHiveMetastoreUris());
options.put(FlinkOptions.HIVE_SYNC_DB.key(),
hudiProperties.getDatabase());
options.put(FlinkOptions.HIVE_SYNC_CONF_DIR.key(),
hudiProperties.getHiveConfDir());
options.put(FlinkOptions.CLUSTERING_SCHEDULE_ENABLED.key(),
String.valueOf(true));
options.put(FlinkOptions.CLUSTERING_ASYNC_ENABLED.key(),
String.valueOf(true));
options.put(FlinkOptions.CLUSTERING_TASKS.key(), String.valueOf(1));
options.put(FlinkOptions.WRITE_BULK_INSERT_SORT_INPUT.key(),
String.valueOf(false));
options.put(FlinkOptions.WRITE_RATE_LIMIT.key(),
String.valueOf(500));
```
**Environment Description**
* Hudi version : 0.15.0
* Hadoop version : 3.3.3
* Storage (HDFS/S3/GCS..) : hdfs
* Running on Docker? (yes/no) : no
**Stacktrace**
```
org.apache.hudi.exception.HoodieException: Timeout(121000ms) while waiting
for instant initialize
at org.apache.hudi.sink.utils.TimeWait.waitFor(TimeWait.java:57)
at
org.apache.hudi.sink.common.AbstractStreamWriteFunction.instantToWrite(AbstractStreamWriteFunction.java:269)
at
org.apache.hudi.sink.append.AppendWriteFunction.initWriterHelper(AppendWriteFunction.java:125)
at
org.apache.hudi.sink.append.AppendWriteFunction.processElement(AppendWriteFunction.java:84)
at
org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66)
at
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:233)
at
org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
at
org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:546)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:837)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:786)
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
at java.lang.Thread.run(Thread.java:750)
```
checkpoint interval: 5minutes
but deltaCommit spend 15minutes or 10 minutes;
```
-rw-rw-r-- 3 yxfcenter hadoop 5455 2025-02-10 05:36
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210052607892.deltacommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 05:26
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210052607892.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 05:26
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210052607892.deltacommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 5455 2025-02-10 05:46
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210053608150.deltacommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 05:36
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210053608150.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 05:36
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210053608150.deltacommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 5455 2025-02-10 05:56
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210054609874.deltacommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 05:46
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210054609874.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 05:46
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210054609874.deltacommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 14324 2025-02-10 06:01
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210055609022.replacecommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:01
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210055609022.replacecommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 10382 2025-02-10 05:56
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210055609022.replacecommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 5455 2025-02-10 06:06
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210055609184.deltacommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 05:56
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210055609184.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 05:56
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210055609184.deltacommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 6786 2025-02-10 06:06
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210060609076.clean
-rw-rw-r-- 3 yxfcenter hadoop 7219 2025-02-10 06:06
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210060609076.clean.inflight
-rw-rw-r-- 3 yxfcenter hadoop 7219 2025-02-10 06:06
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210060609076.clean.requested
-rw-rw-r-- 3 yxfcenter hadoop 5455 2025-02-10 06:16
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210060609288.deltacommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:06
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210060609288.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:06
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210060609288.deltacommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 5455 2025-02-10 06:26
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210061609535.deltacommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:16
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210061609535.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:16
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210061609535.deltacommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 5455 2025-02-10 06:36
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210062610189.deltacommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:26
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210062610189.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:26
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210062610189.deltacommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 14326 2025-02-10 06:41
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210063610893.replacecommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:41
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210063610893.replacecommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 10382 2025-02-10 06:36
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210063610893.replacecommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 5451 2025-02-10 06:46
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210063611067.deltacommit
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:36
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210063611067.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:36
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210063611067.deltacommit.requested
-rw-rw-r-- 3 yxfcenter hadoop 6210 2025-02-10 06:46
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210064610874.clean
-rw-rw-r-- 3 yxfcenter hadoop 6555 2025-02-10 06:46
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210064610874.clean.inflight
-rw-rw-r-- 3 yxfcenter hadoop 6555 2025-02-10 06:46
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210064610874.clean.requested
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:46
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210064611475.deltacommit.inflight
-rw-rw-r-- 3 yxfcenter hadoop 0 2025-02-10 06:46
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/20250210064611475.deltacommit.requested
drwxrwxr-x - yxfcenter hadoop 0 2025-01-24 07:20
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/archived
-rw-rw-r-- 3 yxfcenter hadoop 2352 2025-01-23 20:09
/user/yxfcenter/hudi/tele_tables/tele_table/ods_count_limit_comp_mbr/.hoodie/hoodie.properties
```
The entire task is stuck on instant 20250210064611475
what can I do to fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
