maheshguptags opened a new issue, #12738:
URL: https://github.com/apache/hudi/issues/12738
**Issue**
While performing load testing with METADATA enabled, I encountered a data
loss issue. The issue occurs when deploying the job with Autoscale enabled.
Specifically, if checkpointing fails due to reasons such as TM add-ons or
memory heap issues, all data is discarded, and no further data is processed
after that failure.
Checkpointing failures lead to data loss.
After a failed checkpoint due to lack of resources, a new checkpoint is
triggered but no data is processed.
I tried to replicate this behavior on Hudi 1.0, and the same issue persists.
**Hudi Properties**
``` hoodie.properties
#Updated at 2025-01-20T07:41:05.654545Z
#Mon Jan 20 07:41:05 UTC 2025
hoodie.table.keygenerator.type=COMPLEX_AVRO
hoodie.table.type=COPY_ON_WRITE
hoodie.table.precombine.field=updated_date
hoodie.table.create.schema={}
hoodie.timeline.layout.version=2
hoodie.timeline.history.path=history
hoodie.table.checksum=1292384652
hoodie.datasource.write.drop.partition.columns=false
hoodie.record.merge.strategy.id=00000000-0000-0000-0000-000000000000
hoodie.datasource.write.hive_style_partitioning=false
hoodie.table.metadata.partitions.inflight=
hoodie.database.name=default_database
hoodie.datasource.write.partitionpath.urlencode=false
hoodie.record.merge.mode=CUSTOM
hoodie.table.version=8
hoodie.compaction.payload.class=com.gupshup.cdp.PartialUpdate
hoodie.table.initial.version=8
hoodie.table.metadata.partitions=files
hoodie.table.partition.fields=xyz
hoodie.table.cdc.enabled=false
hoodie.archivelog.folder=history
hoodie.table.name=customer_temp
hoodie.table.recordkey.fields=xyz.abc
hoodie.timeline.path=timeline
```
**Steps to reproduce the behavior**:
1. Create a table with Flink hudi along with MDT Enable
2. Ingest some load
3. Try to delete one of TM Or Ingest heavy load so that it can give memory
issue
4. once it fails it will discard all the data after that checkpointing
**Expected behavior**
After checkpoint failure due to resource issues, the system should continue
processing data once resources are available, without losing previously
processed data.
Environment Description
* Hudi version : 1.0.0
* Flink version: 1.18
* Hive version : NO
* Hadoop version :
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : Yes
* Table Type: COPY_ON_WRITE
**Additional context**
Can the Hudi team assist with troubleshooting this issue? Is this expected
behavior with METADATA enabled, or is there a bug with flink under resource
constraint scenarios?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
