GitHub user yuzhangqin55-bit created a discussion: Paimon's function about
compaction and partial-update
In my business scenarios,I need to use 8 tasks write to the same table, and I
also need only a few columns keep the first non-null value, other be covered.
Now,I set table as:
WITH (
'bucket' = '40',
'path' =
's3://yj-datalake-test/paimon/qinyuzhang.db/Mt5HistoryTradeRecordModel',
'write-mode' = 'WRITE_ONLY',
'merge-engine' = 'partial-update',
'bucket-key' = 'b_id,s_id'
)
and run 8 tasks to write,but it prints logs as:
2025-10-09 05:33:13,865 WARN org.apache.flink.runtime.taskmanager.Task
[] - Writer : Mt5HistoryTradeRecordModel -> Global Committer :
Mt5HistoryTradeRecordModel -> end: Writer (1/1)#677
(70647f047568adcfd35f0ed8f64fd97d_f6dc7f4d2283f4605b127b9364e21148_0_677)
switched from RUNNING to FAILED with failure cause:
java.lang.RuntimeException: File deletion conflicts detected! Give up
committing.
Don't panic!
Conflicts during commits are normal and this failure is intended to resolve the
conflicts.
Conflicts are mainly caused by the following scenarios:
1. Multiple jobs are writing into the same partition at the same time, or you
use STATEMENT SET to execute multiple INSERT statements into the same Paimon
table.
You'll probably see different base commit user and current commit user below.
You can use
https://paimon.apache.org/docs/master/maintenance/dedicated-compaction#dedicated-compaction-job
to support multiple writing.
2. You're recovering from an old savepoint, or you're creating multiple jobs
from a savepoint.
The job will fail continuously in this scenario to protect metadata from
corruption.
You can either recover from the latest savepoint, or you can revert the
table to the snapshot corresponding to the old savepoint.
Base commit user is: d0d36272-9541-497b-8710-0ad75295d42b; Current commit user
is: d0d36272-9541-497b-8710-0ad75295d42b
So table does not save all data.
How should I set up the table to eliminate the problem and retain all the data?
Help me thanks a lot.
GitHub link: https://github.com/apache/incubator-xtable/discussions/751
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
