ravjot28 opened a new issue, #20610:
URL: https://github.com/apache/hudi/issues/20610
### Bug Description
**What happened:**
`HoodieMetadataWriteUtils#createMetadataWriteConfig` builds the metadata
table write config without
`hoodie.client.heartbeat.interval_in_ms` and
`hoodie.client.heartbeat.tolerable.misses`, so the metadata table writer
always uses the defaults (60 s, 10 misses). Heartbeats matter for the
metadata table when it cleans failed writes
lazily: with streaming metadata writes, and in metadata table multi-writer
mode. A pending metadata table write is
then judged expired on the default lease, not on the one configured for the
table. Lengthening the lease to tolerate
slow heartbeat writes has no effect on the metadata table, and shortening it
does not make the metadata table
recover failed writes sooner.
**What you expected:**
The metadata table writer uses the heartbeat settings of the data table
writer it is created from, as it already does
for several other settings (consistency guard, cleaning and archival limits,
metrics).
**Steps to reproduce:**
1. Build a data table write config with
`hoodie.client.heartbeat.interval_in_ms=5000`,
`hoodie.client.heartbeat.tolerable.misses=4` and streaming metadata
writes enabled.
2. Call `HoodieMetadataWriteUtils.createMetadataWriteConfig(...)`.
3. The returned config uses the `LAZY` failed writes cleaning policy, with
heartbeat interval 60000 and 10 tolerable
misses.
**Root cause (master 9a7728c5862):**
`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataWriteUtils.java:200-336`:
the
builder sets the `LAZY` policy for streaming writes and multi-writer mode
(`:224`, `:230`) but no heartbeat setting.
The heartbeat client is created from the metadata write config
(`BaseHoodieClient.java:140`), and lazy cleaning of
failed writes checks expiry with it
(`BaseHoodieTableServiceClient.java:1089`, `:1192`). Heartbeats are started with
it when the policy is lazy (`BaseHoodieWriteClient.java:1167-1168`). With
streaming metadata writes, the metadata
writer reaches that cleaning from
`HoodieBackedTableMetadataWriter.java:1425`.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT at 9a7728c5862), 1.2.1
**Query engine:** any writer with the metadata table enabled
**Relevant configs:** `hoodie.client.heartbeat.interval_in_ms`,
`hoodie.client.heartbeat.tolerable.misses`,
`hoodie.metadata.streaming.write.enabled` or metadata table multi-writer mode
### Logs and Stack Trace
none
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]