ravjot28 opened a new issue, #20610:
URL: https://github.com/apache/hudi/issues/20610

   ### Bug Description
   
   **What happened:**
   
   `HoodieMetadataWriteUtils#createMetadataWriteConfig` builds the metadata 
table write config without
   `hoodie.client.heartbeat.interval_in_ms` and 
`hoodie.client.heartbeat.tolerable.misses`, so the metadata table writer
   always uses the defaults (60 s, 10 misses). Heartbeats matter for the 
metadata table when it cleans failed writes
   lazily: with streaming metadata writes, and in metadata table multi-writer 
mode. A pending metadata table write is
   then judged expired on the default lease, not on the one configured for the 
table. Lengthening the lease to tolerate
   slow heartbeat writes has no effect on the metadata table, and shortening it 
does not make the metadata table
   recover failed writes sooner.
   
   **What you expected:**
   
   The metadata table writer uses the heartbeat settings of the data table 
writer it is created from, as it already does
   for several other settings (consistency guard, cleaning and archival limits, 
metrics).
   
   **Steps to reproduce:**
   
   1. Build a data table write config with 
`hoodie.client.heartbeat.interval_in_ms=5000`,
      `hoodie.client.heartbeat.tolerable.misses=4` and streaming metadata 
writes enabled.
   2. Call `HoodieMetadataWriteUtils.createMetadataWriteConfig(...)`.
   3. The returned config uses the `LAZY` failed writes cleaning policy, with 
heartbeat interval 60000 and 10 tolerable
      misses.
   
   **Root cause (master 9a7728c5862):**
   
   
`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataWriteUtils.java:200-336`:
 the
   builder sets the `LAZY` policy for streaming writes and multi-writer mode 
(`:224`, `:230`) but no heartbeat setting.
   The heartbeat client is created from the metadata write config 
(`BaseHoodieClient.java:140`), and lazy cleaning of
   failed writes checks expiry with it 
(`BaseHoodieTableServiceClient.java:1089`, `:1192`). Heartbeats are started with
   it when the policy is lazy (`BaseHoodieWriteClient.java:1167-1168`). With 
streaming metadata writes, the metadata
   writer reaches that cleaning from 
`HoodieBackedTableMetadataWriter.java:1425`.
   
   ### Environment
   
   **Hudi version:** master (1.3.0-SNAPSHOT at 9a7728c5862), 1.2.1
   **Query engine:** any writer with the metadata table enabled
   **Relevant configs:** `hoodie.client.heartbeat.interval_in_ms`, 
`hoodie.client.heartbeat.tolerable.misses`,
   `hoodie.metadata.streaming.write.enabled` or metadata table multi-writer mode
   
   ### Logs and Stack Trace
   
   none
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to