danny0405 opened a new issue, #20491: URL: https://github.com/apache/hudi/issues/20491
### Task Description **What needs to be done:** Define ownership and reuse of the Spark metadata-table writer's embedded timeline service across streaming micro-batches, including write-client replacement and terminal cleanup (RFC-112 Phase 2 Track B4). Current master constructs `SparkRDDMetadataWriteClient(engineContext, metadataWriteConfig, Option.empty())` in `SparkHoodieBackedTableMetadataWriter.initializeWriteClient()`. `HoodieBackedTableMetadataWriter.close()` closes that client. The data-table sink's query-scoped lifetime work does not by itself establish reuse for this metadata client. **Why this task is needed:** Avoid repeated metadata timeline-service startup/shutdown while preserving fresh metadata views and resource ownership. This tracks the follow-up requested in #20401; it does not claim a newly reproduced leak. Acceptance criteria: - With metadata enabled, measure service creation/closure across multiple micro-batches and demonstrate reuse under an explicit owner. - Verify metadata reads stay current across commits and client/schema changes. - Verify AvailableNow completion, explicit stop and failure release owned resources without stopping another query's service. - Cover metadata compaction/rollback and document reuse limits. ### Task Type Performance optimization ### Related Issues **Parent feature issue:** #20401 (Phase 2 B4) **Related issues:** #20400, #20061. The latter is a separate leak investigation; startup/stop churn alone is not evidence of a leak. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
