danny0405 opened a new pull request, #18756:
URL: https://github.com/apache/hudi/pull/18756
```markdown
### Describe the issue this Pull Request addresses
`AppendWriteFunctionWithBIMBufferSort` can have an in-flight async flush
when the Flink task is being closed. The existing shutdown path stopped
accepting new async work but did not wait for already-submitted work before
releasing the sort buffers, which could allow background writes to overlap with
resource cleanup during failover or cancellation.
In addition, `AppendWriteFunction` did not close a surviving
`BulkInsertWriterHelper` from its `close()` path, leaving helper-owned write
resources open when a task shuts down before the normal checkpoint flush path
closes them.
### Summary and Changelog
This change tightens append-writer shutdown behavior so async buffer
flushing completes before dependent resources are released, and any surviving
writer helper is closed during function shutdown.
#### Working tree: harden append writer shutdown cleanup
- Update `AppendWriteFunction.close()` to close `writerHelper` when it is
still present during task shutdown.
- Update `AppendWriteFunctionWithBIMBufferSort.close()` to shut down the
async executor, wait for in-flight async flush work to finish, and only then
release memory segment pools.
- Extend `TestAppendWriteFunction` with `testCloseClosesWriterHelper`.
- Add `TestAppendWriteFunctionWithBIMBufferSort` with
`testCloseWaitsForAsyncWriteBeforeClosingWriterHelper` to verify close ordering
when async work is still running.
- Validate related append-buffer behavior with:
- `TestAppendWriteFunction`
- `TestAppendWriteFunctionWithBIMBufferSort`
- `TestAppendWriteFunctionWithBufferSort`
### Impact
This change affects Flink append-write task shutdown behavior only. It does
not change public APIs, configuration, storage format, or normal write
semantics.
The main user-facing effect is safer cleanup during cancellation/failover
for append writers using bounded in-memory buffer sort, reducing the chance of
resource cleanup racing with already-submitted async write work.
### Risk Level
low
The change is localized to append-writer shutdown paths and adds explicit
waiting during close, so the primary risk is a longer close path if async work
is still in progress. This is mitigated by targeted unit coverage and
successful validation with:
`mvn -pl hudi-flink-datasource/hudi-flink -DskipITs
-Dtest=TestAppendWriteFunction,TestAppendWriteFunctionWithBIMBufferSort,TestAppendWriteFunctionWithBufferSort
test`
Result: `16` tests run, `0` failures, `0` errors.
### Documentation Update
none
No user-facing configuration, API, or behavior contract changed; this is an
internal lifecycle and cleanup fix.
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
