joeyutong opened a new pull request, #1058:
URL: https://github.com/apache/flink-agents/pull/1058
Linked issue: #1057
### Purpose of change
Pemja values crossing the Java/Python bridge may carry native Python
references. The current resource paths do not share one ownership rule, so
temporary values, nested results, long-lived resources, and partial
initialization failures can retain references differently.
This change introduces `PythonObjectScope` as the common ownership
abstraction:
- an operation scope recursively owns `PyObject` handles reachable through
maps, iterables, and object arrays, deduplicates them by identity, and releases
them in reverse order;
- `release(...)` makes ownership transfer to a long-lived wrapper explicit;
- values that must outlive a bridge call are materialized into detached Java
primitives, lists, and maps before the raw references are closed;
- resource creation and MCP discovery use transactional rollback, closing
both Python lifecycle state and native handles before any failed resource can
remain registered.
The abstraction is applied consistently to chat and embedding models, vector
stores, MCP servers/tools/prompts, Mem0 call-scoped values, Python-backed
`FunctionTool` calls, and Java/Python resource-cache initialization.
This complements #944, which owns executor-lifetime roots and the shutdown
close chain, and #1049, which owns per-Action execution values. This PR covers
the remaining operation-level resource bridges and initialization rollback
paths.
### Tests
- `mvn -pl runtime -am -DskipITs test`
- plan: 294 tests, 1 skipped
- runtime: 721 tests, 0 failures/errors
- Final rollback-path changes: focused
`PythonResourceProviderTest,ResourceCacheTest`
- `bash tools/ut.sh -p`: 935 passed, 13 skipped, 165 deselected
- `./tools/lint.sh -c`
- `bash tools/check-license.sh`
- `git diff --check`
Local Flink 2.2 A/B validation ran six jobs, each with a fresh TaskManager,
one probe subtask, Pemja 0.5.7, and 200 measured operations:
- chat: fixed `0` live bridge values; leak variant retained `600` messages,
`200` tools, `200 MiB`, and increased RSS by `160.3 MiB`;
- vector: fixed `0` live bridge values; leak variant retained `600`
documents, `200` normalized handles, about `100 MiB`, and increased RSS by
`99.1 MiB`;
- resource recreation: fixed `0` live resources with `200` logical closes;
leak variant retained `200` resources and `50 MiB` with no close calls.
The deterministic live-reference, close-count, and reachable-payload
measurements are the primary evidence; RSS is supporting evidence because
JVM/CPython startup and native allocator behavior affect short runs.
### API
This adds an `@Internal` ownership utility and a default materialization
hook to the existing Python resource adapter. It does not change user-facing
resource contracts, and the default method is binary compatible.
### Documentation
- [ ] `doc-needed`
- [x] `doc-not-needed`
- [ ] `doc-included`
### Was this patch authored or co-authored using generative AI tooling?
- [x] Yes
- [ ] No
Generated-by: Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
