weiqingy opened a new pull request, #984: URL: https://github.com/apache/flink-agents/pull/984
Linked issue: #983 ### Purpose of change `test_long_term_memory_async_execution_in_action` asserts `len(items) == 1`, which pins how many items mem0 stores after alice's two inputs. Whether those two facts collapse into one item or stay separate is the extraction model's decision, so the count is not stable. It has been observed at 0, 1, 2, and 4 across four CI failures in three weeks, on `main`, `release-0.2`, and a feature branch. The comment above the assertion already conceded the variability. This asserts on retrievable content instead of an exact count. The empty case is deliberately kept as a failure rather than tolerated. Zero stored items means neither add landed, which may be a real defect in the add path rather than model variance, and it is the only one of the four observed outcomes that carries information. To make the next occurrence diagnosable, bob's memory set is reported alongside the failure: if bob has items and alice does not, the miss is per-key; if both are empty, extraction or the store failed wholesale. That matters because the e2e arm runs with `log_cli_level=OFF` and pytest-rerunfailures reports only the final attempt, so today an occurrence leaves no trace. Two related changes come with it. The `created_at < updated_at` assertion is dropped. On a plain add mem0 sets `updated_at` equal to `created_at`, so the strict comparison holds only when the model picked the update branch. That made it a second instance of the same flake, and it can also fail on a legitimate delete-then-add even when exactly one item survives. An explicit cross-key leak check is added. The exact count was incidentally covering this: alice and bob share a job id and a memory set name, so `agent_id` is the only isolation boundary between them, and a scoping break would have surfaced as an inflated count. Asserting that bob's facts never appear in alice's set restores that coverage without depending on a count. The content match is case-insensitive because the stored text is the model's paraphrase rather than the input, so a title-cased "Bananas" would otherwise fail for the same reason the count does. ### Tests Test-only change. The modified assertions were exercised against the observed and adjacent outcomes: | Case | Result | |---|---| | 0 items | fails, with bob's set in the message | | 1 item, consolidated | passes | | 2 items, kept separate | passes | | 4 items, kept separate | passes | | single item, title-cased "Bananas" | passes | | stale read, only the watermelon fact | fails | | bob's facts present in alice's set | fails | The first four rows are the counts seen in CI. The last three are regressions that must still fail. `ruff check` and `ruff format --check` pass on the file. The test is gated on `ACTION_API_KEY` and needs a Flink cluster plus a local Ollama embedding model, so it does not run locally without those. Verification of the end-to-end path comes from CI. ### API No public API change. ### Documentation - [ ] `doc-needed` - [x] `doc-not-needed` - [ ] `doc-included` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
