wenjin272 commented on code in PR #984:
URL: https://github.com/apache/flink-agents/pull/984#discussion_r3763289970
##########
python/flink_agents/e2e_tests/e2e_tests_integration/long_term_memory_test.py:
##########
@@ -303,13 +303,45 @@ def check_result(*, result_dir: Path) -> None:
for record in actual_result:
records[f"{record.name}.{record.count}"] = record
+ assert "alice.2" in records, f"missing alice.2; got {sorted(records)}"
items = records["alice.2"].items
- # LLMs may treat different review comments as updates to the same
- # fact or as distinct facts.
- assert len(items) == 1
- item: MemorySetItem = items[-1]
- assert item.created_at < item.updated_at
- assert "bananas" in item.value
+ # The extraction model decides whether alice's two facts collapse into a
+ # single item or stay separate, so the item count is not fixed. bob's set
+ # is reported alongside a failure to tell a per-key miss from a store-wide
+ # one.
+ bob_items = records["bob.2"].items if "bob.2" in records else None
Review Comment:
`bob.2` is optional here, so a missing output record, `items is None`, and
an empty item list are all normalized into “no leaked values.” The new
reverse-direction check can therefore pass without ever observing the Bob
memory set. I verified that the current `check_result` passes both when `bob.2`
is absent and when `bob.2.items == []`, provided the Alice set is healthy.
Since this PR is intended to check isolation in both directions—and zero items
are deliberately treated as a failure for Alice—could we assert that `bob.2`
exists and that its item list is non-empty before scanning it for Alice facts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
