wenjin272 commented on issue #1066: URL: https://github.com/apache/flink-agents/issues/1066#issuecomment-5460757763
Thanks for raising this. I think replay is a valuable direction for Flink Agents. It could help users evaluate an agent against realistic state and determine whether changes to prompts, models, parameters, or other components actually improve its behavior. Flink Agents is particularly well positioned for this because durable, stateful execution is already one of its core capabilities. The community has considered this before—for example, the 0.3 brainstorming included the question: “How to compare the performance of two different versions (prompt, model, parameter, ...)?” However, this is a broad topic and has not yet been a near-term priority, so the community has not started working on it. I’m glad to see this brought up again. Before moving toward implementation, I think we should open a Discussion to align on the design, use cases, and an appropriately focused initial scope. With the 0.4 code freeze approaching on September 15, the community will likely prioritize the items already planned for 0.4 in the near term #862, but the Discussion can help us shape this direction for future work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
