xintongsong commented on code in PR #466: URL: https://github.com/apache/flink-agents/pull/466#discussion_r2725089478
########## docs/content/docs/development/memory/sensory_and_short_term_memory.md: ########## @@ -0,0 +1,253 @@ +--- +title: Sensory & Short-Term Memory +weight: 2 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview +### Sensory Memory + +Sensory memory is a temporary storage mechanism in Flink Agents designed for data that only needs to persist during a single agent run. + +Sensory memory will be auto cleaned after an agent run finished, which means isolation between agent runs. It provides a convenient way to store intermediate results, tool call contexts, and other temporary data without the overhead of persistence across multiple runs. + +### Short-Term Memory + +Short-Term Memory is shared across all actions within an agent run, and multiple agent runs with the same input key. This corresponds to Flink’s Keyed State, which is visible to processing of multiple records within the same keyed partition, and is not visible to processing of data in other keyed partitions. + +## When to Use + +### Sensory Memory +Sensory Memory is ideal for: + +- **Intermediate results and temporary information** which will be used later by other actions in the same agent run. +- **Passing data through multiple actions**, reduce unnecessary data copy and serialization. + +{{< hint warning >}} +Do not use Sensory Memory for data that needs to persist across multiple agent runs. Use Short-Term Memory or [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +### Short-Term Memory +Short-Term Memory is ideal for: + +- **Persistent Data**: Data needs to persist across multiple runs. +- **Complete original data retrieval**: User want to retrieve the exact same data they have written to memory. + +{{< hint warning >}} +Short-Term Memory is designed for complete original data retrival. For use case that need get the concise and highly related context, consider using [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +## Data Types & Operations + +Sensory memory and short-term memory have the same data types and operations. They support a hierarchical key-value structure. + +### MemoryObject + +The root of the sensory memory and short-term memory is `MemoryObject`. User can use it to store a series of key-value pairs. + +### Supported Value Types + +The key of the pairs store in `MemoryObject` must be string, and the value can be follow types + +- **Primitive Types**: integer, float, boolean, string +- **Collections**: list, map +- **Java POJOs**: See [Flink POJOs](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#pojos) for details. +- **General Class Types**: Any objects can be serialized by kryo. See [General Class Types](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#general-class-types) for details. +- **Memory Object**: The value can also be a `MemoryObject`, which means user can store nested objects. + +### Read & Write + +{{< tabs "Read & Write" >}} + +{{< tab "Python" >}} +```python +@action(InputEvent) +def process_event(event: InputEvent, ctx: RunnerContext) -> None: + memory: MemoryObject = ctx.sensory_memory # or ctx.short_term_memory + # store primitive + memory.set("primitive", 123) + # store collection + memory.set("collection", [1, 2, 3]) + # store general class types + memory.set("object", LocalPrompt("the test {content}")) + # store memory object + memory.new_object("memory_object") Review Comment: ```suggestion obj1: MemoryObject = memory.new_object("obj1") obj1.set("foo", "bar") ``` ########## docs/content/docs/development/memory/sensory_and_short_term_memory.md: ########## @@ -0,0 +1,253 @@ +--- +title: Sensory & Short-Term Memory +weight: 2 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview +### Sensory Memory + +Sensory memory is a temporary storage mechanism in Flink Agents designed for data that only needs to persist during a single agent run. + +Sensory memory will be auto cleaned after an agent run finished, which means isolation between agent runs. It provides a convenient way to store intermediate results, tool call contexts, and other temporary data without the overhead of persistence across multiple runs. + +### Short-Term Memory + +Short-Term Memory is shared across all actions within an agent run, and multiple agent runs with the same input key. This corresponds to Flink’s Keyed State, which is visible to processing of multiple records within the same keyed partition, and is not visible to processing of data in other keyed partitions. + +## When to Use + +### Sensory Memory +Sensory Memory is ideal for: + +- **Intermediate results and temporary information** which will be used later by other actions in the same agent run. +- **Passing data through multiple actions**, reduce unnecessary data copy and serialization. + +{{< hint warning >}} +Do not use Sensory Memory for data that needs to persist across multiple agent runs. Use Short-Term Memory or [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +### Short-Term Memory +Short-Term Memory is ideal for: + +- **Persistent Data**: Data needs to persist across multiple runs. +- **Complete original data retrieval**: User want to retrieve the exact same data they have written to memory. + +{{< hint warning >}} +Short-Term Memory is designed for complete original data retrival. For use case that need get the concise and highly related context, consider using [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +## Data Types & Operations + +Sensory memory and short-term memory have the same data types and operations. They support a hierarchical key-value structure. + +### MemoryObject + +The root of the sensory memory and short-term memory is `MemoryObject`. User can use it to store a series of key-value pairs. + +### Supported Value Types + +The key of the pairs store in `MemoryObject` must be string, and the value can be follow types + +- **Primitive Types**: integer, float, boolean, string +- **Collections**: list, map +- **Java POJOs**: See [Flink POJOs](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#pojos) for details. +- **General Class Types**: Any objects can be serialized by kryo. See [General Class Types](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#general-class-types) for details. +- **Memory Object**: The value can also be a `MemoryObject`, which means user can store nested objects. + +### Read & Write + +{{< tabs "Read & Write" >}} + +{{< tab "Python" >}} +```python +@action(InputEvent) +def process_event(event: InputEvent, ctx: RunnerContext) -> None: + memory: MemoryObject = ctx.sensory_memory # or ctx.short_term_memory + # store primitive + memory.set("primitive", 123) + # store collection + memory.set("collection", [1, 2, 3]) + # store general class types + memory.set("object", LocalPrompt("the test {content}")) + # store memory object + memory.new_object("memory_object") + + # read values from memory + value1: int = memory.get("primitive") + value2: List[int] = memory.get("collection") + value3: Prompt = memory.get("object") + value4: MemoryObject = memory.get("memory_object") +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +@Action(listenEvents = {InputEvent.class}) +public static void processEvent(InputEvent event, RunnerContext ctx) throws Exception { + MemoryObject memory = ctx.getSensoryMemory(); // ctx.getShortTermMemory(); + // store primitive + memory.set("primitive", 123); + // store collection + memory.set("collection", List.of(1, 2, 3)); + // store pojo + memory.set("pojo", new WordWithCount("hello", "1")) + // store general class types + memory.set("object", new LocalPrompt("the test {content}")); + // store memory object + memory.newObject("memory_object", true); // The second argument means whether to overwrite existing field if it's not a nested object Review Comment: ```suggestion MemoryObject obj1 = memory.newObject("obj1", true); // The second argument means whether to overwrite existing field if it's not a nested object obj1.set("foo", "bar"); ``` ########## docs/content/docs/development/memory/sensory_and_short_term_memory.md: ########## @@ -0,0 +1,253 @@ +--- +title: Sensory & Short-Term Memory +weight: 2 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview +### Sensory Memory + +Sensory memory is a temporary storage mechanism in Flink Agents designed for data that only needs to persist during a single agent run. + +Sensory memory will be auto cleaned after an agent run finished, which means isolation between agent runs. It provides a convenient way to store intermediate results, tool call contexts, and other temporary data without the overhead of persistence across multiple runs. + +### Short-Term Memory + +Short-Term Memory is shared across all actions within an agent run, and multiple agent runs with the same input key. This corresponds to Flink’s Keyed State, which is visible to processing of multiple records within the same keyed partition, and is not visible to processing of data in other keyed partitions. + +## When to Use + +### Sensory Memory +Sensory Memory is ideal for: + +- **Intermediate results and temporary information** which will be used later by other actions in the same agent run. +- **Passing data through multiple actions**, reduce unnecessary data copy and serialization. + +{{< hint warning >}} +Do not use Sensory Memory for data that needs to persist across multiple agent runs. Use Short-Term Memory or [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +### Short-Term Memory +Short-Term Memory is ideal for: + +- **Persistent Data**: Data needs to persist across multiple runs. +- **Complete original data retrieval**: User want to retrieve the exact same data they have written to memory. + +{{< hint warning >}} +Short-Term Memory is designed for complete original data retrival. For use case that need get the concise and highly related context, consider using [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +## Data Types & Operations + +Sensory memory and short-term memory have the same data types and operations. They support a hierarchical key-value structure. + +### MemoryObject + +The root of the sensory memory and short-term memory is `MemoryObject`. User can use it to store a series of key-value pairs. + +### Supported Value Types + +The key of the pairs store in `MemoryObject` must be string, and the value can be follow types + +- **Primitive Types**: integer, float, boolean, string +- **Collections**: list, map +- **Java POJOs**: See [Flink POJOs](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#pojos) for details. +- **General Class Types**: Any objects can be serialized by kryo. See [General Class Types](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#general-class-types) for details. +- **Memory Object**: The value can also be a `MemoryObject`, which means user can store nested objects. + +### Read & Write + +{{< tabs "Read & Write" >}} + +{{< tab "Python" >}} +```python +@action(InputEvent) +def process_event(event: InputEvent, ctx: RunnerContext) -> None: + memory: MemoryObject = ctx.sensory_memory # or ctx.short_term_memory + # store primitive + memory.set("primitive", 123) + # store collection + memory.set("collection", [1, 2, 3]) + # store general class types + memory.set("object", LocalPrompt("the test {content}")) + # store memory object + memory.new_object("memory_object") + + # read values from memory + value1: int = memory.get("primitive") + value2: List[int] = memory.get("collection") + value3: Prompt = memory.get("object") + value4: MemoryObject = memory.get("memory_object") +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +@Action(listenEvents = {InputEvent.class}) +public static void processEvent(InputEvent event, RunnerContext ctx) throws Exception { + MemoryObject memory = ctx.getSensoryMemory(); // ctx.getShortTermMemory(); + // store primitive + memory.set("primitive", 123); + // store collection + memory.set("collection", List.of(1, 2, 3)); + // store pojo + memory.set("pojo", new WordWithCount("hello", "1")) + // store general class types + memory.set("object", new LocalPrompt("the test {content}")); + // store memory object + memory.newObject("memory_object", true); // The second argument means whether to overwrite existing field if it's not a nested object Review Comment: I think we should add an overload method that does not require the second argument and use `false` for default (aligned with python), and use that for the example. ########## docs/content/docs/development/memory/sensory_and_short_term_memory.md: ########## @@ -0,0 +1,253 @@ +--- +title: Sensory & Short-Term Memory +weight: 2 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview +### Sensory Memory + +Sensory memory is a temporary storage mechanism in Flink Agents designed for data that only needs to persist during a single agent run. + +Sensory memory will be auto cleaned after an agent run finished, which means isolation between agent runs. It provides a convenient way to store intermediate results, tool call contexts, and other temporary data without the overhead of persistence across multiple runs. + +### Short-Term Memory + +Short-Term Memory is shared across all actions within an agent run, and multiple agent runs with the same input key. This corresponds to Flink’s Keyed State, which is visible to processing of multiple records within the same keyed partition, and is not visible to processing of data in other keyed partitions. + +## When to Use + +### Sensory Memory +Sensory Memory is ideal for: + +- **Intermediate results and temporary information** which will be used later by other actions in the same agent run. +- **Passing data through multiple actions**, reduce unnecessary data copy and serialization. + +{{< hint warning >}} +Do not use Sensory Memory for data that needs to persist across multiple agent runs. Use Short-Term Memory or [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +### Short-Term Memory +Short-Term Memory is ideal for: + +- **Persistent Data**: Data needs to persist across multiple runs. +- **Complete original data retrieval**: User want to retrieve the exact same data they have written to memory. + +{{< hint warning >}} +Short-Term Memory is designed for complete original data retrival. For use case that need get the concise and highly related context, consider using [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +## Data Types & Operations + +Sensory memory and short-term memory have the same data types and operations. They support a hierarchical key-value structure. + +### MemoryObject + +The root of the sensory memory and short-term memory is `MemoryObject`. User can use it to store a series of key-value pairs. + +### Supported Value Types + +The key of the pairs store in `MemoryObject` must be string, and the value can be follow types + +- **Primitive Types**: integer, float, boolean, string +- **Collections**: list, map +- **Java POJOs**: See [Flink POJOs](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#pojos) for details. +- **General Class Types**: Any objects can be serialized by kryo. See [General Class Types](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#general-class-types) for details. +- **Memory Object**: The value can also be a `MemoryObject`, which means user can store nested objects. + +### Read & Write + +{{< tabs "Read & Write" >}} + +{{< tab "Python" >}} +```python +@action(InputEvent) +def process_event(event: InputEvent, ctx: RunnerContext) -> None: + memory: MemoryObject = ctx.sensory_memory # or ctx.short_term_memory + # store primitive + memory.set("primitive", 123) + # store collection + memory.set("collection", [1, 2, 3]) + # store general class types + memory.set("object", LocalPrompt("the test {content}")) + # store memory object + memory.new_object("memory_object") + + # read values from memory + value1: int = memory.get("primitive") + value2: List[int] = memory.get("collection") + value3: Prompt = memory.get("object") + value4: MemoryObject = memory.get("memory_object") +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +@Action(listenEvents = {InputEvent.class}) +public static void processEvent(InputEvent event, RunnerContext ctx) throws Exception { + MemoryObject memory = ctx.getSensoryMemory(); // ctx.getShortTermMemory(); + // store primitive + memory.set("primitive", 123); + // store collection + memory.set("collection", List.of(1, 2, 3)); + // store pojo + memory.set("pojo", new WordWithCount("hello", "1")) + // store general class types + memory.set("object", new LocalPrompt("the test {content}")); + // store memory object + memory.newObject("memory_object", true); // The second argument means whether to overwrite existing field if it's not a nested object + + // read values from memory + int value1 = (int) memory.get("primitive").getValue(); + List<Integer> value2 = (List<Integer>) memory.get("collection").getValue(); + WordWithCount value3 = (WordWithCount) memory.get("pojo").getValue(); + Prompt value4 = (Prompt) memory.get("object").getValue(); + MemoryObject value5 = memory.get("memory_object"); Review Comment: ```suggestion MemoryObject value5 = memory.get("obj1"); String value6 = value5.get("foo"); ``` ########## docs/content/docs/development/memory/sensory_and_short_term_memory.md: ########## @@ -0,0 +1,253 @@ +--- +title: Sensory & Short-Term Memory +weight: 2 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview +### Sensory Memory + +Sensory memory is a temporary storage mechanism in Flink Agents designed for data that only needs to persist during a single agent run. + +Sensory memory will be auto cleaned after an agent run finished, which means isolation between agent runs. It provides a convenient way to store intermediate results, tool call contexts, and other temporary data without the overhead of persistence across multiple runs. + +### Short-Term Memory + +Short-Term Memory is shared across all actions within an agent run, and multiple agent runs with the same input key. This corresponds to Flink’s Keyed State, which is visible to processing of multiple records within the same keyed partition, and is not visible to processing of data in other keyed partitions. + +## When to Use + +### Sensory Memory +Sensory Memory is ideal for: + +- **Intermediate results and temporary information** which will be used later by other actions in the same agent run. +- **Passing data through multiple actions**, reduce unnecessary data copy and serialization. + +{{< hint warning >}} +Do not use Sensory Memory for data that needs to persist across multiple agent runs. Use Short-Term Memory or [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +### Short-Term Memory +Short-Term Memory is ideal for: + +- **Persistent Data**: Data needs to persist across multiple runs. +- **Complete original data retrieval**: User want to retrieve the exact same data they have written to memory. + +{{< hint warning >}} +Short-Term Memory is designed for complete original data retrival. For use case that need get the concise and highly related context, consider using [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +## Data Types & Operations + +Sensory memory and short-term memory have the same data types and operations. They support a hierarchical key-value structure. + +### MemoryObject + +The root of the sensory memory and short-term memory is `MemoryObject`. User can use it to store a series of key-value pairs. + +### Supported Value Types + +The key of the pairs store in `MemoryObject` must be string, and the value can be follow types + +- **Primitive Types**: integer, float, boolean, string +- **Collections**: list, map +- **Java POJOs**: See [Flink POJOs](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#pojos) for details. +- **General Class Types**: Any objects can be serialized by kryo. See [General Class Types](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#general-class-types) for details. +- **Memory Object**: The value can also be a `MemoryObject`, which means user can store nested objects. + +### Read & Write + +{{< tabs "Read & Write" >}} + +{{< tab "Python" >}} +```python +@action(InputEvent) +def process_event(event: InputEvent, ctx: RunnerContext) -> None: + memory: MemoryObject = ctx.sensory_memory # or ctx.short_term_memory + # store primitive + memory.set("primitive", 123) + # store collection + memory.set("collection", [1, 2, 3]) + # store general class types + memory.set("object", LocalPrompt("the test {content}")) + # store memory object + memory.new_object("memory_object") + + # read values from memory + value1: int = memory.get("primitive") + value2: List[int] = memory.get("collection") + value3: Prompt = memory.get("object") + value4: MemoryObject = memory.get("memory_object") Review Comment: ```suggestion value4: MemoryObject = memory.get("obj1") value5: str = value4.get("foo") ``` ########## docs/content/docs/development/memory/sensory_and_short_term_memory.md: ########## @@ -0,0 +1,253 @@ +--- +title: Sensory & Short-Term Memory +weight: 2 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview +### Sensory Memory + +Sensory memory is a temporary storage mechanism in Flink Agents designed for data that only needs to persist during a single agent run. + +Sensory memory will be auto cleaned after an agent run finished, which means isolation between agent runs. It provides a convenient way to store intermediate results, tool call contexts, and other temporary data without the overhead of persistence across multiple runs. + +### Short-Term Memory + +Short-Term Memory is shared across all actions within an agent run, and multiple agent runs with the same input key. This corresponds to Flink’s Keyed State, which is visible to processing of multiple records within the same keyed partition, and is not visible to processing of data in other keyed partitions. + +## When to Use + +### Sensory Memory +Sensory Memory is ideal for: + +- **Intermediate results and temporary information** which will be used later by other actions in the same agent run. +- **Passing data through multiple actions**, reduce unnecessary data copy and serialization. + +{{< hint warning >}} +Do not use Sensory Memory for data that needs to persist across multiple agent runs. Use Short-Term Memory or [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +### Short-Term Memory +Short-Term Memory is ideal for: + +- **Persistent Data**: Data needs to persist across multiple runs. +- **Complete original data retrieval**: User want to retrieve the exact same data they have written to memory. + +{{< hint warning >}} +Short-Term Memory is designed for complete original data retrival. For use case that need get the concise and highly related context, consider using [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +## Data Types & Operations + +Sensory memory and short-term memory have the same data types and operations. They support a hierarchical key-value structure. + +### MemoryObject + +The root of the sensory memory and short-term memory is `MemoryObject`. User can use it to store a series of key-value pairs. + +### Supported Value Types + +The key of the pairs store in `MemoryObject` must be string, and the value can be follow types + +- **Primitive Types**: integer, float, boolean, string +- **Collections**: list, map +- **Java POJOs**: See [Flink POJOs](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#pojos) for details. +- **General Class Types**: Any objects can be serialized by kryo. See [General Class Types](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#general-class-types) for details. +- **Memory Object**: The value can also be a `MemoryObject`, which means user can store nested objects. + +### Read & Write + +{{< tabs "Read & Write" >}} + +{{< tab "Python" >}} +```python +@action(InputEvent) +def process_event(event: InputEvent, ctx: RunnerContext) -> None: + memory: MemoryObject = ctx.sensory_memory # or ctx.short_term_memory + # store primitive + memory.set("primitive", 123) + # store collection + memory.set("collection", [1, 2, 3]) + # store general class types + memory.set("object", LocalPrompt("the test {content}")) + # store memory object + memory.new_object("memory_object") + + # read values from memory + value1: int = memory.get("primitive") + value2: List[int] = memory.get("collection") + value3: Prompt = memory.get("object") + value4: MemoryObject = memory.get("memory_object") +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +@Action(listenEvents = {InputEvent.class}) +public static void processEvent(InputEvent event, RunnerContext ctx) throws Exception { + MemoryObject memory = ctx.getSensoryMemory(); // ctx.getShortTermMemory(); + // store primitive + memory.set("primitive", 123); + // store collection + memory.set("collection", List.of(1, 2, 3)); + // store pojo + memory.set("pojo", new WordWithCount("hello", "1")) + // store general class types + memory.set("object", new LocalPrompt("the test {content}")); + // store memory object + memory.newObject("memory_object", true); // The second argument means whether to overwrite existing field if it's not a nested object + + // read values from memory + int value1 = (int) memory.get("primitive").getValue(); + List<Integer> value2 = (List<Integer>) memory.get("collection").getValue(); + WordWithCount value3 = (WordWithCount) memory.get("pojo").getValue(); + Prompt value4 = (Prompt) memory.get("object").getValue(); + MemoryObject value5 = memory.get("memory_object"); +} +``` +{{< /tab >}} + +{{< /tabs >}} + +#### Nested Object + +There are two ways to access (get & set) fields of s nested object. Users can choose whichever they like. We don't recommend one over another: +* Use `new_object` to manually create nested field. +* Use dot-separated path as key to created nested field. + +{{< tabs "Nested Object Access" >}} + +{{< tab "Python" >}} +```python +# Use new_object Review Comment: ```suggestion # access fields from the innermost memory object with field names ``` ########## docs/content/docs/development/memory/sensory_and_short_term_memory.md: ########## @@ -0,0 +1,253 @@ +--- +title: Sensory & Short-Term Memory +weight: 2 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview +### Sensory Memory + +Sensory memory is a temporary storage mechanism in Flink Agents designed for data that only needs to persist during a single agent run. + +Sensory memory will be auto cleaned after an agent run finished, which means isolation between agent runs. It provides a convenient way to store intermediate results, tool call contexts, and other temporary data without the overhead of persistence across multiple runs. + +### Short-Term Memory + +Short-Term Memory is shared across all actions within an agent run, and multiple agent runs with the same input key. This corresponds to Flink’s Keyed State, which is visible to processing of multiple records within the same keyed partition, and is not visible to processing of data in other keyed partitions. + +## When to Use + +### Sensory Memory +Sensory Memory is ideal for: + +- **Intermediate results and temporary information** which will be used later by other actions in the same agent run. +- **Passing data through multiple actions**, reduce unnecessary data copy and serialization. + +{{< hint warning >}} +Do not use Sensory Memory for data that needs to persist across multiple agent runs. Use Short-Term Memory or [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +### Short-Term Memory +Short-Term Memory is ideal for: + +- **Persistent Data**: Data needs to persist across multiple runs. +- **Complete original data retrieval**: User want to retrieve the exact same data they have written to memory. + +{{< hint warning >}} +Short-Term Memory is designed for complete original data retrival. For use case that need get the concise and highly related context, consider using [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +## Data Types & Operations + +Sensory memory and short-term memory have the same data types and operations. They support a hierarchical key-value structure. + +### MemoryObject + +The root of the sensory memory and short-term memory is `MemoryObject`. User can use it to store a series of key-value pairs. + +### Supported Value Types + +The key of the pairs store in `MemoryObject` must be string, and the value can be follow types + +- **Primitive Types**: integer, float, boolean, string +- **Collections**: list, map +- **Java POJOs**: See [Flink POJOs](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#pojos) for details. +- **General Class Types**: Any objects can be serialized by kryo. See [General Class Types](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#general-class-types) for details. +- **Memory Object**: The value can also be a `MemoryObject`, which means user can store nested objects. + +### Read & Write + +{{< tabs "Read & Write" >}} + +{{< tab "Python" >}} +```python +@action(InputEvent) +def process_event(event: InputEvent, ctx: RunnerContext) -> None: + memory: MemoryObject = ctx.sensory_memory # or ctx.short_term_memory + # store primitive + memory.set("primitive", 123) + # store collection + memory.set("collection", [1, 2, 3]) + # store general class types + memory.set("object", LocalPrompt("the test {content}")) Review Comment: ```suggestion memory.set("object", Prompt.from_text("the test {content}")) ``` ########## docs/content/docs/development/memory/sensory_and_short_term_memory.md: ########## @@ -0,0 +1,253 @@ +--- +title: Sensory & Short-Term Memory +weight: 2 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview +### Sensory Memory + +Sensory memory is a temporary storage mechanism in Flink Agents designed for data that only needs to persist during a single agent run. + +Sensory memory will be auto cleaned after an agent run finished, which means isolation between agent runs. It provides a convenient way to store intermediate results, tool call contexts, and other temporary data without the overhead of persistence across multiple runs. + +### Short-Term Memory + +Short-Term Memory is shared across all actions within an agent run, and multiple agent runs with the same input key. This corresponds to Flink’s Keyed State, which is visible to processing of multiple records within the same keyed partition, and is not visible to processing of data in other keyed partitions. + +## When to Use + +### Sensory Memory +Sensory Memory is ideal for: + +- **Intermediate results and temporary information** which will be used later by other actions in the same agent run. +- **Passing data through multiple actions**, reduce unnecessary data copy and serialization. + +{{< hint warning >}} +Do not use Sensory Memory for data that needs to persist across multiple agent runs. Use Short-Term Memory or [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +### Short-Term Memory +Short-Term Memory is ideal for: + +- **Persistent Data**: Data needs to persist across multiple runs. +- **Complete original data retrieval**: User want to retrieve the exact same data they have written to memory. + +{{< hint warning >}} +Short-Term Memory is designed for complete original data retrival. For use case that need get the concise and highly related context, consider using [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +## Data Types & Operations + +Sensory memory and short-term memory have the same data types and operations. They support a hierarchical key-value structure. + +### MemoryObject + +The root of the sensory memory and short-term memory is `MemoryObject`. User can use it to store a series of key-value pairs. + +### Supported Value Types + +The key of the pairs store in `MemoryObject` must be string, and the value can be follow types + +- **Primitive Types**: integer, float, boolean, string +- **Collections**: list, map +- **Java POJOs**: See [Flink POJOs](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#pojos) for details. +- **General Class Types**: Any objects can be serialized by kryo. See [General Class Types](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#general-class-types) for details. +- **Memory Object**: The value can also be a `MemoryObject`, which means user can store nested objects. + +### Read & Write + +{{< tabs "Read & Write" >}} + +{{< tab "Python" >}} +```python +@action(InputEvent) +def process_event(event: InputEvent, ctx: RunnerContext) -> None: + memory: MemoryObject = ctx.sensory_memory # or ctx.short_term_memory + # store primitive + memory.set("primitive", 123) + # store collection + memory.set("collection", [1, 2, 3]) + # store general class types + memory.set("object", LocalPrompt("the test {content}")) + # store memory object + memory.new_object("memory_object") + + # read values from memory + value1: int = memory.get("primitive") + value2: List[int] = memory.get("collection") + value3: Prompt = memory.get("object") + value4: MemoryObject = memory.get("memory_object") +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +@Action(listenEvents = {InputEvent.class}) +public static void processEvent(InputEvent event, RunnerContext ctx) throws Exception { + MemoryObject memory = ctx.getSensoryMemory(); // ctx.getShortTermMemory(); + // store primitive + memory.set("primitive", 123); + // store collection + memory.set("collection", List.of(1, 2, 3)); + // store pojo + memory.set("pojo", new WordWithCount("hello", "1")) + // store general class types + memory.set("object", new LocalPrompt("the test {content}")); + // store memory object + memory.newObject("memory_object", true); // The second argument means whether to overwrite existing field if it's not a nested object + + // read values from memory + int value1 = (int) memory.get("primitive").getValue(); + List<Integer> value2 = (List<Integer>) memory.get("collection").getValue(); + WordWithCount value3 = (WordWithCount) memory.get("pojo").getValue(); + Prompt value4 = (Prompt) memory.get("object").getValue(); + MemoryObject value5 = memory.get("memory_object"); +} +``` +{{< /tab >}} + +{{< /tabs >}} + +#### Nested Object + +There are two ways to access (get & set) fields of s nested object. Users can choose whichever they like. We don't recommend one over another: Review Comment: ```suggestion There are two ways to access (get & set) fields of s nested object. Users can choose whichever they like. ``` ########## docs/content/docs/development/memory/sensory_and_short_term_memory.md: ########## @@ -0,0 +1,253 @@ +--- +title: Sensory & Short-Term Memory +weight: 2 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview +### Sensory Memory + +Sensory memory is a temporary storage mechanism in Flink Agents designed for data that only needs to persist during a single agent run. + +Sensory memory will be auto cleaned after an agent run finished, which means isolation between agent runs. It provides a convenient way to store intermediate results, tool call contexts, and other temporary data without the overhead of persistence across multiple runs. + +### Short-Term Memory + +Short-Term Memory is shared across all actions within an agent run, and multiple agent runs with the same input key. This corresponds to Flink’s Keyed State, which is visible to processing of multiple records within the same keyed partition, and is not visible to processing of data in other keyed partitions. + +## When to Use + +### Sensory Memory +Sensory Memory is ideal for: + +- **Intermediate results and temporary information** which will be used later by other actions in the same agent run. +- **Passing data through multiple actions**, reduce unnecessary data copy and serialization. + +{{< hint warning >}} +Do not use Sensory Memory for data that needs to persist across multiple agent runs. Use Short-Term Memory or [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +### Short-Term Memory +Short-Term Memory is ideal for: + +- **Persistent Data**: Data needs to persist across multiple runs. +- **Complete original data retrieval**: User want to retrieve the exact same data they have written to memory. + +{{< hint warning >}} +Short-Term Memory is designed for complete original data retrival. For use case that need get the concise and highly related context, consider using [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +## Data Types & Operations + +Sensory memory and short-term memory have the same data types and operations. They support a hierarchical key-value structure. + +### MemoryObject + +The root of the sensory memory and short-term memory is `MemoryObject`. User can use it to store a series of key-value pairs. + +### Supported Value Types + +The key of the pairs store in `MemoryObject` must be string, and the value can be follow types + +- **Primitive Types**: integer, float, boolean, string +- **Collections**: list, map +- **Java POJOs**: See [Flink POJOs](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#pojos) for details. +- **General Class Types**: Any objects can be serialized by kryo. See [General Class Types](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#general-class-types) for details. +- **Memory Object**: The value can also be a `MemoryObject`, which means user can store nested objects. + +### Read & Write + +{{< tabs "Read & Write" >}} + +{{< tab "Python" >}} +```python +@action(InputEvent) +def process_event(event: InputEvent, ctx: RunnerContext) -> None: + memory: MemoryObject = ctx.sensory_memory # or ctx.short_term_memory + # store primitive + memory.set("primitive", 123) + # store collection + memory.set("collection", [1, 2, 3]) + # store general class types + memory.set("object", LocalPrompt("the test {content}")) + # store memory object + memory.new_object("memory_object") + + # read values from memory + value1: int = memory.get("primitive") + value2: List[int] = memory.get("collection") + value3: Prompt = memory.get("object") + value4: MemoryObject = memory.get("memory_object") +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +@Action(listenEvents = {InputEvent.class}) +public static void processEvent(InputEvent event, RunnerContext ctx) throws Exception { + MemoryObject memory = ctx.getSensoryMemory(); // ctx.getShortTermMemory(); + // store primitive + memory.set("primitive", 123); + // store collection + memory.set("collection", List.of(1, 2, 3)); + // store pojo + memory.set("pojo", new WordWithCount("hello", "1")) + // store general class types + memory.set("object", new LocalPrompt("the test {content}")); + // store memory object + memory.newObject("memory_object", true); // The second argument means whether to overwrite existing field if it's not a nested object + + // read values from memory + int value1 = (int) memory.get("primitive").getValue(); + List<Integer> value2 = (List<Integer>) memory.get("collection").getValue(); + WordWithCount value3 = (WordWithCount) memory.get("pojo").getValue(); + Prompt value4 = (Prompt) memory.get("object").getValue(); + MemoryObject value5 = memory.get("memory_object"); +} +``` +{{< /tab >}} + +{{< /tabs >}} + +#### Nested Object + +There are two ways to access (get & set) fields of s nested object. Users can choose whichever they like. We don't recommend one over another: +* Use `new_object` to manually create nested field. +* Use dot-separated path as key to created nested field. + +{{< tabs "Nested Object Access" >}} + +{{< tab "Python" >}} +```python +# Use new_object +user: MemoryObject = memory.new_object("user") +user.set("name", "john") +user.set("age", 13) + +user: MemoryObject = memory.get_object("user") +name: str = user.get("name") +age: int = user.get("age") + +# Use dot-separated path +memory.set("user.name", "jhon") +memory.set("user.age", 13) + +name: str = memory.get("user.name") +age: int = memory.get("user.age") +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +// Use new_object +MemoryObject user = memory.newObject("user", true); +user.set("name", "john"); +user.set("age", 13); + +user = sensoryMemory.get("user"); +String name = (String) user.get("name").getValue(); +int age = (int) user.get("age").getValue(); + +// Use dot-separated path +sensoryMemory.set("user.name", "john"); +sensoryMemory.set("user.age", 13); + +name = (String) sensoryMemory.get("user.name").getValue(); +age = (int) sensoryMemory.get("user.age").getValue(); +``` +{{< /tab >}} + +{{< /tabs >}} + +### Memory Reference + +`MemoryRef` is a reference of the objects stored in memory. The `set` method memory object will return a `MemoryRef`. Review Comment: ```suggestion `MemoryRef` is a reference of the objects stored in memory. The `set` method of memory object will return a `MemoryRef`. ``` ########## docs/content/docs/development/memory/sensory_and_short_term_memory.md: ########## @@ -0,0 +1,253 @@ +--- +title: Sensory & Short-Term Memory +weight: 2 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview +### Sensory Memory + +Sensory memory is a temporary storage mechanism in Flink Agents designed for data that only needs to persist during a single agent run. + +Sensory memory will be auto cleaned after an agent run finished, which means isolation between agent runs. It provides a convenient way to store intermediate results, tool call contexts, and other temporary data without the overhead of persistence across multiple runs. + +### Short-Term Memory + +Short-Term Memory is shared across all actions within an agent run, and multiple agent runs with the same input key. This corresponds to Flink’s Keyed State, which is visible to processing of multiple records within the same keyed partition, and is not visible to processing of data in other keyed partitions. + +## When to Use + +### Sensory Memory +Sensory Memory is ideal for: + +- **Intermediate results and temporary information** which will be used later by other actions in the same agent run. +- **Passing data through multiple actions**, reduce unnecessary data copy and serialization. + +{{< hint warning >}} +Do not use Sensory Memory for data that needs to persist across multiple agent runs. Use Short-Term Memory or [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +### Short-Term Memory +Short-Term Memory is ideal for: + +- **Persistent Data**: Data needs to persist across multiple runs. +- **Complete original data retrieval**: User want to retrieve the exact same data they have written to memory. + +{{< hint warning >}} +Short-Term Memory is designed for complete original data retrival. For use case that need get the concise and highly related context, consider using [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +## Data Types & Operations + +Sensory memory and short-term memory have the same data types and operations. They support a hierarchical key-value structure. + +### MemoryObject + +The root of the sensory memory and short-term memory is `MemoryObject`. User can use it to store a series of key-value pairs. + +### Supported Value Types + +The key of the pairs store in `MemoryObject` must be string, and the value can be follow types + +- **Primitive Types**: integer, float, boolean, string +- **Collections**: list, map +- **Java POJOs**: See [Flink POJOs](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#pojos) for details. +- **General Class Types**: Any objects can be serialized by kryo. See [General Class Types](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#general-class-types) for details. +- **Memory Object**: The value can also be a `MemoryObject`, which means user can store nested objects. + +### Read & Write + +{{< tabs "Read & Write" >}} + +{{< tab "Python" >}} +```python +@action(InputEvent) +def process_event(event: InputEvent, ctx: RunnerContext) -> None: + memory: MemoryObject = ctx.sensory_memory # or ctx.short_term_memory + # store primitive + memory.set("primitive", 123) + # store collection + memory.set("collection", [1, 2, 3]) + # store general class types + memory.set("object", LocalPrompt("the test {content}")) + # store memory object + memory.new_object("memory_object") + + # read values from memory + value1: int = memory.get("primitive") + value2: List[int] = memory.get("collection") + value3: Prompt = memory.get("object") + value4: MemoryObject = memory.get("memory_object") +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +@Action(listenEvents = {InputEvent.class}) +public static void processEvent(InputEvent event, RunnerContext ctx) throws Exception { + MemoryObject memory = ctx.getSensoryMemory(); // ctx.getShortTermMemory(); + // store primitive + memory.set("primitive", 123); + // store collection + memory.set("collection", List.of(1, 2, 3)); + // store pojo + memory.set("pojo", new WordWithCount("hello", "1")) + // store general class types + memory.set("object", new LocalPrompt("the test {content}")); + // store memory object + memory.newObject("memory_object", true); // The second argument means whether to overwrite existing field if it's not a nested object + + // read values from memory + int value1 = (int) memory.get("primitive").getValue(); + List<Integer> value2 = (List<Integer>) memory.get("collection").getValue(); + WordWithCount value3 = (WordWithCount) memory.get("pojo").getValue(); + Prompt value4 = (Prompt) memory.get("object").getValue(); + MemoryObject value5 = memory.get("memory_object"); +} +``` +{{< /tab >}} + +{{< /tabs >}} + +#### Nested Object + +There are two ways to access (get & set) fields of s nested object. Users can choose whichever they like. We don't recommend one over another: +* Use `new_object` to manually create nested field. +* Use dot-separated path as key to created nested field. Review Comment: This doesn't seem right. "Always using `new_object` for creating memory objects" is something we need to emphasis, but it's not a way of accessing the fields. ########## docs/content/docs/development/memory/sensory_and_short_term_memory.md: ########## @@ -0,0 +1,253 @@ +--- +title: Sensory & Short-Term Memory +weight: 2 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview +### Sensory Memory + +Sensory memory is a temporary storage mechanism in Flink Agents designed for data that only needs to persist during a single agent run. + +Sensory memory will be auto cleaned after an agent run finished, which means isolation between agent runs. It provides a convenient way to store intermediate results, tool call contexts, and other temporary data without the overhead of persistence across multiple runs. + +### Short-Term Memory + +Short-Term Memory is shared across all actions within an agent run, and multiple agent runs with the same input key. This corresponds to Flink’s Keyed State, which is visible to processing of multiple records within the same keyed partition, and is not visible to processing of data in other keyed partitions. + +## When to Use + +### Sensory Memory +Sensory Memory is ideal for: + +- **Intermediate results and temporary information** which will be used later by other actions in the same agent run. +- **Passing data through multiple actions**, reduce unnecessary data copy and serialization. + +{{< hint warning >}} +Do not use Sensory Memory for data that needs to persist across multiple agent runs. Use Short-Term Memory or [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +### Short-Term Memory +Short-Term Memory is ideal for: + +- **Persistent Data**: Data needs to persist across multiple runs. +- **Complete original data retrieval**: User want to retrieve the exact same data they have written to memory. + +{{< hint warning >}} +Short-Term Memory is designed for complete original data retrival. For use case that need get the concise and highly related context, consider using [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +## Data Types & Operations + +Sensory memory and short-term memory have the same data types and operations. They support a hierarchical key-value structure. + +### MemoryObject + +The root of the sensory memory and short-term memory is `MemoryObject`. User can use it to store a series of key-value pairs. + +### Supported Value Types + +The key of the pairs store in `MemoryObject` must be string, and the value can be follow types + +- **Primitive Types**: integer, float, boolean, string +- **Collections**: list, map +- **Java POJOs**: See [Flink POJOs](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#pojos) for details. +- **General Class Types**: Any objects can be serialized by kryo. See [General Class Types](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#general-class-types) for details. +- **Memory Object**: The value can also be a `MemoryObject`, which means user can store nested objects. + +### Read & Write + +{{< tabs "Read & Write" >}} + +{{< tab "Python" >}} +```python +@action(InputEvent) +def process_event(event: InputEvent, ctx: RunnerContext) -> None: + memory: MemoryObject = ctx.sensory_memory # or ctx.short_term_memory + # store primitive + memory.set("primitive", 123) + # store collection + memory.set("collection", [1, 2, 3]) + # store general class types + memory.set("object", LocalPrompt("the test {content}")) + # store memory object + memory.new_object("memory_object") + + # read values from memory + value1: int = memory.get("primitive") + value2: List[int] = memory.get("collection") + value3: Prompt = memory.get("object") + value4: MemoryObject = memory.get("memory_object") +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +@Action(listenEvents = {InputEvent.class}) +public static void processEvent(InputEvent event, RunnerContext ctx) throws Exception { + MemoryObject memory = ctx.getSensoryMemory(); // ctx.getShortTermMemory(); + // store primitive + memory.set("primitive", 123); + // store collection + memory.set("collection", List.of(1, 2, 3)); + // store pojo + memory.set("pojo", new WordWithCount("hello", "1")) + // store general class types + memory.set("object", new LocalPrompt("the test {content}")); + // store memory object + memory.newObject("memory_object", true); // The second argument means whether to overwrite existing field if it's not a nested object + + // read values from memory + int value1 = (int) memory.get("primitive").getValue(); + List<Integer> value2 = (List<Integer>) memory.get("collection").getValue(); + WordWithCount value3 = (WordWithCount) memory.get("pojo").getValue(); + Prompt value4 = (Prompt) memory.get("object").getValue(); + MemoryObject value5 = memory.get("memory_object"); +} +``` +{{< /tab >}} + +{{< /tabs >}} + +#### Nested Object + +There are two ways to access (get & set) fields of s nested object. Users can choose whichever they like. We don't recommend one over another: +* Use `new_object` to manually create nested field. +* Use dot-separated path as key to created nested field. Review Comment: ```suggestion * Use `new_object` to manually create nested field. * Use dot-separated path as key to access nested field. ``` ########## docs/content/docs/development/memory/sensory_and_short_term_memory.md: ########## @@ -0,0 +1,253 @@ +--- +title: Sensory & Short-Term Memory +weight: 2 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +## Overview +### Sensory Memory + +Sensory memory is a temporary storage mechanism in Flink Agents designed for data that only needs to persist during a single agent run. + +Sensory memory will be auto cleaned after an agent run finished, which means isolation between agent runs. It provides a convenient way to store intermediate results, tool call contexts, and other temporary data without the overhead of persistence across multiple runs. + +### Short-Term Memory + +Short-Term Memory is shared across all actions within an agent run, and multiple agent runs with the same input key. This corresponds to Flink’s Keyed State, which is visible to processing of multiple records within the same keyed partition, and is not visible to processing of data in other keyed partitions. + +## When to Use + +### Sensory Memory +Sensory Memory is ideal for: + +- **Intermediate results and temporary information** which will be used later by other actions in the same agent run. +- **Passing data through multiple actions**, reduce unnecessary data copy and serialization. + +{{< hint warning >}} +Do not use Sensory Memory for data that needs to persist across multiple agent runs. Use Short-Term Memory or [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +### Short-Term Memory +Short-Term Memory is ideal for: + +- **Persistent Data**: Data needs to persist across multiple runs. +- **Complete original data retrieval**: User want to retrieve the exact same data they have written to memory. + +{{< hint warning >}} +Short-Term Memory is designed for complete original data retrival. For use case that need get the concise and highly related context, consider using [Long-Term Memory]({{< ref "docs/development/memory/long_term_memory" >}}) instead. +{{< /hint >}} + +## Data Types & Operations + +Sensory memory and short-term memory have the same data types and operations. They support a hierarchical key-value structure. + +### MemoryObject + +The root of the sensory memory and short-term memory is `MemoryObject`. User can use it to store a series of key-value pairs. + +### Supported Value Types + +The key of the pairs store in `MemoryObject` must be string, and the value can be follow types + +- **Primitive Types**: integer, float, boolean, string +- **Collections**: list, map +- **Java POJOs**: See [Flink POJOs](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#pojos) for details. +- **General Class Types**: Any objects can be serialized by kryo. See [General Class Types](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#general-class-types) for details. +- **Memory Object**: The value can also be a `MemoryObject`, which means user can store nested objects. + +### Read & Write + +{{< tabs "Read & Write" >}} + +{{< tab "Python" >}} +```python +@action(InputEvent) +def process_event(event: InputEvent, ctx: RunnerContext) -> None: + memory: MemoryObject = ctx.sensory_memory # or ctx.short_term_memory + # store primitive + memory.set("primitive", 123) + # store collection + memory.set("collection", [1, 2, 3]) + # store general class types + memory.set("object", LocalPrompt("the test {content}")) + # store memory object + memory.new_object("memory_object") + + # read values from memory + value1: int = memory.get("primitive") + value2: List[int] = memory.get("collection") + value3: Prompt = memory.get("object") + value4: MemoryObject = memory.get("memory_object") +``` +{{< /tab >}} + +{{< tab "Java" >}} +```java +@Action(listenEvents = {InputEvent.class}) +public static void processEvent(InputEvent event, RunnerContext ctx) throws Exception { + MemoryObject memory = ctx.getSensoryMemory(); // ctx.getShortTermMemory(); + // store primitive + memory.set("primitive", 123); + // store collection + memory.set("collection", List.of(1, 2, 3)); + // store pojo + memory.set("pojo", new WordWithCount("hello", "1")) + // store general class types + memory.set("object", new LocalPrompt("the test {content}")); + // store memory object + memory.newObject("memory_object", true); // The second argument means whether to overwrite existing field if it's not a nested object + + // read values from memory + int value1 = (int) memory.get("primitive").getValue(); + List<Integer> value2 = (List<Integer>) memory.get("collection").getValue(); + WordWithCount value3 = (WordWithCount) memory.get("pojo").getValue(); + Prompt value4 = (Prompt) memory.get("object").getValue(); + MemoryObject value5 = memory.get("memory_object"); +} +``` +{{< /tab >}} + +{{< /tabs >}} + +#### Nested Object + +There are two ways to access (get & set) fields of s nested object. Users can choose whichever they like. We don't recommend one over another: +* Use `new_object` to manually create nested field. +* Use dot-separated path as key to created nested field. + +{{< tabs "Nested Object Access" >}} + +{{< tab "Python" >}} +```python +# Use new_object +user: MemoryObject = memory.new_object("user") +user.set("name", "john") +user.set("age", 13) + +user: MemoryObject = memory.get_object("user") +name: str = user.get("name") +age: int = user.get("age") + +# Use dot-separated path Review Comment: ```suggestion # access fields from an outer memory object with paths to the fields ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
