[ https://issues.apache.org/jira/browse/APEXMALHAR-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Yan updated APEXMALHAR-2191: ---------------------------------- Description: WindowedKeyedStorage is basically a Map<Window, Map<K, V>>, and we need the capability of getting all K's given a window, and getting V given a window and K. Currently, the spillable implementation of WindowedKeyedStorage uses two spillable data structures -- SpillableByteMapImpl<Pair<Window, K>, V> and SpillableArrayListMultimapImpl<Window, K>. This will not work because we need to be able to remove a key from a window, which SpillableArrayList does not support, and having two separate spillable data structures if it can be achieved by just one should be avoided in general. We will implement the solution that supports prefix scanning (given a window, return all keys), which requires the keys to be stored in order in managed state. Traversing all keys in the spillable map is needed by WindowStateMap. The WindowedOperator needs a way to traverse all windows in its state when firing a trigger. However, this is less urgent since the Window meta info is small and should fit in memory even with millions of windows. This is more for the checkpointing efficiency. was:This is needed by the spillable implementation of WindowStateMap. The WindowedOperator needs a way to traverse all windows in its state. > Implement a way to get all keys with or without prefix from > SpillableByteMapImpl > -------------------------------------------------------------------------------- > > Key: APEXMALHAR-2191 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2191 > Project: Apache Apex Malhar > Issue Type: Sub-task > Reporter: David Yan > Assignee: David Yan > Priority: Critical > > WindowedKeyedStorage is basically a Map<Window, Map<K, V>>, and we need the > capability of getting all K's given a window, and getting V given a window > and K. > Currently, the spillable implementation of WindowedKeyedStorage uses two > spillable data structures -- SpillableByteMapImpl<Pair<Window, K>, V> and > SpillableArrayListMultimapImpl<Window, K>. This will not work because we > need to be able to remove a key from a window, which SpillableArrayList does > not support, and having two separate spillable data structures if it can be > achieved by just one should be avoided in general. > We will implement the solution that supports prefix scanning (given a window, > return all keys), which requires the keys to be stored in order in managed > state. > Traversing all keys in the spillable map is needed by WindowStateMap. The > WindowedOperator needs a way to traverse all windows in its state when firing > a trigger. However, this is less urgent since the Window meta info is small > and should fit in memory even with millions of windows. This is more for the > checkpointing efficiency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)