[jira] [Updated] (APEXMALHAR-2191) Implement a way to get all keys with or without prefix from SpillableByteMapImpl

[David Yan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Yan updated APEXMALHAR-2191:
--
Assignee: (was: David Yan)

> Implement a way to get all keys with or without prefix from 
> SpillableByteMapImpl
> 
>
> Key: APEXMALHAR-2191
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2191
> Project: Apache Apex Malhar
>  Issue Type: Sub-task
>Reporter: David Yan
>Priority: Critical
>
> WindowedKeyedStorage is basically a Map>, and we need the 
> capability of getting all K's given a window, and getting V given a window 
> and K.
> Currently, the spillable implementation of WindowedKeyedStorage uses two 
> spillable data structures -- SpillableByteMapImpl, V> and 
> SpillableArrayListMultimapImpl.  This will not work because we 
> need to be able to remove a key from a window, which SpillableArrayList does 
> not support, and having two separate spillable data structures if it can be 
> achieved by just one should be avoided in general. 
> We will implement the solution that supports prefix scanning (given a window, 
> return all keys), which requires the keys to be stored in order in managed 
> state.
> Traversing all keys in the spillable map is needed by WindowStateMap. The 
> WindowedOperator needs a way to traverse all windows in its state when firing 
> a trigger. However, this is less urgent since the Window meta info is small 
> and should fit in memory even with millions of windows. This is more for the 
> checkpointing efficiency.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2191) Implement a way to get all keys with or without prefix from SpillableByteMapImpl

[David Yan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Yan updated APEXMALHAR-2191:
--
Description: 
WindowedKeyedStorage is basically a Map>, and we need the 
capability of getting all K's given a window, and getting V given a window and 
K.

Currently, the spillable implementation of WindowedKeyedStorage uses two 
spillable data structures -- SpillableByteMapImpl, V> and 
SpillableArrayListMultimapImpl.  This will not work because we need 
to be able to remove a key from a window, which SpillableArrayList does not 
support, and having two separate spillable data structures if it can be 
achieved by just one should be avoided in general. 

We will implement the solution that supports prefix scanning (given a window, 
return all keys), which requires the keys to be stored in order in managed 
state.

Traversing all keys in the spillable map is needed by WindowStateMap. The 
WindowedOperator needs a way to traverse all windows in its state when firing a 
trigger. However, this is less urgent since the Window meta info is small and 
should fit in memory even with millions of windows. This is more for the 
checkpointing efficiency.



  was:This is needed by the spillable implementation of WindowStateMap. The 
WindowedOperator needs a way to traverse all windows in its state.


> Implement a way to get all keys with or without prefix from 
> SpillableByteMapImpl
> 
>
> Key: APEXMALHAR-2191
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2191
> Project: Apache Apex Malhar
>  Issue Type: Sub-task
>Reporter: David Yan
>Assignee: David Yan
>Priority: Critical
>
> WindowedKeyedStorage is basically a Map>, and we need the 
> capability of getting all K's given a window, and getting V given a window 
> and K.
> Currently, the spillable implementation of WindowedKeyedStorage uses two 
> spillable data structures -- SpillableByteMapImpl, V> and 
> SpillableArrayListMultimapImpl.  This will not work because we 
> need to be able to remove a key from a window, which SpillableArrayList does 
> not support, and having two separate spillable data structures if it can be 
> achieved by just one should be avoided in general. 
> We will implement the solution that supports prefix scanning (given a window, 
> return all keys), which requires the keys to be stored in order in managed 
> state.
> Traversing all keys in the spillable map is needed by WindowStateMap. The 
> WindowedOperator needs a way to traverse all windows in its state when firing 
> a trigger. However, this is less urgent since the Window meta info is small 
> and should fit in memory even with millions of windows. This is more for the 
> checkpointing efficiency.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2191) Implement a way to get all keys with or without prefix from SpillableByteMapImpl

[David Yan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Yan updated APEXMALHAR-2191:
--
Summary: Implement a way to get all keys with or without prefix from 
SpillableByteMapImpl  (was: Implement a way to get all keys from 
SpillableByteMapImpl)

> Implement a way to get all keys with or without prefix from 
> SpillableByteMapImpl
> 
>
> Key: APEXMALHAR-2191
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2191
> Project: Apache Apex Malhar
>  Issue Type: Sub-task
>Reporter: David Yan
>Assignee: David Yan
>Priority: Critical
>
> This is needed by the spillable implementation of WindowStateMap. The 
> WindowedOperator needs a way to traverse all windows in its state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)