[
https://issues.apache.org/jira/browse/APEXMALHAR-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bright chen updated APEXMALHAR-2130:
------------------------------------
Comment: was deleted
(was: Here are some of thoughts:
There are two type of actor for this storage, one is the operator which
receive/handle the data( which need operation like query, add, modify and
delete). Another type operator is query/manage the result only( need operation
query, maybe need operation delete). The first type of operator need response
as soon as possible and the second type of operator need accurate result.
The storage can have a cache layer and persistent layer
for the first type of operator, all the operation only apply to cache layer.
The second type of operator apply to persistent layer.
There has separate thread(s) which merge the cache data to persistent data(one
way only).
use the version to manage the conflict.
When do the query from query operator, it's better to sync all cached data to
the persistent first and then do the query. But there need a mechanism to
notify all operators which have the cache. Maybe think about this issue in
future.
)
> implement scalable windowed storage
> -----------------------------------
>
> Key: APEXMALHAR-2130
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2130
> Project: Apache Apex Malhar
> Issue Type: Task
> Reporter: bright chen
> Assignee: bright chen
>
> This feature is used for supporting windowing.
> The storage needs to have the following features:
> 1. Spillable key value storage (integrate with APEXMALHAR-2026)
> 2. Upon checkpoint, it saves a snapshot for the entire data set with the
> checkpointing window id. This should be done incrementally (ManagedState) to
> avoid wasting space with unchanged data
> 3. When recovering, it takes the recovery window id and restores to that
> snapshot
> 4. When a window is committed, all windows with a lower ID should be purged
> from the store.
> 5. It should implement the WindowedStorage and WindowedKeyedStorage
> interfaces, and because of 2 and 3, we may want to add methods to the
> WindowedStorage interface so that the implementation of WindowedOperator can
> notify the storage of checkpointing, recovering and committing of a window.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
