[
https://issues.apache.org/jira/browse/APEXMALHAR-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Yan updated APEXMALHAR-2130:
----------------------------------
Description:
This feature is used for supporting windowing.
The storage needs to have the following features:
1. Spillable key value storage (integrate with APEXMALHAR-2026)
2. Upon checkpoint, it saves a snapshot for the entire data set with the
checkpointing window id
3. When recovering, it takes the recovery window id and restores to that
snapshot
4. It should implement the WindowedStorage and WindowedKeyedStorage interfaces,
and because of 2 and 3, we may want to add methods to the WindowedStorage
interface so that the implementation of WindowedOperator can notify the storage
of checkpointing, recovering and committing of a window.
5. Item 2 should be done incrementally (ManagedState) to avoid wasting space
with unchanged data
6. When a window is committed, all window with a lower ID should be purged from
the store.
was:
This feature is used for supporting windowing.
The storage needs to have the following features:
1. Spillable key value storage (integrate with APEXMALHAR-2026)
2. Upon checkpoint, it saves a snapshot for the entire data set with the
checkpointing window id
3. When recovering, it takes the recovery window id and restores to that
snapshot
4. It should implement the WindowedStorage and WindowedKeyedStorage interfaces,
and because of 2 and 3, we may want to add methods to the WindowedStorage
interface so that the implementation of WindowedOperator can notify the storage
of checkpointing and recovering.
5. Item 2 should be done incrementally (ManagedState) to avoid wasting space
with unchanged data
6. When a window is committed, all window with a lower ID should be purged from
the store.
> implement scalable windowed storage
> -----------------------------------
>
> Key: APEXMALHAR-2130
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2130
> Project: Apache Apex Malhar
> Issue Type: Task
> Reporter: bright chen
> Assignee: bright chen
>
> This feature is used for supporting windowing.
> The storage needs to have the following features:
> 1. Spillable key value storage (integrate with APEXMALHAR-2026)
> 2. Upon checkpoint, it saves a snapshot for the entire data set with the
> checkpointing window id
> 3. When recovering, it takes the recovery window id and restores to that
> snapshot
> 4. It should implement the WindowedStorage and WindowedKeyedStorage
> interfaces, and because of 2 and 3, we may want to add methods to the
> WindowedStorage interface so that the implementation of WindowedOperator can
> notify the storage of checkpointing, recovering and committing of a window.
> 5. Item 2 should be done incrementally (ManagedState) to avoid wasting space
> with unchanged data
> 6. When a window is committed, all window with a lower ID should be purged
> from the store.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
