[
https://issues.apache.org/jira/browse/APEXMALHAR-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200083#comment-15200083
]
ASF GitHub Bot commented on APEXMALHAR-2017:
--------------------------------------------
GitHub user PramodSSImmaneni opened a pull request:
https://github.com/apache/incubator-apex-malhar/pull/218
APEXMALHAR-2017 Use pre checkpoint notification to optimize operator IO
@chandnisingh @siyuanh @DT-Priyanka please review as it touches many
operators
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/PramodSSImmaneni/incubator-apex-malhar
before-checkpoint
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-apex-malhar/pull/218.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #218
----
commit 0339e2cea68b6c10b6082a1c9bbd5bd293ef8612
Author: Pramod Immaneni <[email protected]>
Date: 2016-03-17T03:57:39Z
Using CheckpointNotificationListener and beforeCheckpoint callback to do IO
in a more optimized fashion
commit 3599abbba304f7ab7fa333484ca701d817625c24
Author: Pramod Immaneni <[email protected]>
Date: 2016-03-17T17:21:07Z
Fixed fileoutput operator failing tests
commit ae1c2eabb64382af5e20f905112a7d67b7fe80e1
Author: Pramod Immaneni <[email protected]>
Date: 2016-03-17T17:42:45Z
Fixes for japicmp
----
> Use pre checkpoint notification to optimize operator IO
> -------------------------------------------------------
>
> Key: APEXMALHAR-2017
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2017
> Project: Apache Apex Malhar
> Issue Type: Improvement
> Reporter: Pramod Immaneni
> Assignee: Pramod Immaneni
>
> Currently many output operators enforce persistence of data on endWindow by
> calling flush, hflush or equivalent calls. This was done to help recovery.
> Doing this always ensures that the data corresponding to checkpoint state at
> recovery is always present.
> A recent addition to the engine lets the operators know about an impending
> checkpoint just before it happens using a callback. Operators can now enforce
> persistence of data one time in this in this callback instead of end of every
> window. This results in better performance as data is not being frequently
> written to persistent storage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
