[
https://issues.apache.org/jira/browse/FALCON-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354540#comment-15354540
]
Pallavi Rao commented on FALCON-2051:
-------------------------------------
[~me.venkatr], we were actually planning to get rid of post-processing
altogether and move any logic there in, as a service on Falcon Server. The
reasons are as follows:
1. If the user workflow succeeds and succeeded-post-processing fails or hangs,
the whole instance gets flagged as failed and retries kick in. We have quite
many such instances where there is unnecessary delay because of post-processing
failing.
2. post-processing takes up one container on YARN which can be avoided.
Two things that are happening in post-processing apart from notification:
1. The Hadoop counters - These are not directly from Hadoop. It expects a
counter.txt in the LogDir. I think this is broken as of now. Correct me if I'm
wrong.
2. Log archival - Move oozie and task logs under staging dir. This logic can be
easily moved to Server side (listen to notification and do a hdfs copy).
Given this, lets not continue to depend on post processing. Thoughts?
> Falcon post-processing services are not getting invoked
> -------------------------------------------------------
>
> Key: FALCON-2051
> URL: https://issues.apache.org/jira/browse/FALCON-2051
> Project: Falcon
> Issue Type: Bug
> Components: general
> Affects Versions: 0.9
> Reporter: Peeyush Bishnoi
> Assignee: Venkatesan Ramachandran
> Priority: Blocker
> Fix For: 0.10
>
>
> With latest Falcon, none of the post-processing services like metadata, late
> data handling, retry is getting invoked. Hence created a blocker issue to fix
> this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
