[
https://issues.apache.org/jira/browse/HADOOP-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12470351
]
Owen O'Malley commented on HADOOP-491:
--------------------------------------
This looks good so far, it just needs the default for streaming set. The logic
here is that streaming applications don't have an easy way of reporting to the
framework that they are making progress. So it is far too easy for them to be
killed as being dead when they are just working away. Clearly, disabling the
timeout makes it easy to write applications that just get stuck, but it seems
better to allow applications to get stuck rather than killing productive tasks.
> streaming jobs should allow programs that don't do any IO for a long time
> -------------------------------------------------------------------------
>
> Key: HADOOP-491
> URL: https://issues.apache.org/jira/browse/HADOOP-491
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: arkady borkovsky
> Assigned To: Arun C Murthy
> Fix For: 0.12.0
>
> Attachments: HADOOP-491_20070205_1.patch, HADOOP-491_20070206_2.patch
>
>
> The jobtracker relies on task to send heartbeats to know the tasks are still
> alive.
> There is a 600 seconds timeout preset.
> hadoop streaming also uses input to or output from the program it spawns to
> indicate progress, sending appropriate heartbeats.
> Some spawned programs spend longer that 600 seconds without any output while
> being perfectly healthy.
> It would be good to enhance the interface between hadoop streaming and the
> programs it spawns to track a healthy program in the absense of output.
> There are certain dangers with this protocol: e.g. a task can run a separate
> thread that does nothing but send "i'm alive" message. This would be a user
> bug to abuse the API in such way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
