Streaming should send keep-alive signals to Reporter every 10 seconds, not
every 100 records
--------------------------------------------------------------------------------------------
Key: HADOOP-556
URL: http://issues.apache.org/jira/browse/HADOOP-556
Project: Hadoop
Issue Type: Improvement
Components: contrib/streaming
Reporter: Michel Tourn
Streaming should send keep-alive signals to Reporter every 10 seconds, not
every 100 records.
A first version of this was already implemented but was not satisfactory.
Now we check whether 10sec have passed:
-after reading a *line* of Application's stderr
(so please use as your stderr heartbeat: cerr << ".\n", not cerr << "." )
-and after outputting a record in mapper / combiner / reducer.
If 10 sec have passed then we set Reporter status.
Effects:
-the reporter status changes more often and provides useful feedback in the Web
UI or in another client.
-a Task will not time out after 10 minutes just because it outputs records
slowly.
No artificial heartbeat is introduced in this proposal.
The streaming Application still has to show activity (either on stdout or on
stderr)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
