Owen O'Malley wrote:
On Apr 16, 2008, at 8:28 AM, Chaman Singh Verma wrote:
I am developing one application with MapReduce and in that whenever some
MapTask condition is
met, I would like to broadcast to all other MapTask to abort their
work. I
am not quite sure whether
such broadcasting functionality currently exist in Hadoop MapReduce.
Could
someone give some
hints.
This is pretty atypical behavior, but you could have each map look for
the existence of an hdfs file every 1 minute or so. When the condition
is true, create the file and your maps will exit in the next minute.
Except on very large clusters, that wouldn't be too expensive...
See also HADOOP-490. I use the message queue facility in my applications
(HADOOP-368) but it works only for infrequent communication and smaller
clusters.
I still think that the job control protocol should allow sending
"signals" to all tasks of a job. This would eliminate the need for
polling, because applications could use a simple listener.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
