Re: Multi-level aggregation with combining the result of maps per node/rack

Robert, Thank you for your precious opinion and sharing the related JIRA tickets. The combination consisting of reusing container (MAPREDUCE-3902) and the coordination system in the AM is good idea to minimize implementation cost and ensure fault tolerance. The design can also solve scheduling pr

[jira] [Created] (MAPREDUCE-4502) Multi-level aggregation with combining the result of maps per node/rack

Tsuyoshi OZAWA created MAPREDUCE-4502: - Summary: Multi-level aggregation with combining the result of maps per node/rack Key: MAPREDUCE-4502 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4502

[jira] [Created] (MAPREDUCE-4501) couldn't compile hadoop-2.0 successfully because of errors in build files

Yan Liu created MAPREDUCE-4501: -- Summary: couldn't compile hadoop-2.0 successfully because of errors in build files Key: MAPREDUCE-4501 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4501 Project:

[jira] [Resolved] (MAPREDUCE-4500) TestUlimit is failing locally

[ https://issues.apache.org/jira/browse/MAPREDUCE-4500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla resolved MAPREDUCE-4500. - Resolution: Duplicate Created a Common JIRA, resolving this one as duplicate.

[jira] [Created] (MAPREDUCE-4500) TestUlimit is failing locally

Karthik Kambatla created MAPREDUCE-4500: --- Summary: TestUlimit is failing locally Key: MAPREDUCE-4500 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4500 Project: Hadoop Map/Reduce

[jira] [Created] (MAPREDUCE-4499) Looking for speculative tasks is very expensive in 1.x

Nathan Roberts created MAPREDUCE-4499: - Summary: Looking for speculative tasks is very expensive in 1.x Key: MAPREDUCE-4499 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4499 Project: Hadoop

[jira] [Resolved] (MAPREDUCE-4481) User Log Retention across TT restarts

[ https://issues.apache.org/jira/browse/MAPREDUCE-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony resolved MAPREDUCE-4481. - Resolution: Not A Problem > User Log Retention across TT restarts > -

RE: Multi-level aggregation with combining the result of maps per node/rack

Can you please share a brief note on the design. Just a few sentences on the main changes. What you are saying sounds similar to multi-level aggregation done in the Dryad runtime. That is useful to reduce the input arity (as you suggest) and al

[jira] [Created] (MAPREDUCE-4498) Downgrade hsqldb dependency to 1.8

Robert Kanter created MAPREDUCE-4498: Summary: Downgrade hsqldb dependency to 1.8 Key: MAPREDUCE-4498 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4498 Project: Hadoop Map/Reduce

Hadoop-Mapreduce-trunk - Build # 1153 - Still Failing

See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1153/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 31897 lines...] Results : Failed tests: testFor

Re: Multi-level aggregation with combining the result of maps per node/rack

Tsuyoshi, There has been a lot of work happening in the shuffle phase. It is being made pluggable in both 1.0 and 2.0/trunk (MAPREDUCE-4049). There is also some work being done to reuse containers in trunk/2.0 (MAPREDUCE-3902). This should have a similar, although perhaps more limited result, b