Re: Heads up: branch-2.1-beta

2013-06-19 Thread Arun C Murthy
Ping. Any luck? On Jun 17, 2013, at 4:06 PM, Roman Shaposhnik r...@apache.org wrote: On Sun, Jun 16, 2013 at 5:14 PM, Arun C Murthy a...@hortonworks.com wrote: Roman, Is there a chance you can run the tests with the full stack built against branch-2.1-beta and help us know where we are?

InputFormat to regroup splits of underlying InputFormat to control number of map tasks

2013-06-19 Thread Nicolae Marasoiu
Hi, When running map-reduce with many splits it would be nice from a performance perspective to have fewer splits while maintaining data locality, so that the overhead of running a map task (jvm creation, map executor ramp-up e.g. spring context, etc) be less impactful when frequently running

[jira] [Created] (MAPREDUCE-5332) Support token-preserving restart of history server

2013-06-19 Thread Jason Lowe (JIRA)
Jason Lowe created MAPREDUCE-5332: - Summary: Support token-preserving restart of history server Key: MAPREDUCE-5332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5332 Project: Hadoop Map/Reduce

Hadoop-Mapreduce-trunk - Build # 1462 - Failure

2013-06-19 Thread Apache Jenkins Server
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1462/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 29128 lines...] Running

Re: InputFormat to regroup splits of underlying InputFormat to control number of map tasks

2013-06-19 Thread Robert Evans
This sounds similar to MultiFileInputFormat http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-mapreduce-project/h adoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apach e/hadoop/mapred/MultiFileInputFormat.java?revision=1239482view=markup It would be nice if you could

Re: mapred.child.ulimit in MR2

2013-06-19 Thread Robert Evans
Sandy, I think it was something that was missed in the port to YARN and the dead code was cleaned up as part of HADOOP-8288. If you have a use case for it or are worried about backwards compatibility we can add it back in. It is not that hard, all it did was add 'ulimt -v number' to the shell

Re: InputFormat to regroup splits of underlying InputFormat to control number of map tasks

2013-06-19 Thread Nicolae Marasoiu
Hi, Our intention is to solve this in a generic context, not just file input. Thus the split class should be generic (very similar to CompositeInputSplit from mapred). We also already implement getRecordReader by iterating over record readers created by the decorated input format (this method

Re: Heads up: branch-2.1-beta

2013-06-19 Thread Roman Shaposhnik
On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy a...@hortonworks.com wrote: Ping. Any luck? The full bigtop build failed because of Giraph: http://bigtop01.cloudera.org:8080/view/Upstream-tests/job/Hadoop-2.1.0/label=centos5/4/console Now, I'm pretty sure that's an issue with Giraph

[jira] [Created] (MAPREDUCE-5333) Add test that verifies MRAM works correctly when sending requests with non-normalized capabilities

2013-06-19 Thread Alejandro Abdelnur (JIRA)
Alejandro Abdelnur created MAPREDUCE-5333: - Summary: Add test that verifies MRAM works correctly when sending requests with non-normalized capabilities Key: MAPREDUCE-5333 URL:

[jira] [Created] (MAPREDUCE-5334) TestContainerLauncherImpl is failing

2013-06-19 Thread Vinod Kumar Vavilapalli (JIRA)
Vinod Kumar Vavilapalli created MAPREDUCE-5334: -- Summary: TestContainerLauncherImpl is failing Key: MAPREDUCE-5334 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5334 Project:

Re: Heads up: branch-2.1-beta

2013-06-19 Thread Roman Shaposhnik
On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy a...@hortonworks.com wrote: Ping. Any luck? Unfortunately I've ran into: https://issues.apache.org/jira/browse/HADOOP-9654 which was caused by the unrelated memory pressure on the NN, but it had an unfortunate side effect of making the rest of