Re: Update interval of default counters

2014-04-16 Thread Akira AJISAKA
I'm thinking the reason for hard-coding is to protect Hadoop cluster from high network traffic. If the value is too small, there are too many network traffic between Map/Reduce tasks and MRAppMaster. Please see https://issues.apache.org/jira/browse/MAPREDUCE-4381 also. That's why you need to be

Re: Policy on adding timeouts to tests

2014-04-16 Thread Vinod Kumar Vavilapalli
The other advantage of timeout is early failure - earlier than the uber 10 min timeout that seems to exist in the build files. Usually the test-writer has a general idea of how long the test is supposed to run and if that doesn't happen, we can fail early. Clearly, this involves choosing a reaso

[jira] [Created] (MAPREDUCE-5841) uber job doesn't terminate on getting mapred job kill

2014-04-16 Thread Sangjin Lee (JIRA)
Sangjin Lee created MAPREDUCE-5841: -- Summary: uber job doesn't terminate on getting mapred job kill Key: MAPREDUCE-5841 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5841 Project: Hadoop Map/Re

[jira] [Resolved] (MAPREDUCE-3825) MR should not be getting duplicate tokens for a MR Job.

2014-04-16 Thread Mit Desai (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai resolved MAPREDUCE-3825. -- Resolution: Duplicate Target Version/s: 2.0.0-alpha, 0.23.3, 3.0.0 (was: 0.23.3,

[jira] [Resolved] (MAPREDUCE-3704) Yarn client goes into tight loop upon connection failure

2014-04-16 Thread Daryn Sharp (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-3704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp resolved MAPREDUCE-3704. Resolution: Not a Problem I think this old issue has already been fixed. > Yarn client

Re: Policy on adding timeouts to tests

2014-04-16 Thread Tsuyoshi OZAWA
Hi Karthik, Some tests with servers like MiniCluster or ZK can never end because of unexpected busy loop or something if the tests don't have timeouts. It can blocks the other jobs of Jenkins server. Therefore, IMHO, we should add timeouts when we write tests with them. Thanks, - Tsuyoshi On Wed

[jira] [Created] (MAPREDUCE-5840) Update MapReduce calls to ProxyUsers#authorize.

2014-04-16 Thread Chris Nauroth (JIRA)
Chris Nauroth created MAPREDUCE-5840: Summary: Update MapReduce calls to ProxyUsers#authorize. Key: MAPREDUCE-5840 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5840 Project: Hadoop Map/Redu

[jira] [Resolved] (MAPREDUCE-3406) Add node information to bin/mapred job -list-attempt-ids and other improvements

2014-04-16 Thread Chen He (JIRA)
[ https://issues.apache.org/jira/browse/MAPREDUCE-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He resolved MAPREDUCE-3406. Resolution: Duplicate Target Version/s: 2.0.0-alpha, 0.23.3, 3.0.0 (was: 0.23.3, 2.0.

Re: Update interval of default counters

2014-04-16 Thread Dharmesh Kakadia
Hi Akira, Thanks fir the quick reply. Any particular reason for hard-coding it? Is there a workaround? I want to be able to get the counters as fine as possible. Also can you point me to the relevant source code. I am willing to take the issue and contribute if required. Thanks, Dharmesh On Wed

Re: Update interval of default counters

2014-04-16 Thread Akira AJISAKA
Moved mapreduce-dev@ to Bcc. Hi Dharmesh, The parameter is to set the interval of polling the progress of the MRAppMaster, not the Map/Reduce tasks. The tasks send the progress (includes the counter information) to MRAppMaster every 3000 milliseconds, which is hard-coded. That's why a sudden bi

Re: Policy on adding timeouts to tests

2014-04-16 Thread Steve Loughran
There's a JIRA somewhere that 's never gone in, to add a timeout rule to a base class; this rule gets picked up in that test class and all children to specify the timeout @Rule public final Timeout testTimeout = new Timeout(TEST_TIMEOUT); 1. If we are going to have a timeout everywhere, i