[jira] [Created] (MAPREDUCE-2777) Backport MAPREDUCE-220 to Hadoop 20 security branch

2011-08-04 Thread Jonathan Eagles (JIRA)
Backport MAPREDUCE-220 to Hadoop 20 security branch
---

 Key: MAPREDUCE-2777
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2777
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Affects Versions: 0.20.205.0
Reporter: Jonathan Eagles




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Resolved] (MAPREDUCE-2698) Expired NM's containers aren't being communicated to AM

2011-08-04 Thread Vinod Kumar Vavilapalli (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved MAPREDUCE-2698.


Resolution: Duplicate

The recent ResourceManager cleanup fixed this automatically. Closing this as 
duplicate.

 Expired NM's containers aren't being communicated to AM
 ---

 Key: MAPREDUCE-2698
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2698
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Reporter: Arun C Murthy
Assignee: Arun C Murthy

 Expired NM's containers aren't being communicated to AM - so the AM needs to 
 rely on timeouts. We need to fix this.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (MAPREDUCE-2489) Jobsplits with random hostnames can make the queue unusable

2011-08-04 Thread Jeffrey Naisbitt (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Naisbitt updated MAPREDUCE-2489:


Attachment: MAPREDUCE-2489-0.20s-v5.patch

Updated 20-security patch to fix issues with tests failing.  

test-patch results:
 [exec] +1 overall.  

 [exec] 

 [exec] +1 @author.  The patch does not contain any @author tags.

 [exec] 

 [exec] +1 tests included.  The patch appears to include 6 new or 
modified tests.

 [exec] 

 [exec] +1 javadoc.  The javadoc tool did not generate any warning 
messages.

 [exec] 

 [exec] +1 javac.  The applied patch does not increase the total number 
of javac compiler warnings.

 [exec] 

 [exec] +1 findbugs.  The patch does not introduce any new Findbugs 
(version 1.3.9) warnings.

 [exec] 

==
mvn test results:
All the tests pass that currently pass with the clean checkout.




 Jobsplits with random hostnames can make the queue unusable
 ---

 Key: MAPREDUCE-2489
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2489
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Affects Versions: 0.20.205.0, 0.23.0
Reporter: Jeffrey Naisbitt
Assignee: Jeffrey Naisbitt
 Fix For: 0.20.205.0, 0.23.0

 Attachments: MAPREDUCE-2489-0.20s-v2.patch, 
 MAPREDUCE-2489-0.20s-v3.patch, MAPREDUCE-2489-0.20s-v4.patch, 
 MAPREDUCE-2489-0.20s-v5.patch, MAPREDUCE-2489-0.20s.patch, 
 MAPREDUCE-2489-mapred-v2.patch, MAPREDUCE-2489-mapred-v3.patch, 
 MAPREDUCE-2489-mapred-v4.patch, MAPREDUCE-2489-mapred-v5.patch, 
 MAPREDUCE-2489-mapred.patch


 We saw an issue where a custom InputSplit was returning invalid hostnames for 
 the splits that were then causing the JobTracker to attempt to excessively 
 resolve host names.  This caused a major slowdown for the JobTracker.  We 
 should prevent invalid InputSplit hostnames from affecting everyone else.
 I propose we implement some verification for the hostnames to try to ensure 
 that we only do DNS lookups on valid hostnames (and fail otherwise).  We 
 could also fail the job after a certain number of failures in the resolve.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (MAPREDUCE-2489) Jobsplits with random hostnames can make the queue unusable

2011-08-04 Thread Jeffrey Naisbitt (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeffrey Naisbitt updated MAPREDUCE-2489:


Status: Patch Available  (was: Open)

 Jobsplits with random hostnames can make the queue unusable
 ---

 Key: MAPREDUCE-2489
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2489
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Affects Versions: 0.20.205.0, 0.23.0
Reporter: Jeffrey Naisbitt
Assignee: Jeffrey Naisbitt
 Fix For: 0.20.205.0, 0.23.0

 Attachments: MAPREDUCE-2489-0.20s-v2.patch, 
 MAPREDUCE-2489-0.20s-v3.patch, MAPREDUCE-2489-0.20s-v4.patch, 
 MAPREDUCE-2489-0.20s-v5.patch, MAPREDUCE-2489-0.20s.patch, 
 MAPREDUCE-2489-mapred-v2.patch, MAPREDUCE-2489-mapred-v3.patch, 
 MAPREDUCE-2489-mapred-v4.patch, MAPREDUCE-2489-mapred-v5.patch, 
 MAPREDUCE-2489-mapred.patch


 We saw an issue where a custom InputSplit was returning invalid hostnames for 
 the splits that were then causing the JobTracker to attempt to excessively 
 resolve host names.  This caused a major slowdown for the JobTracker.  We 
 should prevent invalid InputSplit hostnames from affecting everyone else.
 I propose we implement some verification for the hostnames to try to ensure 
 that we only do DNS lookups on valid hostnames (and fail otherwise).  We 
 could also fail the job after a certain number of failures in the resolve.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Resolved] (MAPREDUCE-2730) mr279: Application EXPIRED_PENDING to FAILED state transition incomplete

2011-08-04 Thread Thomas Graves (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Graves resolved MAPREDUCE-2730.
--

Resolution: Invalid

No longer applies with changes made to RM.

 mr279: Application EXPIRED_PENDING to FAILED state transition incomplete
 

 Key: MAPREDUCE-2730
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2730
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.0
Reporter: Thomas Graves
Assignee: Thomas Graves
Priority: Minor
 Fix For: 0.23.0


 in mrv2 if the application fails, it currently does not set the finishTime so 
 it shows up as being 0.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (MAPREDUCE-2729) Reducers are always counted having pending tasks even if they can't be scheduled yet because not enough of their mappers have completed

2011-08-04 Thread Sherry Chen (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079451#comment-13079451
 ] 

Sherry Chen commented on MAPREDUCE-2729:


I manually verified this fix on the 10 nodes cluster.

Verification steps:
1. Replace hadoop-capacity-scheduler.jar with the fix on the cluster gateway
2. Modify the capacity-scheduler.xml to ensure a queue have multiple map  
reduce task slots
3. restart mapred
4. Submit jobs for a user which start reduces when 5% (default) maps complete, 
submit jobs for 2nd user (same queue as 1st user) which start reduces when 50% 
maps complete.
5. Verify that 1st user got all queue reduce capacity whatever the 2nd user 
hasn't used yet, it is greater than user-limit.



 Reducers are always counted having pending tasks even if they can't be 
 scheduled yet because not enough of their mappers have completed
 -

 Key: MAPREDUCE-2729
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2729
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 0.20.205.0
 Environment: 0.20.1xx-Secondary
Reporter: Sherry Chen
Assignee: Sherry Chen
 Fix For: 0.20.205.0

 Attachments: MAPREDUCE-2729.patch


 In capacity scheduler, number of users in a queue needing slots are 
 calculated based on whether users' jobs have any pending tasks.
 This works fine for map tasks. However, for reduce tasks, jobs do not need 
 reduce slots until the minimum number of map tasks have been completed.
 Here, we add checking whether reduce is ready to schedule (i.e. if a job has 
 completed enough map tasks) when we increment number of users in a queue 
 needing reduce slots.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (MAPREDUCE-2489) Jobsplits with random hostnames can make the queue unusable

2011-08-04 Thread Jeffrey Naisbitt (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079472#comment-13079472
 ] 

Jeffrey Naisbitt commented on MAPREDUCE-2489:
-

I meant 'ant test' instead of 'mvn test'.
I was running with ant-1.8.2, which apparently caused the test failures. 
All of the tests pass with ant-1.7.1

 Jobsplits with random hostnames can make the queue unusable
 ---

 Key: MAPREDUCE-2489
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2489
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Affects Versions: 0.20.205.0, 0.23.0
Reporter: Jeffrey Naisbitt
Assignee: Jeffrey Naisbitt
 Fix For: 0.20.205.0, 0.23.0

 Attachments: MAPREDUCE-2489-0.20s-v2.patch, 
 MAPREDUCE-2489-0.20s-v3.patch, MAPREDUCE-2489-0.20s-v4.patch, 
 MAPREDUCE-2489-0.20s-v5.patch, MAPREDUCE-2489-0.20s.patch, 
 MAPREDUCE-2489-mapred-v2.patch, MAPREDUCE-2489-mapred-v3.patch, 
 MAPREDUCE-2489-mapred-v4.patch, MAPREDUCE-2489-mapred-v5.patch, 
 MAPREDUCE-2489-mapred.patch


 We saw an issue where a custom InputSplit was returning invalid hostnames for 
 the splits that were then causing the JobTracker to attempt to excessively 
 resolve host names.  This caused a major slowdown for the JobTracker.  We 
 should prevent invalid InputSplit hostnames from affecting everyone else.
 I propose we implement some verification for the hostnames to try to ensure 
 that we only do DNS lookups on valid hostnames (and fail otherwise).  We 
 could also fail the job after a certain number of failures in the resolve.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (MAPREDUCE-2287) add Kerberos HTTP SPNEGO authentication support to Hadoop JT/NN/DN/TT web-consoles

2011-08-04 Thread Alejandro Abdelnur (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated MAPREDUCE-2287:
--

Resolution: Not A Problem
Status: Resolved  (was: Patch Available)

as per 'v3' HADOOP-7119 this patch is not required.

 add Kerberos HTTP SPNEGO authentication support to Hadoop JT/NN/DN/TT 
 web-consoles
 --

 Key: MAPREDUCE-2287
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2287
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: security
Affects Versions: 0.23.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: ha-mapred-01.patch, ha-mapred-02.patch, ha-mapred.patch


 This JIRA is for the MAPRED portion of HADOOP-7119

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (MAPREDUCE-2489) Jobsplits with random hostnames can make the queue unusable

2011-08-04 Thread Jeffrey Naisbitt (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079540#comment-13079540
 ] 

Jeffrey Naisbitt commented on MAPREDUCE-2489:
-

To clarify, all the tests pass for the 0.20s patch.  I am still looking at the 
patch for trunk.

 Jobsplits with random hostnames can make the queue unusable
 ---

 Key: MAPREDUCE-2489
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2489
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Affects Versions: 0.20.205.0, 0.23.0
Reporter: Jeffrey Naisbitt
Assignee: Jeffrey Naisbitt
 Fix For: 0.20.205.0, 0.23.0

 Attachments: MAPREDUCE-2489-0.20s-v2.patch, 
 MAPREDUCE-2489-0.20s-v3.patch, MAPREDUCE-2489-0.20s-v4.patch, 
 MAPREDUCE-2489-0.20s-v5.patch, MAPREDUCE-2489-0.20s.patch, 
 MAPREDUCE-2489-mapred-v2.patch, MAPREDUCE-2489-mapred-v3.patch, 
 MAPREDUCE-2489-mapred-v4.patch, MAPREDUCE-2489-mapred-v5.patch, 
 MAPREDUCE-2489-mapred.patch


 We saw an issue where a custom InputSplit was returning invalid hostnames for 
 the splits that were then causing the JobTracker to attempt to excessively 
 resolve host names.  This caused a major slowdown for the JobTracker.  We 
 should prevent invalid InputSplit hostnames from affecting everyone else.
 I propose we implement some verification for the hostnames to try to ensure 
 that we only do DNS lookups on valid hostnames (and fail otherwise).  We 
 could also fail the job after a certain number of failures in the resolve.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (MAPREDUCE-1638) Remove MapReduce server depedencies on client code

2011-08-04 Thread Todd Lipcon (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079617#comment-13079617
 ] 

Todd Lipcon commented on MAPREDUCE-1638:


+1 to the patch from 7/22

 Remove MapReduce server depedencies on client code
 --

 Key: MAPREDUCE-1638
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1638
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: build, client
Reporter: Tom White
Assignee: Tom White
 Fix For: 0.23.0

 Attachments: MAPREDUCE-1638.patch, MAPREDUCE-1638.patch, 
 MAPREDUCE-1638.patch, MAPREDUCE-1638.sh


 I think it makes sense to separate the MapReduce source into client and 
 server parts.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

2011-08-04 Thread Jitendra Nath Pandey (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079654#comment-13079654
 ] 

Jitendra Nath Pandey commented on MAPREDUCE-2764:
-

1. The patch introduces some refactoring around token selector, and also 
probably code cleanup in HftpFileSystem.java, but it would be good if that is 
done in a separate jira. This will keep this patch smaller and focused. 
2. I like the idea to store uri in the token which is issuer of the token. So 
going forward we have two things in the token:
 - service: where the token should be used. This should contain rpc port. This 
remains same as we have today, so no changes should be needed to set/getService 
APIs.
 - issuer: The uri of the file system that issued this token and add 
set/getIssuer APIs to the Token.
  Can we rename serviceUri to something like issuer or something else, just to 
differentiate it from service.
  So, the DelegationTokenRenewal looks at the issuer, while token selectors 
continue to use service, as they do today.

3. I will recommend the issuer should be set by every filesystem that 
implements getDelegationToken.
4. So we add only two APIs to Token: set/getIssuer. We don't need to add others 
like getServiceAuthority.
5. The patch didn't compile for me. Probably a typo in 
TestDelegationTokenRenewal.

 Fix renewal of dfs delegation tokens
 

 Key: MAPREDUCE-2764
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Daryn Sharp
Assignee: Daryn Sharp
 Fix For: 0.20.205.0

 Attachments: MAPREDUCE-2764.patch


 The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
 The problem is the JT's delegation token renewal code is built on brittle 
 assumptions.  The token's service field contains only the ip:port pair.  
 The renewal process assumes that the scheme must be hdfs.  If that fails due 
 to a {{VersionMismatchException}}, it tries https based on another assumption 
 that it must be hftp if it's not hdfs.  A number of other exceptions, most 
 commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
 it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

2011-08-04 Thread Daryn Sharp (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079681#comment-13079681
 ] 

Daryn Sharp commented on MAPREDUCE-2764:


The changes aren't entirely independent so I'm not sure if I can easily tease 
them apart, but I'll start trying to break it up into more jiras...  The token 
selectors were all just copy-n-paste, so I would have been further perpetrating 
unnecessary duplication if I didn't commonize them.  The {{HftpFileSystem}} 
changes were the fulcrum of this bug.

To help clarify the approach I took in case there is misunderstanding: I let 
the service be one of two formats: scheme://authority or authority (for 
compatibility).  I added {{getServiceAuthority}} in order to easily get the 
authority irregardless of the format.  The benefits to supporting both formats 
in the same field is to avoid unnecessary duplication of the same data, and 
more importantly to avoid introducing binary incompatibility to the tokens.  
Unless I'm mistaken, your recommendation for a new issuer field would have 
those downsides.  Are there other benefits that would outweigh those issues?


 Fix renewal of dfs delegation tokens
 

 Key: MAPREDUCE-2764
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Daryn Sharp
Assignee: Daryn Sharp
 Fix For: 0.20.205.0

 Attachments: MAPREDUCE-2764.patch


 The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
 The problem is the JT's delegation token renewal code is built on brittle 
 assumptions.  The token's service field contains only the ip:port pair.  
 The renewal process assumes that the scheme must be hdfs.  If that fails due 
 to a {{VersionMismatchException}}, it tries https based on another assumption 
 that it must be hftp if it's not hdfs.  A number of other exceptions, most 
 commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
 it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Created] (MAPREDUCE-2778) Refactor token selectors to leverage common code

2011-08-04 Thread Daryn Sharp (JIRA)
Refactor token selectors to leverage common code


 Key: MAPREDUCE-2778
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2778
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Daryn Sharp
Assignee: Daryn Sharp
 Fix For: 0.20.205.0


The token selectors are copy-n-paste of each other.  Need to create a base 
class to hold the common code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (MAPREDUCE-2778) Refactor token selectors to leverage common code

2011-08-04 Thread Daryn Sharp (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated MAPREDUCE-2778:
---

Attachment: MAPREDUCE-2778.patch

 Refactor token selectors to leverage common code
 

 Key: MAPREDUCE-2778
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2778
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Daryn Sharp
Assignee: Daryn Sharp
 Fix For: 0.20.205.0

 Attachments: MAPREDUCE-2778.patch


 The token selectors are copy-n-paste of each other.  Need to create a base 
 class to hold the common code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Created] (MAPREDUCE-2779) JobSplitWriter.java can't handle large job.split file

2011-08-04 Thread Ming Ma (JIRA)
JobSplitWriter.java can't handle large job.split file
-

 Key: MAPREDUCE-2779
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2779
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: job submission
Reporter: Ming Ma


We use cascading MultiInputFormat. MultiInputFormat sometimes generates big 
job.split used internally by hadoop, sometimes it can go beyond 2GB.

In JobSplitWriter.java, the function that generates such file uses 32bit signed 
integer to compute offset into job.split.


writeNewSplits
...
int prevCount = out.size();
...
int currCount = out.size();

writeOldSplits
...
  long offset = out.size();
...
  int currLen = out.size();


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (MAPREDUCE-2779) JobSplitWriter.java can't handle large job.split file

2011-08-04 Thread Ming Ma (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ming Ma updated MAPREDUCE-2779:
---

Attachment: MAPREDUCE-2779-trunk.patch

 JobSplitWriter.java can't handle large job.split file
 -

 Key: MAPREDUCE-2779
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2779
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: job submission
Reporter: Ming Ma
 Attachments: MAPREDUCE-2779-trunk.patch


 We use cascading MultiInputFormat. MultiInputFormat sometimes generates big 
 job.split used internally by hadoop, sometimes it can go beyond 2GB.
 In JobSplitWriter.java, the function that generates such file uses 32bit 
 signed integer to compute offset into job.split.
 writeNewSplits
 ...
 int prevCount = out.size();
 ...
 int currCount = out.size();
 writeOldSplits
 ...
   long offset = out.size();
 ...
   int currLen = out.size();

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Created] (MAPREDUCE-2780) Standardize the value of token service

2011-08-04 Thread Daryn Sharp (JIRA)
Standardize the value of token service
--

 Key: MAPREDUCE-2780
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2780
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Daryn Sharp
Assignee: Daryn Sharp
 Fix For: 0.20.205.0


The token's service field must (currently) be set to ip:port.  All the 
producers of a token are independently building the service string.  This 
should be done via a common method to reduce the chance of error, and to 
facilitate the field value being easily changed in the (near) future.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Created] (MAPREDUCE-2781) mr279 RM application finishtime not set

2011-08-04 Thread Thomas Graves (JIRA)
mr279 RM application finishtime not set
---

 Key: MAPREDUCE-2781
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2781
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Reporter: Thomas Graves
Assignee: Thomas Graves
Priority: Minor


The RM Application finishTime isn't being set.  Looks like it got lost in the 
RM refactor.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (MAPREDUCE-2781) mr279 RM application finishtime not set

2011-08-04 Thread Thomas Graves (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Graves updated MAPREDUCE-2781:
-

Attachment: finishtime.patch

 mr279 RM application finishtime not set
 ---

 Key: MAPREDUCE-2781
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2781
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Reporter: Thomas Graves
Assignee: Thomas Graves
Priority: Minor
 Attachments: finishtime.patch


 The RM Application finishTime isn't being set.  Looks like it got lost in the 
 RM refactor.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (MAPREDUCE-2781) mr279 RM application finishtime not set

2011-08-04 Thread Mahadev konar (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079791#comment-13079791
 ] 

Mahadev konar commented on MAPREDUCE-2781:
--

Thomas, 
 Would it be possible to add a unit test, that includes unit tests to make sure 
all the application finish time/start time/diagnostics and other are set? We 
can do it in a separate jira as well. 

 mr279 RM application finishtime not set
 ---

 Key: MAPREDUCE-2781
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2781
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Reporter: Thomas Graves
Assignee: Thomas Graves
Priority: Minor
 Attachments: finishtime.patch


 The RM Application finishTime isn't being set.  Looks like it got lost in the 
 RM refactor.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira