[jira] [Commented] (MAPREDUCE-2584) Check for serializers early, and give out more information regarding missing serializers

2011-06-12 Thread jirapos...@reviews.apache.org (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048323#comment-13048323
 ] 

jirapos...@reviews.apache.org commented on MAPREDUCE-2584:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/885/
---

(Updated 2011-06-12 14:00:13.155848)


Review request for hadoop-mapreduce.


Changes
---

Merged exception handling with common, formattable message.


Summary
---

As discussed on HADOOP-7328, MapReduce can handle serializers in a much better 
way in case of bad configuration, improper imports (Some odd Text class instead 
of the Writable Text set as key), etc..

This issue covers the MapReduce parts of the improvements (made to 
MapOutputBuffer and possible early-check of serializer availability pre-submit) 
that provide more information than just an NPE as is the current case.


This addresses bug MAPREDUCE-2584.
http://issues.apache.org/jira/browse/MAPREDUCE-2584


Diffs (updated)
-

  src/java/org/apache/hadoop/mapred/MapTask.java 21599c2 
  src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528 
  src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a 

Diff: https://reviews.apache.org/r/885/diff


Testing
---

Added a test case that expects a failure if no io.serializers are present.


Thanks,

Harsh



> Check for serializers early, and give out more information regarding missing 
> serializers
> 
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: task
>Affects Versions: 0.20.2
>Reporter: Harsh J
>Assignee: Harsh J
>  Labels: serializers, tasks
> Fix For: 0.23.0
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much 
> better way in case of bad configuration, improper imports (Some odd Text 
> class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, 
> MapOutputBuffer, etc. and possible early-check of serializer availability 
> pre-submit) that provide more information than just an NPE as is the current 
> case.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2584) Check for serializers early, and give out more information regarding missing serializers

2011-06-12 Thread jirapos...@reviews.apache.org (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048327#comment-13048327
 ] 

jirapos...@reviews.apache.org commented on MAPREDUCE-2584:
--



bq.  On 2011-06-12 04:39:03, Todd Lipcon wrote:
bq.  > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, line 471
bq.  > 
bq.  >
bq.  > ah, I noticed one more thing. This catch clause is only for the 
exceptions you've thrown above.
bq.  > 
bq.  > Rather than do the two nested exceptions, better to have just one 
layer of exceptions. You could move the generic instructions from the lower 
exception to a constant and then just append it to either of the two more 
specific methods above.

Fixed. Unable to run tests right now locally due to a missing protobuf class 
(which is fixed, but I don't seem to have the fresh jar pulled yet in my maven2 
cache). Will update the test when it works (and if it does not).


- Harsh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/885/#review808
---


On 2011-06-12 14:00:13, Harsh J wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/885/
bq.  ---
bq.  
bq.  (Updated 2011-06-12 14:00:13)
bq.  
bq.  
bq.  Review request for hadoop-mapreduce.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  As discussed on HADOOP-7328, MapReduce can handle serializers in a much 
better way in case of bad configuration, improper imports (Some odd Text class 
instead of the Writable Text set as key), etc..
bq.  
bq.  This issue covers the MapReduce parts of the improvements (made to 
MapOutputBuffer and possible early-check of serializer availability pre-submit) 
that provide more information than just an NPE as is the current case.
bq.  
bq.  
bq.  This addresses bug MAPREDUCE-2584.
bq.  http://issues.apache.org/jira/browse/MAPREDUCE-2584
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/java/org/apache/hadoop/mapred/MapTask.java 21599c2 
bq.src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528 
bq.src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a 
bq.  
bq.  Diff: https://reviews.apache.org/r/885/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Added a test case that expects a failure if no io.serializers are present.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Harsh
bq.  
bq.



> Check for serializers early, and give out more information regarding missing 
> serializers
> 
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: task
>Affects Versions: 0.20.2
>Reporter: Harsh J
>Assignee: Harsh J
>  Labels: serializers, tasks
> Fix For: 0.23.0
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much 
> better way in case of bad configuration, improper imports (Some odd Text 
> class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, 
> MapOutputBuffer, etc. and possible early-check of serializer availability 
> pre-submit) that provide more information than just an NPE as is the current 
> case.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Moved] (MAPREDUCE-2585) Add dumpConfiguration option in hadoop help message

2011-06-12 Thread Harsh J (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J moved HADOOP-6219 to MAPREDUCE-2585:


Fix Version/s: (was: 0.23.0)
   0.23.0
Affects Version/s: (was: 0.20.1)
   0.20.1
   Issue Type: Improvement  (was: Bug)
  Key: MAPREDUCE-2585  (was: HADOOP-6219)
  Project: Hadoop Map/Reduce  (was: Hadoop Common)

> Add dumpConfiguration option in hadoop help message
> ---
>
> Key: MAPREDUCE-2585
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2585
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 0.20.1
>Reporter: Ramya R
>Assignee: V.V.Chaitanya Krishna
>Priority: Trivial
> Fix For: 0.23.0
>
> Attachments: HADOOP-6184-ydist.patch, HADOOP-6219-ydist.patch, 
> MAPREDUCE-919.patch, MAPREDUCE-919.patch
>
>
> Execution of bin/hadoop should show the -dumpConfiguration option introduced 
> in MAPREDUCE-768

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2585) Add dumpConfiguration option in hadoop help message

2011-06-12 Thread Harsh J (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated MAPREDUCE-2585:
---

Status: Patch Available  (was: Reopened)

> Add dumpConfiguration option in hadoop help message
> ---
>
> Key: MAPREDUCE-2585
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2585
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 0.20.1
>Reporter: Ramya R
>Assignee: V.V.Chaitanya Krishna
>Priority: Trivial
> Fix For: 0.23.0
>
> Attachments: HADOOP-6184-ydist.patch, HADOOP-6219-ydist.patch, 
> MAPREDUCE-919.patch, MAPREDUCE-919.patch
>
>
> Execution of bin/hadoop should show the -dumpConfiguration option introduced 
> in MAPREDUCE-768

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2585) Add dumpConfiguration option in hadoop help message

2011-06-12 Thread Harsh J (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048357#comment-13048357
 ] 

Harsh J commented on MAPREDUCE-2585:


(Moved to MAPREDUCE as MAPREDUCE-2585)

> Add dumpConfiguration option in hadoop help message
> ---
>
> Key: MAPREDUCE-2585
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2585
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 0.20.1
>Reporter: Ramya R
>Assignee: V.V.Chaitanya Krishna
>Priority: Trivial
> Fix For: 0.23.0
>
> Attachments: HADOOP-6184-ydist.patch, HADOOP-6219-ydist.patch, 
> MAPREDUCE-919.patch, MAPREDUCE-919.patch
>
>
> Execution of bin/hadoop should show the -dumpConfiguration option introduced 
> in MAPREDUCE-768

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2423) Monitoring the job tracker ui of hadoop using other open source monitoring tools like Nagios

2011-06-12 Thread Harsh J (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048388#comment-13048388
 ] 

Harsh J commented on MAPREDUCE-2423:


Saurabh -- Can this be resolved then?

> Monitoring the job tracker ui of hadoop using other open source monitoring 
> tools like Nagios
> 
>
> Key: MAPREDUCE-2423
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2423
> Project: Hadoop Map/Reduce
>  Issue Type: Wish
>  Components: jobtracker
>Reporter: Saurabh Mishra
>Priority: Trivial
>
> I just wish if there is a way I can write monitors to check my hadoop job 
> tracker UI using my existing Nagios infrastructure. As this would help me in 
> keeping everything centrally located and hence under manageable limits.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-2286) ASF mapreduce

2011-06-12 Thread Harsh J (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J resolved MAPREDUCE-2286.


 Tags: spam
   Resolution: Incomplete
Fix Version/s: (was: 0.23.0)
   (was: 0.22.0)
 Release Note:   (was: If versions are this related then seniority has to 
be in releasable format.)
 Hadoop Flags:   (was: [Reviewed])

Closing as Incomplete. Bot (or w/e), needs more work and provide more info in 
spam next time (so it may send us looking around for a real issue).

> ASF mapreduce
> -
>
> Key: MAPREDUCE-2286
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2286
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>  Components: benchmarks, client, contrib/streaming, jobtracker, pipes
> Environment: 2.2 Commodore
>Reporter: Miguel Ochoa
>Priority: Trivial
>   Original Estimate: 50h
>  Remaining Estimate: 50h
>
> This sub-net ensures versions in description, however projects or 
> manufacturing will have to be in working conditioning in the time of unknown 
> versions.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-1678) Change org.apache.hadoop.mapreduce.Cluster methods to allow for extending

2011-06-12 Thread Harsh J (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated MAPREDUCE-1678:
---

Fix Version/s: 0.23.0
 Assignee: Harsh J
Affects Version/s: 0.23.0
   Status: Patch Available  (was: Open)

> Change org.apache.hadoop.mapreduce.Cluster methods to allow for extending
> -
>
> Key: MAPREDUCE-1678
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1678
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>  Components: job submission
>Affects Versions: 0.23.0
>Reporter: Kyle Ellrott
>Assignee: Harsh J
>Priority: Trivial
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-1678.r1.patch
>
>   Original Estimate: 5m
>  Remaining Estimate: 5m
>
> Change methods in org.apache.hadoop.mapreduce.Cluster from private to 
> protected to allow extension of cluster.
> If the method createRPCProxy is changed from private to protected, then 
> alternate cluster implementations could be written that return other 
> ClientProtocol's. 
> For example, changing the protocol some custom implementation called 
> SimpleClient
>  
> ie:
> public class SimpleCluster extends Cluster {
>   @Override
>   protected ClientProtocol createRPCProxy(InetSocketAddress addr, 
> Configuration conf) throws IOException {
> return new SimpleClient(conf);
>   } 
> }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-504) JobTracker's configuration should be shown on jobtracker.jsp

2011-06-12 Thread Harsh J (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048398#comment-13048398
 ] 

Harsh J commented on MAPREDUCE-504:
---

Isn't this available at ${URL}/conf?

> JobTracker's configuration should be shown on jobtracker.jsp
> 
>
> Key: MAPREDUCE-504
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-504
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Amar Kamat
>Priority: Trivial
>
> A job's configuration is displayed for each job on {{jobdetails.jsp}}.. It 
> would be really useful if {{JobTracker}}'s config is also displayed on 
> jobtracker.jsp.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-423) Remove getNumResolvedTaskTrackers() api from JobTracker

2011-06-12 Thread Harsh J (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated MAPREDUCE-423:
--

Attachment: MAPREDUCE-423.r1.diff

> Remove getNumResolvedTaskTrackers() api from JobTracker
> ---
>
> Key: MAPREDUCE-423
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-423
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Amar Kamat
>Assignee: Harsh J
>Priority: Trivial
> Attachments: MAPREDUCE-423.r1.diff
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-423) Remove getNumResolvedTaskTrackers() api from JobTracker

2011-06-12 Thread Harsh J (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J reassigned MAPREDUCE-423:
-

Assignee: Harsh J

> Remove getNumResolvedTaskTrackers() api from JobTracker
> ---
>
> Key: MAPREDUCE-423
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-423
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Amar Kamat
>Assignee: Harsh J
>Priority: Trivial
> Attachments: MAPREDUCE-423.r1.diff
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-423) Remove getNumResolvedTaskTrackers() api from JobTracker

2011-06-12 Thread Harsh J (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated MAPREDUCE-423:
--

 Tags: api
Fix Version/s: 0.23.0
Affects Version/s: 0.23.0
   Status: Patch Available  (was: Open)

> Remove getNumResolvedTaskTrackers() api from JobTracker
> ---
>
> Key: MAPREDUCE-423
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-423
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Amar Kamat
>Assignee: Harsh J
>Priority: Trivial
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-423.r1.diff
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2178) Race condition in LinuxTaskController permissions handling

2011-06-12 Thread Nigel Daley (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048404#comment-13048404
 ] 

Nigel Daley commented on MAPREDUCE-2178:


Todd, is there a new patch you were planning to upload?  From previous comments 
it looks like he's waiting on that. 

> Race condition in LinuxTaskController permissions handling
> --
>
> Key: MAPREDUCE-2178
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2178
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: security, task-controller
>Affects Versions: 0.22.0
>Reporter: Todd Lipcon
>Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: 
> 0001-Amend-MAPREDUCE-2178.-Fix-racy-check-for-config-file.patch, 
> 0002-Amend-MAPREDUCE-2178.-Check-argc-after-checks-for-pe.patch, 
> 0003-Amend-MAPREDUCE-2178.-Check-result-of-chdir.patch, 
> ac-sys-largefile.patch, mapreduce-2178-test-compile-fix.txt, 
> mr-2178-0.22.txt, mr-2178-error-on-launch-fail.txt, mr-2178-y20-sortof.patch, 
> racy-config-check-test-changes.txt
>
>
> The linux-task-controller executable currently traverses a directory 
> heirarchy and calls chown/chmod on the files inside. There is a race 
> condition here which can be exploited by an attacker, causing the 
> task-controller to improprly chown an arbitrary target file (via a symlink) 
> to the user running a MR job. This can be exploited to escalate to root.
> [this issue was raised and discussed on the security@ list over the last 
> couple of months]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-397) In JobInProgress, failed TIP should be added back to the non-running queue only if the tip has not failed.

2011-06-12 Thread Harsh J (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J reassigned MAPREDUCE-397:
-

Assignee: Harsh J

> In JobInProgress, failed TIP should be added back to the non-running queue 
> only if the tip has not failed.
> --
>
> Key: MAPREDUCE-397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-397
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Amar Kamat
>Assignee: Harsh J
>Priority: Trivial
>
> In case of a task failure, the corresponding TIP is added back to the 
> non-running list via {{failMap()/failReduce()}} api. This reentry should be 
> done only for TIPs that have not failed. Reentry is useless for failed TIPs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-397) In JobInProgress, failed TIP should be added back to the non-running queue only if the tip has not failed.

2011-06-12 Thread jirapos...@reviews.apache.org (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048409#comment-13048409
 ] 

jirapos...@reviews.apache.org commented on MAPREDUCE-397:
-


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/887/
---

Review request for hadoop-mapreduce.


Summary
---

In case of a task failure, the corresponding TIP is added back to the 
non-running list via failMap()/failReduce() api. This reentry should be done 
only for TIPs that have not failed. Reentry is useless for failed TIPs.

Change checks for tip.isFailed() before proceeding.


This addresses bug MAPREDUCE-397.
http://issues.apache.org/jira/browse/MAPREDUCE-397


Diffs
-

  src/java/org/apache/hadoop/mapred/JobInProgress.java 429d2b0 

Diff: https://reviews.apache.org/r/887/diff


Testing
---


Thanks,

Harsh



> In JobInProgress, failed TIP should be added back to the non-running queue 
> only if the tip has not failed.
> --
>
> Key: MAPREDUCE-397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-397
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Amar Kamat
>Assignee: Harsh J
>Priority: Trivial
>
> In case of a task failure, the corresponding TIP is added back to the 
> non-running list via {{failMap()/failReduce()}} api. This reentry should be 
> done only for TIPs that have not failed. Reentry is useless for failed TIPs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-279) Map-Reduce 2.0

2011-06-12 Thread Nigel Daley (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048423#comment-13048423
 ] 

Nigel Daley commented on MAPREDUCE-279:
---

Given these build issues (and just good engineering practice), I'd like to see 
a Jenkins CI build on this branch so we know when merged to trunk the builds 
won't be  (more) broken. 

> Map-Reduce 2.0
> --
>
> Key: MAPREDUCE-279
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-279
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobtracker, tasktracker
>Reporter: Arun C Murthy
>Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MR-279.patch, MR-279.patch, MR-279.sh, 
> MR-279_MR_files_to_move.txt, capacity-scheduler-dark-theme.png, 
> multi-column-stable-sort-default-theme.png, yarn-state-machine.job.dot, 
> yarn-state-machine.job.png, yarn-state-machine.task-attempt.dot, 
> yarn-state-machine.task-attempt.png, yarn-state-machine.task.dot, 
> yarn-state-machine.task.png
>
>
> Re-factor MapReduce into a generic resource scheduler and a per-job, 
> user-defined component that manages the application execution. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-2178) Race condition in LinuxTaskController permissions handling

2011-06-12 Thread Nigel Daley (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Daley reassigned MAPREDUCE-2178:
--

Assignee: Devaraj Das

Assigning to Devaraj since he agreed to work on this. Thanks Devaraj!

> Race condition in LinuxTaskController permissions handling
> --
>
> Key: MAPREDUCE-2178
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2178
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: security, task-controller
>Affects Versions: 0.22.0
>Reporter: Todd Lipcon
>Assignee: Devaraj Das
>Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: 
> 0001-Amend-MAPREDUCE-2178.-Fix-racy-check-for-config-file.patch, 
> 0002-Amend-MAPREDUCE-2178.-Check-argc-after-checks-for-pe.patch, 
> 0003-Amend-MAPREDUCE-2178.-Check-result-of-chdir.patch, 
> ac-sys-largefile.patch, mapreduce-2178-test-compile-fix.txt, 
> mr-2178-0.22.txt, mr-2178-error-on-launch-fail.txt, mr-2178-y20-sortof.patch, 
> racy-config-check-test-changes.txt
>
>
> The linux-task-controller executable currently traverses a directory 
> heirarchy and calls chown/chmod on the files inside. There is a race 
> condition here which can be exploited by an attacker, causing the 
> task-controller to improprly chown an arbitrary target file (via a symlink) 
> to the user running a MR job. This can be exploited to escalate to root.
> [this issue was raised and discussed on the security@ list over the last 
> couple of months]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-2586) mapreduce/contrib/raid not compile due to hdfs changes.

2011-06-12 Thread Bochun Bai (JIRA) 
mapreduce/contrib/raid not compile due to hdfs changes.
---

 Key: MAPREDUCE-2586
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2586
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/raid
Reporter: Bochun Bai
Priority: Blocker
 Attachments: MAPREDUCE-2586.patch



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2586) mapreduce/contrib/raid not compile due to hdfs changes.

2011-06-12 Thread Bochun Bai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bochun Bai updated MAPREDUCE-2586:
--

Attachment: MAPREDUCE-2586.patch

> mapreduce/contrib/raid not compile due to hdfs changes.
> ---
>
> Key: MAPREDUCE-2586
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2586
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: contrib/raid
>Reporter: Bochun Bai
>Priority: Blocker
> Attachments: MAPREDUCE-2586.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2586) mapreduce/contrib/raid not compile due to hdfs changes.

2011-06-12 Thread Bochun Bai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bochun Bai updated MAPREDUCE-2586:
--

Assignee: Bochun Bai
  Status: Patch Available  (was: Open)

> mapreduce/contrib/raid not compile due to hdfs changes.
> ---
>
> Key: MAPREDUCE-2586
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2586
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: contrib/raid
>Reporter: Bochun Bai
>Assignee: Bochun Bai
>Priority: Blocker
> Attachments: MAPREDUCE-2586.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2584) Check for serializers early, and give out more information regarding missing serializers

2011-06-12 Thread jirapos...@reviews.apache.org (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048454#comment-13048454
 ] 

jirapos...@reviews.apache.org commented on MAPREDUCE-2584:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/885/#review811
---


Looks good. Can you upload latest patch to JIRA so QA-bot picks it up?

- Todd


On 2011-06-12 14:00:13, Harsh J wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/885/
bq.  ---
bq.  
bq.  (Updated 2011-06-12 14:00:13)
bq.  
bq.  
bq.  Review request for hadoop-mapreduce.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  As discussed on HADOOP-7328, MapReduce can handle serializers in a much 
better way in case of bad configuration, improper imports (Some odd Text class 
instead of the Writable Text set as key), etc..
bq.  
bq.  This issue covers the MapReduce parts of the improvements (made to 
MapOutputBuffer and possible early-check of serializer availability pre-submit) 
that provide more information than just an NPE as is the current case.
bq.  
bq.  
bq.  This addresses bug MAPREDUCE-2584.
bq.  http://issues.apache.org/jira/browse/MAPREDUCE-2584
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/java/org/apache/hadoop/mapred/MapTask.java 21599c2 
bq.src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528 
bq.src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a 
bq.  
bq.  Diff: https://reviews.apache.org/r/885/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Added a test case that expects a failure if no io.serializers are present.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Harsh
bq.  
bq.



> Check for serializers early, and give out more information regarding missing 
> serializers
> 
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: task
>Affects Versions: 0.20.2
>Reporter: Harsh J
>Assignee: Harsh J
>  Labels: serializers, tasks
> Fix For: 0.23.0
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much 
> better way in case of bad configuration, improper imports (Some odd Text 
> class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, 
> MapOutputBuffer, etc. and possible early-check of serializer availability 
> pre-submit) that provide more information than just an NPE as is the current 
> case.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2563) Gridmix high ram jobs emulation system tests.

2011-06-12 Thread Vinay Kumar Thota (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinay Kumar Thota updated MAPREDUCE-2563:
-

Fix Version/s: 0.23.0
   Status: Patch Available  (was: Open)

> Gridmix high ram jobs emulation system tests.
> -
>
> Key: MAPREDUCE-2563
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2563
> Project: Hadoop Map/Reduce
>  Issue Type: Task
>  Components: contrib/gridmix
>Reporter: Vinay Kumar Thota
>Assignee: Vinay Kumar Thota
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2563-v1.patch, MAPREDUCE-2563-v2.patch, 
> MAPREDUCE-2563.patch
>
>
> 1. Run the Gridmix with a high ram jobs trace and  verify each Gridmix job 
> whether it honors the high ram or not. In
> the trace the jobs should use the high ram for both maps and reduces.
> 2. Run the Gridmix with a high ram jobs trace and  verify each Gridmix job 
> whether it honors the high ram or not. In
> the trace the jobs should use the high ram only for maps.
> 3. Run the Gridmix with a high ram jobs trace and verify each Gridmix job 
> whether it honors the high ram or not. In the
> trace the jobs should use the high ram only for reducers.
> 4. Run the Gridmix with a high ram jobs trace by disabling the emulation of 
> high ram  and verify each Gridmix job
> whether it honors the high ram or not. In disable mode it should should not 
> honor the high ram and run it as a normal
> job.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-279) Map-Reduce 2.0

2011-06-12 Thread Tom White (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048465#comment-13048465
 ] 

Tom White commented on MAPREDUCE-279:
-

{quote}
Both 'classic' and 'nextgen' MR use the same 'runtime' i.e. MapTask,ReduceTask, 
shuffle etc. which is what is currently under 'mr-client'. I'm happy to 
brainstorm on a better name for it... how about 'mr-runtime'?
{quote}

I think 'mr-runtime', or just 'runtime' given that the context is MapReduce, 
would be fine. I guess that there would be a 'classic' submodule to contain 
JobTracker, TaskTracker etc, that currently aren't in any MR-279 Maven modules.

bq. Yep, it would be nice to completely mavenize and I strongly believe it 
should be our goal.

+1

> Map-Reduce 2.0
> --
>
> Key: MAPREDUCE-279
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-279
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobtracker, tasktracker
>Reporter: Arun C Murthy
>Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MR-279.patch, MR-279.patch, MR-279.sh, 
> MR-279_MR_files_to_move.txt, capacity-scheduler-dark-theme.png, 
> multi-column-stable-sort-default-theme.png, yarn-state-machine.job.dot, 
> yarn-state-machine.job.png, yarn-state-machine.task-attempt.dot, 
> yarn-state-machine.task-attempt.png, yarn-state-machine.task.dot, 
> yarn-state-machine.task.png
>
>
> Re-factor MapReduce into a generic resource scheduler and a per-job, 
> user-defined component that manages the application execution. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2547) TestDFSIO fails on a physical cluster

2011-06-12 Thread Todd Lipcon (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048473#comment-13048473
 ] 

Todd Lipcon commented on MAPREDUCE-2547:


The deprecated config ought to work. When was this particular build from? Did 
it include the rather-recent fix for HADOOP-7287 (committed May 23)? If so and 
this still didn't work, it seems like a bug to me.

> TestDFSIO fails on a physical cluster
> -
>
> Key: MAPREDUCE-2547
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2547
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.22.0
> Environment: Physical cluster based on 0.22-SNAPSHOT
>Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
>Priority: Blocker
>
> An attempt to run TestDSFIO on cluster fails because TestDFSIO tries to run 
> MR job with local runner. If JT is explicitly specified via {{-jt}} cmd. arg. 
> then everything is working as expected.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and give out more information regarding missing serializers

2011-06-12 Thread Harsh J (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated MAPREDUCE-2584:
---

Attachment: MAPREDUCE-2584.r3.diff

Review patch attached.

> Check for serializers early, and give out more information regarding missing 
> serializers
> 
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: task
>Affects Versions: 0.20.2
>Reporter: Harsh J
>Assignee: Harsh J
>  Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r3.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much 
> better way in case of bad configuration, improper imports (Some odd Text 
> class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, 
> MapOutputBuffer, etc. and possible early-check of serializer availability 
> pre-submit) that provide more information than just an NPE as is the current 
> case.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and give out more information regarding missing serializers

2011-06-12 Thread Harsh J (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated MAPREDUCE-2584:
---

Release Note: Check for proper Key/Value serializers during submission of 
MR jobs.
  Status: Patch Available  (was: Open)

> Check for serializers early, and give out more information regarding missing 
> serializers
> 
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: task
>Affects Versions: 0.20.2
>Reporter: Harsh J
>Assignee: Harsh J
>  Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r3.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much 
> better way in case of bad configuration, improper imports (Some odd Text 
> class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, 
> MapOutputBuffer, etc. and possible early-check of serializer availability 
> pre-submit) that provide more information than just an NPE as is the current 
> case.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and give out more information regarding missing serializers

2011-06-12 Thread Harsh J (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated MAPREDUCE-2584:
---

Status: Open  (was: Patch Available)

> Check for serializers early, and give out more information regarding missing 
> serializers
> 
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: task
>Affects Versions: 0.20.2
>Reporter: Harsh J
>Assignee: Harsh J
>  Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r3.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much 
> better way in case of bad configuration, improper imports (Some odd Text 
> class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, 
> MapOutputBuffer, etc. and possible early-check of serializer availability 
> pre-submit) that provide more information than just an NPE as is the current 
> case.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2584) Check for serializers early, and give out more information regarding missing serializers

2011-06-12 Thread Harsh J (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048477#comment-13048477
 ] 

Harsh J commented on MAPREDUCE-2584:


Test will fail without https://issues.apache.org/jira/browse/HADOOP-7328 as 
that blocks this.

> Check for serializers early, and give out more information regarding missing 
> serializers
> 
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: task
>Affects Versions: 0.20.2
>Reporter: Harsh J
>Assignee: Harsh J
>  Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r3.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much 
> better way in case of bad configuration, improper imports (Some odd Text 
> class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, 
> MapOutputBuffer, etc. and possible early-check of serializer availability 
> pre-submit) that provide more information than just an NPE as is the current 
> case.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2562) NullPointerException in Jobtracker when it is started without Name Node

2011-06-12 Thread Ravi Teja Ch N V (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048490#comment-13048490
 ] 

Ravi Teja Ch N V commented on MAPREDUCE-2562:
-

org.apache.hadoop.mapred.TestJobTrackerStart passed locally when the patch is 
applied over the trunk code.

> NullPointerException in Jobtracker when it is started without Name Node
> ---
>
> Key: MAPREDUCE-2562
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2562
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobtracker
>Affects Versions: 0.23.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2562.patch
>
>
> It is throwing NullPointerException in job tracker logs when job tracker is 
> started without NameNode.
> {code:title=Bar.java|borderStyle=solid}
> 2011-06-03 01:50:04,304 INFO org.apache.hadoop.ipc.Client: Retrying connect 
> to server: /10.18.52.225:9000. Already tried 7 time(s).
> 2011-06-03 01:50:05,307 INFO org.apache.hadoop.ipc.Client: Retrying connect 
> to server: /10.18.52.225:9000. Already tried 8 time(s).
> 2011-06-03 01:50:06,310 INFO org.apache.hadoop.ipc.Client: Retrying connect 
> to server: /10.18.52.225:9000. Already tried 9 time(s).
> 2011-06-03 01:50:21,243 FATAL org.apache.hadoop.mapred.JobTracker: 
> java.lang.NullPointerException
>   at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:1635)
>   at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:287)
>   at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:279)
>   at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:274)
>   at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4312)
> {code} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira