[jira] [Commented] (MAPREDUCE-6332) Provide facility to users for writting custom MergeManager implementation when custom shuffleconsumerPluggin is used
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14580072#comment-14580072 ] Rohith commented on MAPREDUCE-6332: --- [~devaraj.k] Kindly review the updated patch. -1 for whitespace, this line is not modified in the patch but still QA reports -1. > Provide facility to users for writting custom MergeManager implementation > when custom shuffleconsumerPluggin is used > > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6332.patch, 0002-MAPREDUCE-6332.patch, > 0003-MAPREDUCE-6332.patch, 0004-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Moved] (MAPREDUCE-6393) Application Master and Task Tracker timeouts are applied incorrectly
[ https://issues.apache.org/jira/browse/MAPREDUCE-6393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith moved YARN-3788 to MAPREDUCE-6393: - Affects Version/s: (was: 2.4.1) 2.4.1 Key: MAPREDUCE-6393 (was: YARN-3788) Project: Hadoop Map/Reduce (was: Hadoop YARN) > Application Master and Task Tracker timeouts are applied incorrectly > > > Key: MAPREDUCE-6393 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6393 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.4.1 >Reporter: Dmitry Sivachenko > > I am running a streaming job which requires a big (~50GB) data file to run > (file is attached via hadoop jar <...> -file BigFile.dat). > Most likely this command will fail as follows (note that error message is > rather meaningless): > 2015-05-27 15:55:00,754 WARN [main] streaming.StreamJob > (StreamJob.java:parseArgv(291)) - -file option is deprecated, please use > generic option -files instead. > packageJobJar: [/ssd/mt/lm/en_reorder.ylm, mapper.py, > /tmp/hadoop-mitya/hadoop-unjar3778165585140840383/] [] > /var/tmp/streamjob633547925483233845.jar tmpDir=null > 2015-05-27 19:46:22,942 INFO [main] client.RMProxy > (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at > nezabudka1-00.yandex.ru/5.255.231.129:8032 > 2015-05-27 19:46:23,733 INFO [main] client.RMProxy > (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at > nezabudka1-00.yandex.ru/5.255.231.129:8032 > 2015-05-27 20:13:37,231 INFO [main] mapred.FileInputFormat > (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1 > 2015-05-27 20:13:38,110 INFO [main] mapreduce.JobSubmitter > (JobSubmitter.java:submitJobInternal(396)) - number of splits:1 > 2015-05-27 20:13:38,136 INFO [main] Configuration.deprecation > (Configuration.java:warnOnceIfDeprecated(1009)) - mapred.reduce.tasks is > deprecated. Instead, use mapreduce.job.reduces > 2015-05-27 20:13:38,390 INFO [main] mapreduce.JobSubmitter > (JobSubmitter.java:printTokens(479)) - Submitting tokens for job: > job_1431704916575_2531 > 2015-05-27 20:13:38,689 INFO [main] impl.YarnClientImpl > (YarnClientImpl.java:submitApplication(204)) - Submitted application > application_1431704916575_2531 > 2015-05-27 20:13:38,743 INFO [main] mapreduce.Job (Job.java:submit(1289)) - > The url to track the job: > http://nezabudka1-00.yandex.ru:8088/proxy/application_1431704916575_2531/ > 2015-05-27 20:13:38,746 INFO [main] mapreduce.Job > (Job.java:monitorAndPrintJob(1334)) - Running job: job_1431704916575_2531 > 2015-05-27 21:04:12,353 INFO [main] mapreduce.Job > (Job.java:monitorAndPrintJob(1355)) - Job job_1431704916575_2531 running in > uber mode : false > 2015-05-27 21:04:12,356 INFO [main] mapreduce.Job > (Job.java:monitorAndPrintJob(1362)) - map 0% reduce 0% > 2015-05-27 21:04:12,374 INFO [main] mapreduce.Job > (Job.java:monitorAndPrintJob(1375)) - Job job_1431704916575_2531 failed with > state FAILED due to: Application application_1431704916575_2531 failed 2 > times due to ApplicationMaster for attempt > appattempt_1431704916575_2531_02 timed out. Failing the application. > 2015-05-27 21:04:12,473 INFO [main] mapreduce.Job > (Job.java:monitorAndPrintJob(1380)) - Counters: 0 > 2015-05-27 21:04:12,474 ERROR [main] streaming.StreamJob > (StreamJob.java:submitAndMonitorJob(1019)) - Job not Successful! > Streaming Command Failed! > This is because yarn.am.liveness-monitor.expiry-interval-ms (defaults to 600 > sec) timeout expires before large data file is transferred. > Next step I increase yarn.am.liveness-monitor.expiry-interval-ms. After that > application is successfully initialized and tasks are spawned. > But I encounter another error: the default 600 seconds mapreduce.task.timeout > expires before tasks are initialized and tasks fail. > Error message Task attempt_XXX failed to report status for 600 seconds is > also misleading: this timeout is supposed to kill non-responsive (stuck) > tasks but it rather strikes because auxiliary data files are copying slowly. > So I need to increase mapreduce.task.timeout too and only after that my job > is successful. > At the very least error messages need to be tweaked to indicate that > Application (or Task) is failing because auxiliary files are not copied > during that time, not just generic "timeout expired". > Better solution would be not to account time spent for data files > distribution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6332) Provide facility to users for writting custom MergeManager implementation when custom shuffleconsumerPluggin is used
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6332: -- Status: Open (was: Patch Available) > Provide facility to users for writting custom MergeManager implementation > when custom shuffleconsumerPluggin is used > > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6332.patch, 0002-MAPREDUCE-6332.patch, > 0003-MAPREDUCE-6332.patch, 0004-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6332) Provide facility to users for writting custom MergeManager implementation when custom shuffleconsumerPluggin is used
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6332: -- Attachment: 0004-MAPREDUCE-6332.patch Updated the patch to fix checkstyle & whitespace warnings. > Provide facility to users for writting custom MergeManager implementation > when custom shuffleconsumerPluggin is used > > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6332.patch, 0002-MAPREDUCE-6332.patch, > 0003-MAPREDUCE-6332.patch, 0004-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6332) Provide facility to users for writting custom MergeManager implementation when custom shuffleconsumerPluggin is used
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6332: -- Status: Patch Available (was: Open) > Provide facility to users for writting custom MergeManager implementation > when custom shuffleconsumerPluggin is used > > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6332.patch, 0002-MAPREDUCE-6332.patch, > 0003-MAPREDUCE-6332.patch, 0004-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6332) Provide facility to users for writting custom MergeManager implementation when custom shuffleconsumerPluggin is used
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14576977#comment-14576977 ] Rohith commented on MAPREDUCE-6332: --- Updated the patch rebasing against trunk. Kindly review.. > Provide facility to users for writting custom MergeManager implementation > when custom shuffleconsumerPluggin is used > > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.5.0, 2.6.0, 2.7.0 >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6332.patch, 0002-MAPREDUCE-6332.patch, > 0003-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6332) Provide facility to users for writting custom MergeManager implementation when custom shuffleconsumerPluggin is used
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6332: -- Attachment: 0003-MAPREDUCE-6332.patch > Provide facility to users for writting custom MergeManager implementation > when custom shuffleconsumerPluggin is used > > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.5.0, 2.6.0, 2.7.0 >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6332.patch, 0002-MAPREDUCE-6332.patch, > 0003-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6332) Provide facility to users for writting custom MergeManager implementation when custom shuffleconsumerPluggin is used
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6332: -- Summary: Provide facility to users for writting custom MergeManager implementation when custom shuffleconsumerPluggin is used (was: Add more required API's to MergeManager interface ) > Provide facility to users for writting custom MergeManager implementation > when custom shuffleconsumerPluggin is used > > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.5.0, 2.6.0, 2.7.0 >Reporter: Rohith >Assignee: Rohith > Labels: BB2015-05-TBR > Attachments: 0001-MAPREDUCE-6332.patch, 0002-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4754) Job is marked as FAILED and also throwing the TransitonException instead of KILLED when issues a KILL command
[
https://issues.apache.org/jira/browse/MAPREDUCE-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith updated MAPREDUCE-4754:
--
Issue Type: Sub-task (was: Bug)
Parent: MAPREDUCE-5422
> Job is marked as FAILED and also throwing the TransitonException instead of
> KILLED when issues a KILL command
> -
>
> Key: MAPREDUCE-4754
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4754
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 2.0.1-alpha, 2.0.2-alpha
>Reporter: Nishan Shetty
>
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at KILLED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:695)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:119)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:893)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:889)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6332) Add more required API's to MergeManager interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539669#comment-14539669 ] Rohith commented on MAPREDUCE-6332: --- [~vinodkv] Kindly review the patch .. > Add more required API's to MergeManager interface > -- > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.5.0, 2.6.0, 2.7.0 >Reporter: Rohith >Assignee: Rohith > Labels: BB2015-05-TBR > Attachments: 0001-MAPREDUCE-6332.patch, 0002-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4289) JobStatus.getReduceProgress() and JobStatus.getMapProgress() API's not giving any values
[ https://issues.apache.org/jira/browse/MAPREDUCE-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539586#comment-14539586 ] Rohith commented on MAPREDUCE-4289: --- The issue is still exist in the latest code base. This issue require more discussion on whether to change the hard coded values which may break comatibility for the MR client OR bring out a new design to handle this dielema. If not planning to fix , I think this can be closed as wont fix. > JobStatus.getReduceProgress() and JobStatus.getMapProgress() API's not giving > any values > > > Key: MAPREDUCE-4289 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4289 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Nishan Shetty > > 1.Run a simple job > 2.Invoke JobStatus.getReduceProgress() and JobStatus.getMapProgress() API's > Observe that these API's are giving zeros instead of showing map/reduce > progress -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4288) ClusterStatus.getMapTasks() and ClusterStatus.getReduceTasks() is giving one when no job is running
[ https://issues.apache.org/jira/browse/MAPREDUCE-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539585#comment-14539585 ] Rohith commented on MAPREDUCE-4288: --- Linking to similar issue in MR > ClusterStatus.getMapTasks() and ClusterStatus.getReduceTasks() is giving one > when no job is running > --- > > Key: MAPREDUCE-4288 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4288 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Nishan Shetty > > When no job is running in the cluster invoke the ClusterStatus.getMapTasks() > and ClusterStatus.getReduceTasks() API's > Observed that these API's are returning one instead of zero(as no job is > running) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4288) ClusterStatus.getMapTasks() and ClusterStatus.getReduceTasks() is giving one when no job is running
[ https://issues.apache.org/jira/browse/MAPREDUCE-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539583#comment-14539583 ] Rohith commented on MAPREDUCE-4288: --- The issue is still exist in the latest code base. This issue require more discussion on whether to change the hard coded values which may break comatibility for the MR client OR bring out a new design to handle this dielema. If not planning to fix , I think this can be closed as wont fix. > ClusterStatus.getMapTasks() and ClusterStatus.getReduceTasks() is giving one > when no job is running > --- > > Key: MAPREDUCE-4288 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4288 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Nishan Shetty > > When no job is running in the cluster invoke the ClusterStatus.getMapTasks() > and ClusterStatus.getReduceTasks() API's > Observed that these API's are returning one instead of zero(as no job is > running) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-4461) Resourcemanager UI does not show the queue details in IE
[ https://issues.apache.org/jira/browse/MAPREDUCE-4461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith resolved MAPREDUCE-4461. --- Resolution: Cannot Reproduce IE browser rendering problem should had fixed by YARN-1868. I can not reproduce the issue. Closing as Can not reproduce. Feel free to reopen if problem still exist. > Resourcemanager UI does not show the queue details in IE > > > Key: MAPREDUCE-4461 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4461 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Nishan Shetty > Attachments: ASF.LICENSE.NOT.GRANTED--screenshot-1.jpg > > > Resourcemanager UI does not show the queue details in IE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4754) Job is marked as FAILED and also throwing the TransitonException instead of KILLED when issues a KILL command
[
https://issues.apache.org/jira/browse/MAPREDUCE-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14537834#comment-14537834
]
Rohith commented on MAPREDUCE-4754:
---
I see current JobImpl also does not handle JOB_TASK_COMPLETED at KILLED, is it
potential issue still exist in trunk or branch-2?
> Job is marked as FAILED and also throwing the TransitonException instead of
> KILLED when issues a KILL command
> -
>
> Key: MAPREDUCE-4754
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4754
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.1-alpha, 2.0.2-alpha
>Reporter: Nishan Shetty
>
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at KILLED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:695)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:119)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:893)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:889)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4754) Job is marked as FAILED and also throwing the TransitonException instead of KILLED when issues a KILL command
[
https://issues.apache.org/jira/browse/MAPREDUCE-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14537835#comment-14537835
]
Rohith commented on MAPREDUCE-4754:
---
Any folks has seen this issue recently?
> Job is marked as FAILED and also throwing the TransitonException instead of
> KILLED when issues a KILL command
> -
>
> Key: MAPREDUCE-4754
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4754
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.1-alpha, 2.0.2-alpha
>Reporter: Nishan Shetty
>
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at KILLED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:695)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:119)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:893)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:889)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-4378) hadoop-validate-setup.sh fails to execute kinit command in secure mode
[ https://issues.apache.org/jira/browse/MAPREDUCE-4378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith resolved MAPREDUCE-4378. --- Resolution: Cannot Reproduce In trunk or branch-2, I dont see hadoop-validate-setup.sh script file anymore. Closing as 'Cannot reproduce'. Reopen the issue if any script files fails while execution kinit. > hadoop-validate-setup.sh fails to execute kinit command in secure mode > -- > > Key: MAPREDUCE-4378 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4378 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha, 3.0.0 > Environment: SUSE Linux Enterprise Server 11 (x86_64) > VERSION = 11 > PATCHLEVEL = 1 >Reporter: Nishan Shetty > > hadoop-validate-setup.sh is refering to the invalid kinit location. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-4453) Jobs should be executed as same user in hadoop-validate-setup.sh
[ https://issues.apache.org/jira/browse/MAPREDUCE-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith resolved MAPREDUCE-4453. --- Resolution: Cannot Reproduce In trunk or branch-2 , *su -c* dont exist any more in the script files. Closing as "cannot reporduce' > Jobs should be executed as same user in hadoop-validate-setup.sh > > > Key: MAPREDUCE-4453 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4453 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.1-alpha >Reporter: Nishan Shetty > > 'su -c' command should be removed in hadoop-validate-setup.sh as > TeraGen, Terasort and teravalidate jobs should be executed as same user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6342) Make POM project names consistent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6342: -- Attachment: MAPREDUCE-6342-branch-2.patch Updated patch for branch-2 > Make POM project names consistent > - > > Key: MAPREDUCE-6342 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6342 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Rohith >Assignee: Rohith >Priority: Minor > Labels: BB2015-05-RFC > Attachments: MAPREDUCE-6342-branch-2.patch, MAPREDUCE-6342.patch > > > This is track MR changes for POM changes by name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6342) Make POM project names consistent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6342: -- Labels: BB2015-05-RFC (was: BB2015-05-TBR) > Make POM project names consistent > - > > Key: MAPREDUCE-6342 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6342 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Rohith >Assignee: Rohith >Priority: Minor > Labels: BB2015-05-RFC > Attachments: MAPREDUCE-6342.patch > > > This is track MR changes for POM changes by name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6338) MR AppMaster does not honor ephemeral port range
[ https://issues.apache.org/jira/browse/MAPREDUCE-6338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520744#comment-14520744 ] Rohith commented on MAPREDUCE-6338: --- The patch looks good to me.. ApplicationMaster web port also random, I think it is good to make it also in port range.. Any thoughts? > MR AppMaster does not honor ephemeral port range > > > Key: MAPREDUCE-6338 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6338 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 2.6.0 >Reporter: Frank Nguyen >Assignee: Frank Nguyen > Attachments: MAPREDUCE-6338.002.patch > > > The MR AppMaster should only use port ranges defined in the > yarn.app.mapreduce.am.job.client.port-range property. On initial startup of > the MRAppMaster, it does use the port range defined in the property. > However, it also opens up a listener on a random ephemeral port. This is not > the Jetty listener. It is another listener opened by the MRAppMaster via > another thread and is recognized by the RM. Other nodes will try to > communicate to it via that random port. With firewall settings on, the MR > job will fail because the random port is not opened. This problem has caused > others to have all OS ephemeral ports opened to have MR jobs run. > This is related to MAPREDUCE-4079 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6332) Add more required API's to MergeManager interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516344#comment-14516344 ] Rohith commented on MAPREDUCE-6332: --- bq. MergeThread.java directly uses the merge-manager impl, that should be fixed too? Agree, I missed it bq. Add some javadoc to the new methods? Done. Updated the patch fixing comments. Kindly review the updated patch. > Add more required API's to MergeManager interface > -- > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.5.0, 2.6.0, 2.7.0 >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6332.patch, 0002-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6332) Add more required API's to MergeManager interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6332: -- Attachment: 0002-MAPREDUCE-6332.patch > Add more required API's to MergeManager interface > -- > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.5.0, 2.6.0, 2.7.0 >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6332.patch, 0002-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6332) Add more required API's to MergeManager interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516318#comment-14516318 ] Rohith commented on MAPREDUCE-6332: --- Thanks [~vinodkv] for sharing your thoughts.. bq. This almost seems like a new feature to me, at least given that we have to expose more APIs to the outside world. I will mark the JIRA as New Feature. > Add more required API's to MergeManager interface > -- > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.5.0, 2.6.0, 2.7.0 >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6332) Add more required API's to MergeManager interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6332: -- Issue Type: New Feature (was: Bug) > Add more required API's to MergeManager interface > -- > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.5.0, 2.6.0, 2.7.0 >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6342) Make POM project names consistent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6342: -- Status: Patch Available (was: Open) > Make POM project names consistent > - > > Key: MAPREDUCE-6342 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6342 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Rohith >Assignee: Rohith >Priority: Minor > Attachments: MAPREDUCE-6342.patch > > > This is track MR changes for POM changes by name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6342) Make POM project names consistent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6342: -- Attachment: MAPREDUCE-6342.patch > Make POM project names consistent > - > > Key: MAPREDUCE-6342 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6342 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Rohith >Assignee: Rohith >Priority: Minor > Attachments: MAPREDUCE-6342.patch > > > This is track MR changes for POM changes by name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6342) Make POM project names consistent
Rohith created MAPREDUCE-6342: - Summary: Make POM project names consistent Key: MAPREDUCE-6342 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6342 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Rohith Assignee: Rohith Priority: Minor This is track MR changes for POM changes by name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6332) Add more required API's to MergeManager interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14513831#comment-14513831 ] Rohith commented on MAPREDUCE-6332: --- I'd appreciate if any commiter/PMC member comment on the JIRA. > Add more required API's to MergeManager interface > -- > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.5.0, 2.6.0, 2.7.0 >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6329) Failure of start map task on NM cause job hang
[ https://issues.apache.org/jira/browse/MAPREDUCE-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508847#comment-14508847 ] Rohith commented on MAPREDUCE-6329: --- No need to create new JIRA in YARN..This issue only can be moved.. > Failure of start map task on NM cause job hang > -- > > Key: MAPREDUCE-6329 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6329 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Peng Zhang > Attachments: syslog.tgz, yarn-app.log > > > During rolling update of NM, AM start of container on NM failed. > And then job hang there. > Attach AM logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6329) Failure of start map task on NM cause job hang
[ https://issues.apache.org/jira/browse/MAPREDUCE-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508829#comment-14508829 ] Rohith commented on MAPREDUCE-6329: --- Thanks Peng Zhang for your analysis.. As [~jlowe] said, this is bug in YARN. [~peng.zhang] Would you like to provide patch for this issue? > Failure of start map task on NM cause job hang > -- > > Key: MAPREDUCE-6329 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6329 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Peng Zhang > Attachments: syslog.tgz, yarn-app.log > > > During rolling update of NM, AM start of container on NM failed. > And then job hang there. > Attach AM logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6332) Add more required API's to MergeManager interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6332: -- Status: Patch Available (was: Open) > Add more required API's to MergeManager interface > -- > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.6.0, 2.5.0, 2.7.0 >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6332) Add more required API's to MergeManager interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6332: -- Target Version/s: 2.8.0, 2.7.1 Affects Version/s: 2.7.0 2.5.0 2.6.0 > Add more required API's to MergeManager interface > -- > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.5.0, 2.6.0, 2.7.0 >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6332) Add more required API's to MergeManager interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6332: -- Attachment: 0001-MAPREDUCE-6332.patch Attaching the patch, Kindly review the patch > Add more required API's to MergeManager interface > -- > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6332) Add more required API's to MergeManager interface
Rohith created MAPREDUCE-6332: - Summary: Add more required API's to MergeManager interface Key: MAPREDUCE-6332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Rohith Assignee: Rohith MR provides ability to the user for plugin custom ShuffleConsumerPlugin using *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is allowed to use this configuration as plugin, user also interest in implementing his own MergeManagerImpl. But now , user is forced to use MR provided MergeManagerImpl instead of custom MergeManagerImpl when user is using shuffle.consumer.plugin class. There should be well defined API's in MergeManager that can be used for any implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6329) Failure of start map task on NM cause job hang
[ https://issues.apache.org/jira/browse/MAPREDUCE-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507730#comment-14507730 ] Rohith commented on MAPREDUCE-6329: --- bq. Therefore I don't see how the RM could reasonably be expiring the node, nor should the node be unregistering Agree, practically thinking it won't be possible. bq. Re-registration does not kill containers on the node Without NM work-preserving restart enabled , RM should kill the running containers on re-registration. IIRC, It is legacy behavior. > Failure of start map task on NM cause job hang > -- > > Key: MAPREDUCE-6329 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6329 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Peng Zhang > Attachments: syslog.tgz, yarn-app.log > > > During rolling update of NM, AM start of container on NM failed. > And then job hang there. > Attach AM logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6329) Failure of start map task on NM cause job hang
[ https://issues.apache.org/jira/browse/MAPREDUCE-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507671#comment-14507671 ] Rohith commented on MAPREDUCE-6329: --- But I don't see any Node removed event from attached logs. The question remains unanswered!! > Failure of start map task on NM cause job hang > -- > > Key: MAPREDUCE-6329 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6329 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Peng Zhang > Attachments: syslog.tgz, yarn-app.log > > > During rolling update of NM, AM start of container on NM failed. > And then job hang there. > Attach AM logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6329) Failure of start map task on NM cause job hang
[ https://issues.apache.org/jira/browse/MAPREDUCE-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507637#comment-14507637 ] Rohith commented on MAPREDUCE-6329: --- bq. is there anything in the RM log indicating why the container transitioned from ALLOCATED to KILLED? This would be probably because during rolling upgrade , NM was down for some time. So Node_Removed event might have occurred either because of expiry or reconnected event. Node removed event kills all the running containers which has been done before container is pulled by AM. > Failure of start map task on NM cause job hang > -- > > Key: MAPREDUCE-6329 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6329 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Peng Zhang > Attachments: syslog.tgz, yarn-app.log > > > During rolling update of NM, AM start of container on NM failed. > And then job hang there. > Attach AM logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6311) AM JVM hungs after job unregister and finished
[ https://issues.apache.org/jira/browse/MAPREDUCE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6311: -- Priority: Blocker (was: Major) > AM JVM hungs after job unregister and finished > -- > > Key: MAPREDUCE-6311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6311 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Rohith >Assignee: Rohith >Priority: Blocker > Attachments: 0001-MAPREDUCE-6311.patch, 0001-MAPREDUCE-6311.patch, > MR_TD.out > > > It is observed that MRAppMaster JVM hungs after unregistered with > ResourceManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6311) AM JVM hungs after job unregister and finished
[ https://issues.apache.org/jira/browse/MAPREDUCE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6311: -- Affects Version/s: 2.7.0 > AM JVM hungs after job unregister and finished > -- > > Key: MAPREDUCE-6311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6311 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6311.patch, 0001-MAPREDUCE-6311.patch, > MR_TD.out > > > It is observed that MRAppMaster JVM hungs after unregistered with > ResourceManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6311) AM JVM hungs after job unregister and finished
[ https://issues.apache.org/jira/browse/MAPREDUCE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485773#comment-14485773 ] Rohith commented on MAPREDUCE-6311: --- Linking the issue to which breaks this. > AM JVM hungs after job unregister and finished > -- > > Key: MAPREDUCE-6311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6311 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6311.patch, 0001-MAPREDUCE-6311.patch, > MR_TD.out > > > It is observed that MRAppMaster JVM hungs after unregistered with > ResourceManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6311) AM JVM hungs after job unregister and finished
[ https://issues.apache.org/jira/browse/MAPREDUCE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485754#comment-14485754 ] Rohith commented on MAPREDUCE-6311: --- It is there in 2.7. The issue breaks is HADOOP-11754 which is fixed in 2.7 > AM JVM hungs after job unregister and finished > -- > > Key: MAPREDUCE-6311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6311 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6311.patch, 0001-MAPREDUCE-6311.patch, > MR_TD.out > > > It is observed that MRAppMaster JVM hungs after unregistered with > ResourceManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6311) AM JVM hungs after job unregister and finished
[ https://issues.apache.org/jira/browse/MAPREDUCE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485749#comment-14485749 ] Rohith commented on MAPREDUCE-6311: --- I believe it must be there in 2.7 also. I will check for the check in history and confirm. > AM JVM hungs after job unregister and finished > -- > > Key: MAPREDUCE-6311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6311 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6311.patch, 0001-MAPREDUCE-6311.patch, > MR_TD.out > > > It is observed that MRAppMaster JVM hungs after unregistered with > ResourceManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6311) AM JVM hungs after job unregister and finished
[ https://issues.apache.org/jira/browse/MAPREDUCE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485712#comment-14485712 ] Rohith commented on MAPREDUCE-6311: --- I confirm that it is happening every time in my env. I have verified in both SuSE linux and OS X. It is occurring all the time. > AM JVM hungs after job unregister and finished > -- > > Key: MAPREDUCE-6311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6311 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6311.patch, 0001-MAPREDUCE-6311.patch, > MR_TD.out > > > It is observed that MRAppMaster JVM hungs after unregistered with > ResourceManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6311) AM JVM hungs after job unregister and finished
[ https://issues.apache.org/jira/browse/MAPREDUCE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485686#comment-14485686 ] Rohith commented on MAPREDUCE-6311: --- I am using trunk version for test. I encountered in trunk version. > AM JVM hungs after job unregister and finished > -- > > Key: MAPREDUCE-6311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6311 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6311.patch, 0001-MAPREDUCE-6311.patch, > MR_TD.out > > > It is observed that MRAppMaster JVM hungs after unregistered with > ResourceManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6189) TestMRTimelineEventHandling fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485666#comment-14485666
]
Rohith commented on MAPREDUCE-6189:
---
Hi [~zjshen]
bq. When an application is finished, the AM container is still alive for
minutes. I didn't change my config that was used in 2.6 before. Not sure if it
is a related issue.
I think this is MAPREDUCE-6311. All the time whenever MR job has run, AM jvm
does not shutdown. Everytime AM expired event is triggered.
> TestMRTimelineEventHandling fails in trunk
> --
>
> Key: MAPREDUCE-6189
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6189
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Junping Du
>
> From https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1988/:
> {code}
> REGRESSION:
> org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMRTimelineEventHandling
> Error Message:
> Job didn't finish in 30 seconds
> Stack Trace:
> java.io.IOException: Job didn't finish in 30 seconds
> at
> org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622)
> at
> org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMRTimelineEventHandling(TestMRTimelineEventHandling.java:105)
> REGRESSION:
> org.apache.hadoop.mapred.TestMRTimelineEventHandling.testTimelineServiceStartInMiniCluster
> Error Message:
> Job didn't finish in 30 seconds
> Stack Trace:
> java.io.IOException: Job didn't finish in 30 seconds
> at
> org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622)
> at
> org.apache.hadoop.mapred.TestMRTimelineEventHandling.testTimelineServiceStartInMiniCluster(TestMRTimelineEventHandling.java:61)
> REGRESSION:
> org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMapreduceJobTimelineServiceEnabled
> Error Message:
> Job didn't finish in 30 seconds
> Stack Trace:
> java.io.IOException: Job didn't finish in 30 seconds
> at
> org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622)
> at
> org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMapreduceJobTimelineServiceEnabled(TestMRTimelineEventHandling.java:198)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6311) AM JVM hungs after job unregister and finished
[ https://issues.apache.org/jira/browse/MAPREDUCE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6311: -- Attachment: 0001-MAPREDUCE-6311.patch Updated patch fixing tests failure > AM JVM hungs after job unregister and finished > -- > > Key: MAPREDUCE-6311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6311 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6311.patch, 0001-MAPREDUCE-6311.patch, > MR_TD.out > > > It is observed that MRAppMaster JVM hungs after unregistered with > ResourceManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6311) AM JVM hungs after job unregister and finished
[
https://issues.apache.org/jira/browse/MAPREDUCE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14482936#comment-14482936
]
Rohith commented on MAPREDUCE-6311:
---
Attached the patch for stopping ScheduledThreadPoolExecutor using
{{SignerSecretProvider#destroy}} API. Tested the patch in cluster, it is
working fine.
Kindly review the patch
> AM JVM hungs after job unregister and finished
> --
>
> Key: MAPREDUCE-6311
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6311
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: 0001-MAPREDUCE-6311.patch, MR_TD.out
>
>
> It is observed that MRAppMaster JVM hungs after unregistered with
> ResourceManager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6311) AM JVM hungs after job unregister and finished
[ https://issues.apache.org/jira/browse/MAPREDUCE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6311: -- Status: Patch Available (was: Open) > AM JVM hungs after job unregister and finished > -- > > Key: MAPREDUCE-6311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6311 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6311.patch, MR_TD.out > > > It is observed that MRAppMaster JVM hungs after unregistered with > ResourceManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6311) AM JVM hungs after job unregister and finished
[ https://issues.apache.org/jira/browse/MAPREDUCE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6311: -- Attachment: 0001-MAPREDUCE-6311.patch > AM JVM hungs after job unregister and finished > -- > > Key: MAPREDUCE-6311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6311 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Rohith >Assignee: Rohith > Attachments: 0001-MAPREDUCE-6311.patch, MR_TD.out > > > It is observed that MRAppMaster JVM hungs after unregistered with > ResourceManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6311) AM JVM hungs after job unregister and finished
[ https://issues.apache.org/jira/browse/MAPREDUCE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14482928#comment-14482928 ] Rohith commented on MAPREDUCE-6311: --- While starting HttpServer2, SignerSecretProvider is added to webAppContext in attribute. SignerSecretProvider is creating the ScheduledThreadPoolExecutor and scheduled but never stopped explicitly while stopping HttpServer2 which causes MRAppMaster JVM to hung after unregister with RM. > AM JVM hungs after job unregister and finished > -- > > Key: MAPREDUCE-6311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6311 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Rohith >Assignee: Rohith > Attachments: MR_TD.out > > > It is observed that MRAppMaster JVM hungs after unregistered with > ResourceManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6311) AM JVM hungs after job unregister and finished
[
https://issues.apache.org/jira/browse/MAPREDUCE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14482704#comment-14482704
]
Rohith commented on MAPREDUCE-6311:
---
Extracting thread dump, ScheduledThreadPoolExecutor is started while starting
MRAppMaster but is not stopped during shutdown. The stack trace can be seen for
the same.
{noformat}
"pool-6-thread-1" prio=10 tid=0x7fe3d0dc7000 nid=0xb4a waiting on condition
[0x7fe3d4643000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xf216dc08> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> AM JVM hungs after job unregister and finished
> --
>
> Key: MAPREDUCE-6311
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6311
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MR_TD.out
>
>
> It is observed that MRAppMaster JVM hungs after unregistered with
> ResourceManager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6311) AM JVM hungs after job unregister and finished
[ https://issues.apache.org/jira/browse/MAPREDUCE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6311: -- Attachment: MR_TD.out Attaching thread dump for AM JVM > AM JVM hungs after job unregister and finished > -- > > Key: MAPREDUCE-6311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6311 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Rohith >Assignee: Rohith > Attachments: MR_TD.out > > > It is observed that MRAppMaster JVM hungs after unregistered with > ResourceManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6311) AM JVM hungs after job unregister and finished
Rohith created MAPREDUCE-6311: - Summary: AM JVM hungs after job unregister and finished Key: MAPREDUCE-6311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6311 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Rohith Assignee: Rohith It is observed that MRAppMaster JVM hungs after unregistered with ResourceManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6291) Correct mapred queue usage command
[
https://issues.apache.org/jira/browse/MAPREDUCE-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383385#comment-14383385
]
Rohith commented on MAPREDUCE-6291:
---
I think it is better to strict to [~qwertymaniac] suggestion, all Hadoop script
help message changes can go all together in one jira. I believe this will
reduce work.
> Correct mapred queue usage command
> --
>
> Key: MAPREDUCE-6291
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6291
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: MAPRED-6291-001.patch, MAPRED-6291.patch,
> MAPREDUCE-6291-002.patch
>
>
> *Currently it is like following..*
> Usage: JobQueueClient
> *It should be*
> Usage: queue
> *For more Details check following*
> {noformat}
> hdfs@host1:/hadoop/bin> ./mapred queue
> Usage: JobQueueClient
> [-list]
> [-info [-showJobs]]
> [-showacls]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6291) Correct mapred queue usage command
[
https://issues.apache.org/jira/browse/MAPREDUCE-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383266#comment-14383266
]
Rohith commented on MAPREDUCE-6291:
---
If no commands need to change then I think YARN-3398 can closed as 'Not a
Problem' if you dont have any concern.
> Correct mapred queue usage command
> --
>
> Key: MAPREDUCE-6291
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6291
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: MAPRED-6291-001.patch, MAPRED-6291.patch,
> MAPREDUCE-6291-002.patch
>
>
> *Currently it is like following..*
> Usage: JobQueueClient
> *It should be*
> Usage: queue
> *For more Details check following*
> {noformat}
> hdfs@host1:/hadoop/bin> ./mapred queue
> Usage: JobQueueClient
> [-list]
> [-info [-showJobs]]
> [-showacls]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6291) Correct mapred queue usage command
[
https://issues.apache.org/jira/browse/MAPREDUCE-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14379693#comment-14379693
]
Rohith commented on MAPREDUCE-6291:
---
Probably it is would be good if usage command are in consistent like below.
{{Usage: mapred }}
any thought?
> Correct mapred queue usage command
> --
>
> Key: MAPREDUCE-6291
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6291
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: MAPRED-6291.patch
>
>
> *Currently it is like following..*
> Usage: JobQueueClient
> *It should be*
> Usage: queue
> *For more Details check following*
> {noformat}
> hdfs@host1:/hadoop/bin> ./mapred queue
> Usage: JobQueueClient
> [-list]
> [-info [-showJobs]]
> [-showacls]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6291) Correct mapred queue usage command
[
https://issues.apache.org/jira/browse/MAPREDUCE-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14379687#comment-14379687
]
Rohith commented on MAPREDUCE-6291:
---
typo mistake : *wold* - -> *would*
> Correct mapred queue usage command
> --
>
> Key: MAPREDUCE-6291
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6291
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: MAPRED-6291.patch
>
>
> *Currently it is like following..*
> Usage: JobQueueClient
> *It should be*
> Usage: queue
> *For more Details check following*
> {noformat}
> hdfs@host1:/hadoop/bin> ./mapred queue
> Usage: JobQueueClient
> [-list]
> [-info [-showJobs]]
> [-showacls]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6291) Correct mapred queue usage command
[
https://issues.apache.org/jira/browse/MAPREDUCE-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14379684#comment-14379684
]
Rohith commented on MAPREDUCE-6291:
---
There is another command need to change {{mapred job}}. Similarly, it wold be
better to cross check for other commands.
> Correct mapred queue usage command
> --
>
> Key: MAPREDUCE-6291
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6291
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 2.6.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: MAPRED-6291.patch
>
>
> *Currently it is like following..*
> Usage: JobQueueClient
> *It should be*
> Usage: queue
> *For more Details check following*
> {noformat}
> hdfs@host1:/hadoop/bin> ./mapred queue
> Usage: JobQueueClient
> [-list]
> [-info [-showJobs]]
> [-showacls]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5807) Print usage for TeraSort job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14366932#comment-14366932 ] Rohith commented on MAPREDUCE-5807: --- Thanks for the updated patch. Changes looks good to me +1 (non-binding) > Print usage for TeraSort job. > - > > Key: MAPREDUCE-5807 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5807 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Reporter: Rohith >Assignee: Rohith >Priority: Trivial > Attachments: 0001-MAPREDUCE-5807.patch, 0002-MAPREDUCE-5807.patch, > MAPREDUCE-5807.patch > > > For new to hadoop, try for getting help mesage for examples jobs provided in > mapreduce. These Usage helps them in providing arguements. > terasort job execution does not print Usage message instead throw exception. > ./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar terasort > 14/03/24 15:34:55 INFO terasort.TeraSort: starting > java.lang.ArrayIndexOutOfBoundsException: 0 > at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:283) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) > at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) > at > org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6276) Error in encrypted shuffle
[
https://issues.apache.org/jira/browse/MAPREDUCE-6276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14365041#comment-14365041
]
Rohith commented on MAPREDUCE-6276:
---
I suspect importing the certificates into truststore has problem. Each hosts
all jks need to be imported to all other hosts. May be cross check the import
of certificates.
> Error in encrypted shuffle
> --
>
> Key: MAPREDUCE-6276
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6276
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Environment: Centos 6.5
> Multinode hdp 2.2 cluster
> using mapreduce version 2
> hadoop version 2.6.0
> namenode - hdprndmaster.dm.com (10.200.100.83)
> resourcemanager/datanode/nodemanager - hdprndnode.dm.com (10.200.100.85)
>Reporter: Kuldeep Kulkarni
>Priority: Blocker
> Attachments: capacity-scheduler.xml, commons-logging.properties,
> container-executor.cfg, core-site.xml, dfs.exclude, dfs_data_dir_mount.hist,
> hadoop-env.sh, hadoop-metrics2.properties, hadoop-policy.xml, hdfs-site.xml,
> health_check, log.txt, log4j.properties, mapred-env.sh, mapred-site.xml,
> slaves, ssl-client.xml, ssl-server.xml, task-log4j.properties,
> taskcontroller.cfg, yarn-env.sh, yarn-site.xml
>
>
> Hey Guys,
> After enabling wire encryption my UIs are working fine, I'm able to
> read/write to hdfs securely however encrypted shuffle is not working. I'm
> getting below error, could you please help me ?
> Note - mappers are getting finished successfully however job gets failed
> during shuffle.
> {code}
> 2015-03-17 17:00:54,322 WARN [fetcher#1]
> org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to
> hdprndnode.dm.com:13562 with 8 map outputs
> javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> {code}
> Please find attached full log for more details.
> Thanks,
> Kuldeep
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5807) Print usage for TeraSort job.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14364887#comment-14364887
]
Rohith commented on MAPREDUCE-5807:
---
Usage print as below along with configurations
{noformat}
Usage: terasort
TeraSort configurations are:
Number of rows to generate
Number of partitions used for sampling
sampling size
Wheather to do final sync before the stream
is closed
Wheather to use tera scheduler
Wheather to use simple partitioner
Number of replications to be stored
for output data
{noformat}
> Print usage for TeraSort job.
> -
>
> Key: MAPREDUCE-5807
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5807
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: examples
>Reporter: Rohith
>Assignee: Rohith
>Priority: Trivial
> Attachments: 0001-MAPREDUCE-5807.patch, MAPREDUCE-5807.patch
>
>
> For new to hadoop, try for getting help mesage for examples jobs provided in
> mapreduce. These Usage helps them in providing arguements.
> terasort job execution does not print Usage message instead throw exception.
> ./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar terasort
> 14/03/24 15:34:55 INFO terasort.TeraSort: starting
> java.lang.ArrayIndexOutOfBoundsException: 0
> at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:283)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at
> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5807) Print usage for TeraSort job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-5807: -- Status: Patch Available (was: Open) > Print usage for TeraSort job. > - > > Key: MAPREDUCE-5807 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5807 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Reporter: Rohith >Assignee: Rohith >Priority: Trivial > Attachments: 0001-MAPREDUCE-5807.patch, MAPREDUCE-5807.patch > > > For new to hadoop, try for getting help mesage for examples jobs provided in > mapreduce. These Usage helps them in providing arguements. > terasort job execution does not print Usage message instead throw exception. > ./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar terasort > 14/03/24 15:34:55 INFO terasort.TeraSort: starting > java.lang.ArrayIndexOutOfBoundsException: 0 > at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:283) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) > at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) > at > org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5807) Print usage for TeraSort job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14364883#comment-14364883 ] Rohith commented on MAPREDUCE-5807: --- Updated the patch, kindly review > Print usage for TeraSort job. > - > > Key: MAPREDUCE-5807 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5807 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Reporter: Rohith >Assignee: Rohith >Priority: Trivial > Attachments: 0001-MAPREDUCE-5807.patch, MAPREDUCE-5807.patch > > > For new to hadoop, try for getting help mesage for examples jobs provided in > mapreduce. These Usage helps them in providing arguements. > terasort job execution does not print Usage message instead throw exception. > ./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar terasort > 14/03/24 15:34:55 INFO terasort.TeraSort: starting > java.lang.ArrayIndexOutOfBoundsException: 0 > at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:283) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) > at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) > at > org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5807) Print usage for TeraSort job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-5807: -- Attachment: 0001-MAPREDUCE-5807.patch > Print usage for TeraSort job. > - > > Key: MAPREDUCE-5807 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5807 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Reporter: Rohith >Assignee: Rohith >Priority: Trivial > Attachments: 0001-MAPREDUCE-5807.patch, MAPREDUCE-5807.patch > > > For new to hadoop, try for getting help mesage for examples jobs provided in > mapreduce. These Usage helps them in providing arguements. > terasort job execution does not print Usage message instead throw exception. > ./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar terasort > 14/03/24 15:34:55 INFO terasort.TeraSort: starting > java.lang.ArrayIndexOutOfBoundsException: 0 > at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:283) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) > at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) > at > org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6190) MR Job is stuck because of one mapper stuck in STARTING
[
https://issues.apache.org/jira/browse/MAPREDUCE-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14342806#comment-14342806
]
Rohith commented on MAPREDUCE-6190:
---
[~amalhotra159] [~tannin]
Would mind attaching full log of MRAppMaster. And did anyone observe scenario
mentioned in YARN-1680?
> MR Job is stuck because of one mapper stuck in STARTING
> ---
>
> Key: MAPREDUCE-6190
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6190
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ankit Malhotra
>
> Trying to figure out a weird issue we started seeing on our CDH5.1.0 cluster
> with map reduce jobs on YARN.
> We had a job stuck for hours because one of the mappers never started up
> fully. Basically, the map task had 2 attempts, the first one failed and the
> AM tried to schedule a second one and the second attempt was stuck on STATE:
> STARTING, STATUS: NEW. A node never got assigned and the task along with the
> job was stuck indefinitely.
> The AM logs had this being logged again and again:
> {code}
> 2014-12-09 19:25:12,347 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down 0
> 2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_1408745633994_450952_02_003807
> 2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce preemption
> successful attempt_1408745633994_450952_r_48_1000
> 2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down all
> scheduled reduces:0
> 2014-12-09 19:25:13,352 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting
> attempt_1408745633994_450952_r_50_1000
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=0
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
> completedMapPercent 0.99968 totalMemLimit:1722880 finalMapMemLimit:2560
> finalReduceMemLimit:1720320 netScheduledMapMem:2560
> netScheduledReduceMem:1722880
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down 0
> 2014-12-09 19:25:13,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
> PendingReds:77 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0
> AssignedReds:673 CompletedMaps:3124 CompletedReds:0 ContAlloc:4789
> ContRel:798 HostLocal:2944 RackLocal:155
> 2014-12-09 19:25:14,353 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:78 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0
> AssignedReds:673 CompletedMaps:3124 CompletedReds:0 ContAlloc:4789
> ContRel:798 HostLocal:2944 RackLocal:155
> 2014-12-09 19:25:14,359 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=0
> {code}
> On killing the task manually, the AM started up the task again, scheduled and
> ran it successfully completing the task and the job with it.
> Some quick code grepping led us here:
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-mapreduce-client-app/2.3.0/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java#397
> But still dont quite understand why this would happen once in a while and why
> the job would suddenly be ok once the stuck task is manually killed.
> Note: Other jobs succeed on the cluster while this job is stuck.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6212) Hadoop 2.6.0: Basic error “starting MRAppMaster” after installing
[ https://issues.apache.org/jira/browse/MAPREDUCE-6212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268142#comment-14268142 ] Rohith commented on MAPREDUCE-6212: --- Since issue is in MRAppMaster, moving to MapReduce project. > Hadoop 2.6.0: Basic error “starting MRAppMaster” after installing > - > > Key: MAPREDUCE-6212 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6212 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 > Environment: Ubuntu 64bit >Reporter: Dinh Hoang Mai >Priority: Critical > Fix For: 2.6.0 > > > I have just started to work with Hadoop 2. > After installing with basic configs, I always failed to run any examples. Has > anyone seen this problem and please help me? > This is the log > 2015-01-08 01:52:01,599 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for > application appattempt_1420648881673_0004_01 > 2015-01-08 01:52:01,764 FATAL [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster > java.lang.RuntimeException: java.lang.reflect.InvocationTargetException > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) > at org.apache.hadoop.security.Groups.(Groups.java:70) > at org.apache.hadoop.security.Groups.(Groups.java:66) > at > org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:280) > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:271) > at > org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:299) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1473) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1429) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:408) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129) > ... 7 more > Caused by: java.lang.UnsatisfiedLinkError: > org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative()V > at > org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative(Native > Method) > at > org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:49) > at > org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.(JniBasedUnixGroupsMappingWithFallback.java:39) > ... 12 more > 2015-01-08 01:52:01,767 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting > with status 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Moved] (MAPREDUCE-6212) Hadoop 2.6.0: Basic error “starting MRAppMaster” after installing
[ https://issues.apache.org/jira/browse/MAPREDUCE-6212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith moved YARN-3012 to MAPREDUCE-6212: - Component/s: (was: security) security Fix Version/s: (was: 2.6.0) 2.6.0 Target Version/s: (was: 2.6.0) Affects Version/s: (was: 2.6.0) 2.6.0 Key: MAPREDUCE-6212 (was: YARN-3012) Project: Hadoop Map/Reduce (was: Hadoop YARN) > Hadoop 2.6.0: Basic error “starting MRAppMaster” after installing > - > > Key: MAPREDUCE-6212 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6212 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 > Environment: Ubuntu 64bit >Reporter: Dinh Hoang Mai >Priority: Critical > Fix For: 2.6.0 > > > I have just started to work with Hadoop 2. > After installing with basic configs, I always failed to run any examples. Has > anyone seen this problem and please help me? > This is the log > 2015-01-08 01:52:01,599 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for > application appattempt_1420648881673_0004_01 > 2015-01-08 01:52:01,764 FATAL [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster > java.lang.RuntimeException: java.lang.reflect.InvocationTargetException > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) > at org.apache.hadoop.security.Groups.(Groups.java:70) > at org.apache.hadoop.security.Groups.(Groups.java:66) > at > org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:280) > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:271) > at > org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:299) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1473) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1429) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:408) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129) > ... 7 more > Caused by: java.lang.UnsatisfiedLinkError: > org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative()V > at > org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative(Native > Method) > at > org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:49) > at > org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.(JniBasedUnixGroupsMappingWithFallback.java:39) > ... 12 more > 2015-01-08 01:52:01,767 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting > with status 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6189) TestMRTimelineEventHandling fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith reassigned MAPREDUCE-6189:
-
Assignee: Rohith
> TestMRTimelineEventHandling fails in trunk
> --
>
> Key: MAPREDUCE-6189
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6189
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Rohith
>Priority: Minor
>
> From https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1988/:
> {code}
> REGRESSION:
> org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMRTimelineEventHandling
> Error Message:
> Job didn't finish in 30 seconds
> Stack Trace:
> java.io.IOException: Job didn't finish in 30 seconds
> at
> org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622)
> at
> org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMRTimelineEventHandling(TestMRTimelineEventHandling.java:105)
> REGRESSION:
> org.apache.hadoop.mapred.TestMRTimelineEventHandling.testTimelineServiceStartInMiniCluster
> Error Message:
> Job didn't finish in 30 seconds
> Stack Trace:
> java.io.IOException: Job didn't finish in 30 seconds
> at
> org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622)
> at
> org.apache.hadoop.mapred.TestMRTimelineEventHandling.testTimelineServiceStartInMiniCluster(TestMRTimelineEventHandling.java:61)
> REGRESSION:
> org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMapreduceJobTimelineServiceEnabled
> Error Message:
> Job didn't finish in 30 seconds
> Stack Trace:
> java.io.IOException: Job didn't finish in 30 seconds
> at
> org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622)
> at
> org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMapreduceJobTimelineServiceEnabled(TestMRTimelineEventHandling.java:198)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6183) Fix findbugs warnings in mapreduce-examples
[ https://issues.apache.org/jira/browse/MAPREDUCE-6183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14239046#comment-14239046 ] Rohith commented on MAPREDUCE-6183: --- I believe these findbug error would be due to either change is java version or change in findbug version. Previously these were not failed. > Fix findbugs warnings in mapreduce-examples > --- > > Key: MAPREDUCE-6183 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6183 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Reporter: Akira AJISAKA >Assignee: Rohith > Labels: findbugs, newbie > > Work on MAPREDUCE-5800 has exposed some findbugs warnings. > https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5063//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-examples.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6183) Fix findbugs warnings in mapreduce-examples
[ https://issues.apache.org/jira/browse/MAPREDUCE-6183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith reassigned MAPREDUCE-6183: - Assignee: Rohith > Fix findbugs warnings in mapreduce-examples > --- > > Key: MAPREDUCE-6183 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6183 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Reporter: Akira AJISAKA >Assignee: Rohith > Labels: findbugs, newbie > > Work on MAPREDUCE-5800 has exposed some findbugs warnings. > https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5063//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-examples.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6179) Running any MapReduce jobs is throwing Heap Error
[ https://issues.apache.org/jira/browse/MAPREDUCE-6179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234322#comment-14234322 ] Rohith commented on MAPREDUCE-6179: --- What is the applicationMaster heap memory configured? > Running any MapReduce jobs is throwing Heap Error > - > > Key: MAPREDUCE-6179 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6179 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.2.0 > Environment: Linux 2.6.32-358.el6.x86_64 > RAM : 12 GB >Reporter: Ketan Deshmukh > > I have a hadoop distribution installed on a cluster with 1 namenode and 2 > data nodes. I have been trying to run the default mapreduce example included > in the hadoop package and its been throwing the following error : > "Error occurred during initialization of VM > Could not reserve enough space for object heap" > The complete message test is as following : > [192769@hawq /]$ hadoop jar > /usr/lib/gphd/hadoop-mapreduce/hadoop-mapreduce-examples-2.2.0-gphd-3.0.1.0.jar > pi 10 100 > Number of Maps = 10 > Samples per Map = 100 > Wrote input for Map #0 > Wrote input for Map #1 > Wrote input for Map #2 > Wrote input for Map #3 > Wrote input for Map #4 > Wrote input for Map #5 > Wrote input for Map #6 > Wrote input for Map #7 > Wrote input for Map #8 > Wrote input for Map #9 > Starting Job > 14/12/04 17:28:44 INFO client.RMProxy: Connecting to ResourceManager at > dn2.tcsgegdc.com/3.209.124.208:8032 > 14/12/04 17:28:44 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 50 > for 192769 on 3.209.124.204:8020 > 14/12/04 17:28:44 INFO security.TokenCache: Got dt for > hdfs://pcc.tcsgegdc.com:8020; Kind: HDFS_DELEGATION_TOKEN, Service: > 3.209.124.204:8020, Ident: (HDFS_DELEGATION_TOKEN token 50 for 192769) > 14/12/04 17:28:44 INFO input.FileInputFormat: Total input paths to process : > 10 > 14/12/04 17:28:44 INFO mapreduce.JobSubmitter: number of splits:10 > 14/12/04 17:28:44 INFO Configuration.deprecation: user.name is deprecated. > Instead, use mapreduce.job.user.name > 14/12/04 17:28:44 INFO Configuration.deprecation: mapred.jar is deprecated. > Instead, use mapreduce.job.jar > 14/12/04 17:28:44 INFO Configuration.deprecation: > mapred.map.tasks.speculative.execution is deprecated. Instead, use > mapreduce.map.speculative > 14/12/04 17:28:44 INFO Configuration.deprecation: mapred.reduce.tasks is > deprecated. Instead, use mapreduce.job.reduces > 14/12/04 17:28:44 INFO Configuration.deprecation: mapred.output.value.class > is deprecated. Instead, use mapreduce.job.output.value.class > 14/12/04 17:28:44 INFO Configuration.deprecation: > mapred.reduce.tasks.speculative.execution is deprecated. Instead, use > mapreduce.reduce.speculative > 14/12/04 17:28:44 INFO Configuration.deprecation: mapreduce.map.class is > deprecated. Instead, use mapreduce.job.map.class > 14/12/04 17:28:44 INFO Configuration.deprecation: mapred.job.name is > deprecated. Instead, use mapreduce.job.name > 14/12/04 17:28:44 INFO Configuration.deprecation: mapreduce.reduce.class is > deprecated. Instead, use mapreduce.job.reduce.class > 14/12/04 17:28:44 INFO Configuration.deprecation: mapreduce.inputformat.class > is deprecated. Instead, use mapreduce.job.inputformat.class > 14/12/04 17:28:44 INFO Configuration.deprecation: mapred.input.dir is > deprecated. Instead, use mapreduce.input.fileinputformat.inputdir > 14/12/04 17:28:44 INFO Configuration.deprecation: mapred.output.dir is > deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir > 14/12/04 17:28:44 INFO Configuration.deprecation: > mapreduce.outputformat.class is deprecated. Instead, use > mapreduce.job.outputformat.class > 14/12/04 17:28:44 INFO Configuration.deprecation: mapred.map.tasks is > deprecated. Instead, use mapreduce.job.maps > 14/12/04 17:28:44 INFO Configuration.deprecation: mapred.output.key.class is > deprecated. Instead, use mapreduce.job.output.key.class > 14/12/04 17:28:44 INFO Configuration.deprecation: mapred.working.dir is > deprecated. Instead, use mapreduce.job.working.dir > 14/12/04 17:28:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: > job_1413806590994_0039 > 14/12/04 17:28:44 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, > Service: 3.209.124.204:8020, Ident: (HDFS_DELEGATION_TOKEN token 50 for > 192769) > 14/12/04 17:28:44 INFO impl.YarnClientImpl: Submitted application > application_1413806590994_0039 to ResourceManager at > dn2.tcsgegdc.com/3.209.124.208:8032 > 14/12/04 17:28:44 INFO mapreduce.Job: The url to track the job: > http://dn2.tcsgegdc.com:8088/proxy/application_1413806590994_0039/ > 14/12/04 17:28:44 INFO mapreduce.Job: Running job: job_1413806590994_0039 > 14/12/04 17:28:48 INFO mapreduce.Job: Job job_1413806590994_0039
[jira] [Commented] (MAPREDUCE-3902) MR AM should reuse containers for map tasks, there-by allowing fine-grained control on num-maps for users without need for CombineFileInputFormat etc.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14233932#comment-14233932 ] Rohith commented on MAPREDUCE-3902: --- I will update earliest > MR AM should reuse containers for map tasks, there-by allowing fine-grained > control on num-maps for users without need for CombineFileInputFormat etc. > -- > > Key: MAPREDUCE-3902 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3902 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Arun C Murthy >Assignee: Kannan Rajah > Attachments: AMContainerRefactorNotes.pdf, AM_ContainerRefactor.pdf, > MAPREDUCE-3902.2.patch, MAPREDUCE-3902.patch > > > The MR AM is now in a great position to reuse containers across (map) tasks. > This is something similar to JVM re-use we had in 0.20.x, but in a > significantly better manner: > # Consider data-locality when re-using containers > # Consider the new shuffle - ensure that reduces fetch output of the whole > container at once (i.e. all maps) : MAPREDUCE-4525 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3902) MR AM should reuse containers for map tasks, there-by allowing fine-grained control on num-maps for users without need for CombineFileInputFormat etc.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232635#comment-14232635 ] Rohith commented on MAPREDUCE-3902: --- I wonder this jira has become stale for long time and would like to know the reason. I personally think this feature would be helpfull in terms of latency container allocation latency.We have done few analysis and implemented support for JVM reuse on branch-2 without breaking existing AM functionality. We would be ready to share prototype patch along with design doc. > MR AM should reuse containers for map tasks, there-by allowing fine-grained > control on num-maps for users without need for CombineFileInputFormat etc. > -- > > Key: MAPREDUCE-3902 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3902 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Arun C Murthy >Assignee: Siddharth Seth > Attachments: AMContainerRefactorNotes.pdf, AM_ContainerRefactor.pdf, > MAPREDUCE-3902.2.patch, MAPREDUCE-3902.patch > > > The MR AM is now in a great position to reuse containers across (map) tasks. > This is something similar to JVM re-use we had in 0.20.x, but in a > significantly better manner: > # Consider data-locality when re-using containers > # Consider the new shuffle - ensure that reduces fetch output of the whole > container at once (i.e. all maps) : MAPREDUCE-4525 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith updated MAPREDUCE-6160:
--
Status: Patch Available (was: Open)
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.1.patch, MAPREDUCE-6160.2.patch,
> MAPREDUCE-6160.3.patch, MAPREDUCE-6160.patch, MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith updated MAPREDUCE-6160:
--
Status: Open (was: Patch Available)
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.1.patch, MAPREDUCE-6160.2.patch,
> MAPREDUCE-6160.3.patch, MAPREDUCE-6160.patch, MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224764#comment-14224764
]
Rohith commented on MAPREDUCE-6160:
---
bq. just one nit. Should we just say "Unknown Job " + jobId for the error
message?
I changed log as above.
I updated the patch with changing comment message for consistency. Please
review.
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.1.patch, MAPREDUCE-6160.2.patch,
> MAPREDUCE-6160.3.patch, MAPREDUCE-6160.patch, MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith updated MAPREDUCE-6160:
--
Attachment: MAPREDUCE-6160.3.patch
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.1.patch, MAPREDUCE-6160.2.patch,
> MAPREDUCE-6160.3.patch, MAPREDUCE-6160.patch, MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith updated MAPREDUCE-6160:
--
Attachment: MAPREDUCE-6160.2.patch
I updated the patch as per discussion.Please review
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.1.patch, MAPREDUCE-6160.2.patch,
> MAPREDUCE-6160.patch, MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223245#comment-14223245
]
Rohith commented on MAPREDUCE-6160:
---
Thanks [~jlowe] for your inputs. Given client APIs already throwing
IOExceptions, leaving getJobReport() others API's can throw IOExceptions with
valid message.
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.1.patch, MAPREDUCE-6160.patch,
> MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14222456#comment-14222456
]
Rohith commented on MAPREDUCE-6160:
---
Oops can't imagine these many test cases are failing!!:-( Deeply need to look
into individual tests failures are really caused by patch or something else
problem.
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.1.patch, MAPREDUCE-6160.patch,
> MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14222394#comment-14222394
]
Rohith commented on MAPREDUCE-6160:
---
Please review the patch.
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.1.patch, MAPREDUCE-6160.patch,
> MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14222391#comment-14222391
]
Rohith commented on MAPREDUCE-6160:
---
Attached patch with following fix
1. If invalid JobId is detected at AM/HistoryServer, the API's return with null
i.e consistent with getJobReport(). There is no change made to getJobReport().
2. ClientServiceDelegate identifies response having null for corresponding
api's and throw IOException with message "Unknown job " + jobId.
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.1.patch, MAPREDUCE-6160.patch,
> MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith updated MAPREDUCE-6160:
--
Attachment: MAPREDUCE-6160.1.patch
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.1.patch, MAPREDUCE-6160.patch,
> MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-5429) App Master throw OutOfMemoryErrors.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith resolved MAPREDUCE-5429.
---
Resolution: Not a Problem
Closing the issue since problem was with am memory configuration. After
increasing memory to 5GB, it worked well.
Thanks all who looked into this issue and for their suggestions.
> App Master throw OutOfMemoryErrors.
> ---
>
> Key: MAPREDUCE-5429
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5429
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.5-alpha
>Reporter: Rohith
>
> While running job , got OOM in app master and exitted the app master jvm.
> {noformat}
> 2013-07-28 13:45:21,937 ERROR [IPC Server handler 14 on 59522]
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:job_1374920247399_0422 (auth:TOKEN) cause:java.io.IOException:
> java.lang.OutOfMemoryError: Java heap space
> 2013-07-28 13:45:21,937 INFO [IPC Server handler 22 on 59522]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1374920247399_0422_r_000384_0
> 2013-07-28 13:45:46,100 INFO [IPC Server handler 22 on 59522]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1374920247399_0422_r_000384_0 is : 0.22976667
> 2013-07-28 13:45:21,937 ERROR [IPC Server handler 15 on 59522]
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:job_1374920247399_0422 (auth:TOKEN) cause:java.io.IOException:
> java.lang.OutOfMemoryError: Java heap space
> 2013-07-28 13:45:21,937 ERROR [IPC Server handler 13 on 59522]
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:job_1374920247399_0422 (auth:TOKEN) cause:java.io.IOException:
> java.lang.OutOfMemoryError: Java heap space
> 2013-07-28 13:45:54,522 INFO [IPC Server handler 15 on 59522]
> org.apache.hadoop.ipc.Server: IPC Server handler 15 on 59522, call
> statusUpdate(attempt_1374920247399_0422_r_000225_0,
> org.apache.hadoop.mapred.ReduceTaskStatus@dd89c26), rpc version=2, client
> version=19, methodsFingerPrint=937413979 from 10.71.115.238:59691: error:
> java.io.IOException: java.lang.OutOfMemoryError: Java heap space
> java.io.IOException: java.lang.OutOfMemoryError: Java heap space
> 2013-07-28 13:45:21,937 INFO [IPC Server handler 19 on 59522]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1374920247399_0422_r_000307_0
> 2013-07-28 13:45:21,937 INFO [IPC Server handler 16 on 59522]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1374920247399_0422_r_000552_0
> 2013-07-28 13:46:09,900 INFO [IPC Server handler 16 on 59522]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1374920247399_0422_r_000552_0 is : 0.17983334
> 2013-07-28 13:45:14,870 ERROR [IPC Server handler 6 on 59522]
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:job_1374920247399_0422 (auth:TOKEN) cause:java.io.IOException:
> java.lang.OutOfMemoryError: Java heap space
> 2013-07-28 13:45:14,870 FATAL [ResponseProcessor for block
> BP-myhacluster-25656:blk_-2026966945468195799_12352]
> org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
> Thread[ResponseProcessor for block
> BP-myhacluster-25656:blk_-2026966945468195799_12352,5,main] threw an Error.
> Shutting down now...
> java.lang.OutOfMemoryError: Java heap space
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14220402#comment-14220402
]
Rohith commented on MAPREDUCE-6160:
---
I mean leaving getJobReport for others APIs.
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.patch, MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219625#comment-14219625
]
Rohith commented on MAPREDUCE-6160:
---
bq. However it may be better to return null instead if that would make the
interface more self-consistent.
I understand this way, let ClientServiceDelegate check for response null and
handle it by throwing IOException.Is it so?
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.patch, MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219582#comment-14219582
]
Rohith commented on MAPREDUCE-6160:
---
The only API that expect null in the MRClientProtocol getJobReport. null has
been checked in the trailing call hierarchy of getJobReport.I agree this breaks
compatibility.
But other API's implementation is vulnerable to throw NPE if queried with
invalid job id. This would be better if either it should be RuntimeException or
API defined IOException with proper message to client.Any thoughts?
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.patch, MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219049#comment-14219049
]
Rohith commented on MAPREDUCE-6160:
---
bq. Is there any reason why the job is not available in History? I see in your
above comment the Job got succeeded, Do you see any exception in in MR AM/JHS
log for the same job?
MRAppMaster was failed to write to HDFS because of safemode and safemode was
recovered later.
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.patch, MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218912#comment-14218912
]
Rohith commented on MAPREDUCE-6160:
---
[~jlowe] Could you please look at this jira whenever you have free time?
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.patch, MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith updated MAPREDUCE-6160:
--
Attachment: MAPREDUCE-6160.patch
Update the patch for correcting tests. Please review the patch
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.patch, MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14217759#comment-14217759
]
Rohith commented on MAPREDUCE-6160:
---
NPE trace in {{MRClientProtocol#getTaskAttemptCompletionEvents()}}
{code}
14/11/07 15:09:38 INFO mapreduce.Job: map 92% reduce 25%
14/11/07 15:09:40 INFO mapreduce.Job: map 96% reduce 25%
14/11/07 15:09:42 INFO mapreduce.Job: map 100% reduce 25%
14/11/07 15:09:43 INFO mapreduce.Job: map 100% reduce 100%
14/11/07 15:09:43 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
14/11/07 15:09:43 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
14/11/07 15:09:43 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:277)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:173)
at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:283)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1612)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
{code}
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6049) AM JVM does not exit if MRClientService gracefull shutdown fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14216418#comment-14216418
]
Rohith commented on MAPREDUCE-6049:
---
Thanks [~devaraj.k] for reviewing and commiting patch:-)
> AM JVM does not exit if MRClientService gracefull shutdown fails
>
>
> Key: MAPREDUCE-6049
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6049
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, resourcemanager
>Affects Versions: 2.5.0
>Reporter: Nishan Shetty
>Assignee: Rohith
> Fix For: 2.7.0
>
> Attachments: AM_ThreadDump.td, MAPREDUCE-6049.1.patch,
> MAPREDUCE-6049.patch, MAPREDUCE-6049.patch
>
>
> Eventhough job got FAILED, AM process still not exiting
> ThreadDump of AM process is below
> {noformat}
> "Job Fail Wait Timeout Monitor #0" daemon prio=10 tid=0x00aa9000
> nid=0x41fa waiting on condition [0x7f0e0d1d]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xc104c688> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith updated MAPREDUCE-6160:
--
Attachment: MAPREDUCE-6160.patch
Attached the patch for handling NPE for invalid job id's. Please review.
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith updated MAPREDUCE-6160:
--
Status: Patch Available (was: Open)
> Potential NullPointerException in MRClientProtocol interface implementation.
>
>
> Key: MAPREDUCE-6160
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Rohith
>Assignee: Rohith
> Attachments: MAPREDUCE-6160.patch
>
>
> In the implementation of MRClientProtocol, many methods can throw
> NullPointerExceptions. Instead of NullPointerExceptions, better to throw
> IOException with proper message.
> In the HistoryClientService class and MRClientService class has
> #verifyAndGetJob() method that return job object as null.
> {code}
> getTaskReport(GetTaskReportRequest request) throws IOException;
> getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
> getCounters(GetCountersRequest request) throws IOException;
> getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
> throws IOException;
> getTaskReports(GetTaskReportsRequest request) throws IOException;
> getDiagnostics(GetDiagnosticsRequest request) throws IOException;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6160) Potential NullPointerException in MRClientProtocol interface implementation.
Rohith created MAPREDUCE-6160:
-
Summary: Potential NullPointerException in MRClientProtocol
interface implementation.
Key: MAPREDUCE-6160
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6160
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Rohith
Assignee: Rohith
In the implementation of MRClientProtocol, many methods can throw
NullPointerExceptions. Instead of NullPointerExceptions, better to throw
IOException with proper message.
In the HistoryClientService class and MRClientService class has
#verifyAndGetJob() method that return job object as null.
{code}
getTaskReport(GetTaskReportRequest request) throws IOException;
getTaskAttemptReport(GetTaskAttemptReportRequest request) throws IOException;
getCounters(GetCountersRequest request) throws IOException;
getTaskAttemptCompletionEvents(GetTaskAttemptCompletionEventsRequest request)
throws IOException;
getTaskReports(GetTaskReportsRequest request) throws IOException;
getDiagnostics(GetDiagnosticsRequest request) throws IOException;
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6049) AM JVM does not exit if MRClientService gracefull shutdown fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14198002#comment-14198002
]
Rohith commented on MAPREDUCE-6049:
---
Hi [~vinodkv] [~jianhe]
Can this issue fix go into 2.6 release? Here also ApplicationMaster JVM
does not exit for ever.
> AM JVM does not exit if MRClientService gracefull shutdown fails
>
>
> Key: MAPREDUCE-6049
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6049
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, resourcemanager
>Affects Versions: 2.5.0
>Reporter: Nishan Shetty
>Assignee: Rohith
> Attachments: AM_ThreadDump.td, MAPREDUCE-6049.1.patch,
> MAPREDUCE-6049.patch, MAPREDUCE-6049.patch
>
>
> Eventhough job got FAILED, AM process still not exiting
> ThreadDump of AM process is below
> {noformat}
> "Job Fail Wait Timeout Monitor #0" daemon prio=10 tid=0x00aa9000
> nid=0x41fa waiting on condition [0x7f0e0d1d]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xc104c688> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5542) Killing a job just as it finishes can generate an NPE in client
[ https://issues.apache.org/jira/browse/MAPREDUCE-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175815#comment-14175815 ] Rohith commented on MAPREDUCE-5542: --- Thanks Jason Lowe for review and committing patch:-) > Killing a job just as it finishes can generate an NPE in client > --- > > Key: MAPREDUCE-5542 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5542 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client, mrv2 >Affects Versions: 2.1.0-beta, 0.23.9 >Reporter: Jason Lowe >Assignee: Rohith > Fix For: 2.6.0 > > Attachments: MAPREDUCE-5542.1.patch, MAPREDUCE-5542.2.patch, > MAPREDUCE-5542.3.patch, MAPREDUCE-5542.4.patch, MAPREDUCE-5542.5.patch, > MAPREDUCE-5542.6.patch > > > If a client tries to kill a job just as the job is finishing then the client > can crash with an NPE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5542) Killing a job just as it finishes can generate an NPE in client
[ https://issues.apache.org/jira/browse/MAPREDUCE-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175184#comment-14175184 ] Rohith commented on MAPREDUCE-5542: --- I updated patch addressing above comments, please review.. > Killing a job just as it finishes can generate an NPE in client > --- > > Key: MAPREDUCE-5542 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5542 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client, mrv2 >Affects Versions: 2.1.0-beta, 0.23.9 >Reporter: Jason Lowe >Assignee: Rohith > Attachments: MAPREDUCE-5542.1.patch, MAPREDUCE-5542.2.patch, > MAPREDUCE-5542.3.patch, MAPREDUCE-5542.4.patch, MAPREDUCE-5542.5.patch, > MAPREDUCE-5542.6.patch > > > If a client tries to kill a job just as the job is finishing then the client > can crash with an NPE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5542) Killing a job just as it finishes can generate an NPE in client
[ https://issues.apache.org/jira/browse/MAPREDUCE-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-5542: -- Attachment: MAPREDUCE-5542.6.patch > Killing a job just as it finishes can generate an NPE in client > --- > > Key: MAPREDUCE-5542 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5542 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client, mrv2 >Affects Versions: 2.1.0-beta, 0.23.9 >Reporter: Jason Lowe >Assignee: Rohith > Attachments: MAPREDUCE-5542.1.patch, MAPREDUCE-5542.2.patch, > MAPREDUCE-5542.3.patch, MAPREDUCE-5542.4.patch, MAPREDUCE-5542.5.patch, > MAPREDUCE-5542.6.patch > > > If a client tries to kill a job just as the job is finishing then the client > can crash with an NPE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
