[jira] [Created] (MAPREDUCE-6050) Upgrade JUnit3 TestCase to JUnit 4
Chen He created MAPREDUCE-6050: -- Summary: Upgrade JUnit3 TestCase to JUnit 4 Key: MAPREDUCE-6050 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6050 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Chen He Priority: Trivial There are still test classes that extend from junit.framework.TestCase. upgrade them to JUnit4. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-5961) Job start time setting to Thu Jan 01 05:29:59 IST 1970
[
https://issues.apache.org/jira/browse/MAPREDUCE-5961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen He resolved MAPREDUCE-5961.
Resolution: Duplicate
Job start time setting to Thu Jan 01 05:29:59 IST 1970
Key: MAPREDUCE-5961
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5961
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 2.4.1
Reporter: Nishan Shetty, Huawei
Priority: Minor
Induce RM switchover while job is in progress
Observe that job start time setting to Thu Jan 01 05:29:59 IST 1970 saying
below error
{code}
2014-07-05 21:38:12,415 INFO
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Moving
hdfs://mycluster:8020/home/testos/staging-dir/history/done_intermediate/testos/job_1404572770516_0056_conf.xml
to
hdfs://mycluster:8020/home/testos/staging-dir/history/done/2014/07/05/00/job_1404572770516_0056_conf.xml
2014-07-05 21:41:12,289 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory:
Starting scan to move intermediate done files
2014-07-05 21:41:12,294 WARN
org.apache.hadoop.mapreduce.v2.jobhistory.FileNameIndexUtils: Unable to parse
launch time from job history file
job_1404572770516_0057-1404576372149-testos-word+count-1404576499406-85-10-SUCCEEDED-default--1.jhist
: java.lang.NumberFormatException: For input string:
2014-07-05 21:41:12,297 INFO
org.apache.hadoop.mapreduce.jobhistory.JobSummary:
jobId=job_1404572770516_0057,submitTime=1404576372149,launchTime=-1,firstMapTaskLaunchTime=1404576442635,firstReduceTaskLaunchTime=1404576492243,finishTime=1404576499406,resourcesPerMap=1024,resourcesPerReduce=1024,numMaps=85,numReduces=10,user=testos,queue=default,status=SUCCEEDED,mapSlotSeconds=690,reduceSlotSeconds=39,jobName=word
count
2014-07-05 21:41:12,298 INFO
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Deleting JobSummary
file:
{code}
AM LOG
{code}
2014-07-05 21:38:19,432 INFO [Thread-74]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
JobHistoryEventHandler notified that forceJobCompletion is true
2014-07-05 21:38:19,432 INFO [Thread-74]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the
services
2014-07-05 21:38:19,433 INFO [Thread-74]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
JobHistoryEventHandler. Size of the outstanding queue size is 0
2014-07-05 21:38:19,556 INFO [eventHandlingThread]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying
hdfs://mycluster/home/testos/staging-dir/testos/.staging/job_1404572770516_0057/job_1404572770516_0057_2.jhist
to
hdfs://mycluster/home/testos/staging-dir/history/done_intermediate/testos/job_1404572770516_0057-1404576372149-testos-word+count-1404576499406-85-10-SUCCEEDED-default--1.jhist_tmp
2014-07-05 21:38:19,770 INFO [eventHandlingThread]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done
location:
hdfs://mycluster/home/testos/staging-dir/history/done_intermediate/testos/job_1404572770516_0057-1404576372149-testos-word+count-1404576499406-85-10-SUCCEEDED-default--1.jhist_tmp
2014-07-05 21:38:19,785 INFO [eventHandlingThread]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying
hdfs://mycluster/home/testos/staging-dir/testos/.staging/job_1404572770516_0057/job_1404572770516_0057_2_conf.xml
to
hdfs://mycluster/home/testos/staging-dir/history/done_intermediate/testos/job_1404572770516_0057_conf.xml_tmp
2014-07-05 21:38:19,862 INFO [eventHandlingThread]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done
location:
hdfs://mycluster/home/testos/staging-dir/history/done_intermediate/testos/job_1404572770516_0057_conf.xml_tmp
2014-07-05 21:38:19,886 INFO [eventHandlingThread]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to
done:
hdfs://mycluster/home/testos/staging-dir/history/done_intermediate/testos/job_1404572770516_0057.summary_tmp
to
hdfs://mycluster/home/testos/staging-dir/history/done_intermediate/testos/job_1404572770516_0057.summary
2014-07-05 21:38:19,898 INFO [eventHandlingThread]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to
done:
hdfs://mycluster/home/testos/staging-dir/history/done_intermediate/testos/job_1404572770516_0057_conf.xml_tmp
to
hdfs://mycluster/home/testos/staging-dir/history/done_intermediate/testos/job_1404572770516_0057_conf.xml
2014-07-05 21:38:19,910 INFO [eventHandlingThread]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to
done:
[jira] [Created] (MAPREDUCE-5923) org.apache.hadoop.mapred.pipes.TestPipeApplication timeouts intermittently
Chen He created MAPREDUCE-5923: -- Summary: org.apache.hadoop.mapred.pipes.TestPipeApplication timeouts intermittently Key: MAPREDUCE-5923 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5923 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: trunk Reporter: Chen He Priority: Minor -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-3677) If hadoop.security.authorization is set to true, NM is not starting.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He resolved MAPREDUCE-3677. Resolution: Not a Problem If hadoop.security.authorization is set to true, NM is not starting. -- Key: MAPREDUCE-3677 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3677 Project: Hadoop Map/Reduce Issue Type: Bug Components: nodemanager Affects Versions: 0.23.0 Reporter: Ramgopal N Assignee: Chen He I have the hadoop cluster setup with root user.Accidentally i have set hadoop.security.authorization to true.I have not set any permissions in policy.xml.When i am trying to start the NM with root user ...it is throwing the following error Exception in thread main java.lang.NoClassDefFoundError: nodemanager Caused by: java.lang.ClassNotFoundException: nodemanager at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:303) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316) Could not find the main class: nodemanager. Program will exit. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-4339) pi example job hangs on when run on hadoop 0.23.0 when capacity scheduler is included in the setting environment.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He resolved MAPREDUCE-4339. Resolution: Cannot Reproduce pi example job hangs on when run on hadoop 0.23.0 when capacity scheduler is included in the setting environment. - Key: MAPREDUCE-4339 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4339 Project: Hadoop Map/Reduce Issue Type: Bug Components: examples, job submission, mrv2, scheduler Affects Versions: 0.23.0 Environment: Ubuntu Server 11.04, Hadoop 0.23.0, Reporter: srikanth ayalasomayajulu Labels: hadoop Fix For: 0.23.0 Original Estimate: 48h Remaining Estimate: 48h Tried to include default capacity scheduler in hadoop and tried to run an example pi program. The job hangs and no more output is getting displayed. Starting Job 2012-06-12 22:10:02,524 INFO ipc.YarnRPC (YarnRPC.java:create(47)) - Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC 2012-06-12 22:10:02,538 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:init(95)) - Connecting to ResourceManager at localhost/127.0.0.1:8030 2012-06-12 22:10:02,539 INFO ipc.HadoopYarnRPC (HadoopYarnProtoRPC.java:getProxy(48)) - Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol 2012-06-12 22:10:02,665 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:init(99)) - Connected to ResourceManager at localhost/127.0.0.1:8030 2012-06-12 22:10:02,727 WARN conf.Configuration (Configuration.java:handleDeprecation(326)) - fs.default.name is deprecated. Instead, use fs.defaultFS 2012-06-12 22:10:02,728 WARN conf.Configuration (Configuration.java:handleDeprecation(343)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used 2012-06-12 22:10:02,831 INFO input.FileInputFormat (FileInputFormat.java:listStatus(245)) - Total input paths to process : 10 2012-06-12 22:10:02,900 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(362)) - number of splits:10 2012-06-12 22:10:03,044 INFO mapred.YARNRunner (YARNRunner.java:createApplicationSubmissionContext(279)) - AppMaster capability = memory: 2048 2012-06-12 22:10:03,286 INFO mapred.YARNRunner (YARNRunner.java:createApplicationSubmissionContext(355)) - Command to launch container for ApplicationMaster is : $JAVA_HOME/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.mapreduce.container.log.dir=LOG_DIR -Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1536m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1LOG_DIR/stdout 2LOG_DIR/stderr 2012-06-12 22:10:03,370 INFO mapred.ResourceMgrDelegate (ResourceMgrDelegate.java:submitApplication(304)) - Submitted application application_1339507608976_0002 to ResourceManager 2012-06-12 22:10:03,432 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1207)) - Running job: job_1339507608976_0002 2012-06-12 22:10:04,443 INFO mapreduce.Job (Job.java:monitorAndPrintJob(1227)) - map 0% reduce 0% -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-3406) Add node information to bin/mapred job -list-attempt-ids and other improvements
[ https://issues.apache.org/jira/browse/MAPREDUCE-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He resolved MAPREDUCE-3406. Resolution: Duplicate Target Version/s: 2.0.0-alpha, 0.23.3, 3.0.0 (was: 0.23.3, 2.0.0-alpha, 3.0.0) Add node information to bin/mapred job -list-attempt-ids and other improvements --- Key: MAPREDUCE-3406 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3406 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Ravi Prakash Fix For: 0.24.0 From [~rramya] Providing the NM information where the containers are scheduled in bin/mapred job -list-attempt-ids will be helpful in automation, debugging and to avoid grepping through the AM logs. From my own observation, the list-attempt-ids should list the attempt ids and not require the arguments. The arguments if given, can be used to filter the results. From the usage: bq. [-list-attempt-ids job-id task-type task-state]. Valid values for task-type are MAP REDUCE JOB_SETUP JOB_CLEANUP TASK_CLEANUP. Valid values for task-state are running, completed -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5677) Hadoop 2.2 historyviewer report NPE when read 0.23 fail job's history
Chen He created MAPREDUCE-5677: -- Summary: Hadoop 2.2 historyviewer report NPE when read 0.23 fail job's history Key: MAPREDUCE-5677 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5677 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.2.0 Reporter: Chen He 2013-12-10 12:49:39,394 WARN org.apache.hadoop.yarn.webapp.GenericExceptionHandler: INTERNAL_SERVER_ERROR java.lang.NullPointerException at org.apache.hadoop.mapreduce.jobhistory.EventReader.fromAvro(EventReader.java:174) at org.apache.hadoop.mapreduce.jobhistory.TaskFailedEvent.setDatum(TaskFailedEvent.java:111) at org.apache.hadoop.mapreduce.jobhistory.EventReader.getNextEvent(EventReader.java:156) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:111) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:153) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139) at org.apache.hadoop.mapreduce.v2.hs.CompletedJob.loadFullHistoryData(CompletedJob.java:337) at org.apache.hadoop.mapreduce.v2.hs.CompletedJob.init(CompletedJob.java:101) at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager$HistoryFileInfo.loadJob(HistoryFileManager.java:410) at org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.loadJob(CachedHistoryStorage.java:106) at org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage.getFullJob(CachedHistoryStorage.java:137) at org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:217) at org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices.getJobFromJobIdString(AMWebServices.java:120) at org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJob(HsWebServices.java:223) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1310) at
[jira] [Created] (MAPREDUCE-5669) No Submit time on AM web pages
Chen He created MAPREDUCE-5669: -- Summary: No Submit time on AM web pages Key: MAPREDUCE-5669 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5669 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chen He Assignee: Chen He Similar to MAPREDUCE-5052, This is a fix on AM side. Add submitTime field to the AM's web services REST API -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5671) NaN can be created by client and assign to Progress
Chen He created MAPREDUCE-5671: -- Summary: NaN can be created by client and assign to Progress Key: MAPREDUCE-5671 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5671 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chen He Assignee: Chen He MapReduce should filter illegal progress values that are not fall into (0,1) interval when the progress value is given. If it is Float.NaN, Float.NEGATIVE_INFINITY, or smaller than 0: set progress to be 0; If its is Float.POSITIVE_INFINITY or larger than 1: set progress to be 1; -- This message was sent by Atlassian JIRA (v6.1#6144)
