[jira] [Created] (MAPREDUCE-3186) User jobs are getting hanged if the Resource manager process goes down and comes up while job is getting executed.
User jobs are getting hanged if the Resource manager process goes down and comes up while job is getting executed. -- Key: MAPREDUCE-3186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3186 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Environment: linux Reporter: Ramgopal N If the resource manager is restarted while the job execution is in progress, the job is getting hanged. UI shows the job as running. In the RM log, it is throwing an error ERROR org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: AppAttemptId doesnt exist in cache appattempt_1318579738195_0004_01 In the console MRAppMaster and Runjar processes are not getting killed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3187) Add names for various unnamed threads in MR2
Add names for various unnamed threads in MR2 Key: MAPREDUCE-3187 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3187 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Simple patch to add thread names for all the places we use Executors, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3190) bin/yarn should barf early if HADOOP_COMMON_HOME or HADOOP_HDFS_HOME are not set
bin/yarn should barf early if HADOOP_COMMON_HOME or HADOOP_HDFS_HOME are not set
Key: MAPREDUCE-3190
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3190
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Attachments: mr-3190.txt
Currently, if these env vars are not set when you run bin/yarn, it will crash
with various ClassNotFoundExceptions, having added {{/share/hadoop/hdfs}} to
the classpath. Rather, we should check for these env vars in the wrapper script
and display a reasonable error message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3191) docs for map output compression incorrectly reference SequenceFile
docs for map output compression incorrectly reference SequenceFile -- Key: MAPREDUCE-3191 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3191 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Todd Lipcon Priority: Trivial The documentation currently says that map output compression uses SequenceFile compression. This hasn't been true in several years, since we use IFile for intermediate data now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3192) Fix Javadoc warning in JobClient.java and Cluster.java
Fix Javadoc warning in JobClient.java and Cluster.java -- Key: MAPREDUCE-3192 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3192 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Javadoc warnings in JobClient.java and Cluster.java need to be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: 0.23 trunk tars, we'll we publishing 1 tar per component or a single tar? What about source tar?
Other approach would be asking which tar to build. mapred-tar (mapred and common) hdfs-tar (hdfs and common) hadoop-tar (all) In this case, hbase can just use hdfs-tar. -Bharath From: Ravi Teja ravit...@huawei.com To: mapreduce-dev@hadoop.apache.org; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org Sent: Wednesday, October 12, 2011 9:43 PM Subject: RE: 0.23 trunk tars, we'll we publishing 1 tar per component or a single tar? What about source tar? I feel #4 as a better option. Regards, Ravi Teja -Original Message- From: Alejandro Abdelnur [mailto:t...@cloudera.com] Sent: Wednesday, October 12, 2011 9:38 PM To: common-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org Subject: 0.23 trunk tars, we'll we publishing 1 tar per component or a single tar? What about source tar? Currently common, hdfs and mapred create partial tars which are not usable unless they are stitched together into a single tar. With HADOOP-7642 the stitching happens as part of the build. The build currently produces the following tars: 1* common TAR 2* hdfs (partial) TAR 3* mapreduce (partial) TAR 4* hadoop (full, the stitched one) TAR #1 on its own does not run anything, #2 and #3 on their own don't run. #4 runs hdfs mapreduce. Questions: Q1. Does it make sense to publish #1, #2 #3? Or #4 is sufficient and you start the services you want (i.e. Hbase would just use HDFS)? Q2. And what about a source TAR, does it make sense to have source TAR per component or a single TAR for the whole? For simplicity (for the build system and for users) I'd prefer a single binary TAR and a single source TAR. Thanks. Alejandro
[jira] [Resolved] (MAPREDUCE-3160) Merge -r 1177530:1177531 from trunk to branch-0.23 to fix MAPREDUCE-2996 broke ant test compilation
[
https://issues.apache.org/jira/browse/MAPREDUCE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash resolved MAPREDUCE-3160.
-
Resolution: Invalid
This was a problem with my ivy cache. After clearing it, and rebuilding
everything, this problem went away.
Merge -r 1177530:1177531 from trunk to branch-0.23 to fix MAPREDUCE-2996
broke ant test compilation
---
Key: MAPREDUCE-3160
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3160
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ravi Prakash
I git bisected and the problem starts from
adb810babaf25b9f9dae75b43d4beac782deaa01 . ant
{noformat}
[jsp-compile] log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[javac]
/home/raviprak/Code/hadoop/hadoop-all/hadoop-mapreduce-project/build.xml:398:
warning: 'includeantruntime' was not set, defaulting to
build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 2 source files to
/home/raviprak/Code/hadoop/hadoop-all/hadoop-mapreduce-project/build/classes
[javac]
/home/raviprak/Code/hadoop/hadoop-all/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java:697:
cannot find symbol
[javac] symbol : constructor
JobInitedEvent(org.apache.hadoop.mapred.JobID,long,int,int,java.lang.String,boolean)
[javac] location: class
org.apache.hadoop.mapreduce.jobhistory.JobInitedEvent
[javac] JobInitedEvent jie = new JobInitedEvent(
[javac] ^
[javac] Note:
/home/raviprak/Code/hadoop/hadoop-all/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java
uses or overrides a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 1 error
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3107) DFSIO tests are failing intermittently
[ https://issues.apache.org/jira/browse/MAPREDUCE-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George resolved MAPREDUCE-3107. Resolution: Cannot Reproduce It seems like this was fixed somehow. I confirmed with Rajit and he does not see it anymore as well. Hence, closing this as cannot be reproduced. DFSIO tests are failing intermittently -- Key: MAPREDUCE-3107 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3107 Project: Hadoop Map/Reduce Issue Type: Bug Components: benchmarks, mrv2 Affects Versions: 0.23.0 Reporter: Rajit Saha Assignee: John George Priority: Blocker Fix For: 0.23.0 Intermittently DFSIO tests are failing either in read or write operations. $HADOOP_COMMON_HOME/bin/hadoop --config $HADOOP_CONF_DIR jar $HADOOP_MAPRED_HOME/hadoop-mapreduce-test-*.jar TestDFSIO -read -nrFiles 36 -fileSize 320 11/09/27 03:21:03 INFO fs.TestDFSIO: TestDFSIO.0.0.6 11/09/27 03:21:03 INFO fs.TestDFSIO: nrFiles = 36 11/09/27 03:21:03 INFO fs.TestDFSIO: fileSize (MB) = 320.0 11/09/27 03:21:03 INFO fs.TestDFSIO: bufferSize = 100 11/09/27 03:21:03 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO 11/09/27 03:21:03 INFO fs.TestDFSIO: creating control file: 335544320 bytes, 36 files 11/09/27 03:21:05 INFO fs.TestDFSIO: created control files for: 36 files .. .. .. .. .. 11/09/27 03:24:04 INFO mapreduce.Job: map 88% reduce 25% 11/09/27 03:24:05 INFO mapreduce.Job: map 90% reduce 25% 11/09/27 03:24:06 INFO mapreduce.Job: map 92% reduce 25% 11/09/27 03:24:09 INFO mapreduce.Job: map 93% reduce 25% 11/09/27 03:24:10 INFO mapreduce.Job: map 93% reduce 26% 11/09/27 03:24:42 INFO mapreduce.Job: map 94% reduce 26% 11/09/27 03:24:43 INFO mapreduce.Job: map 94% reduce 27% 11/09/27 03:24:56 INFO mapreduce.Job: map 95% reduce 27% 11/09/27 03:24:58 INFO mapreduce.Job: map 95% reduce 28% 11/09/27 03:25:13 INFO mapreduce.Job: map 96% reduce 28% 11/09/27 03:25:15 INFO mapreduce.Job: map 97% reduce 28% 11/09/27 03:25:16 INFO mapreduce.Job: map 97% reduce 30% 11/09/27 03:25:16 INFO mapreduce.Job: map 98% reduce 30% 11/09/27 03:25:16 INFO mapreduce.Job: Job job_1317092846056_0012 failed with state FAILED 11/09/27 03:25:16 INFO mapreduce.Job: Counters: 44 File System Counters FILE: BYTES_READ=20931 FILE: BYTES_WRITTEN=2192966 FILE: READ_OPS=0 FILE: LARGE_READ_OPS=0 FILE: WRITE_OPS=0 HDFS: BYTES_READ=10454250228 HDFS: BYTES_WRITTEN=0 HDFS: READ_OPS=172 HDFS: LARGE_READ_OPS=0 HDFS: WRITE_OPS=0 org.apache.hadoop.mapreduce.JobCounter NUM_FAILED_MAPS=6 TOTAL_LAUNCHED_MAPS=44 TOTAL_LAUNCHED_REDUCES=1 DATA_LOCAL_MAPS=10 RACK_LOCAL_MAPS=34 SLOTS_MILLIS_MAPS=2340307 org.apache.hadoop.mapreduce.TaskCounter MAP_INPUT_RECORDS=36 MAP_OUTPUT_RECORDS=140 MAP_OUTPUT_BYTES=2103 MAP_OUTPUT_MATERIALIZED_BYTES=2579 SPLIT_RAW_BYTES=5210 COMBINE_INPUT_RECORDS=0 COMBINE_OUTPUT_RECORDS=0 REDUCE_INPUT_GROUPS=0 REDUCE_SHUFFLE_BYTES=3054 REDUCE_INPUT_RECORDS=0 REDUCE_OUTPUT_RECORDS=0 SPILLED_RECORDS=140 SHUFFLED_MAPS=33 FAILED_SHUFFLE=0 MERGED_MAP_OUTPUTS=0 GC_TIME_MILLIS=59933 CPU_MILLISECONDS=159470 PHYSICAL_MEMORY_BYTES=11310596096 VIRTUAL_MEMORY_BYTES=31425290240 COMMITTED_HEAP_BYTES=12728664064 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter BYTES_READ=4058 org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter BYTES_WRITTEN=0 java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:781) at org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:340) at org.apache.hadoop.fs.TestDFSIO.readTest(TestDFSIO.java:418) at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:522) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:445) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
Hadoop-Mapreduce-0.23-Commit - Build # 1 - Failure
See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/1/ ### ## LAST 60 LINES OF THE CONSOLE ### Started by user gkesavan Building remotely on hadoop7 Location 'http://svn.apache.org/repos/asf/hadoop/common/braches/branch-0.23' does not exist Cleaning workspace /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit Checking out http://svn.apache.org/repos/asf/hadoop/common/braches/branch-0.23 ERROR: Failed to check out http://svn.apache.org/repos/asf/hadoop/common/braches/branch-0.23 org.tmatesoft.svn.core.SVNException: svn: URL 'http://svn.apache.org/repos/asf/hadoop/common/braches/branch-0.23' doesn't exist at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64) at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:910) at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:90) at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:136) at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:773) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:754) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:738) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1995) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:287) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.tmatesoft.svn.core.SVNErrorMessage: svn: URL 'http://svn.apache.org/repos/asf/hadoop/common/braches/branch-0.23' doesn't exist at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:163) at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:118) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:909) ... 15 more Recording test results Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## All tests passed
