[jira] [Updated] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated MAPREDUCE-6025: -- Attachment: mapreduce-6025-v2.txt native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090454#comment-14090454 ] Manu Zhang commented on MAPREDUCE-6025: --- Your approach is better. I've updated the patch accordingly which is modified from hadoop-project-dist script. native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090526#comment-14090526 ] Binglin Chang commented on MAPREDUCE-6025: -- Hi Manu, patch looks good, comments: 1. use space rather than tabs 2. I can see you reuse some code in hadoop-project-dist, I think *cp -R* is much simpler than *tar xxx*, look at hadoop-dist/pom.xml, which already uses cp -r native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6021) MR AM should have working directory in LD_LIBRARY_PATH
[ https://issues.apache.org/jira/browse/MAPREDUCE-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090684#comment-14090684 ] Hudson commented on MAPREDUCE-6021: --- FAILURE: Integrated in Hadoop-Yarn-trunk #638 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/638/]) MAPREDUCE-6021. MR AM should have working directory in LD_LIBRARY_PATH. Contributed by Jason Lowe (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1616585) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java MR AM should have working directory in LD_LIBRARY_PATH -- Key: MAPREDUCE-6021 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6021 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.4.1 Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 3.0.0, 2.6.0 Attachments: MAPREDUCE-6021-v2.patch, MAPREDUCE-6021.patch Tasks implicitly pick up shared libraries added to the job because the task launch context explicitly adds the container working directory to LD_LIBRARY_PATH. However the same is not done for the AM container which is inconsistent. User code can run in the AM via output committer, speculator, uber job, etc., so the AM's LD_LIBRARY_PATH should have the container work directory for consistency with tasks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090817#comment-14090817
]
Karthik Kambatla commented on MAPREDUCE-6027:
-
HADOOP-10876 is reverted.
mr jobs with relative paths can fail
Key: MAPREDUCE-6027
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission
Reporter: Wing Yew Poon
Assignee: Wing Yew Poon
Attachments: MAPREDUCE-6027.patch
I built hadoop from branch-2 and tried to run terasort as follows:
{noformat}
wypoon$ bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
sort-input sort-output
14/08/07 08:57:55 INFO terasort.TeraSort: starting
2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
SCDynamicStore
14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
Spent 156ms computing base-splits.
Spent 2ms computing TeraScheduler splits.
Computing input splits took 159ms
Sampling 2 splits of 2
Making 1 from 10 sampled records
Computing parititions took 626ms
Spent 789ms computing partitions.
14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
localhost/127.0.0.1:8032
14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
/tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
java.lang.IllegalArgumentException: Can not create a Path from an empty URI
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
at org.apache.hadoop.fs.Path.init(Path.java:192)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:316)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
{noformat}
If I used absolute paths for the input and out directories, the job runs fine.
This breakage is due to HADOOP-10876.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-6021) MR AM should have working directory in LD_LIBRARY_PATH
[ https://issues.apache.org/jira/browse/MAPREDUCE-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090826#comment-14090826 ] Hudson commented on MAPREDUCE-6021: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #1831 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1831/]) MAPREDUCE-6021. MR AM should have working directory in LD_LIBRARY_PATH. Contributed by Jason Lowe (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1616585) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java MR AM should have working directory in LD_LIBRARY_PATH -- Key: MAPREDUCE-6021 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6021 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.4.1 Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 3.0.0, 2.6.0 Attachments: MAPREDUCE-6021-v2.patch, MAPREDUCE-6021.patch Tasks implicitly pick up shared libraries added to the job because the task launch context explicitly adds the container working directory to LD_LIBRARY_PATH. However the same is not done for the AM container which is inconsistent. User code can run in the AM via output committer, speculator, uber job, etc., so the AM's LD_LIBRARY_PATH should have the container work directory for consistency with tasks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6021) MR AM should have working directory in LD_LIBRARY_PATH
[ https://issues.apache.org/jira/browse/MAPREDUCE-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090863#comment-14090863 ] Hudson commented on MAPREDUCE-6021: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1857 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1857/]) MAPREDUCE-6021. MR AM should have working directory in LD_LIBRARY_PATH. Contributed by Jason Lowe (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1616585) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java MR AM should have working directory in LD_LIBRARY_PATH -- Key: MAPREDUCE-6021 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6021 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.4.1 Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 3.0.0, 2.6.0 Attachments: MAPREDUCE-6021-v2.patch, MAPREDUCE-6021.patch Tasks implicitly pick up shared libraries added to the job because the task launch context explicitly adds the container working directory to LD_LIBRARY_PATH. However the same is not done for the AM container which is inconsistent. User code can run in the AM via output committer, speculator, uber job, etc., so the AM's LD_LIBRARY_PATH should have the container work directory for consistency with tasks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-6029) TestCommitterEventHandler fails in trunk
Ted Yu created MAPREDUCE-6029:
-
Summary: TestCommitterEventHandler fails in trunk
Key: MAPREDUCE-6029
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6029
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ted Yu
From https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1857/console :
{code}
Running org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.429 sec
FAILURE! - in
org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
testFailure(org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler)
Time elapsed: 1.198 sec FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertNotNull(Assert.java:621)
at org.junit.Assert.assertNotNull(Assert.java:631)
at
org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler.testFailure(TestCommitterEventHandler.java:314)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (MAPREDUCE-6029) TestCommitterEventHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mit Desai reassigned MAPREDUCE-6029:
Assignee: Mit Desai
TestCommitterEventHandler fails in trunk
Key: MAPREDUCE-6029
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6029
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ted Yu
Assignee: Mit Desai
From https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1857/console :
{code}
Running org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.429 sec
FAILURE! - in
org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
testFailure(org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler)
Time elapsed: 1.198 sec FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertNotNull(Assert.java:621)
at org.junit.Assert.assertNotNull(Assert.java:631)
at
org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler.testFailure(TestCommitterEventHandler.java:314)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-6029) TestCommitterEventHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090895#comment-14090895
]
Mit Desai commented on MAPREDUCE-6029:
--
I'm aware of this issue. I'll post a patch this afternoon.
TestCommitterEventHandler fails in trunk
Key: MAPREDUCE-6029
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6029
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ted Yu
Assignee: Mit Desai
From https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1857/console :
{code}
Running org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.429 sec
FAILURE! - in
org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
testFailure(org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler)
Time elapsed: 1.198 sec FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertNotNull(Assert.java:621)
at org.junit.Assert.assertNotNull(Assert.java:631)
at
org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler.testFailure(TestCommitterEventHandler.java:314)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-4052) Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090954#comment-14090954 ] Jay commented on MAPREDUCE-4052: I am still having the same issue after upgrading to hadoop2.4 from 2.2. I assume 2.4 has this fix in it. I am running hadoop on local windows box. Trying to use spring client from Eclipse to run the wordcount example. My cluster is configured correctly as I can run wordcount from the hdfs command line. I get in to issue when I try to run a mapreduce job from Eclipse on Windows. I have modified mapred--site.xml to use the proeprty mapreduce.app-submission.cross-platform to be true. I also removed yarn.application.classpath entry from my yarn-site.xml. I tried adding it back fro Windows. All scenarios seem to not work. Is there a documentation on what needs to take place to make it work. Will appreciate any help. jsaraiya( container_1407515618489_0002_01_01╤╣ ♠stderr ♦1440java.lang.NoClassDefFoundError: org/apache/hadoop/service/CompositeService at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) at java.lang.ClassLoader.defineClass(ClassLoader.java:616) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.service.CompositeService at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) ... 12 more Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. Exception in thread main ♠stdout ☺0( container_1407515618489_0002_02_01╤╣ ♠stderr ♦1440java.lang.NoClassDefFoundError: org/apache/hadoop/service/CompositeService at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) at java.lang.ClassLoader.defineClass(ClassLoader.java:616) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.service.CompositeService at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) ... 12 more Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster. --- Key: MAPREDUCE-4052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 0.23.1, 2.2.0 Environment: client on the Windows, the the cluster on the suse Reporter: xieguiming Assignee: Jian He Fix For: 2.4.0 Attachments: MAPREDUCE-4052-0.patch, MAPREDUCE-4052.1.patch, MAPREDUCE-4052.2.patch, MAPREDUCE-4052.3.patch, MAPREDUCE-4052.4.patch, MAPREDUCE-4052.5.patch, MAPREDUCE-4052.6.patch, MAPREDUCE-4052.7.patch,
[jira] [Resolved] (MAPREDUCE-6008) Update distcp docs to include new option that suppresses preservation of RAW.* namespace extended attributes
[ https://issues.apache.org/jira/browse/MAPREDUCE-6008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved MAPREDUCE-6008. Resolution: Not a Problem We took care of the docs in parent JIRA, no need for this one. Update distcp docs to include new option that suppresses preservation of RAW.* namespace extended attributes Key: MAPREDUCE-6008 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6008 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: distcp Affects Versions: fs-encryption Reporter: Charles Lamb Assignee: Charles Lamb Update the docs to include this new option. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091101#comment-14091101
]
Wing Yew Poon commented on MAPREDUCE-6027:
--
[~kkambatl], with the revert of HADOOP-10876, the change to
ClientDistributedCacheManager is no longer needed (although works fine with the
revert). However, the unit test could still be useful, since there was no test
for ClientDistributedCacheManager.determineCacheVisibilities() before; the test
is very simple and does not do much, but it could detect the problem caused by
something like HADOOP-10876. I can submit a new patch with just the test, if
you think that is still of interest.
mr jobs with relative paths can fail
Key: MAPREDUCE-6027
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission
Reporter: Wing Yew Poon
Assignee: Wing Yew Poon
Attachments: MAPREDUCE-6027.patch
I built hadoop from branch-2 and tried to run terasort as follows:
{noformat}
wypoon$ bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
sort-input sort-output
14/08/07 08:57:55 INFO terasort.TeraSort: starting
2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
SCDynamicStore
14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
Spent 156ms computing base-splits.
Spent 2ms computing TeraScheduler splits.
Computing input splits took 159ms
Sampling 2 splits of 2
Making 1 from 10 sampled records
Computing parititions took 626ms
Spent 789ms computing partitions.
14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
localhost/127.0.0.1:8032
14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
/tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
java.lang.IllegalArgumentException: Can not create a Path from an empty URI
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
at org.apache.hadoop.fs.Path.init(Path.java:192)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:316)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091110#comment-14091110
]
Karthik Kambatla commented on MAPREDUCE-6027:
-
Agree that the test is useful. Looking at the patch again, I am actually
tempted to leave the fix in there as well since it is benign.
mr jobs with relative paths can fail
Key: MAPREDUCE-6027
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission
Reporter: Wing Yew Poon
Assignee: Wing Yew Poon
Attachments: MAPREDUCE-6027.patch
I built hadoop from branch-2 and tried to run terasort as follows:
{noformat}
wypoon$ bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
sort-input sort-output
14/08/07 08:57:55 INFO terasort.TeraSort: starting
2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
SCDynamicStore
14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
Spent 156ms computing base-splits.
Spent 2ms computing TeraScheduler splits.
Computing input splits took 159ms
Sampling 2 splits of 2
Making 1 from 10 sampled records
Computing parititions took 626ms
Spent 789ms computing partitions.
14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
localhost/127.0.0.1:8032
14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
/tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
java.lang.IllegalArgumentException: Can not create a Path from an empty URI
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
at org.apache.hadoop.fs.Path.init(Path.java:192)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:316)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
{noformat}
If I used absolute paths for the input and out directories, the job runs fine.
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091131#comment-14091131
]
Wing Yew Poon commented on MAPREDUCE-6027:
--
Yes, the change to ClientDistributedCacheManager itself does not hurt anything.
Let me know if you'd like me to submit a new patch with just the test.
mr jobs with relative paths can fail
Key: MAPREDUCE-6027
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission
Reporter: Wing Yew Poon
Assignee: Wing Yew Poon
Attachments: MAPREDUCE-6027.patch
I built hadoop from branch-2 and tried to run terasort as follows:
{noformat}
wypoon$ bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
sort-input sort-output
14/08/07 08:57:55 INFO terasort.TeraSort: starting
2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
SCDynamicStore
14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
Spent 156ms computing base-splits.
Spent 2ms computing TeraScheduler splits.
Computing input splits took 159ms
Sampling 2 splits of 2
Making 1 from 10 sampled records
Computing parititions took 626ms
Spent 789ms computing partitions.
14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
localhost/127.0.0.1:8032
14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
/tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
java.lang.IllegalArgumentException: Can not create a Path from an empty URI
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
at org.apache.hadoop.fs.Path.init(Path.java:192)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:316)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
{noformat}
If I used absolute paths for the input and out directories, the job
[jira] [Updated] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-5130: -- Attachment: MAPREDUCE-5130-6.patch Updated to be compatible with trunk. Add missing job config options to mapred-default.xml Key: MAPREDUCE-5130 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130-1.patch, MAPREDUCE-5130-2.patch, MAPREDUCE-5130-3.patch, MAPREDUCE-5130-4.patch, MAPREDUCE-5130-5.patch, MAPREDUCE-5130-6.patch, MAPREDUCE-5130.patch, MAPREDUCE-5130.patch I came across that mapreduce.map.java.opts and mapreduce.reduce.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. List so far: mapreduce.map/reduce.java.opts mapreduce.map/reduce.memory.mb mapreduce.job.jvm.numtasks mapreduce.input.lineinputformat.linespermap mapreduce.task.combine.progress.records mapreduce.map/reduce.env -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-5130: -- Status: Patch Available (was: Open) Submitted updated patch for testing. Add missing job config options to mapred-default.xml Key: MAPREDUCE-5130 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130-1.patch, MAPREDUCE-5130-2.patch, MAPREDUCE-5130-3.patch, MAPREDUCE-5130-4.patch, MAPREDUCE-5130-5.patch, MAPREDUCE-5130-6.patch, MAPREDUCE-5130.patch, MAPREDUCE-5130.patch I came across that mapreduce.map.java.opts and mapreduce.reduce.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. List so far: mapreduce.map/reduce.java.opts mapreduce.map/reduce.memory.mb mapreduce.job.jvm.numtasks mapreduce.input.lineinputformat.linespermap mapreduce.task.combine.progress.records mapreduce.map/reduce.env -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091147#comment-14091147 ] Ray Chiang commented on MAPREDUCE-5130: --- FYI, I've put in a simple unit test in YARN-2284 that does a comparison of class member variables of a Configuration class and compares it against the corresponding .xml file. There are another 123 variables in MRJobConfig that are missing in mapred-default.xml. I'm sure some are missing due to being deprecated or being commented out. I'm also sure that there are some more actually missing, but it would take time to document properly. I wanted to get this one out of the way before starting on any other documentation in the .xml file. Add missing job config options to mapred-default.xml Key: MAPREDUCE-5130 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130-1.patch, MAPREDUCE-5130-2.patch, MAPREDUCE-5130-3.patch, MAPREDUCE-5130-4.patch, MAPREDUCE-5130-5.patch, MAPREDUCE-5130-6.patch, MAPREDUCE-5130.patch, MAPREDUCE-5130.patch I came across that mapreduce.map.java.opts and mapreduce.reduce.java.opts were missing in mapred-default.xml. I'll do a fuller sweep to see what else is missing before posting a patch. List so far: mapreduce.map/reduce.java.opts mapreduce.map/reduce.memory.mb mapreduce.job.jvm.numtasks mapreduce.input.lineinputformat.linespermap mapreduce.task.combine.progress.records mapreduce.map/reduce.env -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5918) LineRecordReader can return the same decompressor to CodecPool multiple times
[ https://issues.apache.org/jira/browse/MAPREDUCE-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091228#comment-14091228 ] Ravi Prakash commented on MAPREDUCE-5918: - Hi Sergey! Thanks for your contribution! :-) This is a good find. Sorry for taking so long to come around to it. # Should we do this for the mapreduce api as well (org.apache.hadoop.mapreduce.lib.input) ? # To me it seems the problem really is that CodecPool is not checking if the instance of Codec being returned is not already in the pool. Would you agree? I agree that we should also set decompressor to null after we have returned it. LineRecordReader can return the same decompressor to CodecPool multiple times - Key: MAPREDUCE-5918 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5918 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.3.0 Reporter: Sergey Murylev Assignee: Sergey Murylev Priority: Minor Fix For: trunk Attachments: MAPREDUCE-5918.1.patch LineRecordReader can return the same decompressor to CodecPool multiple times if method close() called multiple times. In this case CodecPool doesn't guarantee that it always return different decompressors. This issue can cause some difficult reproducible and difficult diagnosable bugs in Hadoop based programs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5918) LineRecordReader can return the same decompressor to CodecPool multiple times
[ https://issues.apache.org/jira/browse/MAPREDUCE-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-5918: Priority: Major (was: Minor) LineRecordReader can return the same decompressor to CodecPool multiple times - Key: MAPREDUCE-5918 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5918 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.3.0 Reporter: Sergey Murylev Assignee: Sergey Murylev Fix For: trunk Attachments: MAPREDUCE-5918.1.patch LineRecordReader can return the same decompressor to CodecPool multiple times if method close() called multiple times. In this case CodecPool doesn't guarantee that it always return different decompressors. This issue can cause some difficult reproducible and difficult diagnosable bugs in Hadoop based programs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6029) TestCommitterEventHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mit Desai updated MAPREDUCE-6029:
-
Target Version/s: 3.0.0
Affects Version/s: 3.0.0
Status: Patch Available (was: Open)
TestCommitterEventHandler fails in trunk
Key: MAPREDUCE-6029
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6029
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Ted Yu
Assignee: Mit Desai
Attachments: MAPREDUCE-6029.patch
From https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1857/console :
{code}
Running org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.429 sec
FAILURE! - in
org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
testFailure(org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler)
Time elapsed: 1.198 sec FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertNotNull(Assert.java:621)
at org.junit.Assert.assertNotNull(Assert.java:631)
at
org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler.testFailure(TestCommitterEventHandler.java:314)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-6029) TestCommitterEventHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mit Desai updated MAPREDUCE-6029:
-
Attachment: MAPREDUCE-6029.patch
The test allows 100msec for a response, and if the node is too slow the test
fails. 100msec is quite a bit of time, but I made it a 5 second wait. The
test will only wait that long if it really takes that long for the event to be
posted. If the event shows up sooner the timeout is short-circuited so there
shouldn't be a lot of downside to making this much larger.
TestCommitterEventHandler fails in trunk
Key: MAPREDUCE-6029
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6029
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Ted Yu
Assignee: Mit Desai
Attachments: MAPREDUCE-6029.patch
From https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1857/console :
{code}
Running org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.429 sec
FAILURE! - in
org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
testFailure(org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler)
Time elapsed: 1.198 sec FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertNotNull(Assert.java:621)
at org.junit.Assert.assertNotNull(Assert.java:631)
at
org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler.testFailure(TestCommitterEventHandler.java:314)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-6028) java.lang.ArithmeticException: / by zero
[ https://issues.apache.org/jira/browse/MAPREDUCE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091445#comment-14091445 ] Ravi Prakash commented on MAPREDUCE-6028: - eagles! JIRA is for fixing reported bugs. Please use the mailing list at u...@hive.apache.org or u...@hadoop.apache.org first to figure out whether this is indeed a bug I'll be closing this. java.lang.ArithmeticException: / by zero Key: MAPREDUCE-6028 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6028 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.2.1 Environment: hadoop version:1.2.1 Reporter: eagle Run any sql through hive with following error message: 2014-08-07 10:22:28,061 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201407251033_24476_m_02_0: Error initializing attempt_201407251033_24476_m_02_0: java.lang.ArithmeticException: / by zero and after restart hadoop cluster,the problem resolved。how to find the root casue for the problem? thks。 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-6028) java.lang.ArithmeticException: / by zero
[ https://issues.apache.org/jira/browse/MAPREDUCE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash resolved MAPREDUCE-6028. - Resolution: Invalid java.lang.ArithmeticException: / by zero Key: MAPREDUCE-6028 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6028 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.2.1 Environment: hadoop version:1.2.1 Reporter: eagle Run any sql through hive with following error message: 2014-08-07 10:22:28,061 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201407251033_24476_m_02_0: Error initializing attempt_201407251033_24476_m_02_0: java.lang.ArithmeticException: / by zero and after restart hadoop cluster,the problem resolved。how to find the root casue for the problem? thks。 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091465#comment-14091465 ] Manu Zhang commented on MAPREDUCE-6025: --- thanks for the comment. A new patch is uploaded. native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025-v3.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6025) native-task: fix native library distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-6025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated MAPREDUCE-6025: -- Attachment: mapreduce-6025-v3.txt native-task: fix native library distribution Key: MAPREDUCE-6025 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6025 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Attachments: mapreduce-6025-v2.txt, mapreduce-6025-v3.txt, mapreduce-6025.txt currently running mvn install -Pdist fails and nativetask native library is not distributed to hadoop tar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091466#comment-14091466
]
Hadoop QA commented on MAPREDUCE-5130:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660692/MAPREDUCE-5130-6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-tools/hadoop-gridmix.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4794//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4794//console
This message is automatically generated.
Add missing job config options to mapred-default.xml
Key: MAPREDUCE-5130
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: documentation
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130-1.patch,
MAPREDUCE-5130-2.patch, MAPREDUCE-5130-3.patch, MAPREDUCE-5130-4.patch,
MAPREDUCE-5130-5.patch, MAPREDUCE-5130-6.patch, MAPREDUCE-5130.patch,
MAPREDUCE-5130.patch
I came across that mapreduce.map.java.opts and mapreduce.reduce.java.opts
were missing in mapred-default.xml. I'll do a fuller sweep to see what else
is missing before posting a patch.
List so far:
mapreduce.map/reduce.java.opts
mapreduce.map/reduce.memory.mb
mapreduce.job.jvm.numtasks
mapreduce.input.lineinputformat.linespermap
mapreduce.task.combine.progress.records
mapreduce.map/reduce.env
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-6026) native-task: fix logging
[ https://issues.apache.org/jira/browse/MAPREDUCE-6026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091468#comment-14091468 ] Manu Zhang commented on MAPREDUCE-6026: --- Hi [~tlipcon], [~decster] and [~clockfly], please help to review that patch. native-task: fix logging Key: MAPREDUCE-6026 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6026 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Manu Zhang Assignee: Manu Zhang Priority: Minor Attachments: mapreduce-6026.txt nativetask should use commons-logging and add log4j.properties in test configuration as per hadoop standard -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6029) TestCommitterEventHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091497#comment-14091497
]
Hadoop QA commented on MAPREDUCE-6029:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660707/MAPREDUCE-6029.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4795//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4795//console
This message is automatically generated.
TestCommitterEventHandler fails in trunk
Key: MAPREDUCE-6029
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6029
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Ted Yu
Assignee: Mit Desai
Attachments: MAPREDUCE-6029.patch
From https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1857/console :
{code}
Running org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.429 sec
FAILURE! - in
org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler
testFailure(org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler)
Time elapsed: 1.198 sec FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertNotNull(Assert.java:621)
at org.junit.Assert.assertNotNull(Assert.java:631)
at
org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler.testFailure(TestCommitterEventHandler.java:314)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
