this looks like it belongs to my problem, right? https://issues.apache.org/jira/browse/MAPREDUCE-50
Cheers Wolli 2013/10/11 DSuiter RDX <dsui...@rdx.com> > It looks like you are correct, and I did not have the right solution, I > apologize. I'm not sure if the other nodes need to be involved either. Now > I'm hoping someone with deeper knowledge will step in, because I'm curious > also! Some of the most knowledgeable people on here are on US Pacific Time, > so you will probably get more responses in a few hours. Sorry I couldn't be > of more assistance. > > Sincerely, > *Devin Suiter* > Jr. Data Solutions Software Engineer > 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212 > Google Voice: 412-256-8556 | www.rdx.com > > > On Fri, Oct 11, 2013 at 8:52 AM, fab wol <darkwoll...@gmail.com> wrote: > >> this line: >> >> 2013-10-11 10:24:53,033 ERROR >> org.apache.hadoop.security.UserGroupInformation: >> PriviledgedActionException as:mapred (auth:SIMPLE) >> cause:java.io.IOException: java.lang.NullPointerException >> >> is imho indicating that i am using the user "mapred" for executing (fyi: >> submitting the job from the CLI ( hadoop jar >> /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar >> wordcount hdfs_input_path hdfs_output_path) from another node) ... the file >> permissions for this file are: >> >> -rwxr-x--x 1 mapred hadoop 1382 Oct 10 15:02 topology.py* >> >> i temporarly had set the permissions to 777 to see if something changes, >> but it didn't ... I checked only the jobtracker, are the other nodes >> important for this as well? >> >> thx already in advance, especially for the quick response! >> Wolli >> >> >> 2013/10/11 DSuiter RDX <dsui...@rdx.com> >> >>> The user running the job (might not be your username depending on your >>> setup) does not appear to have executable permissions on the jobtracker >>> cluster topology python script - I'm basing this on the lines: >>> >>> 2013-10-11 10:24:53,035 WARN org.apache.hadoop.net.ScriptBasedMapping: >>> Exception running >>> /run/cloudera-scm-agent/process/556-mapreduce-JOBTRACKER/topology.py >>> 10.160.25.249 >>> java.io.IOException: Cannot run program >>> "/run/cloudera-scm-agent/process/556-mapreduce-JOBTRACKER/topology.py" (in >>> directory "/run/cloudera-scm-agent/process/556-mapreduce-JOBTRACKER"): >>> java.io.IOException: error=13, Permission denied >>> >>> So checking on the permissions for that file, determining what user is >>> kicking off your job, which depends on how you submit it, and making sure >>> that user has the execute permission on that file will probably fix this. >>> >>> If you are using a management console, such as Cloudera SCM, when you >>> submit jobs, they are run as an application user, so, Flume services run >>> under the "Flume" user, HBase jobs will typically run under the HBase user, >>> and so on. It can cause some surprises if you do not expect it. >>> >>> *Devin Suiter* >>> Jr. Data Solutions Software Engineer >>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212 >>> Google Voice: 412-256-8556 | www.rdx.com >>> >>> >>> On Fri, Oct 11, 2013 at 7:59 AM, fab wol <darkwoll...@gmail.com> wrote: >>> >>>> Hey everyone, I've got supplied with a decent ten node CDH 4.4 >>>> cluster, only 7 days old, and someone tried some HBase stuff on it. Now I >>>> wanted to try some MR Stuff on it, but starting a Job is already not >>>> possible (even the wordcount example). The error log of the jobtracker >>>> produces a log 700k lines long but it consists mainly of these lines >>>> repeatedly: >>>> >>>> 2013-10-11 10:24:53,033 INFO org.apache.hadoop.mapred.JobTracker: Lost >>>> tracker 'tracker_z-asanode02:localhost/127.0.0.1:53712' >>>> 2013-10-11 10:24:53,033 ERROR >>>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException >>>> as:mapred (auth:SIMPLE) cause:java.io.IOException: >>>> java.lang.NullPointerException >>>> 2013-10-11 10:24:53,034 INFO org.apache.hadoop.ipc.Server: IPC Server >>>> handler 22 on 8021, call >>>> heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@13b31acd, true, >>>> true, true, -1), rpc version=2, client version=32, >>>> methodsFingerPrint=-159967141 from 10.160.25.250:44389: error: >>>> java.io.IOException: java.lang.NullPointerException >>>> java.io.IOException: java.lang.NullPointerException >>>> at >>>> org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2751) >>>> at >>>> org.apache.hadoop.mapred.JobTracker.addNewTracker(JobTracker.java:2731) >>>> at >>>> org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3227) >>>> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2931) >>>> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>> at >>>> org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474) >>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002) >>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1751) >>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1747) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at javax.security.auth.Subject.doAs(Subject.java:396) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) >>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1745) >>>> 2013-10-11 10:24:53,035 WARN org.apache.hadoop.net.ScriptBasedMapping: >>>> Exception running >>>> /run/cloudera-scm-agent/process/556-mapreduce-JOBTRACKER/topology.py >>>> 10.160.25.249 >>>> java.io.IOException: Cannot run program >>>> "/run/cloudera-scm-agent/process/556-mapreduce-JOBTRACKER/topology.py" (in >>>> directory "/run/cloudera-scm-agent/process/556-mapreduce-JOBTRACKER"): >>>> java.io.IOException: error=13, Permission denied >>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460) >>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:206) >>>> at org.apache.hadoop.util.Shell.run(Shell.java:188) >>>> at >>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) >>>> at >>>> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242) >>>> at >>>> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180) >>>> at >>>> org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119) >>>> at >>>> org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750) >>>> at >>>> org.apache.hadoop.mapred.JobTracker.addNewTracker(JobTracker.java:2731) >>>> at >>>> org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3227) >>>> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2931) >>>> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>> at >>>> org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474) >>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002) >>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1751) >>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1747) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at javax.security.auth.Subject.doAs(Subject.java:396) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) >>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1745) >>>> Caused by: java.io.IOException: java.io.IOException: error=13, >>>> Permission denied >>>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148) >>>> at java.lang.ProcessImpl.start(ProcessImpl.java:65) >>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453) >>>> ... 21 more >>>> >>>> it doesn't matter if it is a pure hadoop job or a oozie submitted job. >>>> there seems to be something wrong in the basic configuration. Anyone an >>>> idea? >>>> >>>> Cheers >>>> Wolli >>>> >>> >>> >> >
