I would, but bdatadev is not one of my servers, it seems like a random host name. I can't figure out how or where this name got generated. That's what puzzling me.
On 8/31/13 5:43 AM, "Shekhar Sharma" <shekhar2...@gmail.com> wrote: >: java.net.UnknownHostException: bdatadev > > >edit your /etc/hosts file >Regards, >Som Shekhar Sharma >+91-8197243810 > > >On Sat, Aug 31, 2013 at 2:05 AM, Narlin M <hpn...@gmail.com> wrote: >> Looks like I was pointing to incorrect ports. After correcting the port >> numbers, >> >> conf.set("fs.defaultFS", "hdfs://<server_address>:8020"); >> conf.set("mapred.job.tracker", "<server_address>:8021"); >> >> I am now getting the following exception: >> >> 2880 [Thread-15] INFO >> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob - >> java.lang.IllegalArgumentException: java.net.UnknownHostException: >>bdatadev >> at >> >>org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.ja >>va:414) >> at >> >>org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.j >>ava:164) >> at >> >>org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:1 >>29) >> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:389) >> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:356) >> at >> >>org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSy >>stem.java:124) >> at >>org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2218) >> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80) >> at >>org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2252) >> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2234) >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:300) >> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194) >> at >> >>org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissio >>nFiles.java:103) >> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:902) >> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:396) >> at >> >>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation >>.java:1332) >> at >>org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896) >> at org.apache.hadoop.mapreduce.Job.submit(Job.java:531) >> at >> >>org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.sub >>mit(CrunchControlledJob.java:305) >> at >> >>org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.startR >>eadyJobs(CrunchJobControl.java:180) >> at >> >>org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJo >>bStatusAndStartNewOnes(CrunchJobControl.java:209) >> at >> >>org.apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:100 >>) >> at >>org.apache.crunch.impl.mr.exec.MRExecutor.access$000(MRExecutor.java:51) >> at org.apache.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:75) >> at java.lang.Thread.run(Thread.java:680) >> Caused by: java.net.UnknownHostException: bdatadev >> ... 27 more >> >> However nowhere in my code a host named "bdatadev" is mentioned, and I >> cannot ping this host. >> >> Thanks for the help. >> >> >> On Fri, Aug 30, 2013 at 3:04 PM, Narlin M <hpn...@gmail.com> wrote: >>> >>> I am getting following exception while trying to submit a crunch >>>pipeline >>> job to a remote hadoop cluster: >>> >>> Exception in thread "main" java.lang.RuntimeException: Cannot create >>>job >>> output directory /tmp/crunch-324987940 >>> at >>> >>>org.apache.crunch.impl.mr.MRPipeline.createTempDirectory(MRPipeline.java >>>:344) >>> at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:125) >>> at test.CrunchTest.setup(CrunchTest.java:98) >>> at test.CrunchTest.main(CrunchTest.java:367) >>> Caused by: java.io.IOException: Failed on local exception: >>> com.google.protobuf.InvalidProtocolBufferException: Protocol message >>> end-group tag did not match expected tag.; Host Details : local host >>>is: >>> "NARLIN/127.0.0.1"; destination host is: "<server_address>":50070; >>> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:759) >>> at org.apache.hadoop.ipc.Client.call(Client.java:1164) >>> at >>> >>>org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine >>>.java:202) >>> at com.sun.proxy.$Proxy11.mkdirs(Unknown Source) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> >>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav >>>a:39) >>> at >>> >>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor >>>Impl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at >>> >>>org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvo >>>cationHandler.java:164) >>> at >>> >>>org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocation >>>Handler.java:83) >>> at com.sun.proxy.$Proxy11.mkdirs(Unknown Source) >>> at >>> >>>org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkd >>>irs(ClientNamenodeProtocolTranslatorPB.java:425) >>> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:1943) >>> at >>> >>>org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSyste >>>m.java:523) >>> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1799) >>> at >>> >>>org.apache.crunch.impl.mr.MRPipeline.createTempDirectory(MRPipeline.java >>>:342) >>> ... 3 more >>> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol >>> message end-group tag did not match expected tag. >>> at >>> >>>com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(Invalid >>>ProtocolBufferException.java:73) >>> at >>> >>>com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.ja >>>va:124) >>> at >>> >>>com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessag >>>eLite.java:213) >>> at >>> >>>com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.ja >>>va:746) >>> at >>> >>>com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.ja >>>va:238) >>> at >>> >>>com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(Abstr >>>actMessageLite.java:282) >>> at >>> >>>com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractM >>>essage.java:760) >>> at >>> >>>com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(Abstr >>>actMessageLite.java:288) >>> at >>> >>>com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractM >>>essage.java:752) >>> at >>> >>>org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderP >>>roto.parseDelimitedFrom(RpcPayloadHeaderProtos.java:985) >>> at >>> >>>org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:882) >>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:813) >>> 0 [Thread-3] WARN org.apache.hadoop.util.ShutdownHookManager - >>> ShutdownHook 'ClientFinalizer' failed, java.lang.NoSuchMethodError: >>> com.google.common.collect.LinkedListMultimap.values()Ljava/util/List; >>> java.lang.NoSuchMethodError: >>> com.google.common.collect.LinkedListMultimap.values()Ljava/util/List; >>> at org.apache.hadoop.hdfs.SocketCache.clear(SocketCache.java:135) >>> at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:672) >>> at >>> >>>org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem >>>.java:539) >>> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2308) >>> at >>> >>>org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.jav >>>a:2324) >>> at >>> >>>org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.jav >>>a:54) >>> >>> Google search on this error yielded solutions that asked to confirm >>>that >>> /etc/hosts file contained the entry for NARLIN which it does in my >>>case. >>> >>> Here's the code that I am using to set up the MRPipeline: >>> >>> Configuration conf = HBaseConfiguration.create(); >>> >>> conf.set("fs.defaultFS", "hdfs://<server_address>:50070"); >>> conf.set("mapred.job.tracker", "<server_address>:50030"); >>> >>> System.out.println("Hadoop configuration created."); >>> System.out.println("Initializing crunch pipeline ..."); >>> >>> conf.set("mapred.jar", "<path_to_jar_file>"); >>> >>> pipeline = new MRPipeline(getClass(), "crunchjobtest", conf); >>> >>> Has anyone faced this issue before and knows how to resolve it/point >>>out >>> if I am missing anything? >>> >>> Thanks for the help. >> >>