RE: Hadoop - impersonation doubts/issues while accessing from remote machine
Thanks :) Regards, Omkar Joshi -Original Message- From: Harsh J [mailto:ha...@cloudera.com] Sent: Friday, August 23, 2013 3:52 PM To: user@hadoop.apache.org Subject: Re: Hadoop - impersonation doubts/issues while accessing from remote machine I've answered this on the stackoverflow link: http://stackoverflow.com/questions/18354664/spring-data-hadoop-connectivity On Thu, Aug 22, 2013 at 1:29 PM, Omkar Joshi omkar.jo...@lntinfotech.com wrote: For readability, I haven't posted the code, output etc. in this mail - please check the thread below : http://stackoverflow.com/questions/18354664/spring-data-hadoop-connectivity I'm trying to connect to a remote hadoop(1.1.2) cluster from my local Windows machine via Spring data(later, eclipse plug-in may also be used). In future, multiple such connections from several Windows machines are expected. On my remote(single-node) cluster, bigdata is the user for Hadoop etc. bigdata@cloudx-843-770:~$ groups bigdata bigdata : bigdata On my local Windows machine D:\echo %username% 298790 D:\hostname INFVA03351 Now if I refer to Hadoop Secure Impersonation., does it mean I need to create a user 298790 on the cluster, add the hostname in core-site.xml etc. ??? Any less-cumbersome ways out? I tried that too on the cluster but the (partial given)output error still persists : Aug 22, 2013 12:29:20 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh INFO: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@1815338: startup date [Thu Aug 22 12:29:20 IST 2013]; root of context hierarchy Aug 22, 2013 12:29:20 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions INFO: Loading XML bean definitions from class path resource [com/hadoop/basics/applicationContext.xml] Aug 22, 2013 12:29:20 PM org.springframework.core.io.support.PropertiesLoaderSupport loadProperties INFO: Loading properties file from class path resource [resources/hadoop.properties] Aug 22, 2013 12:29:20 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7c197e: defining beans [org.springframework.context.support.PropertySourcesPlaceholderConfigurer#0,hadoopConfiguration,wc-job,myjobs-runner,resourceLoader]; root of factory hierarchy Aug 22, 2013 12:29:21 PM org.springframework.data.hadoop.mapreduce.JobExecutor$2 run INFO: Starting job [wc-job] Aug 22, 2013 12:29:21 PM org.apache.hadoop.security.UserGroupInformation doAs SEVERE: PriviledgedActionException as:bigdata via 298790 cause:org.apache.hadoop.ipc.RemoteException: User: 298790 is not allowed to impersonate bigdata Aug 22, 2013 12:29:21 PM org.springframework.data.hadoop.mapreduce.JobExecutor$2 run WARNING: Cannot start job [wc-job] org.apache.hadoop.ipc.RemoteException: User: 298790 is not allowed to impersonate bigdata at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at org.apache.hadoop.mapred.$Proxy2.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:411) at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:499) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:490) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:473) at org.apache.hadoop.mapreduce.Job$1.run(Job.java:513) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Unknown Source) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapreduce.Job.connect(Job.java:511) at org.apache.hadoop.mapreduce.Job.submit(Job.java:499) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530) at org.springframework.data.hadoop.mapreduce.JobExecutor$2.run(JobExecutor.java:197) at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:49) at org.springframework.data.hadoop.mapreduce.JobExecutor.startJobs(JobExecutor.java:168) at org.springframework.data.hadoop.mapreduce.JobExecutor.startJobs(JobExecutor.java:160) at org.springframework.data.hadoop.mapreduce.JobRunner.call(JobRunner.java:52) at org.springframework.data.hadoop.mapreduce.JobRunner.afterPropertiesSet(JobRunner.java:44) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1541) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1479
Hadoop - impersonation doubts/issues while accessing from remote machine
For readability, I haven't posted the code, output etc. in this mail - please check the thread below : http://stackoverflow.com/questions/18354664/spring-data-hadoop-connectivity I'm trying to connect to a remote hadoop(1.1.2) cluster from my local Windows machine via Spring data(later, eclipse plug-in may also be used). In future, multiple such connections from several Windows machines are expected. On my remote(single-node) cluster, bigdata is the user for Hadoop etc. bigdata@cloudx-843-770:~$ groups bigdata bigdata : bigdata On my local Windows machine D:\echo %username% 298790 D:\hostname INFVA03351 Now if I refer to Hadoop Secure Impersonation.http://hadoop.apache.org/docs/stable/Secure_Impersonation.html#Configurations, does it mean I need to create a user 298790 on the cluster, add the hostname in core-site.xml etc. ??? Any less-cumbersome ways out? I tried that too on the cluster but the (partial given)output error still persists : Aug 22, 2013 12:29:20 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh INFO: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@1815338: startup date [Thu Aug 22 12:29:20 IST 2013]; root of context hierarchy Aug 22, 2013 12:29:20 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions INFO: Loading XML bean definitions from class path resource [com/hadoop/basics/applicationContext.xml] Aug 22, 2013 12:29:20 PM org.springframework.core.io.support.PropertiesLoaderSupport loadProperties INFO: Loading properties file from class path resource [resources/hadoop.properties] Aug 22, 2013 12:29:20 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@7c197e: defining beans [org.springframework.context.support.PropertySourcesPlaceholderConfigurer#0,hadoopConfiguration,wc-job,myjobs-runner,resourceLoader]; root of factory hierarchy Aug 22, 2013 12:29:21 PM org.springframework.data.hadoop.mapreduce.JobExecutor$2 run INFO: Starting job [wc-job] Aug 22, 2013 12:29:21 PM org.apache.hadoop.security.UserGroupInformation doAs SEVERE: PriviledgedActionException as:bigdata via 298790 cause:org.apache.hadoop.ipc.RemoteException: User: 298790 is not allowed to impersonate bigdata Aug 22, 2013 12:29:21 PM org.springframework.data.hadoop.mapreduce.JobExecutor$2 run WARNING: Cannot start job [wc-job] org.apache.hadoop.ipc.RemoteException: User: 298790 is not allowed to impersonate bigdata at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at org.apache.hadoop.mapred.$Proxy2.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:411) at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:499) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:490) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:473) at org.apache.hadoop.mapreduce.Job$1.run(Job.java:513) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Unknown Source) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapreduce.Job.connect(Job.java:511) at org.apache.hadoop.mapreduce.Job.submit(Job.java:499) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530) at org.springframework.data.hadoop.mapreduce.JobExecutor$2.run(JobExecutor.java:197) at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:49) at org.springframework.data.hadoop.mapreduce.JobExecutor.startJobs(JobExecutor.java:168) at org.springframework.data.hadoop.mapreduce.JobExecutor.startJobs(JobExecutor.java:160) at org.springframework.data.hadoop.mapreduce.JobRunner.call(JobRunner.java:52) at org.springframework.data.hadoop.mapreduce.JobRunner.afterPropertiesSet(JobRunner.java:44) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1541) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1479) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223) at
