All,
Still working on getting Oozie 3.3 integrated with EMR with most of my time
so far spent resolving the security group config needed for VPC. The EC2
configuration was pretty simple but the main blocker right now is getting
past the error below:
Caused by: org.apache.hadoop.ipc.RemoteException: User: hadoop is not
allowed to impersonate hadoop
at org.apache.hadoop.ipc.Client.call(Client.java:1070)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy24.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at
org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
at
org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:411)
at
org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:409)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at
org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:409)
... 26 more
I know this is usually related to not having the correct proxy configs in
the core site but my current core site proxy configs are below (and I have
bounced both the NN and the JT since applying them):
<property><name>dfs.permissions</name><value>false</value></property>
<property><name>oozie.service.ProxyUserService.proxyuser.hadoop.hosts</name><value>*</value></property>
<property><name>oozie.service.ProxyUserService.proxyuser.hadoop.groups</name><value>*</value></property>
If I recall correctly this authorization is only checked at the the JT/NN
level and therefore shouldn't need to be pushed to the core site on the
slave machines right? Also, would there be any reason the wildcard would be
incompatible across hadoop distros (we are currently using 1.0.3 from EMR)?
Lastly, just for the sake of clarity is the proxy hosts config based on the
box submitting the oozie request (edge node) or based on the boxes actually
running the jobs (data/task nodes)?
--
Matt
On Thu, Dec 13, 2012 at 6:19 PM, Alejandro Abdelnur <t...@cloudera.com>wrote:
> Matt,
>
> It is not matter of bundling native code or not. Officially we suppose to
> do source releases only. As convenience we could do binaries, but there are
> discussions about that, if the could be signed or not.
>
> Regarding installing/running oozie in EC2. I never done it. Would you mind
> writing up a wiki on it once you figure it out?
>
> Cheers
>
>
> On Thu, Dec 13, 2012 at 4:02 PM, Matt Goeke <goeke.matt...@gmail.com>
> wrote:
>
> > Thank you both for the follow-up.
> >
> > 2 other questions that pertain to this:
> > 1) I don't remember any natives being required for Oozie so is there a
> > reason why we don't release with a -bin like most other apache projects?
> > 2) Are there any issues I might expect to run into when trying to run
> this
> > on EC2 backed by EMR?
> >
> > --
> > Matt
> >
> >
> > On Thu, Dec 13, 2012 at 5:48 PM, Alejandro Abdelnur <t...@cloudera.com
> > >wrote:
> >
> > > Matt,
> > >
> > > Apache Oozie release artifacts are sources only. The easiest way to
> build
> > > the TARBALL is:
> > >
> > > * install Maven
> > > * run bin/mkdistro.sh -DskipTests
> > >
> > > Then follow the Quick Start instructions.
> > >
> > > I'll open a JIRA to add this to the docs.
> > >
> > >
> > > On Thu, Dec 13, 2012 at 3:36 PM, Matt Goeke <goeke.matt...@gmail.com>
> > > wrote:
> > >
> > > > All,
> > > >
> > > > I am falling back to Oozie 3.2 for now but can someone possibly
> explain
> > > how
> > > > Oozie 3.3 is supposed to be configured? I was hoping to just follow
> the
> > > > quick start guide but it seems like the packaging does not match up
> at
> > > all.
> > > >
> > > > Trying to work through it I ended up downloading maven and running a
> > 'mvn
> > > > install' on the folder which built some of the hadooplibs but I am
> > still
> > > > missing all of the bin scripts.
> > > >
> > > > --
> > > > Matt
> > > >
> > >
> > >
> > >
> > > --
> > > Alejandro
> > >
> >
>
>
>
> --
> Alejandro
>
