Hitesh Tons of thanks. I am able to run Tez job from a hive query on mapR cluster. I.e. hive query now fires a tez job on mapR.
As pointed out by you I made 2 mistakes. Not configuring mapR repository in tez pom.xml, somehow I thought I had done it but .. Second I was not using minimal tar ball instead was using full tar, now using minimal tar. With these 2 changes, things worked. Also, I ran tez-examples.jar with hadoop jar command, this also worked and executed a tez job. Now I will have to read tez docs to find out if there is a way to see the tez graph in a tez view on mapR like I see in Ambari on HDP? On Sat, Jul 30, 2016 at 10:25 PM, Sandeep Khurana <[email protected]> wrote: > Sorry, by mistake sent half written message. > > I will try both now and update this thread. > > About trying tez bundle, I tried with the steps at > http://doc.mapr.com/display/MapR41/Installing+and+Configuring+Tez+0.5.3 > .But hive job gave some NumberFormatError and found out by googling that > there is version mismatch between tez and hadoop libs. > > On Sat, Jul 30, 2016 at 10:22 PM, Sandeep Khurana <[email protected]> > wrote: > >> Hitesh >> >> Both of things >> >> On Sat, Jul 30, 2016 at 10:20 PM, Hitesh Shah <[email protected]> wrote: >> >>> Hello Sandeep, >>> >>> 2 things to check: >>> - When compiling Tez, is the hadoop.version in the top-level pom ( >>> and addition of mapr’s maven repo ) being used to compile against MapR’s >>> hadoop distribution and not the std. apache release? The Tez AM cannot seem >>> to do a handshake with the YARN RM. If MapR changed anything in that code >>> path, this issue could crop up. >>> - For tez.lib.uris, are you using the minimal tez tarball and not the >>> full one? The full one has hadoop jars in it so it should not be used in >>> your deployment setup as you are using MapR specific jars. You may want to >>> check for jar clashes in terms of the content of the tarball vs the jars >>> from the cluster. The only ones we do bundle in the minimal one are some >>> mapreduce jars I believe. >>> >>> FWIW, I think MapR does provide a Tez bundle for their clusters. You may >>> wish to check on their user lists too to see if they made some changes. If >>> you do find out anything, please do let us know in case this requires some >>> fixes in Tez for better compatibility across all distros. >>> >>> thanks >>> — Hitesh >>> >>> >>> >>> > On Jul 30, 2016, at 8:55 AM, Sandeep Khurana <[email protected]> >>> wrote: >>> > >>> > Hello >>> > >>> > >>> > On mapR single node cluster, I compiled Tez as per the documentation >>> on apache tez site. >>> > >>> > The only thing I changed from documentation was in tez-site.xml, where >>> I gave the value of tez.use.cluster.hadoop-libs as true. I did it because >>> mapR has its own jars for hadoop components. And without these tez might >>> not work. >>> > >>> > When I run any MR job or hive query I get this below error in the job >>> on RM link. (DIGEST-MD5: digest response format violation. Mismatched URI: >>> default/; expecting: null/default) >>> > >>> > I am using mapr 4.1 and tez 0.7.0. mapR 4.1 uses hive 1.2.0. >>> > >>> > Any pointer or suggestion what might be causing this issue or where >>> should I look? >>> > >>> > 2016-07-30 09:58:38,737 INFO [main] ipc.Server: Stopping server on >>> 45868 >>> > 2016-07-30 09:58:38,737 INFO [IPC Server Responder] ipc.Server: >>> Stopping IPC Server Responder >>> > 2016-07-30 09:58:38,739 INFO [IPC Server listener on 45868] >>> ipc.Server: Stopping IPC Server listener on 45868 >>> > 2016-07-30 09:58:38,739 INFO [IPC Server Responder] ipc.Server: >>> Stopping IPC Server Responder >>> > 2016-07-30 09:58:38,739 ERROR [main] app.DAGAppMaster: Error starting >>> DAGAppMaster >>> > org.apache.tez.dag.api.TezUncheckedException: >>> javax.security.sasl.SaslException: DIGEST-MD5: digest response format >>> violation. Mismatched URI: default/; expecting: null/default [Caused by >>> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): >>> DIGEST-MD5: digest response format violation. Mismatched URI: default/; >>> expecting: null/default] >>> > at >>> org.apache.tez.dag.app.rm.YarnTaskSchedulerService.serviceStart(YarnTaskSchedulerService.java:384) >>> > at >>> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) >>> > at >>> org.apache.tez.dag.app.rm.TaskSchedulerEventHandler.serviceStart(TaskSchedulerEventHandler.java:353) >>> > at >>> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) >>> > at >>> org.apache.tez.dag.app.DAGAppMaster$ServiceWithDependency.start(DAGAppMaster.java:1573) >>> > at >>> org.apache.tez.dag.app.DAGAppMaster$ServiceThread.run(DAGAppMaster.java:1591) >>> > Caused by: javax.security.sasl.SaslException: DIGEST-MD5: digest >>> response format violation. Mismatched URI: default/; expecting: >>> null/default [Caused by >>> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): >>> DIGEST-MD5: digest response format violation. Mismatched URI: default/; >>> expecting: null/default] >>> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>> Method) >>> > at >>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>> > at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>> > at >>> java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>> > at >>> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) >>> > at >>> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104) >>> > at >>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109) >>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> > at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> > at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> > at java.lang.reflect.Method.invoke(Method.java:606) >>> > at >>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) >>> > at >>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) >>> > at com.sun.proxy.$Proxy24.registerApplicationMaster(Unknown >>> Source) >>> > at >>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:222) >>> > at >>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:214) >>> > at >>> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.registerApplicationMaster(AMRMClientAsyncImpl.java:138) >>> > at >>> org.apache.tez.dag.app.rm.YarnTaskSchedulerService.serviceStart(YarnTaskSchedulerService.java:368) >>> > ... 5 more >>> > Caused by: >>> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): >>> DIGEST-MD5: digest response format violation. Mismatched URI: default/; >>> expecting: null/default >>> > at org.apache.hadoop.ipc.Client.call(Client.java:1468) >>> > at org.apache.hadoop.ipc.Client.call(Client.java:1399) >>> > at >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) >>> > at com.sun.proxy.$Proxy23.registerApplicationMaster(Unknown >>> Source) >>> > at >>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106) >>> > ... 16 more >>> > 2016-07-30 09:58:38,741 INFO [Thread-2] app.DAGAppMaster: >>> DAGAppMasterShutdownHook invoked >>> > >>> > >>> > -- >>> > Thanks and regards >>> > Sandeep Khurana >>> >>> >> >> >> -- >> Thanks and regards >> Sandeep Khurana >> > > > > -- > Thanks and regards > Sandeep Khurana > -- Thanks and regards Sandeep Khurana
