Re: Enabling Tez-UI on CDH 5.8

[Jan Morlock]

Hi Manoj,

sorry for the late reply. Your problem appears to be similar to

https://issues.apache.org/jira/browse/HIVE-8271

So please check whether you have both jackson-1.8 and jackson-1.9 somewhere
in your classpath.

That being said, you should know, that you can make a lot of friends if you
succeed in getting the Tez UI running on CDH.
I also once tried but didn't pursue further. You can find my post here:

http://mail-archives.apache.org/mod_mbox/tez-user/201608.mbox/%3ccao25edbahyckal+pbp2wr0tyxocamasju+ydvemfvnyqer6...@mail.gmail.com%3e

I hope that helps.

Cheers
Jan




2017-02-17 0:02 GMT+01:00 Manoj Murumkar :

> Also, hive version that's bundled with CDH5.8 is 1.1.0 which is also
> bundled with same class. Perhaps that's the one in effect? If that's the
> case, is there no way to use timeline server for history with this version?
>
> On Thu, Feb 16, 2017 at 2:53 PM, Manoj Murumkar 
> wrote:
>
>> Btw, here's the environment info:
>>
>> Tez: 0.8.4
>> Timeline server: bundled with apache 2.6.5 version
>>
>> I have added following libraries (that are used by timeline server) in
>> the path, so there's no mis-match, but no luck. I am still getting same
>> error.
>>
>> HADOOP_CLASSPATH=/usr/local/tez/client/lib/old/jackson-core-
>> asl-1.9.13.jar:/usr/local/tez/client/lib/old/jackson-mapper-
>> asl-1.9.13.jar:
>> HADOOP_USER_CLASSPATH_FIRST=true
>>
>>
>>
>> On Thu, Feb 16, 2017 at 12:02 PM, Manoj Murumkar <
>> manoj.murum...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Wondering if anyone has successfully enabled Tez-UI (by running ATS from
>>> a separate apache installation) on CDH 5.8 cluster. Would appreciate any
>>> information on this. We are have trouble with API incompatibility (error
>>> pasted below):
>>>
>>> 2017-02-16 19:40:13,874 [FATAL] [HistoryEventHandlingThread] 
>>> |yarn.YarnUncaughtExceptionHandler|: Thread 
>>> Thread[HistoryEventHandlingThread,5,main] threw an Error.  Shutting down 
>>> now...
>>> java.lang.AbstractMethodError: 
>>> org.codehaus.jackson.map.AnnotationIntrospector.findSerializer(Lorg/codehaus/jackson/map/introspect/Annotated;)Ljava/lang/Object;
>>> at 
>>> org.codehaus.jackson.map.ser.BasicSerializerFactory.findSerializerFromAnnotation(BasicSerializerFactory.java:362)
>>> at 
>>> org.codehaus.jackson.map.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:252)
>>> at 
>>> org.codehaus.jackson.map.ser.StdSerializerProvider._createUntypedSerializer(StdSerializerProvider.java:782)
>>> at 
>>> org.codehaus.jackson.map.ser.StdSerializerProvider._createAndCacheUntypedSerializer(StdSerializerProvider.java:735)
>>> at 
>>> org.codehaus.jackson.map.ser.StdSerializerProvider.findValueSerializer(StdSerializerProvider.java:344)
>>> at 
>>> org.codehaus.jackson.map.ser.StdSerializerProvider.findTypedValueSerializer(StdSerializerProvider.java:420)
>>> at 
>>> org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:601)
>>> at 
>>> org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256)
>>> at 
>>> org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:1604)
>>> at 
>>> org.codehaus.jackson.jaxrs.JacksonJsonProvider.writeTo(JacksonJsonProvider.java:527)
>>> at 
>>> com.sun.jersey.api.client.RequestWriter.writeRequestEntity(RequestWriter.java:300)
>>> at 
>>> com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:204)
>>> at 
>>> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
>>> at 
>>> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineJerseyRetryFilter$1.run(TimelineClientImpl.java:226)
>>> at 
>>> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:162)
>>> at 
>>> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineJerseyRetryFilter.handle(TimelineClientImpl.java:237)
>>> at com.sun.jersey.api.client.Client.handle(Client.java:648)
>>> at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
>>> at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
>>> at 
>>> com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:563)
>>> at 
>>> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPostingObject(TimelineClientImpl.java:472)
>>> at 
>>> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPosting(TimelineClientImpl.java:321)
>>> at 
>>> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:301)
>>> at 
>>> org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService.handleEvents(ATSHistoryLoggingService.java:357)
>>> at 
>>> 

Origin of failed tasks

[Jan Morlock]

Hi,

currently failed tasks occur during the execution of my Hive/Tez job.
However in the end, the overall job succeeds. Is it possible to find out
afterwards about the origin of those failed tasks (without using the Tez
UI) just by analyzing the output log files?

Best regards
Jan

Tez UI with CDH

[Jan Morlock]

Hi,

we are using Tez 0.8.4 successfully both with Hive and Pig on our
Cloudera CDH 5.7.1 cluster. However so far we haven't managed yet to
properly integrate the Tez UI in our infrastructure.

The problem is not the UI itself but rather the fact, that it relies on
the Application Timeline Server:
the Application Timeline Server does not seem to be an essential part of
the CDH distribution. By default it seems to be switched off. Although
we managed to activate it, we failed to integrate it in our system in a
way, that it gets automatically populated with application data.

Similar problem descriptions from other users can be found at

http://stackoverflow.com/questions/34221540/cloudera-cdh-can-not-start-yarn-timeline-server-for-tez-ui


Does anybody here have similar experiences and perhaps even find a
solution?

I am aware that the problem is slightly off-topic. None the less any
help is appreciated.

With best regards
Jan

Parallelism settings in Tez

[Jan Morlock]

Hi,

removing the statement

set default_parallel 24;

and all other information about parallelism from my script, causes Pig
on Tez (0.8.2) to fail with the following stack trace:


  org.apache.tez.dag.api.TezException: Vertex failed,
vertexName=scope-5824, vertexId=vertex_1456239615940_0236_1_61,
diagnostics=[Vertex vertex_1456239615940_0236_1_61 [scope-5824]
killed/failed due to:AM_USERCODE_FAILURE, Exception in VertexManager,
vertex:vertex_1456239615940_0236_1_61 [scope-5824],
org.apache.tez.dag.api.TezUncheckedException:
org.apache.pig.impl.plan.VisitorException: ERROR 0:
java.lang.NullPointerException
  at
org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigGraceShuffleVertexManager.onVertexStateUpdated(PigGraceShuffleVertexManager.java:162)
  at
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEventOnVertexStateUpdate.invoke(VertexManager.java:564)
  at
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:647)
  at
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:642)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
  at
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:642)
  at
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:631)
  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
  Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 0:
java.lang.NullPointerException
  at
org.apache.pig.backend.hadoop.executionengine.tez.plan.optimizer.ParallelismSetter.visitTezOp(ParallelismSetter.java:201)
  at
org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:246)
  at
org.apache.pig.backend.hadoop.executionengine.tez.plan.TezOperator.visit(TezOperator.java:53)
  at
org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:87)
  at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46)
  at
org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigGraceShuffleVertexManager.onVertexStateUpdated(PigGraceShuffleVertexManager.java:159)
  ... 12 more
  Caused by: java.lang.NullPointerException
  at
org.apache.pig.backend.hadoop.executionengine.tez.plan.optimizer.TezOperDependencyParallelismEstimator.estimateParallelism(TezOperDependencyParallelismEstimator.java:114)
  at
org.apache.pig.backend.hadoop.executionengine.tez.plan.optimizer.ParallelismSetter.visitTezOp(ParallelismSetter.java:138)
  ... 17 more
  ]


My primary intention behind removing that statement was, that I thought,
it would be best to leave the steering of parallelism up to Tez.
Is this basic assumption correct? And if yes, how can I avoid getting
the exception shown above?

Thank you very much in advance.
With best regards
Jan

Re: jansi dependendency?

[Jan Morlock]

Hi Stephen,

the code inside org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor intends
to use org.fusesource.jansi.Ansi. The Project hive-exec (where
TezJobMonitor is part of) therefore has a dependency on jline (where
Ansi is a part of). However on your class path a jar file containing an
older version of Ansi is found first. This version lacks the bold()
method and therefore the NoSuchMethodError is thrown.

What you have to do is identifying the jar file containing the older
version of Ansi. Afterwards you have to organize the class path in a way
that the more up-to-date version is found first.

I hope this helps.
Cheers
Jan


Am 15.02.2016 um 20:27 schrieb Stephen Sprague:
> hey guys,
> I'm looking to run Hive on Tez and have followed the instructions to a
> tee - but i just can't seem to get around this Jansi error despite
> everything i've tried.  Now given i'm not exactly a Java programmer what
> may appear to you as something pretty trivial i'm at an impasse - but
> not for lack of trying!
> 
> would anyone here have any tips?
> 
> thanks,
> Stephen
> 
> PS Here's my traceback and logging.
> 
> $ hive
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/spragues/downloads/apache-tez-0.7.0-src/tez-dist/target/tez-0.7.0-minimal/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/spragues/downloads/apache-tez-0.7.0-src/tez-dist/target/tez-0.7.0-minimal/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 
> Logging initialized using configuration in
> jar:file:/usr/lib/hive-1.2.1-standalone/lib/hive-common-1.2.1.jar!/hive-log4j.properties
> *hive> set hive.execution.engine=tez;*
> 
> *hive> select count(*) from omniture.hit_data where date_key=20160210;*
> Query ID = spragues_20160215111912_f4b6bc39-d29d-42bb-b0bc-262f8c99f58c
> Total jobs = 1
> Launching Job 1 out of 1
> 
> 
> Status: Running (Executing on YARN cluster with App id
> application_1453472707474_6031)
> 
> 
> java.lang.NoSuchMethodError:
> org.fusesource.jansi.Ansi.bold()Lorg/fusesource/jansi/Ansi;
> at
> org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor.reprintLineWithColorAsBold(TezJobMonitor.java:205)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor.printStatusInPlace(TezJobMonitor.java:611)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor.monitorExecution(TezJobMonitor.java:320)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:168)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> FAILED: Execution Error, return code -101 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask.
> org.fusesource.jansi.Ansi.bold()Lorg/fusesource/jansi/Ansi;

Different results between MapReduce and Tez as execution engines

[Jan Morlock]

Hi,

running Apache Pig, we are currently observing different results between
MapReduce and Tez as execution engines. A description of the problem
including a minimal example can be found here:

https://issues.apache.org/jira/browse/PIG-4789

Can anybody judge whether this is a Pig or a Tez bug?

Any help is appreciated.
Thank you very much in advance.
Jan