[jira] [Commented] (TEZ-3165) Parallel initialization of inputs, outputs, and processor can cause NoSuchMethodException
[
https://issues.apache.org/jira/browse/TEZ-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194701#comment-15194701
]
Jonathan Eagles commented on TEZ-3165:
--
Unfortunately, this does not appear to be a simple case of multiple conflicting
jars. SequenceFileConfig and TextConverter are verified to be loaded from the
same jar using '-verbose:class' jvm option. Somehow, between
PigProcessor.initialize, the HBase Storage input and elephant bird sequence
file output, there is a race condition that exhibits itself as a
NoSuchMethodError. I can't really explain what is causing this. I'll try to dig
deeper tomorrow to understand the root cause.
> Parallel initialization of inputs, outputs, and processor can cause
> NoSuchMethodException
> -
>
> Key: TEZ-3165
> URL: https://issues.apache.org/jira/browse/TEZ-3165
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3165.1.patch
>
>
> 2016-03-13 23:55:17,162 [INFO] [main]
> |runtime.LogicalIOProcessorRuntimeTask|: Initializing
> LogicalIOProcessorRuntimeTask with TaskSpec: DAGName :
> PigLatin:Script.pig-0_scope-0, VertexName: scope-203, VertexParallelism:
> 2707, TaskAttemptID:attempt_1,
> processorName=org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor,
> inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{
> sourceVertexName=scope-0, physicalEdgeCount=1,
> inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ],
> outputSpecList=[{{ destinationVertexName=scope-28, physicalEdgeCount=0,
> outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
> 2016-03-13 23:55:17,164 [INFO] [main] |resources.MemoryDistributor|:
> InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1,
> numOutputs=1, JVM.maxFree=1059061760,
> allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
> 2016-03-13 23:55:17,175 [INFO] [TezChild] |task.TezTaskRunner|: Initializing
> task, taskAttemptId=attempt_1
> 2016-03-13 23:55:17,182 [INFO] [TaskHeartbeatThread] |task.TaskReporter|:
> Routing events from heartbeat response to task,
> currentTaskAttemptId=attempt_1, eventCount=1 fromEventId=0 nextFromEventId=0
> 2016-03-13 23:55:17,212 [INFO] [I/O Setup 1 Initialize: {scope-28}]
> |Configuration.deprecation|: mapreduce.inputformat.class is deprecated.
> Instead, use mapreduce.job.inputformat.class
> 2016-03-13 23:55:17,214 [INFO] [I/O Setup 1 Initialize: {scope-28}]
> |Configuration.deprecation|: fs.default.name is deprecated. Instead, use
> fs.defaultFS
> 2016-03-13 23:55:17,223 [INFO] [I/O Setup 1 Initialize: {scope-28}]
> |counters.Limits|: Counter limits initialized with parameters:
> GROUP_NAME_MAX=256, MAX_GROUPS=1000, COUNTER_NAME_MAX=128, MAX_COUNTERS=5000
> 2016-03-13 23:55:17,228 [INFO] [I/O Setup 0 Initialize: {scope-0}]
> |input.MRInput|: scope-0 using newmapreduce API=true, split via event=true,
> numPhysicalInputs=1
> 2016-03-13 23:55:17,233 [INFO] [I/O Setup 0 Initialize: {scope-0}]
> |input.MRInput|: Initialized MRInput: scope-0
> 2016-03-13 23:55:17,345 [INFO] [TezChild] |data.SchemaTupleBackend|: Key
> [pig.schematuple] was not set... will not generate code.
> 2016-03-13 23:55:17,400 [INFO] [TezChild]
> |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor
> 2016-03-13 23:55:17,400 [INFO] [TezChild]
> |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish
> 2016-03-13 23:55:17,400 [INFO] [TezChild]
> |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish
> 2016-03-13 23:55:17,400 [INFO] [TezChild] |task.TezTaskRunner|: Encounted an
> error while executing task: attempt_1
> java.lang.RuntimeException: could not instantiate
> 'com.twitter.elephantbird.pig.store.SequenceFileStorage' with arguments '[-c
> com.twitter.elephantbird.pig.util.TextConverter, -c
> com.twitter.elephantbird.pig.util.TextConverter]'
> at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:766)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:250)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:76)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigOutputFormatTez.getRecordWriter(PigOutputFormatTez.java:43)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:399)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable._callInternal(LogicalIOProcessorRuntimeTask.java:506)
> at
>
Failed: TEZ-2936 PreCommit Build #1563
Jira: https://issues.apache.org/jira/browse/TEZ-2936
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1563/
###
## LAST 60 LINES OF THE CONSOLE
###
Started by remote host 127.0.0.1
[EnvInject] - Loading node environment variables.
Building remotely on H7 (Mapreduce Falcon Hadoop Pig Zookeeper Tez Hdfs
yahoo-not-h2) in workspace
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://git-wip-us.apache.org/repos/asf/tez.git
> # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/tez.git
> git --version # timeout=10
> git -c core.askpass=true fetch --tags --progress
> https://git-wip-us.apache.org/repos/asf/tez.git
> +refs/heads/*:refs/remotes/origin/*
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 42b61f43f1e4e0cb47a55249fb21378b62bde2c4
(refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 42b61f43f1e4e0cb47a55249fb21378b62bde2c4
> git rev-list dbd763fd479ccebf3988d23f3284fe1ec2f16d64 # timeout=10
No emails were triggered.
[PreCommit-TEZ-Build] $ /bin/bash /tmp/hudson3773472962083433625.sh
Running in Jenkins mode
==
==
Testing patch for TEZ-2936.
==
==
HEAD is now at 42b61f4 TEZ-2936. Create ATS implementation that enables support
for YARN-4265 (ATSv1.5) (hitesh)
Previous HEAD position was 42b61f4... TEZ-2936. Create ATS implementation that
enables support for YARN-4265 (ATSv1.5) (hitesh)
Switched to branch 'master'
Your branch is behind 'origin/master' by 5 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
First, rewinding head to replay your work on top of it...
Fast-forwarded master to 42b61f43f1e4e0cb47a55249fb21378b62bde2c4.
TEZ-2936 is not "Patch Available". Exiting.
==
==
Finished build.
==
==
Archiving artifacts
ERROR: No artifacts found that match the file pattern "patchprocess/*.*".
Configuration error?
ERROR: ?patchprocess/*.*? doesn?t match anything, but ?*.*? does. Perhaps
that?s what you mean?
Build step 'Archive the artifacts' changed build result to FAILURE
[description-setter] Could not determine description.
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
No tests ran.
[jira] [Commented] (TEZ-2686) TestFaultTolerance fails frequently
[ https://issues.apache.org/jira/browse/TEZ-2686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194477#comment-15194477 ] Jeff Zhang commented on TEZ-2686: - [~aplusplus] Assign to you, thanks for taking this. > TestFaultTolerance fails frequently > > > Key: TEZ-2686 > URL: https://issues.apache.org/jira/browse/TEZ-2686 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jeff Zhang >Assignee: Zhiyuan Yang > Attachments: log.tar, syslog_dag_1451372520174_0001_18, > syslog_dag_1451372520174_0001_18_post > > > TestFaultTolerance will fail with a very little possibility. But it fails > frequently recently, need to take a look at it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2686) TestFaultTolerance fails frequently
[ https://issues.apache.org/jira/browse/TEZ-2686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2686: Assignee: Zhiyuan Yang > TestFaultTolerance fails frequently > > > Key: TEZ-2686 > URL: https://issues.apache.org/jira/browse/TEZ-2686 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jeff Zhang >Assignee: Zhiyuan Yang > Attachments: log.tar, syslog_dag_1451372520174_0001_18, > syslog_dag_1451372520174_0001_18_post > > > TestFaultTolerance will fail with a very little possibility. But it fails > frequently recently, need to take a look at it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2686) TestFaultTolerance fails frequently
[ https://issues.apache.org/jira/browse/TEZ-2686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194440#comment-15194440 ] Zhiyuan Yang commented on TEZ-2686: --- [~zjffdu] This problem still happens, like in TEZ-3155 BUILD#1552. Would you mind assigning this JIRA to me? I'd like to have a try. > TestFaultTolerance fails frequently > > > Key: TEZ-2686 > URL: https://issues.apache.org/jira/browse/TEZ-2686 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jeff Zhang > Attachments: log.tar, syslog_dag_1451372520174_0001_18, > syslog_dag_1451372520174_0001_18_post > > > TestFaultTolerance will fail with a very little possibility. But it fails > frequently recently, need to take a look at it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3165) Parallel initialization of inputs, outputs, and processor can cause NoSuchMethodException
[
https://issues.apache.org/jira/browse/TEZ-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194399#comment-15194399
]
Siddharth Seth commented on TEZ-3165:
-
[~jeagles] - is this happening when relocalization kicks in ? Otherwise are
other versions of the library in the classpath ? Don't really see how moving
the initializeLogicalIOProcessor() call helps.
> Parallel initialization of inputs, outputs, and processor can cause
> NoSuchMethodException
> -
>
> Key: TEZ-3165
> URL: https://issues.apache.org/jira/browse/TEZ-3165
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3165.1.patch
>
>
> 2016-03-13 23:55:17,162 [INFO] [main]
> |runtime.LogicalIOProcessorRuntimeTask|: Initializing
> LogicalIOProcessorRuntimeTask with TaskSpec: DAGName :
> PigLatin:Script.pig-0_scope-0, VertexName: scope-203, VertexParallelism:
> 2707, TaskAttemptID:attempt_1,
> processorName=org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor,
> inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{
> sourceVertexName=scope-0, physicalEdgeCount=1,
> inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ],
> outputSpecList=[{{ destinationVertexName=scope-28, physicalEdgeCount=0,
> outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
> 2016-03-13 23:55:17,164 [INFO] [main] |resources.MemoryDistributor|:
> InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1,
> numOutputs=1, JVM.maxFree=1059061760,
> allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
> 2016-03-13 23:55:17,175 [INFO] [TezChild] |task.TezTaskRunner|: Initializing
> task, taskAttemptId=attempt_1
> 2016-03-13 23:55:17,182 [INFO] [TaskHeartbeatThread] |task.TaskReporter|:
> Routing events from heartbeat response to task,
> currentTaskAttemptId=attempt_1, eventCount=1 fromEventId=0 nextFromEventId=0
> 2016-03-13 23:55:17,212 [INFO] [I/O Setup 1 Initialize: {scope-28}]
> |Configuration.deprecation|: mapreduce.inputformat.class is deprecated.
> Instead, use mapreduce.job.inputformat.class
> 2016-03-13 23:55:17,214 [INFO] [I/O Setup 1 Initialize: {scope-28}]
> |Configuration.deprecation|: fs.default.name is deprecated. Instead, use
> fs.defaultFS
> 2016-03-13 23:55:17,223 [INFO] [I/O Setup 1 Initialize: {scope-28}]
> |counters.Limits|: Counter limits initialized with parameters:
> GROUP_NAME_MAX=256, MAX_GROUPS=1000, COUNTER_NAME_MAX=128, MAX_COUNTERS=5000
> 2016-03-13 23:55:17,228 [INFO] [I/O Setup 0 Initialize: {scope-0}]
> |input.MRInput|: scope-0 using newmapreduce API=true, split via event=true,
> numPhysicalInputs=1
> 2016-03-13 23:55:17,233 [INFO] [I/O Setup 0 Initialize: {scope-0}]
> |input.MRInput|: Initialized MRInput: scope-0
> 2016-03-13 23:55:17,345 [INFO] [TezChild] |data.SchemaTupleBackend|: Key
> [pig.schematuple] was not set... will not generate code.
> 2016-03-13 23:55:17,400 [INFO] [TezChild]
> |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor
> 2016-03-13 23:55:17,400 [INFO] [TezChild]
> |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish
> 2016-03-13 23:55:17,400 [INFO] [TezChild]
> |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish
> 2016-03-13 23:55:17,400 [INFO] [TezChild] |task.TezTaskRunner|: Encounted an
> error while executing task: attempt_1
> java.lang.RuntimeException: could not instantiate
> 'com.twitter.elephantbird.pig.store.SequenceFileStorage' with arguments '[-c
> com.twitter.elephantbird.pig.util.TextConverter, -c
> com.twitter.elephantbird.pig.util.TextConverter]'
> at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:766)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:250)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:76)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigOutputFormatTez.getRecordWriter(PigOutputFormatTez.java:43)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:399)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable._callInternal(LogicalIOProcessorRuntimeTask.java:506)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:489)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:474)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
>
[jira] [Updated] (TEZ-2936) Create ATS implementation that enables support for YARN-4265
[ https://issues.apache.org/jira/browse/TEZ-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-2936: - Attachment: TEZ-2936.5.addendum-build.patch > Create ATS implementation that enables support for YARN-4265 > > > Key: TEZ-2936 > URL: https://issues.apache.org/jira/browse/TEZ-2936 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: TEZ-2936.2.patch, TEZ-2936.3.patch, TEZ-2936.4.patch, > TEZ-2936.5.addendum-build.patch, TEZ-2936.5.patch, TEZ-2936.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2936) Create ATS implementation that enables support for YARN-4265
[ https://issues.apache.org/jira/browse/TEZ-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194380#comment-15194380 ] Hitesh Shah commented on TEZ-2936: -- I will modify the build readme for the 2.8 profile. Also 2.8.0-SNAPSHOT should also likely work as as long as hadoop is built locally ( I think the uploaded snapshots are old for 2.8.0-SNAPSHOT \cc [~gtCarrera9] ) > Create ATS implementation that enables support for YARN-4265 > > > Key: TEZ-2936 > URL: https://issues.apache.org/jira/browse/TEZ-2936 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: TEZ-2936.2.patch, TEZ-2936.3.patch, TEZ-2936.4.patch, > TEZ-2936.5.patch, TEZ-2936.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3166) Counters aren't fully updated and sent for failed application
[ https://issues.apache.org/jira/browse/TEZ-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-3166: -- Description: On master branch, if an application fails very soon, counters aren't fully updated so on tez-ui we can only see part of counters. We should publish all counters even task attempt fails. (was: On master branch, if an application fails very soon, counters aren't fully updated so on tez-ui we can only see part of counters. We should publish all counters even task attempt failed.) > Counters aren't fully updated and sent for failed application > - > > Key: TEZ-3166 > URL: https://issues.apache.org/jira/browse/TEZ-3166 > Project: Apache Tez > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > On master branch, if an application fails very soon, counters aren't fully > updated so on tez-ui we can only see part of counters. We should publish all > counters even task attempt fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3166) Counters aren't fully updated and sent for failed application
[ https://issues.apache.org/jira/browse/TEZ-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-3166: -- Description: On master branch, if an application fails very soon, counters aren't fully updated so on tez-ui we can only see part of counters. We should publish all counters even task attempt failed. (was: On master branch, if an application failed very soon, counters aren't fully updated so on tez-ui we can only see part of counters. We should publish all counters even task attempt failed.) > Counters aren't fully updated and sent for failed application > - > > Key: TEZ-3166 > URL: https://issues.apache.org/jira/browse/TEZ-3166 > Project: Apache Tez > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > On master branch, if an application fails very soon, counters aren't fully > updated so on tez-ui we can only see part of counters. We should publish all > counters even task attempt failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3166) Counters aren't fully updated and sent for failed application
Zhiyuan Yang created TEZ-3166: - Summary: Counters aren't fully updated and sent for failed application Key: TEZ-3166 URL: https://issues.apache.org/jira/browse/TEZ-3166 Project: Apache Tez Issue Type: Bug Reporter: Zhiyuan Yang Assignee: Zhiyuan Yang On master branch, if an application failed very soon, counters aren't fully updated so on tez-ui we can only see part of counters. We should publish all counters even task attempt failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2954) Container launch timeouts should count towards node blacklisting
[ https://issues.apache.org/jira/browse/TEZ-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194355#comment-15194355 ] Siddharth Seth commented on TEZ-2954: - [~ozawa] - the problem is highlighted in https://issues.apache.org/jira/browse/TEZ-925?focusedCommentId=13932292=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13932292 If we receive a container timeout - we would have received a task timeout as well - which is factored in. The problem is that a launch failure on the NM will be reported back via the RM. When that happens, we lose track of the fact that the launch failed. If there's a timoue while talking to the NM - that will register as a task failure. The jira description should have been better. > Container launch timeouts should count towards node blacklisting > > > Key: TEZ-2954 > URL: https://issues.apache.org/jira/browse/TEZ-2954 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-2954.001.patch > > > Currently, only task failures count towards blacklisting. A container timing > out should do the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3163) Reuse and tune Inflaters and Deflaters to speed DME processing
[ https://issues.apache.org/jira/browse/TEZ-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194327#comment-15194327 ] Jonathan Eagles commented on TEZ-3163: -- I have also prototype a version where the emptyPartitions bitmap is replaced by a roaring bitmap. The speed is much much faster. Trying to get a sense of what approach or combination of approaches is best here. > Reuse and tune Inflaters and Deflaters to speed DME processing > -- > > Key: TEZ-3163 > URL: https://issues.apache.org/jira/browse/TEZ-3163 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: TEZ-3163.1-branch-0.7.patch, TEZ-3163.1.patch, > TEZ-3163.PERF.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3165) Parallel initialization of inputs, outputs, and processor can cause NoSuchMethodException
[
https://issues.apache.org/jira/browse/TEZ-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-3165:
-
Attachment: TEZ-3165.1.patch
[~sseth], attached a prototype patch that avoids the NoSuchMethodException to
give some insight into this issue. Do you know if something about the
Input/Output Callables is causing an issue with classloading?
> Parallel initialization of inputs, outputs, and processor can cause
> NoSuchMethodException
> -
>
> Key: TEZ-3165
> URL: https://issues.apache.org/jira/browse/TEZ-3165
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3165.1.patch
>
>
> 2016-03-13 23:55:17,162 [INFO] [main]
> |runtime.LogicalIOProcessorRuntimeTask|: Initializing
> LogicalIOProcessorRuntimeTask with TaskSpec: DAGName :
> PigLatin:Script.pig-0_scope-0, VertexName: scope-203, VertexParallelism:
> 2707, TaskAttemptID:attempt_1,
> processorName=org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor,
> inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{
> sourceVertexName=scope-0, physicalEdgeCount=1,
> inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ],
> outputSpecList=[{{ destinationVertexName=scope-28, physicalEdgeCount=0,
> outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
> 2016-03-13 23:55:17,164 [INFO] [main] |resources.MemoryDistributor|:
> InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1,
> numOutputs=1, JVM.maxFree=1059061760,
> allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
> 2016-03-13 23:55:17,175 [INFO] [TezChild] |task.TezTaskRunner|: Initializing
> task, taskAttemptId=attempt_1
> 2016-03-13 23:55:17,182 [INFO] [TaskHeartbeatThread] |task.TaskReporter|:
> Routing events from heartbeat response to task,
> currentTaskAttemptId=attempt_1, eventCount=1 fromEventId=0 nextFromEventId=0
> 2016-03-13 23:55:17,212 [INFO] [I/O Setup 1 Initialize: {scope-28}]
> |Configuration.deprecation|: mapreduce.inputformat.class is deprecated.
> Instead, use mapreduce.job.inputformat.class
> 2016-03-13 23:55:17,214 [INFO] [I/O Setup 1 Initialize: {scope-28}]
> |Configuration.deprecation|: fs.default.name is deprecated. Instead, use
> fs.defaultFS
> 2016-03-13 23:55:17,223 [INFO] [I/O Setup 1 Initialize: {scope-28}]
> |counters.Limits|: Counter limits initialized with parameters:
> GROUP_NAME_MAX=256, MAX_GROUPS=1000, COUNTER_NAME_MAX=128, MAX_COUNTERS=5000
> 2016-03-13 23:55:17,228 [INFO] [I/O Setup 0 Initialize: {scope-0}]
> |input.MRInput|: scope-0 using newmapreduce API=true, split via event=true,
> numPhysicalInputs=1
> 2016-03-13 23:55:17,233 [INFO] [I/O Setup 0 Initialize: {scope-0}]
> |input.MRInput|: Initialized MRInput: scope-0
> 2016-03-13 23:55:17,345 [INFO] [TezChild] |data.SchemaTupleBackend|: Key
> [pig.schematuple] was not set... will not generate code.
> 2016-03-13 23:55:17,400 [INFO] [TezChild]
> |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor
> 2016-03-13 23:55:17,400 [INFO] [TezChild]
> |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish
> 2016-03-13 23:55:17,400 [INFO] [TezChild]
> |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish
> 2016-03-13 23:55:17,400 [INFO] [TezChild] |task.TezTaskRunner|: Encounted an
> error while executing task: attempt_1
> java.lang.RuntimeException: could not instantiate
> 'com.twitter.elephantbird.pig.store.SequenceFileStorage' with arguments '[-c
> com.twitter.elephantbird.pig.util.TextConverter, -c
> com.twitter.elephantbird.pig.util.TextConverter]'
> at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:766)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:250)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:76)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigOutputFormatTez.getRecordWriter(PigOutputFormatTez.java:43)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:399)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable._callInternal(LogicalIOProcessorRuntimeTask.java:506)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:489)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:474)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
>
[jira] [Updated] (TEZ-3165) Parallel initialization of inputs, outputs, and processor can cause NoSuchMethodException
[
https://issues.apache.org/jira/browse/TEZ-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-3165:
-
Description:
2016-03-13 23:55:17,162 [INFO] [main] |runtime.LogicalIOProcessorRuntimeTask|:
Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName :
PigLatin:Script.pig-0_scope-0, VertexName: scope-203, VertexParallelism: 2707,
TaskAttemptID:attempt_1,
processorName=org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor,
inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{
sourceVertexName=scope-0, physicalEdgeCount=1,
inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{
destinationVertexName=scope-28, physicalEdgeCount=0,
outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-03-13 23:55:17,164 [INFO] [main] |resources.MemoryDistributor|:
InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1,
numOutputs=1, JVM.maxFree=1059061760,
allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-03-13 23:55:17,175 [INFO] [TezChild] |task.TezTaskRunner|: Initializing
task, taskAttemptId=attempt_1
2016-03-13 23:55:17,182 [INFO] [TaskHeartbeatThread] |task.TaskReporter|:
Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1,
eventCount=1 fromEventId=0 nextFromEventId=0
2016-03-13 23:55:17,212 [INFO] [I/O Setup 1 Initialize: {scope-28}]
|Configuration.deprecation|: mapreduce.inputformat.class is deprecated.
Instead, use mapreduce.job.inputformat.class
2016-03-13 23:55:17,214 [INFO] [I/O Setup 1 Initialize: {scope-28}]
|Configuration.deprecation|: fs.default.name is deprecated. Instead, use
fs.defaultFS
2016-03-13 23:55:17,223 [INFO] [I/O Setup 1 Initialize: {scope-28}]
|counters.Limits|: Counter limits initialized with parameters:
GROUP_NAME_MAX=256, MAX_GROUPS=1000, COUNTER_NAME_MAX=128, MAX_COUNTERS=5000
2016-03-13 23:55:17,228 [INFO] [I/O Setup 0 Initialize: {scope-0}]
|input.MRInput|: scope-0 using newmapreduce API=true, split via event=true,
numPhysicalInputs=1
2016-03-13 23:55:17,233 [INFO] [I/O Setup 0 Initialize: {scope-0}]
|input.MRInput|: Initialized MRInput: scope-0
2016-03-13 23:55:17,345 [INFO] [TezChild] |data.SchemaTupleBackend|: Key
[pig.schematuple] was not set... will not generate code.
2016-03-13 23:55:17,400 [INFO] [TezChild]
|runtime.LogicalIOProcessorRuntimeTask|: Initialized processor
2016-03-13 23:55:17,400 [INFO] [TezChild]
|runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish
2016-03-13 23:55:17,400 [INFO] [TezChild]
|runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish
2016-03-13 23:55:17,400 [INFO] [TezChild] |task.TezTaskRunner|: Encounted an
error while executing task: attempt_1
java.lang.RuntimeException: could not instantiate
'com.twitter.elephantbird.pig.store.SequenceFileStorage' with arguments '[-c
com.twitter.elephantbird.pig.util.TextConverter, -c
com.twitter.elephantbird.pig.util.TextConverter]'
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:766)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:250)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:76)
at
org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigOutputFormatTez.getRecordWriter(PigOutputFormatTez.java:43)
at
org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:399)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable._callInternal(LogicalIOProcessorRuntimeTask.java:506)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:489)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:474)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
[jira] [Commented] (TEZ-2936) Create ATS implementation that enables support for YARN-4265
[ https://issues.apache.org/jira/browse/TEZ-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194298#comment-15194298 ] Siddharth Seth commented on TEZ-2936: - +1. The 2.8 profile would be built as follows, correct ? mvn clean install -DskipTest -Phadoop28 -Dhadoop.version=2.9.0-SNAPSHOT -DskipTests Should we rename the profile to 2.9 - it does not build against 2.8.0-SNAPSHOT. 2.9 has an issue with SystemClock incompatibility, but that should get addressed soon via YARN-4816. > Create ATS implementation that enables support for YARN-4265 > > > Key: TEZ-2936 > URL: https://issues.apache.org/jira/browse/TEZ-2936 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: TEZ-2936.2.patch, TEZ-2936.3.patch, TEZ-2936.4.patch, > TEZ-2936.5.patch, TEZ-2936.wip.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3165) Parallel initialization of inputs, outputs, and processor can cause NoSuchMethodException
Jonathan Eagles created TEZ-3165: Summary: Parallel initialization of inputs, outputs, and processor can cause NoSuchMethodException Key: TEZ-3165 URL: https://issues.apache.org/jira/browse/TEZ-3165 Project: Apache Tez Issue Type: Bug Reporter: Jonathan Eagles Assignee: Jonathan Eagles -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3164) Surface error histograms from the AM
[ https://issues.apache.org/jira/browse/TEZ-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194133#comment-15194133 ] Siddharth Seth commented on TEZ-3164: - Big +1 for doing this. An external script could be used for such diagnostics, but Tez, MR etc will likely already have a lot of this information from running jobs. > Surface error histograms from the AM > > > Key: TEZ-3164 > URL: https://issues.apache.org/jira/browse/TEZ-3164 > Project: Apache Tez > Issue Type: Improvement >Reporter: Bikas Saha > > Job tasks are constantly probing the cluster. So if there are some issues in > the cluster then jobs would be the first to notice that. If we can make these > observations surface to the user then we could quickly identify cluster > issues. > Lets say a set of bad machines got added to the cluster and tasks started > seeing shuffle errors from those machines. This can slow down or hang the > job. If the AM can surface increased errors counts from source and > destination machines then that could pin point the bad machines vs having to > arrive at those machines from first principles and log searching. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3164) Surface error histograms from the AM
Bikas Saha created TEZ-3164: --- Summary: Surface error histograms from the AM Key: TEZ-3164 URL: https://issues.apache.org/jira/browse/TEZ-3164 Project: Apache Tez Issue Type: Improvement Reporter: Bikas Saha Job tasks are constantly probing the cluster. So if there are some issues in the cluster then jobs would be the first to notice that. If we can make these observations surface to the user then we could quickly identify cluster issues. Lets say a set of bad machines got added to the cluster and tasks started seeing shuffle errors from those machines. This can slow down or hang the job. If the AM can surface increased errors counts from source and destination machines then that could pin point the bad machines vs having to arrive at those machines from first principles and log searching. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3148) Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
[
https://issues.apache.org/jira/browse/TEZ-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated TEZ-3148:
-
Affects Version/s: 0.8.2
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
>
>
> Key: TEZ-3148
> URL: https://issues.apache.org/jira/browse/TEZ-3148
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.0-alpha
>Reporter: Rajesh Balamohan
>Assignee: Tsuyoshi Ozawa
> Fix For: 0.8.3
>
> Attachments: TEZ-3148.001.patch, TEZ-3148.002.patch, am.log.gz,
> dag.dot
>
>
> Got the following when executing one of the DAG.
> Tez details:
> versionInfo=[ component=tez-dag, version=0.8.3-SNAPSHOT,
> revision=3e409ae0ee7233b4cf631cac1bc366679a08b7d1,
> SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
> buildTime=20160227-1912]
> {noformat}
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
> attempt_1455662455106_2317_27_02_000339_0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE
> on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
> Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main"
> java.lang.AbstractMethodError:
> org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
> at org.apache.tez.client.TezClient.stop(TezClient.java:562)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:474)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:436)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:338)
> at
> org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1469)
> at
> org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:719)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Additional note for later reference: Q51 in tpcds can possibly be used to
> reproduce this at 10 TB scale.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3148) Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
[
https://issues.apache.org/jira/browse/TEZ-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated TEZ-3148:
-
Affects Version/s: (was: 0.8.2)
0.8.0-alpha
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
>
>
> Key: TEZ-3148
> URL: https://issues.apache.org/jira/browse/TEZ-3148
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.0-alpha
>Reporter: Rajesh Balamohan
>Assignee: Tsuyoshi Ozawa
> Fix For: 0.8.3
>
> Attachments: TEZ-3148.001.patch, TEZ-3148.002.patch, am.log.gz,
> dag.dot
>
>
> Got the following when executing one of the DAG.
> Tez details:
> versionInfo=[ component=tez-dag, version=0.8.3-SNAPSHOT,
> revision=3e409ae0ee7233b4cf631cac1bc366679a08b7d1,
> SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
> buildTime=20160227-1912]
> {noformat}
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
> attempt_1455662455106_2317_27_02_000339_0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE
> on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
> Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main"
> java.lang.AbstractMethodError:
> org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
> at org.apache.tez.client.TezClient.stop(TezClient.java:562)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:474)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:436)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:338)
> at
> org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1469)
> at
> org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:719)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Additional note for later reference: Q51 in tpcds can possibly be used to
> reproduce this at 10 TB scale.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3148) Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
[
https://issues.apache.org/jira/browse/TEZ-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated TEZ-3148:
-
Fix Version/s: 0.8.3
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
>
>
> Key: TEZ-3148
> URL: https://issues.apache.org/jira/browse/TEZ-3148
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Tsuyoshi Ozawa
> Fix For: 0.8.3
>
> Attachments: TEZ-3148.001.patch, TEZ-3148.002.patch, am.log.gz,
> dag.dot
>
>
> Got the following when executing one of the DAG.
> Tez details:
> versionInfo=[ component=tez-dag, version=0.8.3-SNAPSHOT,
> revision=3e409ae0ee7233b4cf631cac1bc366679a08b7d1,
> SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
> buildTime=20160227-1912]
> {noformat}
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
> attempt_1455662455106_2317_27_02_000339_0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE
> on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
> Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main"
> java.lang.AbstractMethodError:
> org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
> at org.apache.tez.client.TezClient.stop(TezClient.java:562)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:474)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:436)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:338)
> at
> org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1469)
> at
> org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:719)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Additional note for later reference: Q51 in tpcds can possibly be used to
> reproduce this at 10 TB scale.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3148) Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
[
https://issues.apache.org/jira/browse/TEZ-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193862#comment-15193862
]
Zhiyuan Yang commented on TEZ-3148:
---
Thanks! Your patch looks good to me. I've tried out and can confirm the test
does test the fix.
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
>
>
> Key: TEZ-3148
> URL: https://issues.apache.org/jira/browse/TEZ-3148
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-3148.001.patch, TEZ-3148.002.patch, am.log.gz,
> dag.dot
>
>
> Got the following when executing one of the DAG.
> Tez details:
> versionInfo=[ component=tez-dag, version=0.8.3-SNAPSHOT,
> revision=3e409ae0ee7233b4cf631cac1bc366679a08b7d1,
> SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
> buildTime=20160227-1912]
> {noformat}
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
> attempt_1455662455106_2317_27_02_000339_0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE
> on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
> Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main"
> java.lang.AbstractMethodError:
> org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
> at org.apache.tez.client.TezClient.stop(TezClient.java:562)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:474)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:436)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:338)
> at
> org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1469)
> at
> org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:719)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Additional note for later reference: Q51 in tpcds can possibly be used to
> reproduce this at 10 TB scale.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193822#comment-15193822 ] Hitesh Shah commented on TEZ-3155: -- Thanks [~aplusplus]. Committed to master. > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Fix For: 0.8.3 > > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch, TEZ-3155.5.patch, TEZ-3155.6.patch, TEZ-3155.7.patch, > TEZ-3155.8.patch, TEZ-3155.9.patch, TEZ-3155.additional.2.patch, > TEZ-3155.additional.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3105) TezMxBeanResourceCalculator does not run on IBM JDK 7 or 8 causing Tez failures
[ https://issues.apache.org/jira/browse/TEZ-3105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-3105: - Summary: TezMxBeanResourceCalculator does not run on IBM JDK 7 or 8 causing Tez failures (was: Tez does not run on IBM JDK 7 or 8) > TezMxBeanResourceCalculator does not run on IBM JDK 7 or 8 causing Tez > failures > --- > > Key: TEZ-3105 > URL: https://issues.apache.org/jira/browse/TEZ-3105 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Greg Senia >Assignee: Greg Senia > Labels: ibm, ibm-jdk > Attachments: TEZ-3105.patch, TEZ-3105.patch > > > When testing Hive on Tez with IBM JDK 7 and 8. The following issue was > discovered: > 2016-02-08 22:25:22,869 [ERROR] [main] |app.DAGAppMaster|: Error starting > DAGAppMaster > java.lang.RuntimeException: java.lang.reflect.InvocationTargetException > at > org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(ResourceCalculatorProcessTree.java:225) > at > org.apache.tez.dag.app.DAGAppMaster.initResourceCalculatorPlugins(DAGAppMaster.java:347) > at > org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:371) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at org.apache.tez.dag.app.DAGAppMaster$6.run(DAGAppMaster.java:2274) > at > java.security.AccessController.doPrivileged(AccessController.java:686) > at javax.security.auth.Subject.doAs(Subject.java:569) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2271) > at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2086) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:88) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57) > at java.lang.reflect.Constructor.newInstance(Constructor.java:436) > at > org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(ResourceCalculatorProcessTree.java:221) > ... 9 more > Caused by: java.lang.ClassCastException: > com.ibm.lang.management.ExtendedOperatingSystem incompatible with > com.sun.management.OperatingSystemMXBean > at > org.apache.tez.util.TezMxBeanResourceCalculator.(TezMxBeanResourceCalculator.java:44) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3105) TezMxBeanResourceCalculator does not work on IBM JDK 7 or 8 causing Tez failures
[ https://issues.apache.org/jira/browse/TEZ-3105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-3105: - Summary: TezMxBeanResourceCalculator does not work on IBM JDK 7 or 8 causing Tez failures (was: TezMxBeanResourceCalculator does not run on IBM JDK 7 or 8 causing Tez failures) > TezMxBeanResourceCalculator does not work on IBM JDK 7 or 8 causing Tez > failures > > > Key: TEZ-3105 > URL: https://issues.apache.org/jira/browse/TEZ-3105 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Greg Senia >Assignee: Greg Senia > Labels: ibm, ibm-jdk > Attachments: TEZ-3105.patch, TEZ-3105.patch > > > When testing Hive on Tez with IBM JDK 7 and 8. The following issue was > discovered: > 2016-02-08 22:25:22,869 [ERROR] [main] |app.DAGAppMaster|: Error starting > DAGAppMaster > java.lang.RuntimeException: java.lang.reflect.InvocationTargetException > at > org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(ResourceCalculatorProcessTree.java:225) > at > org.apache.tez.dag.app.DAGAppMaster.initResourceCalculatorPlugins(DAGAppMaster.java:347) > at > org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:371) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at org.apache.tez.dag.app.DAGAppMaster$6.run(DAGAppMaster.java:2274) > at > java.security.AccessController.doPrivileged(AccessController.java:686) > at javax.security.auth.Subject.doAs(Subject.java:569) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2271) > at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2086) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:88) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57) > at java.lang.reflect.Constructor.newInstance(Constructor.java:436) > at > org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(ResourceCalculatorProcessTree.java:221) > ... 9 more > Caused by: java.lang.ClassCastException: > com.ibm.lang.management.ExtendedOperatingSystem incompatible with > com.sun.management.OperatingSystemMXBean > at > org.apache.tez.util.TezMxBeanResourceCalculator.(TezMxBeanResourceCalculator.java:44) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3105) Tez does not run on IBM JDK 7 or 8
[ https://issues.apache.org/jira/browse/TEZ-3105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193808#comment-15193808 ] Hitesh Shah commented on TEZ-3105: -- Thanks for the follow [~gss2002]. +1. Committing shortly. > Tez does not run on IBM JDK 7 or 8 > -- > > Key: TEZ-3105 > URL: https://issues.apache.org/jira/browse/TEZ-3105 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Greg Senia >Assignee: Greg Senia > Labels: ibm, ibm-jdk > Attachments: TEZ-3105.patch, TEZ-3105.patch > > > When testing Hive on Tez with IBM JDK 7 and 8. The following issue was > discovered: > 2016-02-08 22:25:22,869 [ERROR] [main] |app.DAGAppMaster|: Error starting > DAGAppMaster > java.lang.RuntimeException: java.lang.reflect.InvocationTargetException > at > org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(ResourceCalculatorProcessTree.java:225) > at > org.apache.tez.dag.app.DAGAppMaster.initResourceCalculatorPlugins(DAGAppMaster.java:347) > at > org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:371) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at org.apache.tez.dag.app.DAGAppMaster$6.run(DAGAppMaster.java:2274) > at > java.security.AccessController.doPrivileged(AccessController.java:686) > at javax.security.auth.Subject.doAs(Subject.java:569) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2271) > at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2086) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:88) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57) > at java.lang.reflect.Constructor.newInstance(Constructor.java:436) > at > org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(ResourceCalculatorProcessTree.java:221) > ... 9 more > Caused by: java.lang.ClassCastException: > com.ibm.lang.management.ExtendedOperatingSystem incompatible with > com.sun.management.OperatingSystemMXBean > at > org.apache.tez.util.TezMxBeanResourceCalculator.(TezMxBeanResourceCalculator.java:44) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193786#comment-15193786 ] Zhiyuan Yang commented on TEZ-3155: --- Additional patch updated. Make log line more user-friendly. > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Fix For: 0.8.3 > > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch, TEZ-3155.5.patch, TEZ-3155.6.patch, TEZ-3155.7.patch, > TEZ-3155.8.patch, TEZ-3155.9.patch, TEZ-3155.additional.2.patch, > TEZ-3155.additional.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3160) Tez UI 2: Swimlane - Create swimlane page & component
[ https://issues.apache.org/jira/browse/TEZ-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193758#comment-15193758 ] Hitesh Shah commented on TEZ-3160: -- +1 > Tez UI 2: Swimlane - Create swimlane page & component > - > > Key: TEZ-3160 > URL: https://issues.apache.org/jira/browse/TEZ-3160 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3160.1.patch > > > - Create swimlane page > - Load the basic vertex data with events available from ATS > - Create swimlane component & display available data in the swimlane page > -- Draw swimlane for each vertex with the default events > -- Show event lines & event bubbles > - Implement fullscreen, like in graphical view > - Show vertex names to the left, with ellipsis if required -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-3155: -- Attachment: TEZ-3155.additional.patch add log line > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Fix For: 0.8.3 > > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch, TEZ-3155.5.patch, TEZ-3155.6.patch, TEZ-3155.7.patch, > TEZ-3155.8.patch, TEZ-3155.9.patch, TEZ-3155.additional.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193661#comment-15193661 ] Zhiyuan Yang edited comment on TEZ-3155 at 3/14/16 5:15 PM: Already added, please check out additional patch. was (Author: aplusplus): add log line > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Fix For: 0.8.3 > > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch, TEZ-3155.5.patch, TEZ-3155.6.patch, TEZ-3155.7.patch, > TEZ-3155.8.patch, TEZ-3155.9.patch, TEZ-3155.additional.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3105) Tez does not run on IBM JDK 7 or 8
[
https://issues.apache.org/jira/browse/TEZ-3105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193330#comment-15193330
]
TezQA commented on TEZ-3105:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12793304/TEZ-3105.patch
against master revision dbd763f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1562//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1562//console
This message is automatically generated.
> Tez does not run on IBM JDK 7 or 8
> --
>
> Key: TEZ-3105
> URL: https://issues.apache.org/jira/browse/TEZ-3105
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Greg Senia
>Assignee: Greg Senia
> Labels: ibm, ibm-jdk
> Attachments: TEZ-3105.patch, TEZ-3105.patch
>
>
> When testing Hive on Tez with IBM JDK 7 and 8. The following issue was
> discovered:
> 2016-02-08 22:25:22,869 [ERROR] [main] |app.DAGAppMaster|: Error starting
> DAGAppMaster
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(ResourceCalculatorProcessTree.java:225)
> at
> org.apache.tez.dag.app.DAGAppMaster.initResourceCalculatorPlugins(DAGAppMaster.java:347)
> at
> org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:371)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at org.apache.tez.dag.app.DAGAppMaster$6.run(DAGAppMaster.java:2274)
> at
> java.security.AccessController.doPrivileged(AccessController.java:686)
> at javax.security.auth.Subject.doAs(Subject.java:569)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2271)
> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2086)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:88)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:436)
> at
> org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(ResourceCalculatorProcessTree.java:221)
> ... 9 more
> Caused by: java.lang.ClassCastException:
> com.ibm.lang.management.ExtendedOperatingSystem incompatible with
> com.sun.management.OperatingSystemMXBean
> at
> org.apache.tez.util.TezMxBeanResourceCalculator.(TezMxBeanResourceCalculator.java:44)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-3105 PreCommit Build #1562
Jira: https://issues.apache.org/jira/browse/TEZ-3105
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1562/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4568 lines...]
[INFO] Total time: 54:02 min
[INFO] Finished at: 2016-03-14T14:06:01+00:00
[INFO] Final Memory: 78M/1060M
[INFO]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12793304/TEZ-3105.patch
against master revision dbd763f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1562//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1562//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
7a642833f71ca82c124e6a725117eaea7f68c43f logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Compressed 3.38 MB of artifacts by 25.9% relative to #1556
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3105) Tez does not run on IBM JDK 7 or 8
[ https://issues.apache.org/jira/browse/TEZ-3105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193207#comment-15193207 ] Greg Senia commented on TEZ-3105: - Looks like this: -1 core tests. The patch failed these unit tests in : org.apache.tez.test.TestFaultTolerance was not introduced by the patch for this jira > Tez does not run on IBM JDK 7 or 8 > -- > > Key: TEZ-3105 > URL: https://issues.apache.org/jira/browse/TEZ-3105 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Greg Senia >Assignee: Greg Senia > Labels: ibm, ibm-jdk > Attachments: TEZ-3105.patch, TEZ-3105.patch > > > When testing Hive on Tez with IBM JDK 7 and 8. The following issue was > discovered: > 2016-02-08 22:25:22,869 [ERROR] [main] |app.DAGAppMaster|: Error starting > DAGAppMaster > java.lang.RuntimeException: java.lang.reflect.InvocationTargetException > at > org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(ResourceCalculatorProcessTree.java:225) > at > org.apache.tez.dag.app.DAGAppMaster.initResourceCalculatorPlugins(DAGAppMaster.java:347) > at > org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:371) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at org.apache.tez.dag.app.DAGAppMaster$6.run(DAGAppMaster.java:2274) > at > java.security.AccessController.doPrivileged(AccessController.java:686) > at javax.security.auth.Subject.doAs(Subject.java:569) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2271) > at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2086) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:88) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57) > at java.lang.reflect.Constructor.newInstance(Constructor.java:436) > at > org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(ResourceCalculatorProcessTree.java:221) > ... 9 more > Caused by: java.lang.ClassCastException: > com.ibm.lang.management.ExtendedOperatingSystem incompatible with > com.sun.management.OperatingSystemMXBean > at > org.apache.tez.util.TezMxBeanResourceCalculator.(TezMxBeanResourceCalculator.java:44) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3105) Tez does not run on IBM JDK 7 or 8
[ https://issues.apache.org/jira/browse/TEZ-3105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Senia updated TEZ-3105: Attachment: TEZ-3105.patch > Tez does not run on IBM JDK 7 or 8 > -- > > Key: TEZ-3105 > URL: https://issues.apache.org/jira/browse/TEZ-3105 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Greg Senia >Assignee: Greg Senia > Labels: ibm, ibm-jdk > Attachments: TEZ-3105.patch, TEZ-3105.patch > > > When testing Hive on Tez with IBM JDK 7 and 8. The following issue was > discovered: > 2016-02-08 22:25:22,869 [ERROR] [main] |app.DAGAppMaster|: Error starting > DAGAppMaster > java.lang.RuntimeException: java.lang.reflect.InvocationTargetException > at > org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(ResourceCalculatorProcessTree.java:225) > at > org.apache.tez.dag.app.DAGAppMaster.initResourceCalculatorPlugins(DAGAppMaster.java:347) > at > org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:371) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at org.apache.tez.dag.app.DAGAppMaster$6.run(DAGAppMaster.java:2274) > at > java.security.AccessController.doPrivileged(AccessController.java:686) > at javax.security.auth.Subject.doAs(Subject.java:569) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2271) > at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2086) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:88) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57) > at java.lang.reflect.Constructor.newInstance(Constructor.java:436) > at > org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(ResourceCalculatorProcessTree.java:221) > ... 9 more > Caused by: java.lang.ClassCastException: > com.ibm.lang.management.ExtendedOperatingSystem incompatible with > com.sun.management.OperatingSystemMXBean > at > org.apache.tez.util.TezMxBeanResourceCalculator.(TezMxBeanResourceCalculator.java:44) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
