[jira] [Commented] (TEZ-3152) Tez UI 2: Build fails when run by multiple users or when node_modules is old
[ https://issues.apache.org/jira/browse/TEZ-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209657#comment-15209657 ] Hitesh Shah commented on TEZ-3152: -- +1 > Tez UI 2: Build fails when run by multiple users or when node_modules is old > > > Key: TEZ-3152 > URL: https://issues.apache.org/jira/browse/TEZ-3152 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3152.1.patch, TEZ-3152.2.patch, TEZ-3152.3.patch, > TEZ-3152.4.patch, TEZ-3152.5.patch, TEZ-3152.wip.1.patch, err2 > > > We have two separate cases in which the build fails: > #1 The build uses Nodejs v0.12.2. When the webapp folder already have a > node_modules folder installed by an old version of node, build fails. > - To repro, just manually run npm install in webapp with an old node (Like > v0.10.36). Then run mvn package from tez. > #2 async-disk-cache package creates files in tmpDir (/tmp). When run from a > different user, because of user permission on there files, the build fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3171) Tez UI: Swimlane - Tooltip, zoom & redirection
[ https://issues.apache.org/jira/browse/TEZ-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209509#comment-15209509 ] Hitesh Shah commented on TEZ-3171: -- +1 pending pre-commit > Tez UI: Swimlane - Tooltip, zoom & redirection > -- > > Key: TEZ-3171 > URL: https://issues.apache.org/jira/browse/TEZ-3171 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3171.1.patch > > > - Implement tooltip > -- Create a tooltip component with dynamic (Update at realtime) data display > --- Create tooltip definition class - in line with tabel definition to use > column definition classes > --- Create tooltip component - Must display title, text & a list of values > --- Attach tooltip to entities displayed > --- When multiple event bubbles come together, the tooltip must display all > of them > - Ensure the data displayed can be changed with the column selector - Ensure > the selector window have a meaningful title > - Implement horizontal zoom & scroll -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3172) Tez UI: Swimlane - In progress & Shadow
[ https://issues.apache.org/jira/browse/TEZ-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-3172: - Priority: Major (was: Blocker) > Tez UI: Swimlane - In progress & Shadow > --- > > Key: TEZ-3172 > URL: https://issues.apache.org/jira/browse/TEZ-3172 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > > - Ensure events are getting updated at realtime for runing DAGs > - Add running animated bubbe for running vertices, display the progress > percentage > - Add shadow line -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3152) Tez UI 2: Build fails when run by multiple users or when node_modules is old
[ https://issues.apache.org/jira/browse/TEZ-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3152: Attachment: TEZ-3152.5.patch Have moved into pluginManagement. [~hitesh] Please look into the patch. > Tez UI 2: Build fails when run by multiple users or when node_modules is old > > > Key: TEZ-3152 > URL: https://issues.apache.org/jira/browse/TEZ-3152 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3152.1.patch, TEZ-3152.2.patch, TEZ-3152.3.patch, > TEZ-3152.4.patch, TEZ-3152.5.patch, TEZ-3152.wip.1.patch, err2 > > > We have two separate cases in which the build fails: > #1 The build uses Nodejs v0.12.2. When the webapp folder already have a > node_modules folder installed by an old version of node, build fails. > - To repro, just manually run npm install in webapp with an old node (Like > v0.10.36). Then run mvn package from tez. > #2 async-disk-cache package creates files in tmpDir (/tmp). When run from a > different user, because of user permission on there files, the build fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-3172) Tez UI: Swimlane - In progress & Shadow
[ https://issues.apache.org/jira/browse/TEZ-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram reassigned TEZ-3172: --- Assignee: Sreenath Somarajapuram > Tez UI: Swimlane - In progress & Shadow > --- > > Key: TEZ-3172 > URL: https://issues.apache.org/jira/browse/TEZ-3172 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram >Priority: Blocker > > - Ensure events are getting updated at realtime for runing DAGs > - Add running animated bubbe for running vertices, display the progress > percentage > - Add shadow line -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3172) Tez UI: Swimlane - In progress & Shadow
[ https://issues.apache.org/jira/browse/TEZ-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-3172: Target Version/s: 0.8.3 Priority: Blocker (was: Major) > Tez UI: Swimlane - In progress & Shadow > --- > > Key: TEZ-3172 > URL: https://issues.apache.org/jira/browse/TEZ-3172 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Priority: Blocker > > - Ensure events are getting updated at realtime for runing DAGs > - Add running animated bubbe for running vertices, display the progress > percentage > - Add shadow line -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3173) Update Tez AM REST APIs for more information for each vertex
[
https://issues.apache.org/jira/browse/TEZ-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209433#comment-15209433
]
TezQA commented on TEZ-3173:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12795061/TEZ-3173.3.patch
against master revision 19280cf.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 34 javac
compiler warnings (more than the master's current 33 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1587//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1587//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1587//console
This message is automatically generated.
> Update Tez AM REST APIs for more information for each vertex
>
>
> Key: TEZ-3173
> URL: https://issues.apache.org/jira/browse/TEZ-3173
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Sreenath Somarajapuram
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3173.1.patch, TEZ-3173.2.patch, TEZ-3173.3.patch
>
>
> `/ui/ws/v2/tez/verticesInfo` endpoint in WebUIService must be upgraded to
> send back the following fields
> # initTime
> # startTime
> # finishTime
> # firstTaskStartTime
> # lastTaskFinishTime
> in addition to the existing id, status, progress, totalTasks,
> succeededTasks, runningTasks, pendingTasks, failedTaskAttempts &
> killedTaskAttempts fields.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-3173 PreCommit Build #1587
Jira: https://issues.apache.org/jira/browse/TEZ-3173
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1587/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4454 lines...]
[ERROR] [Help 2]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-tests
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12795061/TEZ-3173.3.patch
against master revision 19280cf.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 34 javac
compiler warnings (more than the master's current 33 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1587//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1587//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1587//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
c2349425bc6e7f3e14a202c0bbcba4de0af423c4 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
7 tests failed.
FAILED: org.apache.tez.test.TestFaultTolerance.testRandomFailingInputs
Error Message:
expected: but was:
Stack Trace:
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:141)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:124)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:120)
at
org.apache.tez.test.TestFaultTolerance.testRandomFailingInputs(TestFaultTolerance.java:763)
FAILED: org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit
Error Message:
TezSession has already shutdown. No cluster diagnostics found.
Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:849)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:129)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:124)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:120)
at
org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit(TestFaultTolerance.java:261)
FAILED:
org.apache.tez.test.TestFaultTolerance.testInputFailureRerunCanSendOutputToTwoDownstreamVertices
Error Message:
TezSession has already shutdown. No cluster diagnostics found.
Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:849)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:129)
[jira] [Commented] (TEZ-3108) Add support for external services to local mode
[ https://issues.apache.org/jira/browse/TEZ-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209342#comment-15209342 ] Siddharth Seth commented on TEZ-3108: - bq. External services and local mode seems a bit confusing. If local mode is meant to signify that everything is running locally, then external services should not exist. If we are saying that only "yarn" containers are running inline in local mode then local mode should be renamed to something else? Any suggestion on this. I'm looking at local mode as 1) no AM launched, 2) with service specified as 'container' - tasks will run inline. bq. Havent reviewed patch fully - noticed that "Preconditions.checkState(foundUberServiceName);" should be outside the for loop? Yes. This should be outside. Will move it out in the next patch, along with other forthcoming review comments. > Add support for external services to local mode > --- > > Key: TEZ-3108 > URL: https://issues.apache.org/jira/browse/TEZ-3108 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-3108.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3108) Add support for external services to local mode
[ https://issues.apache.org/jira/browse/TEZ-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209242#comment-15209242 ] Hitesh Shah commented on TEZ-3108: -- Havent reviewed patch fully - noticed that "Preconditions.checkState(foundUberServiceName);" should be outside the for loop? > Add support for external services to local mode > --- > > Key: TEZ-3108 > URL: https://issues.apache.org/jira/browse/TEZ-3108 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-3108.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3108) Add support for external services to local mode
[ https://issues.apache.org/jira/browse/TEZ-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209240#comment-15209240 ] Hitesh Shah commented on TEZ-3108: -- External services and local mode seems a bit confusing. If local mode is meant to signify that everything is running locally, then external services should not exist. If we are saying that only "yarn" containers are running inline in local mode then local mode should be renamed to something else? > Add support for external services to local mode > --- > > Key: TEZ-3108 > URL: https://issues.apache.org/jira/browse/TEZ-3108 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-3108.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3173) Update Tez AM REST APIs for more information for each vertex
[ https://issues.apache.org/jira/browse/TEZ-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209221#comment-15209221 ] Hitesh Shah commented on TEZ-3173: -- Comments for patch 3 - mostly looks good. Minor nit: Typo in "// TODO this finishTime is not consent to ATS logged time, will fix it later" - s/consent/consistent/ - also if this is not meant to be addressed in this jira, it is better to file a new jira for this and also modify the todo comment with the actual jira id > Update Tez AM REST APIs for more information for each vertex > > > Key: TEZ-3173 > URL: https://issues.apache.org/jira/browse/TEZ-3173 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Zhiyuan Yang > Attachments: TEZ-3173.1.patch, TEZ-3173.2.patch, TEZ-3173.3.patch > > > `/ui/ws/v2/tez/verticesInfo` endpoint in WebUIService must be upgraded to > send back the following fields > # initTime > # startTime > # finishTime > # firstTaskStartTime > # lastTaskFinishTime > in addition to the existing id, status, progress, totalTasks, > succeededTasks, runningTasks, pendingTasks, failedTaskAttempts & > killedTaskAttempts fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3161) Allow task to report different kinds of errors - fatal / kill
[ https://issues.apache.org/jira/browse/TEZ-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-3161: Attachment: TEZ-3161.3.txt Updated to fix the unit test. Can't do much about the deprecation. > Allow task to report different kinds of errors - fatal / kill > - > > Key: TEZ-3161 > URL: https://issues.apache.org/jira/browse/TEZ-3161 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-3161.1.txt, TEZ-3161.2.txt, TEZ-3161.3.txt > > > In some cases, task failures will be the same across all attempts - e.g. > exceeding memory utilization on an operation. In this case, there's no point > in running another attempt of the same task. > There's other cases where a task may want to mark itself as KILLED - i.e. a > temporary error. An example of this is pipelined shuffle. > Tez should allow both operations. > cc [~vikram.dixit], [~rajesh.balamohan] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3165) Parallel initialization of inputs, outputs, and processor can cause NoSuchMethodException
[
https://issues.apache.org/jira/browse/TEZ-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209205#comment-15209205
]
Siddharth Seth commented on TEZ-3165:
-
Also minor - please make the new boolean fields final before commit, if that's
possible.
> Parallel initialization of inputs, outputs, and processor can cause
> NoSuchMethodException
> -
>
> Key: TEZ-3165
> URL: https://issues.apache.org/jira/browse/TEZ-3165
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3165.1.patch, TEZ-3165.2.patch, TEZ-3165.3.patch
>
>
> 2016-03-13 23:55:17,162 [INFO] [main]
> |runtime.LogicalIOProcessorRuntimeTask|: Initializing
> LogicalIOProcessorRuntimeTask with TaskSpec: DAGName :
> PigLatin:Script.pig-0_scope-0, VertexName: scope-203, VertexParallelism:
> 2707, TaskAttemptID:attempt_1,
> processorName=org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor,
> inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{
> sourceVertexName=scope-0, physicalEdgeCount=1,
> inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ],
> outputSpecList=[{{ destinationVertexName=scope-28, physicalEdgeCount=0,
> outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
> 2016-03-13 23:55:17,164 [INFO] [main] |resources.MemoryDistributor|:
> InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1,
> numOutputs=1, JVM.maxFree=1059061760,
> allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
> 2016-03-13 23:55:17,175 [INFO] [TezChild] |task.TezTaskRunner|: Initializing
> task, taskAttemptId=attempt_1
> 2016-03-13 23:55:17,182 [INFO] [TaskHeartbeatThread] |task.TaskReporter|:
> Routing events from heartbeat response to task,
> currentTaskAttemptId=attempt_1, eventCount=1 fromEventId=0 nextFromEventId=0
> 2016-03-13 23:55:17,212 [INFO] [I/O Setup 1 Initialize: {scope-28}]
> |Configuration.deprecation|: mapreduce.inputformat.class is deprecated.
> Instead, use mapreduce.job.inputformat.class
> 2016-03-13 23:55:17,214 [INFO] [I/O Setup 1 Initialize: {scope-28}]
> |Configuration.deprecation|: fs.default.name is deprecated. Instead, use
> fs.defaultFS
> 2016-03-13 23:55:17,223 [INFO] [I/O Setup 1 Initialize: {scope-28}]
> |counters.Limits|: Counter limits initialized with parameters:
> GROUP_NAME_MAX=256, MAX_GROUPS=1000, COUNTER_NAME_MAX=128, MAX_COUNTERS=5000
> 2016-03-13 23:55:17,228 [INFO] [I/O Setup 0 Initialize: {scope-0}]
> |input.MRInput|: scope-0 using newmapreduce API=true, split via event=true,
> numPhysicalInputs=1
> 2016-03-13 23:55:17,233 [INFO] [I/O Setup 0 Initialize: {scope-0}]
> |input.MRInput|: Initialized MRInput: scope-0
> 2016-03-13 23:55:17,345 [INFO] [TezChild] |data.SchemaTupleBackend|: Key
> [pig.schematuple] was not set... will not generate code.
> 2016-03-13 23:55:17,400 [INFO] [TezChild]
> |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor
> 2016-03-13 23:55:17,400 [INFO] [TezChild]
> |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish
> 2016-03-13 23:55:17,400 [INFO] [TezChild]
> |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish
> 2016-03-13 23:55:17,400 [INFO] [TezChild] |task.TezTaskRunner|: Encounted an
> error while executing task: attempt_1
> java.lang.RuntimeException: could not instantiate
> 'com.twitter.elephantbird.pig.store.SequenceFileStorage' with arguments '[-c
> com.twitter.elephantbird.pig.util.TextConverter, -c
> com.twitter.elephantbird.pig.util.TextConverter]'
> at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:766)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:250)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:76)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigOutputFormatTez.getRecordWriter(PigOutputFormatTez.java:43)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:399)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable._callInternal(LogicalIOProcessorRuntimeTask.java:506)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:489)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:474)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
[jira] [Updated] (TEZ-3165) Allow Inputs/Outputs to be initialized serially, control processor initialization relative to Inputs/Outputs
[
https://issues.apache.org/jira/browse/TEZ-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated TEZ-3165:
Summary: Allow Inputs/Outputs to be initialized serially, control processor
initialization relative to Inputs/Outputs (was: Parallel initialization of
inputs, outputs, and processor can cause NoSuchMethodException)
> Allow Inputs/Outputs to be initialized serially, control processor
> initialization relative to Inputs/Outputs
>
>
> Key: TEZ-3165
> URL: https://issues.apache.org/jira/browse/TEZ-3165
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3165.1.patch, TEZ-3165.2.patch, TEZ-3165.3.patch
>
>
> 2016-03-13 23:55:17,162 [INFO] [main]
> |runtime.LogicalIOProcessorRuntimeTask|: Initializing
> LogicalIOProcessorRuntimeTask with TaskSpec: DAGName :
> PigLatin:Script.pig-0_scope-0, VertexName: scope-203, VertexParallelism:
> 2707, TaskAttemptID:attempt_1,
> processorName=org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor,
> inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{
> sourceVertexName=scope-0, physicalEdgeCount=1,
> inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ],
> outputSpecList=[{{ destinationVertexName=scope-28, physicalEdgeCount=0,
> outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
> 2016-03-13 23:55:17,164 [INFO] [main] |resources.MemoryDistributor|:
> InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1,
> numOutputs=1, JVM.maxFree=1059061760,
> allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
> 2016-03-13 23:55:17,175 [INFO] [TezChild] |task.TezTaskRunner|: Initializing
> task, taskAttemptId=attempt_1
> 2016-03-13 23:55:17,182 [INFO] [TaskHeartbeatThread] |task.TaskReporter|:
> Routing events from heartbeat response to task,
> currentTaskAttemptId=attempt_1, eventCount=1 fromEventId=0 nextFromEventId=0
> 2016-03-13 23:55:17,212 [INFO] [I/O Setup 1 Initialize: {scope-28}]
> |Configuration.deprecation|: mapreduce.inputformat.class is deprecated.
> Instead, use mapreduce.job.inputformat.class
> 2016-03-13 23:55:17,214 [INFO] [I/O Setup 1 Initialize: {scope-28}]
> |Configuration.deprecation|: fs.default.name is deprecated. Instead, use
> fs.defaultFS
> 2016-03-13 23:55:17,223 [INFO] [I/O Setup 1 Initialize: {scope-28}]
> |counters.Limits|: Counter limits initialized with parameters:
> GROUP_NAME_MAX=256, MAX_GROUPS=1000, COUNTER_NAME_MAX=128, MAX_COUNTERS=5000
> 2016-03-13 23:55:17,228 [INFO] [I/O Setup 0 Initialize: {scope-0}]
> |input.MRInput|: scope-0 using newmapreduce API=true, split via event=true,
> numPhysicalInputs=1
> 2016-03-13 23:55:17,233 [INFO] [I/O Setup 0 Initialize: {scope-0}]
> |input.MRInput|: Initialized MRInput: scope-0
> 2016-03-13 23:55:17,345 [INFO] [TezChild] |data.SchemaTupleBackend|: Key
> [pig.schematuple] was not set... will not generate code.
> 2016-03-13 23:55:17,400 [INFO] [TezChild]
> |runtime.LogicalIOProcessorRuntimeTask|: Initialized processor
> 2016-03-13 23:55:17,400 [INFO] [TezChild]
> |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 2 initializers to finish
> 2016-03-13 23:55:17,400 [INFO] [TezChild]
> |runtime.LogicalIOProcessorRuntimeTask|: Waiting for 1 initializers to finish
> 2016-03-13 23:55:17,400 [INFO] [TezChild] |task.TezTaskRunner|: Encounted an
> error while executing task: attempt_1
> java.lang.RuntimeException: could not instantiate
> 'com.twitter.elephantbird.pig.store.SequenceFileStorage' with arguments '[-c
> com.twitter.elephantbird.pig.util.TextConverter, -c
> com.twitter.elephantbird.pig.util.TextConverter]'
> at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:766)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:250)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:76)
> at
> org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigOutputFormatTez.getRecordWriter(PigOutputFormatTez.java:43)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:399)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable._callInternal(LogicalIOProcessorRuntimeTask.java:506)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:489)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:474)
> at
[jira] [Commented] (TEZ-3182) linux superuser use maven compile bower always fail
[ https://issues.apache.org/jira/browse/TEZ-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209185#comment-15209185 ] Hitesh Shah commented on TEZ-3182: -- Updated target versions so that this is backported to both branch 0.7 and master. > linux superuser use maven compile bower always fail > --- > > Key: TEZ-3182 > URL: https://issues.apache.org/jira/browse/TEZ-3182 > Project: Apache Tez > Issue Type: Bug > Components: UI >Affects Versions: 0.6.2, 0.8.2 > Environment: linux rh6 >Reporter: shenxianqiang >Assignee: shenxianqiang >Priority: Trivial > Attachments: TEZ-3182.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > When I am root. Using 'mvn clean package -DskipTests=true' command always > fail. > [INFO] --- exec-maven-plugin:1.3.2:exec (Bower install) @ tez-ui --- > bower ESUDO Cannot be run with sudo > Additional error details: > Since bower is a user command, there is no need to execute it with superuser > permissions. > If you're having permission errors when using bower without sudo, please > spend a few minutes learning more about how your system should work and make > any necessary repairs. > http://www.joyent.com/blog/installing-node-and-npm > https://gist.github.com/isaacs/579814 > You can however run a command with sudo using --allow-root option > I have to modify pom.xml.Why not modify pom.xml in future? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3182) linux superuser use maven compile bower always fail
[ https://issues.apache.org/jira/browse/TEZ-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-3182: - Assignee: shenxianqiang > linux superuser use maven compile bower always fail > --- > > Key: TEZ-3182 > URL: https://issues.apache.org/jira/browse/TEZ-3182 > Project: Apache Tez > Issue Type: Bug > Components: UI >Affects Versions: 0.6.2, 0.8.2 > Environment: linux rh6 >Reporter: shenxianqiang >Assignee: shenxianqiang >Priority: Trivial > Attachments: TEZ-3182.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > When I am root. Using 'mvn clean package -DskipTests=true' command always > fail. > [INFO] --- exec-maven-plugin:1.3.2:exec (Bower install) @ tez-ui --- > bower ESUDO Cannot be run with sudo > Additional error details: > Since bower is a user command, there is no need to execute it with superuser > permissions. > If you're having permission errors when using bower without sudo, please > spend a few minutes learning more about how your system should work and make > any necessary repairs. > http://www.joyent.com/blog/installing-node-and-npm > https://gist.github.com/isaacs/579814 > You can however run a command with sudo using --allow-root option > I have to modify pom.xml.Why not modify pom.xml in future? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3182) linux superuser use maven compile bower always fail
[ https://issues.apache.org/jira/browse/TEZ-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-3182: - Target Version/s: 0.7.1, 0.8.3 (was: 0.8.3) > linux superuser use maven compile bower always fail > --- > > Key: TEZ-3182 > URL: https://issues.apache.org/jira/browse/TEZ-3182 > Project: Apache Tez > Issue Type: Bug > Components: UI >Affects Versions: 0.6.2, 0.8.2 > Environment: linux rh6 >Reporter: shenxianqiang >Assignee: shenxianqiang >Priority: Trivial > Attachments: TEZ-3182.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > When I am root. Using 'mvn clean package -DskipTests=true' command always > fail. > [INFO] --- exec-maven-plugin:1.3.2:exec (Bower install) @ tez-ui --- > bower ESUDO Cannot be run with sudo > Additional error details: > Since bower is a user command, there is no need to execute it with superuser > permissions. > If you're having permission errors when using bower without sudo, please > spend a few minutes learning more about how your system should work and make > any necessary repairs. > http://www.joyent.com/blog/installing-node-and-npm > https://gist.github.com/isaacs/579814 > You can however run a command with sudo using --allow-root option > I have to modify pom.xml.Why not modify pom.xml in future? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3182) linux superuser use maven compile bower always fail
[ https://issues.apache.org/jira/browse/TEZ-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209183#comment-15209183 ] Hitesh Shah commented on TEZ-3182: -- Thanks for the patch [~shenxianqiang]. Lets wait for [~Sreenath] to review and commit if the fix looks good to him > linux superuser use maven compile bower always fail > --- > > Key: TEZ-3182 > URL: https://issues.apache.org/jira/browse/TEZ-3182 > Project: Apache Tez > Issue Type: Bug > Components: UI >Affects Versions: 0.6.2, 0.8.2 > Environment: linux rh6 >Reporter: shenxianqiang >Assignee: shenxianqiang >Priority: Trivial > Attachments: TEZ-3182.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > When I am root. Using 'mvn clean package -DskipTests=true' command always > fail. > [INFO] --- exec-maven-plugin:1.3.2:exec (Bower install) @ tez-ui --- > bower ESUDO Cannot be run with sudo > Additional error details: > Since bower is a user command, there is no need to execute it with superuser > permissions. > If you're having permission errors when using bower without sudo, please > spend a few minutes learning more about how your system should work and make > any necessary repairs. > http://www.joyent.com/blog/installing-node-and-npm > https://gist.github.com/isaacs/579814 > You can however run a command with sudo using --allow-root option > I have to modify pom.xml.Why not modify pom.xml in future? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3173) Update Tez AM REST APIs for more information for each vertex
[ https://issues.apache.org/jira/browse/TEZ-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-3173: -- Attachment: TEZ-3173.3.patch > Update Tez AM REST APIs for more information for each vertex > > > Key: TEZ-3173 > URL: https://issues.apache.org/jira/browse/TEZ-3173 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Zhiyuan Yang > Attachments: TEZ-3173.1.patch, TEZ-3173.2.patch, TEZ-3173.3.patch > > > `/ui/ws/v2/tez/verticesInfo` endpoint in WebUIService must be upgraded to > send back the following fields > # initTime > # startTime > # finishTime > # firstTaskStartTime > # lastTaskFinishTime > in addition to the existing id, status, progress, totalTasks, > succeededTasks, runningTasks, pendingTasks, failedTaskAttempts & > killedTaskAttempts fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-3161 PreCommit Build #1586
Jira: https://issues.apache.org/jira/browse/TEZ-3161
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1586/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4482 lines...]
[ERROR] [Help 2]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-runtime-library
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12795036/TEZ-3161.2.txt
against master revision 19280cf.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 35 javac
compiler warnings (more than the master's current 33 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestShuffle
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1586//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1586//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1586//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
ff03da8e80482958ebeb9904431974e69992f9bd logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
1 tests failed.
FAILED:
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestShuffle.testSchedulerTerminatesOnException
Error Message:
Invalid use of argument matchers!
3 matchers expected, 2 recorded:
-> at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestShuffle.testSchedulerTerminatesOnException(TestShuffle.java:80)
-> at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestShuffle.testSchedulerTerminatesOnException(TestShuffle.java:80)
This exception may occur if matchers are combined with raw values:
//incorrect:
someMethod(anyObject(), "raw String");
When using matchers, all arguments have to be provided by matchers.
For example:
//correct:
someMethod(anyObject(), eq("String by matcher"));
For more info see javadoc for Matchers class.
Stack Trace:
org.mockito.exceptions.misusing.InvalidUseOfMatchersException:
Invalid use of argument matchers!
3 matchers expected, 2 recorded:
-> at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestShuffle.testSchedulerTerminatesOnException(TestShuffle.java:80)
-> at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestShuffle.testSchedulerTerminatesOnException(TestShuffle.java:80)
This exception may occur if matchers are combined with raw values:
//incorrect:
someMethod(anyObject(), "raw String");
When using matchers, all arguments have to be provided by matchers.
For example:
//correct:
someMethod(anyObject(), eq("String by matcher"));
For more info see javadoc for Matchers class.
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestShuffle.testSchedulerTerminatesOnException(TestShuffle.java:80)
[jira] [Commented] (TEZ-3161) Allow task to report different kinds of errors - fatal / kill
[
https://issues.apache.org/jira/browse/TEZ-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209029#comment-15209029
]
TezQA commented on TEZ-3161:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12795036/TEZ-3161.2.txt
against master revision 19280cf.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 35 javac
compiler warnings (more than the master's current 33 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestShuffle
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1586//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1586//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1586//console
This message is automatically generated.
> Allow task to report different kinds of errors - fatal / kill
> -
>
> Key: TEZ-3161
> URL: https://issues.apache.org/jira/browse/TEZ-3161
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-3161.1.txt, TEZ-3161.2.txt
>
>
> In some cases, task failures will be the same across all attempts - e.g.
> exceeding memory utilization on an operation. In this case, there's no point
> in running another attempt of the same task.
> There's other cases where a task may want to mark itself as KILLED - i.e. a
> temporary error. An example of this is pipelined shuffle.
> Tez should allow both operations.
> cc [~vikram.dixit], [~rajesh.balamohan]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2442) Support DFS based shuffle in addition to HTTP shuffle
[ https://issues.apache.org/jira/browse/TEZ-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208898#comment-15208898 ] Hitesh Shah commented on TEZ-2442: -- [~rkannan82] Are you still looking to contribute/implement this feature? It seems like [~shanyu] has also started taking a look at this. Would you mind if we re-assign this jira to him if you are not planning to look at this in the near future? > Support DFS based shuffle in addition to HTTP shuffle > - > > Key: TEZ-2442 > URL: https://issues.apache.org/jira/browse/TEZ-2442 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.3 >Reporter: Kannan Rajah >Assignee: Kannan Rajah > Attachments: HDFS_based_shuffle_v2.pdf, Tez Shuffle using DFS.pdf, > hdfs_broadcast_hack.txt, tez_hdfs_shuffle.patch > > > In Tez, Shuffle is a mechanism by which intermediate data can be shared > between stages. Shuffle data is written to local disk and fetched from any > remote node using HTTP. A DFS like MapR file system can support writing this > shuffle data directly to its DFS using a notion of local volumes and retrieve > it using HDFS API from remote node. The current Shuffle implementation > assumes local data can only be managed by LocalFileSystem. So it uses > RawLocalFileSystem and LocalDirAllocator. If we can remove this assumption > and introduce an abstraction to manage local disks, then we can reuse most of > the shuffle logic (store, sort) and inject a HDFS API based retrieval instead > of HTTP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2442) Support DFS based shuffle in addition to HTTP shuffle
[ https://issues.apache.org/jira/browse/TEZ-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208891#comment-15208891 ] Bikas Saha commented on TEZ-2442: - We typically use fs instead of dfs and DistributedFileSystem is actually the name of the HDFS impl of the FileSystem API. > Support DFS based shuffle in addition to HTTP shuffle > - > > Key: TEZ-2442 > URL: https://issues.apache.org/jira/browse/TEZ-2442 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.3 >Reporter: Kannan Rajah >Assignee: Kannan Rajah > Attachments: HDFS_based_shuffle_v2.pdf, Tez Shuffle using DFS.pdf, > hdfs_broadcast_hack.txt, tez_hdfs_shuffle.patch > > > In Tez, Shuffle is a mechanism by which intermediate data can be shared > between stages. Shuffle data is written to local disk and fetched from any > remote node using HTTP. A DFS like MapR file system can support writing this > shuffle data directly to its DFS using a notion of local volumes and retrieve > it using HDFS API from remote node. The current Shuffle implementation > assumes local data can only be managed by LocalFileSystem. So it uses > RawLocalFileSystem and LocalDirAllocator. If we can remove this assumption > and introduce an abstraction to manage local disks, then we can reuse most of > the shuffle logic (store, sort) and inject a HDFS API based retrieval instead > of HTTP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2442) Support DFS based shuffle in addition to HTTP shuffle
[ https://issues.apache.org/jira/browse/TEZ-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1520#comment-1520 ] Bikas Saha commented on TEZ-2442: - IIRC, this is the same for both kinds of shuffle. Because consumers can fetch and merge spills as they happen in a pipelined manner as they get the DME for each spilled output. The physical fetch method (HTTP or FS) is likely not relevant. [~rajesh.balamohan] can correct me if this is inaccurate. > Support DFS based shuffle in addition to HTTP shuffle > - > > Key: TEZ-2442 > URL: https://issues.apache.org/jira/browse/TEZ-2442 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.3 >Reporter: Kannan Rajah >Assignee: Kannan Rajah > Attachments: HDFS_based_shuffle_v2.pdf, Tez Shuffle using DFS.pdf, > hdfs_broadcast_hack.txt, tez_hdfs_shuffle.patch > > > In Tez, Shuffle is a mechanism by which intermediate data can be shared > between stages. Shuffle data is written to local disk and fetched from any > remote node using HTTP. A DFS like MapR file system can support writing this > shuffle data directly to its DFS using a notion of local volumes and retrieve > it using HDFS API from remote node. The current Shuffle implementation > assumes local data can only be managed by LocalFileSystem. So it uses > RawLocalFileSystem and LocalDirAllocator. If we can remove this assumption > and introduce an abstraction to manage local disks, then we can reuse most of > the shuffle logic (store, sort) and inject a HDFS API based retrieval instead > of HTTP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3161) Allow task to report different kinds of errors - fatal / kill
[ https://issues.apache.org/jira/browse/TEZ-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-3161: Attachment: TEZ-3161.2.txt Updated patch with findbugs and javac warnings fixed. > Allow task to report different kinds of errors - fatal / kill > - > > Key: TEZ-3161 > URL: https://issues.apache.org/jira/browse/TEZ-3161 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-3161.1.txt, TEZ-3161.2.txt > > > In some cases, task failures will be the same across all attempts - e.g. > exceeding memory utilization on an operation. In this case, there's no point > in running another attempt of the same task. > There's other cases where a task may want to mark itself as KILLED - i.e. a > temporary error. An example of this is pipelined shuffle. > Tez should allow both operations. > cc [~vikram.dixit], [~rajesh.balamohan] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3185) Treat certain kinds of failures in task execution as fatal errors by default
Siddharth Seth created TEZ-3185: --- Summary: Treat certain kinds of failures in task execution as fatal errors by default Key: TEZ-3185 URL: https://issues.apache.org/jira/browse/TEZ-3185 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Failures like ClassNotFound, etc can be treated as fatal errors. We could potentially define this list as configurable with a sane set of defaults. Also potentially introduce a FatalTezException - which users can throw to indicate such errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3184) Make use of fail fatal method in various Inputs / Outputs
Siddharth Seth created TEZ-3184: --- Summary: Make use of fail fatal method in various Inputs / Outputs Key: TEZ-3184 URL: https://issues.apache.org/jira/browse/TEZ-3184 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth TEZ-3161 allows a task to be marked as a fatal failure, thus disallowing retries. Make use of this in various Inputs and Outputs where appropriate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-3161 PreCommit Build #1584
Jira: https://issues.apache.org/jira/browse/TEZ-3161
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1584/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4523 lines...]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-tests
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12794922/TEZ-3161.1.txt
against master revision 19280cf.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 13 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 59 javac
compiler warnings (more than the master's current 33 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1584//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1584//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1584//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1584//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
44e6eba441e98587b73ac79e6bcdbdf49a225b5f logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Compressed 3.42 MB of artifacts by 18.3% relative to #1572
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
7 tests failed.
FAILED: org.apache.tez.test.TestFaultTolerance.testRandomFailingInputs
Error Message:
expected: but was:
Stack Trace:
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:141)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:124)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:120)
at
org.apache.tez.test.TestFaultTolerance.testRandomFailingInputs(TestFaultTolerance.java:763)
FAILED: org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit
Error Message:
TezSession has already shutdown. No cluster diagnostics found.
Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:849)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:129)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:124)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:120)
at
org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit(TestFaultTolerance.java:261)
FAILED:
org.apache.tez.test.TestFaultTolerance.testInputFailureRerunCanSendOutputToTwoDownstreamVertices
Error Message:
TezSession has already shutdown. No cluster diagnostics found.
Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:849)
at
[jira] [Commented] (TEZ-3173) Update Tez AM REST APIs for more information for each vertex
[
https://issues.apache.org/jira/browse/TEZ-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207990#comment-15207990
]
TezQA commented on TEZ-3173:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12794888/TEZ-3173.2.patch
against master revision 19280cf.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.dag.impl.TestVertexImpl
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1583//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1583//console
This message is automatically generated.
> Update Tez AM REST APIs for more information for each vertex
>
>
> Key: TEZ-3173
> URL: https://issues.apache.org/jira/browse/TEZ-3173
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Sreenath Somarajapuram
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3173.1.patch, TEZ-3173.2.patch
>
>
> `/ui/ws/v2/tez/verticesInfo` endpoint in WebUIService must be upgraded to
> send back the following fields
> # initTime
> # startTime
> # finishTime
> # firstTaskStartTime
> # lastTaskFinishTime
> in addition to the existing id, status, progress, totalTasks,
> succeededTasks, runningTasks, pendingTasks, failedTaskAttempts &
> killedTaskAttempts fields.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-3173 PreCommit Build #1583
Jira: https://issues.apache.org/jira/browse/TEZ-3173
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1583/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4390 lines...]
[ERROR] [Help 2]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-dag
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12794888/TEZ-3173.2.patch
against master revision 19280cf.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.dag.impl.TestVertexImpl
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1583//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1583//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
6610204b73a9a146ef1fb090fea1ba66195c6f65 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Compressed 3.42 MB of artifacts by 20.1% relative to #1572
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
1 tests failed.
FAILED: org.apache.tez.dag.app.dag.impl.TestVertexImpl.testLastTaskFinishTime
Error Message:
null
Stack Trace:
java.lang.NullPointerException: null
at
org.apache.tez.dag.app.dag.impl.VertexImpl.getLastTaskFinishTime(VertexImpl.java:1260)
at
org.apache.tez.dag.app.dag.impl.TestVertexImpl.testLastTaskFinishTime(TestVertexImpl.java:6755)
