[jira] [Updated] (TEZ-3351) Handle ATS performance issues for Hive-LLAP.
[ https://issues.apache.org/jira/browse/TEZ-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3351: --- Attachment: TEZ-3351.01.patch * Adding config option to enable granular log level for ATSv15. * Changed grouping to use one file for 1000 dags in a session. > Handle ATS performance issues for Hive-LLAP. > > > Key: TEZ-3351 > URL: https://issues.apache.org/jira/browse/TEZ-3351 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3351.01.patch, TEZ-3351.WIP.01.patch > > > With Hive-LLAP, we have subsecond queries and hence can run lots of queries > in a small interval. This can overload ATS with lot of events and create > large number of file hdfs. This jira is to reduce performance impact of ATS > logging by Tez. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Success: TEZ-3351 PreCommit Build #1857
Jira: https://issues.apache.org/jira/browse/TEZ-3351
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1857/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4127 lines...]
[INFO] Tez ... SUCCESS [ 0.035 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 56:29 min
[INFO] Finished at: 2016-07-18T12:09:50+00:00
[INFO] Final Memory: 68M/853M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12818523/TEZ-3351.01.patch
against master revision 55f5186.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1857//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1857//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
4b23956f5fc51d6a96424f783d872f079b419b23 logged out
==
==
Finished build.
==
==
Archiving artifacts
[description-setter] Description set: TEZ-3351
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3351) Handle ATS performance issues for Hive-LLAP.
[
https://issues.apache.org/jira/browse/TEZ-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382170#comment-15382170
]
TezQA commented on TEZ-3351:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12818523/TEZ-3351.01.patch
against master revision 55f5186.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1857//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1857//console
This message is automatically generated.
> Handle ATS performance issues for Hive-LLAP.
>
>
> Key: TEZ-3351
> URL: https://issues.apache.org/jira/browse/TEZ-3351
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3351.01.patch, TEZ-3351.WIP.01.patch
>
>
> With Hive-LLAP, we have subsecond queries and hence can run lots of queries
> in a small interval. This can overload ATS with lot of events and create
> large number of file hdfs. This jira is to reduce performance impact of ATS
> logging by Tez.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3284) Synchronization for every write in UnorderdKVWriter
[
https://issues.apache.org/jira/browse/TEZ-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-3284:
-
Attachment: TEZ-3284.3.patch
> Synchronization for every write in UnorderdKVWriter
> ---
>
> Key: TEZ-3284
> URL: https://issues.apache.org/jira/browse/TEZ-3284
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Gopal V
>Assignee: Jonathan Eagles
>Priority: Critical
> Labels: Performance
> Attachments: TEZ-3284.1.patch, TEZ-3284.2.patch, TEZ-3284.3.patch
>
>
> {code}
> baos = new ByteArrayOutputStream();
> dos = new DataOutputStream(baos);
> keySerializer.open(dos);
> valSerializer.open(dos);
> {code}
> This is a known performance issue as documented in HADOOP-10694
> Both ByteArrayOutputStream::write() and DataOutputStream::write() have lock
> prefix calls in them, because they are object synchronized methods.
> Recommended solution is to replicate the Hive NonSync impls similar to
> HADOOP-10694
> {code}
> TezTaskRunner [RUNNABLE]
> *** java.io.DataOutputStream.write(byte[], int, int) DataOutputStream.java:107
> org.apache.tez.runtime.library.common.serializer.TezBytesWritableSerialization$TezBytesWritableSerializer.serialize(Writable)
> TezBytesWritableSerialization.java:123
> org.apache.tez.runtime.library.common.serializer.TezBytesWritableSerialization$TezBytesWritableSerializer.serialize(Object)
> TezBytesWritableSerialization.java:110
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(Object,
> Object, int) UnorderedPartitionedKVWriter.java:295
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(Object,
> Object) UnorderedPartitionedKVWriter.java:257
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(Object,
> Object) TezProcessor.java:232
> org.apache.hadoop.hive.ql.exec.vector.reducesink.VectorReduceSinkCommonOperator.collect(BytesWritable,
> Writable) VectorReduceSinkCommonOperator.java:432
> org.apache.hadoop.hive.ql.exec.vector.reducesink.VectorReduceSinkCommonOperator.process(Object,
> int) VectorReduceSinkCommonOperator.java:397
> org.apache.hadoop.hive.ql.exec.Operator.forward(Object, ObjectInspector)
> Operator.java:837
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(Object,
> int) VectorSelectOperator.java:144
> org.apache.hadoop.hive.ql.exec.Operator.forward(Object, ObjectInspector)
> Operator.java:837
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(Object,
> int) VectorFilterOperator.java:121
> org.apache.hadoop.hive.ql.exec.Operator.forward(Object, ObjectInspector)
> Operator.java:837
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(Object, int)
> TableScanOperator.java:130
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(Writable)
> VectorMapOperator.java:796
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(Object)
> MapRecordSource.java:86
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord()
> MapRecordSource.java:70
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run()
> MapRecordProcessor.java:361
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(Map,
> Map) TezProcessor.java:172
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(Map, Map)
> TezProcessor.java:160
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run()
> LogicalIOProcessorRuntimeTask.java:370
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run()
> TaskRunner2Callable.java:73
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run()
> TaskRunner2Callable.java:61
> java.security.AccessController.doPrivileged(PrivilegedExceptionAction,
> AccessControlContext) AccessController.java (native)
> javax.security.auth.Subject.doAs(Subject, PrivilegedExceptionAction)
> Subject.java:422
> org.apache.hadoop.security.UserGroupInformation.doAs(PrivilegedExceptionAction)
> UserGroupInformation.java:1657
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal()
> TaskRunner2Callable.java:61
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal()
> TaskRunner2Callable.java:37
> org.apache.tez.common.CallableWithNdc.call() CallableWithNdc.java:36
> java.util.concurrent.FutureTask.run() FutureTask.java:266
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
> ThreadPoolExecutor.java:1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run()
> ThreadPoolExecutor.java:617
> java.lang.Thread.run() Thread.java:745
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (TEZ-3354) Tez UI gets "Adapter operation failed" when RM failover happens
Yesha Vora created TEZ-3354:
---
Summary: Tez UI gets "Adapter operation failed" when RM failover
happens
Key: TEZ-3354
URL: https://issues.apache.org/jira/browse/TEZ-3354
Project: Apache Tez
Issue Type: Bug
Components: UI
Reporter: Yesha Vora
Assignee: Sreenath Somarajapuram
Scenario:
* Run Tez application
* Open Dag detail page
* Kill active RM and restart the RM ( RM failover)
* Validate Progress of Dag detail page.
Noticing, "Adapter operation failed" error message when RM failover occurs.
{code}
Details:
{
"trace": "java.net.ConnectException: Connection refused\n\tat
java.net.PlainSocketImpl.socketConnect(Native Method)\n\tat
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)\n\tat
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)\n\tat
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)\n\tat
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)\n\tat
java.net.Socket.connect(Socket.java:589)\n\tat
sun.net.NetworkClient.doConnect(NetworkClient.java:175)\n\tat
sun.net.www.http.HttpClient.openServer(HttpClient.java:432)\n\tat
sun.net.www.http.HttpClient.openServer(HttpClient.java:527)\n\tat
sun.net.www.http.HttpClient.(HttpClient.java:211)\n\tat
sun.net.www.http.HttpClient.New(HttpClient.java:308)\n\tat
sun.net.www.http.HttpClient.New(HttpClient.java:326)\n\tat
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)\n\tat
sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)\n\tat
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)\n\tat
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)\n\tat
sun.net.www.protocol.http.HttpURLConnection.followRedirect0(HttpURLConnection.java:2662)\n\tat
sun.net.www.protocol.http.HttpURLConnection.followRedirect(HttpURLConnection.java:2584)\n\tat
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1770)\n\tat
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)\n\tat
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)\n\tat
org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:218)\n\tat
org.apache.ambari.server.view.ViewURLStreamProvider.getHttpURLConnection(ViewURLStreamProvider.java:239)\n\tat
org.apache.ambari.server.view.ViewURLStreamProvider.getConnection(ViewURLStreamProvider.java:146)\n\tat
org.apache.ambari.view.tez.utils.ProxyHelper.getResponse(ProxyHelper.java:59)\n\tat
org.apache.ambari.view.tez.rest.BaseProxyResource.getData(BaseProxyResource.java:64)\n\tat
sun.reflect.GeneratedMethodAccessor382.invoke(Unknown Source)\n\tat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
java.lang.reflect.Method.invoke(Method.java:498)\n\tat
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)\n\tat
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)\n\tat
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)\n\tat
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)\n\tat
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\n\tat
com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)\n\tat
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\n\tat
com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)\n\tat
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\n\tat
com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)\n\tat
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\n\tat
com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)\n\tat
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\n\tat
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)\n\tat
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\n\tat
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)\n\tat
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)\n\tat
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)\n\tat
com.sun.jersey.serv
[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat
[
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piyush Narang updated TEZ-3348:
---
Attachment: (was: 11.patch)
> NullPointerException in Tez MROutput while trying to write using Parquet's
> DeprecatedParquetOutputFormat
>
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.(Path.java:105)
> at org.apache.hadoop.fs.Path.(Path.java:94)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the
> workOutputPath -
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) -
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR
> ("mapreduce.task.output.dir") constant.
> 4) This ends up being null and in the Parquet code we end up with an NPE in
> the Path class.
> Looking at the Tez code, we are setting the workOutputPath in the
> MROutput.initCommitter method -
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>
> This call however, is made after the call to access the workOutputPath as
> part of outputFormat.getRecordWriter().
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
> oldApiTaskAttemptContext =
> new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
> jobConf, taskAttemptId,
> new MRTaskReporter(getContext()));
> initCommitter(jobConf, useNewApi); // before the getRecordWriter call
> oldOutputFormat = jobConf.getOutputFormat();
> outputFormatClassName = oldOutputFormat.getClass().getName();
> FileSystem fs = FileSystem.get(jobConf);
> String finalName = getOutputName();
> oldRecordWriter =
> oldOutputFormat.getRecordWriter(
> fs, jobConf, finalName, new
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds
> reasonable, I can cut a PR.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat
[
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piyush Narang updated TEZ-3348:
---
Attachment: (was: 11.patch)
> NullPointerException in Tez MROutput while trying to write using Parquet's
> DeprecatedParquetOutputFormat
>
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.(Path.java:105)
> at org.apache.hadoop.fs.Path.(Path.java:94)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the
> workOutputPath -
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) -
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR
> ("mapreduce.task.output.dir") constant.
> 4) This ends up being null and in the Parquet code we end up with an NPE in
> the Path class.
> Looking at the Tez code, we are setting the workOutputPath in the
> MROutput.initCommitter method -
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>
> This call however, is made after the call to access the workOutputPath as
> part of outputFormat.getRecordWriter().
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
> oldApiTaskAttemptContext =
> new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
> jobConf, taskAttemptId,
> new MRTaskReporter(getContext()));
> initCommitter(jobConf, useNewApi); // before the getRecordWriter call
> oldOutputFormat = jobConf.getOutputFormat();
> outputFormatClassName = oldOutputFormat.getClass().getName();
> FileSystem fs = FileSystem.get(jobConf);
> String finalName = getOutputName();
> oldRecordWriter =
> oldOutputFormat.getRecordWriter(
> fs, jobConf, finalName, new
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds
> reasonable, I can cut a PR.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat
[
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piyush Narang updated TEZ-3348:
---
Attachment: (was: 11.patch)
> NullPointerException in Tez MROutput while trying to write using Parquet's
> DeprecatedParquetOutputFormat
>
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.(Path.java:105)
> at org.apache.hadoop.fs.Path.(Path.java:94)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the
> workOutputPath -
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) -
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR
> ("mapreduce.task.output.dir") constant.
> 4) This ends up being null and in the Parquet code we end up with an NPE in
> the Path class.
> Looking at the Tez code, we are setting the workOutputPath in the
> MROutput.initCommitter method -
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>
> This call however, is made after the call to access the workOutputPath as
> part of outputFormat.getRecordWriter().
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
> oldApiTaskAttemptContext =
> new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
> jobConf, taskAttemptId,
> new MRTaskReporter(getContext()));
> initCommitter(jobConf, useNewApi); // before the getRecordWriter call
> oldOutputFormat = jobConf.getOutputFormat();
> outputFormatClassName = oldOutputFormat.getClass().getName();
> FileSystem fs = FileSystem.get(jobConf);
> String finalName = getOutputName();
> oldRecordWriter =
> oldOutputFormat.getRecordWriter(
> fs, jobConf, finalName, new
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds
> reasonable, I can cut a PR.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3284) Synchronization for every write in UnorderdKVWriter
[
https://issues.apache.org/jira/browse/TEZ-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382803#comment-15382803
]
TezQA commented on TEZ-3284:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12818583/TEZ-3284.3.patch
against master revision 55f5186.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.rm.TestContainerReuse
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1858//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1858//console
This message is automatically generated.
> Synchronization for every write in UnorderdKVWriter
> ---
>
> Key: TEZ-3284
> URL: https://issues.apache.org/jira/browse/TEZ-3284
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Gopal V
>Assignee: Jonathan Eagles
>Priority: Critical
> Labels: Performance
> Attachments: TEZ-3284.1.patch, TEZ-3284.2.patch, TEZ-3284.3.patch
>
>
> {code}
> baos = new ByteArrayOutputStream();
> dos = new DataOutputStream(baos);
> keySerializer.open(dos);
> valSerializer.open(dos);
> {code}
> This is a known performance issue as documented in HADOOP-10694
> Both ByteArrayOutputStream::write() and DataOutputStream::write() have lock
> prefix calls in them, because they are object synchronized methods.
> Recommended solution is to replicate the Hive NonSync impls similar to
> HADOOP-10694
> {code}
> TezTaskRunner [RUNNABLE]
> *** java.io.DataOutputStream.write(byte[], int, int) DataOutputStream.java:107
> org.apache.tez.runtime.library.common.serializer.TezBytesWritableSerialization$TezBytesWritableSerializer.serialize(Writable)
> TezBytesWritableSerialization.java:123
> org.apache.tez.runtime.library.common.serializer.TezBytesWritableSerialization$TezBytesWritableSerializer.serialize(Object)
> TezBytesWritableSerialization.java:110
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(Object,
> Object, int) UnorderedPartitionedKVWriter.java:295
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(Object,
> Object) UnorderedPartitionedKVWriter.java:257
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(Object,
> Object) TezProcessor.java:232
> org.apache.hadoop.hive.ql.exec.vector.reducesink.VectorReduceSinkCommonOperator.collect(BytesWritable,
> Writable) VectorReduceSinkCommonOperator.java:432
> org.apache.hadoop.hive.ql.exec.vector.reducesink.VectorReduceSinkCommonOperator.process(Object,
> int) VectorReduceSinkCommonOperator.java:397
> org.apache.hadoop.hive.ql.exec.Operator.forward(Object, ObjectInspector)
> Operator.java:837
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(Object,
> int) VectorSelectOperator.java:144
> org.apache.hadoop.hive.ql.exec.Operator.forward(Object, ObjectInspector)
> Operator.java:837
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(Object,
> int) VectorFilterOperator.java:121
> org.apache.hadoop.hive.ql.exec.Operator.forward(Object, ObjectInspector)
> Operator.java:837
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(Object, int)
> TableScanOperator.java:130
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(Writable)
> VectorMapOperator.java:796
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(Object)
> MapRecordSource.java:86
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord()
> MapRecordSource.java:70
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run()
> MapRecordProcessor.java:361
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(Map,
> Map) TezProcessor.java:172
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(Map, Map)
> TezProcessor.java:160
> org.apache.tez.runtime.LogicalIOProcessorRuntimeT
Failed: TEZ-3284 PreCommit Build #1858
Jira: https://issues.apache.org/jira/browse/TEZ-3284
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1858/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4137 lines...]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-dag
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12818583/TEZ-3284.3.patch
against master revision 55f5186.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.rm.TestContainerReuse
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1858//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1858//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
7472ac110032e6e4e3ee417abfefb3b05a29ecd9 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Compressed 3.20 MB of artifacts by 27.3% relative to #1857
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
1 tests failed.
FAILED:
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources
Error Message:
Wanted but not invoked:
taskSchedulerManagerForTest.taskAllocated(
0,
Mock for TA attempt_0_0001_0_01_03_1,
,
Container: [ContainerId: container_1_0001_01_01, NodeId: host1:0,
NodeHttpAddress: host1:0, Resource: , Priority: 1,
Token: null, ]
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1254)
However, there were other interactions with this mock:
taskSchedulerManagerForTest.init(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1143)
taskSchedulerManagerForTest.setConfig(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1143)
taskSchedulerManagerForTest.serviceInit(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1143)
taskSchedulerManagerForTest.start();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.serviceStart();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.instantiateSchedulers(
"host",
0,
"",
Mock for AppContext, hashCode: 1800762433
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.getContainerSignatureMatcher();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManage
[jira] [Updated] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat
[
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piyush Narang updated TEZ-3348:
---
Attachment: 11.patch
> NullPointerException in Tez MROutput while trying to write using Parquet's
> DeprecatedParquetOutputFormat
>
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch, 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.(Path.java:105)
> at org.apache.hadoop.fs.Path.(Path.java:94)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the
> workOutputPath -
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) -
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR
> ("mapreduce.task.output.dir") constant.
> 4) This ends up being null and in the Parquet code we end up with an NPE in
> the Path class.
> Looking at the Tez code, we are setting the workOutputPath in the
> MROutput.initCommitter method -
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>
> This call however, is made after the call to access the workOutputPath as
> part of outputFormat.getRecordWriter().
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
> oldApiTaskAttemptContext =
> new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
> jobConf, taskAttemptId,
> new MRTaskReporter(getContext()));
> initCommitter(jobConf, useNewApi); // before the getRecordWriter call
> oldOutputFormat = jobConf.getOutputFormat();
> outputFormatClassName = oldOutputFormat.getClass().getName();
> FileSystem fs = FileSystem.get(jobConf);
> String finalName = getOutputName();
> oldRecordWriter =
> oldOutputFormat.getRecordWriter(
> fs, jobConf, finalName, new
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds
> reasonable, I can cut a PR.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat
[
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382816#comment-15382816
]
Piyush Narang commented on TEZ-3348:
[~hitesh] - Sorry I'll keep that in mind the next time :-). Realized that the
last patch I uploaded was generated incorrectly which is why the tests &
findbugs failed. Uploaded a new one (which is the same as the PR). Hopefully
Jenkins won't crib this time.
> NullPointerException in Tez MROutput while trying to write using Parquet's
> DeprecatedParquetOutputFormat
>
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch, 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.(Path.java:105)
> at org.apache.hadoop.fs.Path.(Path.java:94)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the
> workOutputPath -
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) -
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR
> ("mapreduce.task.output.dir") constant.
> 4) This ends up being null and in the Parquet code we end up with an NPE in
> the Path class.
> Looking at the Tez code, we are setting the workOutputPath in the
> MROutput.initCommitter method -
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>
> This call however, is made after the call to access the workOutputPath as
> part of outputFormat.getRecordWriter().
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
> oldApiTaskAttemptContext =
> new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
> jobConf, taskAttemptId,
> new MRTaskReporter(getContext()));
> initCommitter(jobConf, useNewApi); // before the getRecordWriter call
> oldOutputFormat = jobConf.getOutputFormat();
> outputFormatClassName = oldOutputFormat.getClass().getName();
> FileSystem fs = FileSystem.get(jobConf);
> String finalName = getOutputName();
> oldRecordWriter =
> oldOutputFormat.getRecordWriter(
> fs, jobConf, finalName, new
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds
> reasonable, I can cut a PR.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (TEZ-3355) Tez Custom Shuffle Handler POC
Jonathan Eagles created TEZ-3355: Summary: Tez Custom Shuffle Handler POC Key: TEZ-3355 URL: https://issues.apache.org/jira/browse/TEZ-3355 Project: Apache Tez Issue Type: Sub-task Reporter: Jonathan Eagles -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3317) Speculative execution starts too early due to 0 progress
[
https://issues.apache.org/jira/browse/TEZ-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382998#comment-15382998
]
Kuhu Shukla commented on TEZ-3317:
--
There seems to be a disconnect between Processors and LogicalInput (and its
derivatives) where the RecordReader progress is not used by {{RuntimeTask}} .
Adding the progress calls to {{Input}}/{{LogicalInput}} class would break
anyone who uses that interface, although {{AbstractLogicalInput}} can be
augmented. Requesting for some comments from [~bikassaha], [~sseth] on how to
propagate Input specific progress to each Processor.
> Speculative execution starts too early due to 0 progress
>
>
> Key: TEZ-3317
> URL: https://issues.apache.org/jira/browse/TEZ-3317
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Jonathan Eagles
>
> Don't know at this point if this is a tez or a PigProcessor issue. There is
> some setProgress chain that is keeping task progress from being correctly
> reported. Task status is always zero, so as soon as the first task finishes,
> tasks up to the speculation limit are always launched.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat
[
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383039#comment-15383039
]
TezQA commented on TEZ-3348:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12818611/11.patch
against master revision 55f5186.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1859//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1859//console
This message is automatically generated.
> NullPointerException in Tez MROutput while trying to write using Parquet's
> DeprecatedParquetOutputFormat
>
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch, 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.(Path.java:105)
> at org.apache.hadoop.fs.Path.(Path.java:94)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the
> workOutputPath -
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) -
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR
> ("mapreduce.task.output.dir") constant.
> 4) This ends up being null and in the Parquet code we end up with an NPE in
> the Path class.
> Looking at the Tez code, we are setting the workOutputPath in the
> MROutput.initCommitter method -
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>
> This call however, is made after the call to access the workOutputPath as
> part of outputFormat.getRecordWriter().
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
> oldApiTaskAttemptContext =
> new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
> jobConf, taskAttemptId,
> new MRTaskReporter(getContext()));
> initCommitter(jobConf, useNewApi); // before the getRecordWriter call
> oldOutputFormat = jobConf.getOutputFormat();
> outputFormatClassName = oldOutputFormat.getClass().getName();
> FileSystem fs = FileSystem.get(jobConf);
> String finalName = getOutputName();
> oldRecordWriter =
> oldOutputFormat.getRecordWriter(
> fs, jobConf, finalName, new
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds
> reasonable, I can cut a PR.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3351) Handle ATS performance issues for Hive-LLAP.
[
https://issues.apache.org/jira/browse/TEZ-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383038#comment-15383038
]
Hitesh Shah commented on TEZ-3351:
--
General questions:
- Is ATSHistoryLogLevel sufficient to control the amount of data pushed to
ATS? What if dags have 5000 counters each? Would that be handled easily by ATS?
Do we need a knob to stop publishing counters at certain levels? Additionally,
why do we need so many logging levels?
Comments on the patch:
- ATSHistoryLogLevel sounds ATS specific. ATS is actually a plugin and
therefore it seems wrong to add a plugin specific API to the generic APIs of
TezClient and DAGClient. A generic API to set logging level makes sense though.
- "private static final int DAGS_PER_GROUP = 1000;" - why a 1000? What kind
of perf impact will we see if we scale this value upwards or downwards? Can
this be changed dynamically per app?
- Why does a TIMELINE_GROUPID related vars belong to TezDAGId?
{code}
} else {
367 // dagId does not exist, lets check at AM level.
368 if
(!amAtsHistoryLogLevel.shouldLog(eventType.getAtsHistoryLogLevel())) {
369 return false;
370 }
371 }
{code}
- this approach is incorrect for recovery cases. If the dag object exists, it
should be easy enough to retrieve the log level and add it back to the map.
This does raise an issue for cache misses though.
- testATSLogLevelNone() - not sure how this is actually testing that there is
no data being generated by the history logger?
- there needs to be additional testing for a level that does not match
ALL/NONE
- ATSVHistoryLoggingService has not been changed?
- "Find better way to wait for the events to be drained." - timing based
tests have a tendency to be flaky. Would be good to change this to more
definitive.
> Handle ATS performance issues for Hive-LLAP.
>
>
> Key: TEZ-3351
> URL: https://issues.apache.org/jira/browse/TEZ-3351
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: TEZ-3351.01.patch, TEZ-3351.WIP.01.patch
>
>
> With Hive-LLAP, we have subsecond queries and hence can run lots of queries
> in a small interval. This can overload ATS with lot of events and create
> large number of file hdfs. This jira is to reduce performance impact of ATS
> logging by Tez.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-3348 PreCommit Build #1859
Jira: https://issues.apache.org/jira/browse/TEZ-3348
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1859/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4126 lines...]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 57:00 min
[INFO] Finished at: 2016-07-18T20:53:08+00:00
[INFO] Final Memory: 68M/846M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12818611/11.patch
against master revision 55f5186.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1859//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1859//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
baebfa4df5fc2ae7661cfcb46c0a15f019d71829 logged out
==
==
Finished build.
==
==
Archiving artifacts
Compressed 3.20 MB of artifacts by 27.4% relative to #1857
[description-setter] Description set: TEZ-3348
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3348) NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat
[
https://issues.apache.org/jira/browse/TEZ-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383053#comment-15383053
]
Piyush Narang commented on TEZ-3348:
[~hitesh] - all green now :-). Do take a look when you get the time. Thanks.
> NullPointerException in Tez MROutput while trying to write using Parquet's
> DeprecatedParquetOutputFormat
>
>
> Key: TEZ-3348
> URL: https://issues.apache.org/jira/browse/TEZ-3348
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
> Attachments: 11.patch, 11.patch
>
>
> Trying to run some Tez MR jobs that write out some data using Parquet to
> HDFS. When I try to do so, end up seeing a NPE in the Parquet code:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.(Path.java:105)
> at org.apache.hadoop.fs.Path.(Path.java:94)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getDefaultWorkFile(DeprecatedParquetOutputFormat.java:69)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.access$100(DeprecatedParquetOutputFormat.java:36)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat$RecordWriterWrapper.(DeprecatedParquetOutputFormat.java:89)
> at
> org.apache.parquet.hadoop.mapred.DeprecatedParquetOutputFormat.getRecordWriter(DeprecatedParquetOutputFormat.java:77)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:416)
> {code}
> The flow seems to be:
> 1) The Parquet deprecated output format class tries to read the
> workOutputPath -
> https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/mapred/DeprecatedParquetOutputFormat.java#L69
> 2) This calls FileOutputFormat.getWorkOutputPath(...) -
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java#L229
> 3) That in turn tries to read the JobContext.TASK_OUTPUT_DIR
> ("mapreduce.task.output.dir") constant.
> 4) This ends up being null and in the Parquet code we end up with an NPE in
> the Path class.
> Looking at the Tez code, we are setting the workOutputPath in the
> MROutput.initCommitter method -
> https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/output/MROutput.java#L445.
>
> This call however, is made after the call to access the workOutputPath as
> part of outputFormat.getRecordWriter().
> I tried out a run where I moved this initCommitter call up:
> {code}
> else {
> oldApiTaskAttemptContext =
> new org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl(
> jobConf, taskAttemptId,
> new MRTaskReporter(getContext()));
> initCommitter(jobConf, useNewApi); // before the getRecordWriter call
> oldOutputFormat = jobConf.getOutputFormat();
> outputFormatClassName = oldOutputFormat.getClass().getName();
> FileSystem fs = FileSystem.get(jobConf);
> String finalName = getOutputName();
> oldRecordWriter =
> oldOutputFormat.getRecordWriter(
> fs, jobConf, finalName, new
> MRReporter(getContext().getCounters()));
> }
> {code}
> I tried out a run with this and it seems to succeed. If this sounds
> reasonable, I can cut a PR.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3353) Tez should adjust processor memory on demand
[ https://issues.apache.org/jira/browse/TEZ-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383268#comment-15383268 ] Gunther Hagleitner commented on TEZ-3353: - The request is to get a proper API. Setting "tez.task.scale.memory.reserve-fraction" based on a computation that has to involve various internal Tez variables (fraction of xmx to container, fraction reserved by default for processor, etc) is just bogus. Hive should just be able to tell Tez that it needs 500mb and tez can layout the mem as it sees fit with this in mind. > Tez should adjust processor memory on demand > > > Key: TEZ-3353 > URL: https://issues.apache.org/jira/browse/TEZ-3353 > Project: Apache Tez > Issue Type: Bug >Reporter: Wei Zheng > > Hive requests some amount of memory for its map join, which sometimes is not > allocated as much as expected. Tez should make adjustment and satisfy that > request as is. > This is related to HIVE-13934. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3351) Handle ATS performance issues for Hive-LLAP.
[ https://issues.apache.org/jira/browse/TEZ-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383267#comment-15383267 ] Hitesh Shah commented on TEZ-3351: -- bq. private static final int DAGS_PER_GROUP = 1000; Additional question: given that the timeline cache plugin is hosted within Timeline and could potentially need to handle multiple different deployments of Tez, how would it handle different buidls where the group count is different? > Handle ATS performance issues for Hive-LLAP. > > > Key: TEZ-3351 > URL: https://issues.apache.org/jira/browse/TEZ-3351 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3351.01.patch, TEZ-3351.WIP.01.patch > > > With Hive-LLAP, we have subsecond queries and hence can run lots of queries > in a small interval. This can overload ATS with lot of events and create > large number of file hdfs. This jira is to reduce performance impact of ATS > logging by Tez. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3353) Tez should adjust processor memory on demand
[ https://issues.apache.org/jira/browse/TEZ-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383269#comment-15383269 ] Gunther Hagleitner commented on TEZ-3353: - cc [~hitesh], [~sseth] > Tez should adjust processor memory on demand > > > Key: TEZ-3353 > URL: https://issues.apache.org/jira/browse/TEZ-3353 > Project: Apache Tez > Issue Type: Bug >Reporter: Wei Zheng > > Hive requests some amount of memory for its map join, which sometimes is not > allocated as much as expected. Tez should make adjustment and satisfy that > request as is. > This is related to HIVE-13934. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3356) Fix initializing of stats when custom ShuffleVertexManager is used
Peter Slawski created TEZ-3356: -- Summary: Fix initializing of stats when custom ShuffleVertexManager is used Key: TEZ-3356 URL: https://issues.apache.org/jira/browse/TEZ-3356 Project: Apache Tez Issue Type: Bug Affects Versions: 0.8.4 Reporter: Peter Slawski When using a custom ShuffleVertexManager to set a vertex’s parallelism, the partition stats field will be left uninitialized even after the manager itself gets initialized. This results in a IllegalStateException to be thrown as the stats field will not yet be initialized when VertexManagerEvents are processed upon the start of the vertex. Note that these events contain partition sizes which are aggregated and stored in this stats field. Apache Pig’s grace auto-parallelism feature uses a custom ShuffleVertexManager which sets a vertex’s parallelism upon the completion of one of its parent’s parents. Thus, this corner case is hit and pig scripts with grace parallelism enabled would fail if the DAG consists of at least one vertex having grandparents. The fix should be straight forward. Before rather than after VertexManagerEvents are processed, simply update pending tasks to ensure the partition stats field will be initialized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3356) Fix initializing of stats when custom ShuffleVertexManager is used
[ https://issues.apache.org/jira/browse/TEZ-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Slawski updated TEZ-3356: --- Attachment: TEZ-3356.1.patch > Fix initializing of stats when custom ShuffleVertexManager is used > -- > > Key: TEZ-3356 > URL: https://issues.apache.org/jira/browse/TEZ-3356 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.4 >Reporter: Peter Slawski > Attachments: TEZ-3356.1.patch > > > When using a custom ShuffleVertexManager to set a vertex’s parallelism, the > partition stats field will be left uninitialized even after the manager > itself gets initialized. This results in a IllegalStateException to be thrown > as the stats field will not yet be initialized when VertexManagerEvents are > processed upon the start of the vertex. Note that these events contain > partition sizes which are aggregated and stored in this stats field. > > Apache Pig’s grace auto-parallelism feature uses a custom > ShuffleVertexManager which sets a vertex’s parallelism upon the completion of > one of its parent’s parents. Thus, this corner case is hit and pig scripts > with grace parallelism enabled would fail if the DAG consists of at least one > vertex having grandparents. > > The fix should be straight forward. Before rather than after > VertexManagerEvents are processed, simply update pending tasks to ensure the > partition stats field will be initialized. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3356) Fix initializing of stats when custom ShuffleVertexManager is used
[ https://issues.apache.org/jira/browse/TEZ-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383337#comment-15383337 ] Peter Slawski commented on TEZ-3356: I've attached patch which fixes this issue and adds a test case which illustrates how this bug could be hit with a custom ShuffleVertexManager that is roughly a much simpler version of PigGraceShuffleVertexManager. > Fix initializing of stats when custom ShuffleVertexManager is used > -- > > Key: TEZ-3356 > URL: https://issues.apache.org/jira/browse/TEZ-3356 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.4 >Reporter: Peter Slawski > Attachments: TEZ-3356.1.patch > > > When using a custom ShuffleVertexManager to set a vertex’s parallelism, the > partition stats field will be left uninitialized even after the manager > itself gets initialized. This results in a IllegalStateException to be thrown > as the stats field will not yet be initialized when VertexManagerEvents are > processed upon the start of the vertex. Note that these events contain > partition sizes which are aggregated and stored in this stats field. > > Apache Pig’s grace auto-parallelism feature uses a custom > ShuffleVertexManager which sets a vertex’s parallelism upon the completion of > one of its parent’s parents. Thus, this corner case is hit and pig scripts > with grace parallelism enabled would fail if the DAG consists of at least one > vertex having grandparents. > > The fix should be straight forward. Before rather than after > VertexManagerEvents are processed, simply update pending tasks to ensure the > partition stats field will be initialized. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3356) Fix initializing of stats when custom ShuffleVertexManager is used
[ https://issues.apache.org/jira/browse/TEZ-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Slawski updated TEZ-3356: --- Release Note: (was: I've attached patch which fixes this issue and adds a test case which illustrates how this bug could be hit with a custom ShuffleVertexManager that is roughly a much simpler version of PigGraceShuffleVertexManager.) > Fix initializing of stats when custom ShuffleVertexManager is used > -- > > Key: TEZ-3356 > URL: https://issues.apache.org/jira/browse/TEZ-3356 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.4 >Reporter: Peter Slawski > Attachments: TEZ-3356.1.patch > > > When using a custom ShuffleVertexManager to set a vertex’s parallelism, the > partition stats field will be left uninitialized even after the manager > itself gets initialized. This results in a IllegalStateException to be thrown > as the stats field will not yet be initialized when VertexManagerEvents are > processed upon the start of the vertex. Note that these events contain > partition sizes which are aggregated and stored in this stats field. > > Apache Pig’s grace auto-parallelism feature uses a custom > ShuffleVertexManager which sets a vertex’s parallelism upon the completion of > one of its parent’s parents. Thus, this corner case is hit and pig scripts > with grace parallelism enabled would fail if the DAG consists of at least one > vertex having grandparents. > > The fix should be straight forward. Before rather than after > VertexManagerEvents are processed, simply update pending tasks to ensure the > partition stats field will be initialized. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3356) Fix initializing of stats when custom ShuffleVertexManager is used
[ https://issues.apache.org/jira/browse/TEZ-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383357#comment-15383357 ] Peter Slawski commented on TEZ-3356: This is similar to TEZ-2684 but this issue targets fixing another edge case where the stats field is not initialized. > Fix initializing of stats when custom ShuffleVertexManager is used > -- > > Key: TEZ-3356 > URL: https://issues.apache.org/jira/browse/TEZ-3356 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.4 >Reporter: Peter Slawski > Attachments: TEZ-3356.1.patch > > > When using a custom ShuffleVertexManager to set a vertex’s parallelism, the > partition stats field will be left uninitialized even after the manager > itself gets initialized. This results in a IllegalStateException to be thrown > as the stats field will not yet be initialized when VertexManagerEvents are > processed upon the start of the vertex. Note that these events contain > partition sizes which are aggregated and stored in this stats field. > > Apache Pig’s grace auto-parallelism feature uses a custom > ShuffleVertexManager which sets a vertex’s parallelism upon the completion of > one of its parent’s parents. Thus, this corner case is hit and pig scripts > with grace parallelism enabled would fail if the DAG consists of at least one > vertex having grandparents. > > The fix should be straight forward. Before rather than after > VertexManagerEvents are processed, simply update pending tasks to ensure the > partition stats field will be initialized. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3317) Speculative execution starts too early due to 0 progress
[ https://issues.apache.org/jira/browse/TEZ-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383436#comment-15383436 ] Bikas Saha commented on TEZ-3317: - Sorry I did not understand whats the issue here from the above comment. Is task progress = 0 always or occasionally? What the is flow in the buggy situation? Is it for cases where the processor makes no progress because the input is slow and because input progress is not available to the processor, it reports 0 progress overall for a long time. > Speculative execution starts too early due to 0 progress > > > Key: TEZ-3317 > URL: https://issues.apache.org/jira/browse/TEZ-3317 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Eagles > > Don't know at this point if this is a tez or a PigProcessor issue. There is > some setProgress chain that is keeping task progress from being correctly > reported. Task status is always zero, so as soon as the first task finishes, > tasks up to the speculation limit are always launched. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3317) Speculative execution starts too early due to 0 progress
[
https://issues.apache.org/jira/browse/TEZ-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383464#comment-15383464
]
Kuhu Shukla commented on TEZ-3317:
--
[~bikassaha], Sorry for not elaborating earlier.
Task Attempt progress stays 0 while the attempt is in running state. It becomes
1.0 upon completion but those are the only two values that the progress for the
task can take. This is when, in fact, the Task attempt is making real progress.
For example, a simple MRRSleep job shows 0 progress per task attempt until it
finishes, even though the record reader's progress does change(value of
{{MRReaderMapReduce#recordReader.getProgress()}}).
{code}
hadoop jar tez-tests-0.7.x.y.z.jar mrrsleep -Dtez.am.speculation.enabled=true
-mt 3 -rt 3 -m 100 -r 100
{code}
I understand that for this specific case, the taskReporter.setProgress is
called only upon completion which explains this behavior.
If I add the following change to MapProcessor, the progress updates show up for
map task attempts.
{code}
@Override
public boolean nextKeyValue() throws IOException,
InterruptedException {
mrReporter.setProgress(((MRReader)reader).getProgress());
return reader.next();
}
{code}
But for a non-MR example, the notion of progress seems incomplete between the
RecordReader/LogicalInput layer and Processor layer. One way could be to add
{{getProgress()}} to AbstractLogicalInput and have Readers give out their
progress values to the Inputs based on appropriate calculation (based on number
of records/ number of bytes read may be). Appreciate any comments on how to
reconcile this in a generic way so that progress is available irrespective of
the Processor and the associated Input/Reader(s).
> Speculative execution starts too early due to 0 progress
>
>
> Key: TEZ-3317
> URL: https://issues.apache.org/jira/browse/TEZ-3317
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Jonathan Eagles
>
> Don't know at this point if this is a tez or a PigProcessor issue. There is
> some setProgress chain that is keeping task progress from being correctly
> reported. Task status is always zero, so as soon as the first task finishes,
> tasks up to the speculation limit are always launched.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-3356 PreCommit Build #1860
Jira: https://issues.apache.org/jira/browse/TEZ-3356
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1860/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4123 lines...]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 57:15 min
[INFO] Finished at: 2016-07-19T01:59:33+00:00
[INFO] Final Memory: 70M/878M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12818707/TEZ-3356.1.patch
against master revision a4247a7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1860//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1860//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
b6646e9bfb4316d5fd01f9d3e430c5bdcb340b47 logged out
==
==
Finished build.
==
==
Archiving artifacts
Compressed 3.20 MB of artifacts by 29.3% relative to #1859
[description-setter] Description set: TEZ-3356
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3356) Fix initializing of stats when custom ShuffleVertexManager is used
[
https://issues.apache.org/jira/browse/TEZ-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383472#comment-15383472
]
TezQA commented on TEZ-3356:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12818707/TEZ-3356.1.patch
against master revision a4247a7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1860//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1860//console
This message is automatically generated.
> Fix initializing of stats when custom ShuffleVertexManager is used
> --
>
> Key: TEZ-3356
> URL: https://issues.apache.org/jira/browse/TEZ-3356
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.4
>Reporter: Peter Slawski
> Attachments: TEZ-3356.1.patch
>
>
> When using a custom ShuffleVertexManager to set a vertex’s parallelism, the
> partition stats field will be left uninitialized even after the manager
> itself gets initialized. This results in a IllegalStateException to be thrown
> as the stats field will not yet be initialized when VertexManagerEvents are
> processed upon the start of the vertex. Note that these events contain
> partition sizes which are aggregated and stored in this stats field.
>
> Apache Pig’s grace auto-parallelism feature uses a custom
> ShuffleVertexManager which sets a vertex’s parallelism upon the completion of
> one of its parent’s parents. Thus, this corner case is hit and pig scripts
> with grace parallelism enabled would fail if the DAG consists of at least one
> vertex having grandparents.
>
> The fix should be straight forward. Before rather than after
> VertexManagerEvents are processed, simply update pending tasks to ensure the
> partition stats field will be initialized.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3356) Fix initializing of stats when custom ShuffleVertexManager is used
[ https://issues.apache.org/jira/browse/TEZ-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-3356: - Assignee: Peter Slawski > Fix initializing of stats when custom ShuffleVertexManager is used > -- > > Key: TEZ-3356 > URL: https://issues.apache.org/jira/browse/TEZ-3356 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.4 >Reporter: Peter Slawski >Assignee: Peter Slawski > Attachments: TEZ-3356.1.patch > > > When using a custom ShuffleVertexManager to set a vertex’s parallelism, the > partition stats field will be left uninitialized even after the manager > itself gets initialized. This results in a IllegalStateException to be thrown > as the stats field will not yet be initialized when VertexManagerEvents are > processed upon the start of the vertex. Note that these events contain > partition sizes which are aggregated and stored in this stats field. > > Apache Pig’s grace auto-parallelism feature uses a custom > ShuffleVertexManager which sets a vertex’s parallelism upon the completion of > one of its parent’s parents. Thus, this corner case is hit and pig scripts > with grace parallelism enabled would fail if the DAG consists of at least one > vertex having grandparents. > > The fix should be straight forward. Before rather than after > VertexManagerEvents are processed, simply update pending tasks to ensure the > partition stats field will be initialized. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3356) Fix initializing of stats when custom ShuffleVertexManager is used
[ https://issues.apache.org/jira/browse/TEZ-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383490#comment-15383490 ] Hitesh Shah commented on TEZ-3356: -- \cc [~bikassaha] [~rajesh.balamohan] > Fix initializing of stats when custom ShuffleVertexManager is used > -- > > Key: TEZ-3356 > URL: https://issues.apache.org/jira/browse/TEZ-3356 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.4 >Reporter: Peter Slawski >Assignee: Peter Slawski > Attachments: TEZ-3356.1.patch > > > When using a custom ShuffleVertexManager to set a vertex’s parallelism, the > partition stats field will be left uninitialized even after the manager > itself gets initialized. This results in a IllegalStateException to be thrown > as the stats field will not yet be initialized when VertexManagerEvents are > processed upon the start of the vertex. Note that these events contain > partition sizes which are aggregated and stored in this stats field. > > Apache Pig’s grace auto-parallelism feature uses a custom > ShuffleVertexManager which sets a vertex’s parallelism upon the completion of > one of its parent’s parents. Thus, this corner case is hit and pig scripts > with grace parallelism enabled would fail if the DAG consists of at least one > vertex having grandparents. > > The fix should be straight forward. Before rather than after > VertexManagerEvents are processed, simply update pending tasks to ensure the > partition stats field will be initialized. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3356) Fix initializing of stats when custom ShuffleVertexManager is used
[ https://issues.apache.org/jira/browse/TEZ-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383551#comment-15383551 ] Rajesh Balamohan commented on TEZ-3356: --- [~petersla] - Thanks for uploading the patch. updatePendingTasks() which initializes the stats is called from ShuffleVertexManager::initialize() itself. So I am trying to understand the scenario in which this would happen. > Fix initializing of stats when custom ShuffleVertexManager is used > -- > > Key: TEZ-3356 > URL: https://issues.apache.org/jira/browse/TEZ-3356 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.4 >Reporter: Peter Slawski >Assignee: Peter Slawski > Attachments: TEZ-3356.1.patch > > > When using a custom ShuffleVertexManager to set a vertex’s parallelism, the > partition stats field will be left uninitialized even after the manager > itself gets initialized. This results in a IllegalStateException to be thrown > as the stats field will not yet be initialized when VertexManagerEvents are > processed upon the start of the vertex. Note that these events contain > partition sizes which are aggregated and stored in this stats field. > > Apache Pig’s grace auto-parallelism feature uses a custom > ShuffleVertexManager which sets a vertex’s parallelism upon the completion of > one of its parent’s parents. Thus, this corner case is hit and pig scripts > with grace parallelism enabled would fail if the DAG consists of at least one > vertex having grandparents. > > The fix should be straight forward. Before rather than after > VertexManagerEvents are processed, simply update pending tasks to ensure the > partition stats field will be initialized. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3356) Fix initializing of stats when custom ShuffleVertexManager is used
[
https://issues.apache.org/jira/browse/TEZ-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383568#comment-15383568
]
Peter Slawski commented on TEZ-3356:
Hi [~rajesh.balamohan],
Thanks for the quick reply. The case is when parallelism for a vertex is not
yet known when ShuffleVertexManager::initialize() is called, i.e. numTasks is
set to {{-1}}. Then, the stats field is not initialized when the vertex manager
is initialized as pendingTasks() exits early as shown below.
{code:java}
int tasks = getContext().getVertexNumTasks(getContext().getVertexName());
if (tasks == pendingTasks.size() || tasks <= 0) {
return;
}
{code}
To clarify, in this case, we are using a manager plugin which extends a
ShuffleVertexManager. This manager will set the parallelism during the runtime
of the DAG, which would occur after ShuffleVertexManager::initialize() is
called for a vertex with that custom manager.
See Pig's PigGraceShuffleVertexManager which will set a vertex parallelism
during the DAG execution,
[here|https://github.com/apache/pig/blob/40300960d2edaa8097e551dd8692e29d6018ffc4/src/org/apache/pig/backend/hadoop/executionengine/tez/runtime/PigGraceShuffleVertexManager.java#L172].
When Pig submits a job with grace parallelism, the parallelism for vertices
would be {{-1}}, which is done
[here|https://github.com/apache/pig/blob/7cf1a945772f49ff620d7eab75bf2c7e635ab2ae/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java#L871].
So, in this case, ShuffleVertexManager::initialize() will not initialize the
stats field.
> Fix initializing of stats when custom ShuffleVertexManager is used
> --
>
> Key: TEZ-3356
> URL: https://issues.apache.org/jira/browse/TEZ-3356
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.4
>Reporter: Peter Slawski
>Assignee: Peter Slawski
> Attachments: TEZ-3356.1.patch
>
>
> When using a custom ShuffleVertexManager to set a vertex’s parallelism, the
> partition stats field will be left uninitialized even after the manager
> itself gets initialized. This results in a IllegalStateException to be thrown
> as the stats field will not yet be initialized when VertexManagerEvents are
> processed upon the start of the vertex. Note that these events contain
> partition sizes which are aggregated and stored in this stats field.
>
> Apache Pig’s grace auto-parallelism feature uses a custom
> ShuffleVertexManager which sets a vertex’s parallelism upon the completion of
> one of its parent’s parents. Thus, this corner case is hit and pig scripts
> with grace parallelism enabled would fail if the DAG consists of at least one
> vertex having grandparents.
>
> The fix should be straight forward. Before rather than after
> VertexManagerEvents are processed, simply update pending tasks to ensure the
> partition stats field will be initialized.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3356) Fix initializing of stats when custom ShuffleVertexManager is used
[ https://issues.apache.org/jira/browse/TEZ-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383589#comment-15383589 ] Rajesh Balamohan commented on TEZ-3356: --- Thanks for sharing the details. In this case, updatePendingTasks rearrangement in ShuffleVertexManager.onVertexStarted makes sense. +1. LGTM. > Fix initializing of stats when custom ShuffleVertexManager is used > -- > > Key: TEZ-3356 > URL: https://issues.apache.org/jira/browse/TEZ-3356 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.4 >Reporter: Peter Slawski >Assignee: Peter Slawski > Attachments: TEZ-3356.1.patch > > > When using a custom ShuffleVertexManager to set a vertex’s parallelism, the > partition stats field will be left uninitialized even after the manager > itself gets initialized. This results in a IllegalStateException to be thrown > as the stats field will not yet be initialized when VertexManagerEvents are > processed upon the start of the vertex. Note that these events contain > partition sizes which are aggregated and stored in this stats field. > > Apache Pig’s grace auto-parallelism feature uses a custom > ShuffleVertexManager which sets a vertex’s parallelism upon the completion of > one of its parent’s parents. Thus, this corner case is hit and pig scripts > with grace parallelism enabled would fail if the DAG consists of at least one > vertex having grandparents. > > The fix should be straight forward. Before rather than after > VertexManagerEvents are processed, simply update pending tasks to ensure the > partition stats field will be initialized. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3356) Fix initializing of stats when custom ShuffleVertexManager is used
[ https://issues.apache.org/jira/browse/TEZ-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383592#comment-15383592 ] Peter Slawski commented on TEZ-3356: Thank you [~rajesh.balamohan] for the review! > Fix initializing of stats when custom ShuffleVertexManager is used > -- > > Key: TEZ-3356 > URL: https://issues.apache.org/jira/browse/TEZ-3356 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.4 >Reporter: Peter Slawski >Assignee: Peter Slawski > Attachments: TEZ-3356.1.patch > > > When using a custom ShuffleVertexManager to set a vertex’s parallelism, the > partition stats field will be left uninitialized even after the manager > itself gets initialized. This results in a IllegalStateException to be thrown > as the stats field will not yet be initialized when VertexManagerEvents are > processed upon the start of the vertex. Note that these events contain > partition sizes which are aggregated and stored in this stats field. > > Apache Pig’s grace auto-parallelism feature uses a custom > ShuffleVertexManager which sets a vertex’s parallelism upon the completion of > one of its parent’s parents. Thus, this corner case is hit and pig scripts > with grace parallelism enabled would fail if the DAG consists of at least one > vertex having grandparents. > > The fix should be straight forward. Before rather than after > VertexManagerEvents are processed, simply update pending tasks to ensure the > partition stats field will be initialized. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3355) Tez Custom Shuffle Handler POC
[ https://issues.apache.org/jira/browse/TEZ-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-3355: - Attachment: TEZ-3355.1.patch > Tez Custom Shuffle Handler POC > -- > > Key: TEZ-3355 > URL: https://issues.apache.org/jira/browse/TEZ-3355 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Eagles > Attachments: TEZ-3355.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3355) Tez Custom Shuffle Handler POC
[ https://issues.apache.org/jira/browse/TEZ-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-3355: - Description: This jira is effectively a starter jira to port the mapreduce shuffle handler into the tez namespace. > Tez Custom Shuffle Handler POC > -- > > Key: TEZ-3355 > URL: https://issues.apache.org/jira/browse/TEZ-3355 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Eagles > Attachments: TEZ-3355.1.patch > > > This jira is effectively a starter jira to port the mapreduce shuffle handler > into the tez namespace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3355) Tez Custom Shuffle Handler POC
[ https://issues.apache.org/jira/browse/TEZ-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-3355: - Description: This jira is effectively a starter jira to port the mapreduce shuffle handler into the tez namespace. Answers needed for this jira to be finished: - how to package this artifact without polluting the tez runtime distribution - what directory/artifact to place this code in - how to minimize reliance on hadoop internals - what to do with previous port in tez-ext-service-tests was:This jira is effectively a starter jira to port the mapreduce shuffle handler into the tez namespace. > Tez Custom Shuffle Handler POC > -- > > Key: TEZ-3355 > URL: https://issues.apache.org/jira/browse/TEZ-3355 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Eagles > Attachments: TEZ-3355.1.patch > > > This jira is effectively a starter jira to port the mapreduce shuffle handler > into the tez namespace. > Answers needed for this jira to be finished: > - how to package this artifact without polluting the tez runtime distribution > - what directory/artifact to place this code in > - how to minimize reliance on hadoop internals > - what to do with previous port in tez-ext-service-tests -- This message was sent by Atlassian JIRA (v6.3.4#6332)
