[jira] [Assigned] (TEZ-391) SharedEdge - Support for passing same output from a vertex as input to two different vertices
[ https://issues.apache.org/jira/browse/TEZ-391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang reassigned TEZ-391: -- Assignee: Jeff Zhang SharedEdge - Support for passing same output from a vertex as input to two different vertices - Key: TEZ-391 URL: https://issues.apache.org/jira/browse/TEZ-391 Project: Apache Tez Issue Type: Sub-task Reporter: Rohini Palaniswamy Assignee: Jeff Zhang We need this for lot of usecases. For cases where multi-query is turned off and for optimizing unions. Currently those are BROADCAST or ONE-ONE edges and we write the output multiple times. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1421) MRCombiner throws NPE in MapredWordCount on master branch
[
https://issues.apache.org/jira/browse/TEZ-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14279972#comment-14279972
]
Tsuyoshi OZAWA commented on TEZ-1421:
-
Feel free to contact me if you cannot reproduce the problem. Again, thanks!
MRCombiner throws NPE in MapredWordCount on master branch
-
Key: TEZ-1421
URL: https://issues.apache.org/jira/browse/TEZ-1421
Project: Apache Tez
Issue Type: Bug
Reporter: Tsuyoshi OZAWA
Priority: Blocker
I tested MapredWordCount against 70GB generated by RandowTextWriter. When a
Combiner runs, it throws NPE. It looks setCombinerClass doesn't work
correctly.
{quote}
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.tez.mapreduce.combine.MRCombiner.runOldCombiner(MRCombiner.java:122)
at org.apache.tez.mapreduce.combine.MRCombiner.combine(MRCombiner.java:112)
at
org.apache.tez.runtime.library.common.shuffle.impl.MergeManager.runCombineProcessor(MergeManager.java:472)
at
org.apache.tez.runtime.library.common.shuffle.impl.MergeManager$InMemoryMerger.merge(MergeManager.java:605)
at
org.apache.tez.runtime.library.common.shuffle.impl.MergeThread.run(MergeThread.java:89)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-1593) Refactor PipelinedSorter to remove all MMAP based ByteBuffer references
[
https://issues.apache.org/jira/browse/TEZ-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-1593:
--
Attachment: TEZ-1593.2.patch
Refactor PipelinedSorter to remove all MMAP based ByteBuffer references
---
Key: TEZ-1593
URL: https://issues.apache.org/jira/browse/TEZ-1593
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.6.0
Reporter: Gopal V
Assignee: Gopal V
Labels: Performance
Attachments: TEZ-1593.1.patch, TEZ-1593.2-WIP.patch, TEZ-1593.2.patch
The current implementation of PipelinedSorter has a slow section which
revolves around key comparisons - this was relevant when the implementation
used direct byte buffers to back the kvbuffer.
{code}
kvbuffer.position(istart);
kvbuffer.get(ki, 0, ilen);
kvbuffer.position(jstart);
kvbuffer.get(kj, 0, jlen);
// sort by key
final int cmp = comparator.compare(ki, 0, ilen, kj, 0, jlen);
{code}
The kvbuffer.get into the arrays ki and kj are the slowest part of the
comparator operation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-1934) TestAMRecovery may fail due to the execution order is not determined
[ https://issues.apache.org/jira/browse/TEZ-1934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280116#comment-14280116 ] Jeff Zhang commented on TEZ-1934: - committed to master TestAMRecovery may fail due to the execution order is not determined - Key: TEZ-1934 URL: https://issues.apache.org/jira/browse/TEZ-1934 Project: Apache Tez Issue Type: Bug Reporter: Jeff Zhang Assignee: Jeff Zhang Attachments: TEZ-1934-1.patch task_1 is not guaranteed to been scheduled before task_0, so task_1 may finished before task_0. While in the current TestAMRecovery, the finish of task_1 is treated as the finished signal of vertex ( only 2 tasks in this vertex) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1661) LocalTaskScheduler hangs when shutdown
[
https://issues.apache.org/jira/browse/TEZ-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280117#comment-14280117
]
Jeff Zhang commented on TEZ-1661:
-
committed to master
LocalTaskScheduler hangs when shutdown
--
Key: TEZ-1661
URL: https://issues.apache.org/jira/browse/TEZ-1661
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.5.0
Environment: Local Mode
Reporter: Oleg Zhurakousky
Assignee: Jeff Zhang
Fix For: 0.7.0
Attachments: TEZ-1661-1.patch, TEZ-1661-2.patch
LocalTaskScheduler hangs on 'take' from the 'taskRequestQueue ' when
TezClient shuts down (e.g., TezClient.stop).
Below is jstack output observed when running in Tez local mode:
{code}
Thread-53 prio=5 tid=0x7fc876d8f800 nid=0xac07 runnable
[0x00011df9]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
- locked 0x0007b6ce60a0 (a java.lang.InterruptedException)
at java.lang.Throwable.init(Throwable.java:250)
at java.lang.Exception.init(Exception.java:54)
at java.lang.InterruptedException.init(InterruptedException.java:57)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at
java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535)
at
org.apache.tez.dag.app.rm.LocalTaskSchedulerService$AsyncDelegateRequestHandler.processRequest(LocalTaskSchedulerService.java:310)
at
org.apache.tez.dag.app.rm.LocalTaskSchedulerService$AsyncDelegateRequestHandler.run(LocalTaskSchedulerService.java:304)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-1934) TestAMRecovery may fail due to the execution order is not determined
[ https://issues.apache.org/jira/browse/TEZ-1934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-1934: Fix Version/s: 0.7.0 TestAMRecovery may fail due to the execution order is not determined - Key: TEZ-1934 URL: https://issues.apache.org/jira/browse/TEZ-1934 Project: Apache Tez Issue Type: Bug Reporter: Jeff Zhang Assignee: Jeff Zhang Fix For: 0.7.0 Attachments: TEZ-1934-1.patch task_1 is not guaranteed to been scheduled before task_0, so task_1 may finished before task_0. While in the current TestAMRecovery, the finish of task_1 is treated as the finished signal of vertex ( only 2 tasks in this vertex) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-1968) Tez UI - All vertices of DAG are not listed in vertices page
[ https://issues.apache.org/jira/browse/TEZ-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran reassigned TEZ-1968: - Assignee: Prakash Ramachandran Tez UI - All vertices of DAG are not listed in vertices page Key: TEZ-1968 URL: https://issues.apache.org/jira/browse/TEZ-1968 Project: Apache Tez Issue Type: Bug Reporter: Rajesh Balamohan Assignee: Prakash Ramachandran Attachments: Screen Shot 2015-01-16 at 5.11.14 AM.png, Screen Shot 2015-01-16 at 5.11.27 AM.png DAG view and vertices view can be a little misleading at times. I can see that Reducer 9 is part of the DAG and is listed in DAG view. However, it is not getting listed in Vertices page. I will attach the screen shots soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1855) Avoid scanning for previously written files within Inputs / Outputs
[ https://issues.apache.org/jira/browse/TEZ-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1855: -- Attachment: TEZ-1855.1.patch [~sseth], [~gopalv] - Please review. Avoid scanning for previously written files within Inputs / Outputs --- Key: TEZ-1855 URL: https://issues.apache.org/jira/browse/TEZ-1855 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Attachments: TEZ-1855.1.patch TezTaskOutput has a bunch of methods - getOutputFile, getOutputIndexFile, getSpillIndexFile - which are used within an Output to scan for files written earlier by the same Output. This should be avoided in favour of keeping track of previously written files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-1855) Avoid scanning for previously written files within Inputs / Outputs
[ https://issues.apache.org/jira/browse/TEZ-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan reassigned TEZ-1855: - Assignee: Rajesh Balamohan Avoid scanning for previously written files within Inputs / Outputs --- Key: TEZ-1855 URL: https://issues.apache.org/jira/browse/TEZ-1855 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Assignee: Rajesh Balamohan Attachments: TEZ-1855.1.patch TezTaskOutput has a bunch of methods - getOutputFile, getOutputIndexFile, getSpillIndexFile - which are used within an Output to scan for files written earlier by the same Output. This should be avoided in favour of keeping track of previously written files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1942) Number of tasks show in Tez UI with auto-reduce parallelism is misleading
[ https://issues.apache.org/jira/browse/TEZ-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-1942: -- Attachment: TEZ-1942.3.patch patch 3 * added test cases Number of tasks show in Tez UI with auto-reduce parallelism is misleading - Key: TEZ-1942 URL: https://issues.apache.org/jira/browse/TEZ-1942 Project: Apache Tez Issue Type: Bug Affects Versions: 0.5.2 Reporter: Rajesh Balamohan Assignee: Prakash Ramachandran Priority: Blocker Attachments: Screen Shot 2015-01-14 at 9.18.21 AM.png, Screen Shot 2015-01-14 at 9.18.54 AM.png, TEZ-1942.1.patch, TEZ-1942.2.patch, TEZ-1942.3.patch, output.json, result_with_direct_vertex.png, result_with_primary_filter.png Ran a simple hive query (with tez) and --hiveconf hive.tez.auto.reducer.parallelism=true . This internally turns on tez's auto reduce parallelism. - Job started off with 1009 reduce tasks - Tez reduces the number of reducers to 253 - Job completes successfully, but TEZ UI shows 1009 as the number of reducers (and 253 tasks as successful tasks). This can be a little misleading. I will attach the screenshots soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1968) Tez UI - All vertices of DAG are not listed in vertices page
[ https://issues.apache.org/jira/browse/TEZ-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-1968: -- Attachment: TEZ-1968.1.patch The list of vertexes are paginated. so it would appear in the next page. however the pagination is broken due to start time. fixing the issue. [~Sreenath] can you review. Tez UI - All vertices of DAG are not listed in vertices page Key: TEZ-1968 URL: https://issues.apache.org/jira/browse/TEZ-1968 Project: Apache Tez Issue Type: Bug Reporter: Rajesh Balamohan Assignee: Prakash Ramachandran Attachments: Screen Shot 2015-01-16 at 5.11.14 AM.png, Screen Shot 2015-01-16 at 5.11.27 AM.png, TEZ-1968.1.patch DAG view and vertices view can be a little misleading at times. I can see that Reducer 9 is part of the DAG and is listed in DAG view. However, it is not getting listed in Vertices page. I will attach the screen shots soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1968) Tez UI - All vertices of DAG are not listed in vertices page
[ https://issues.apache.org/jira/browse/TEZ-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-1968: -- Priority: Blocker (was: Major) Tez UI - All vertices of DAG are not listed in vertices page Key: TEZ-1968 URL: https://issues.apache.org/jira/browse/TEZ-1968 Project: Apache Tez Issue Type: Bug Reporter: Rajesh Balamohan Assignee: Prakash Ramachandran Priority: Blocker Attachments: Screen Shot 2015-01-16 at 5.11.14 AM.png, Screen Shot 2015-01-16 at 5.11.27 AM.png, TEZ-1968.1.patch DAG view and vertices view can be a little misleading at times. I can see that Reducer 9 is part of the DAG and is listed in DAG view. However, it is not getting listed in Vertices page. I will attach the screen shots soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1968) Tez UI - All vertices of DAG are not listed in vertices page
[ https://issues.apache.org/jira/browse/TEZ-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-1968: -- Target Version/s: 0.6.0 Tez UI - All vertices of DAG are not listed in vertices page Key: TEZ-1968 URL: https://issues.apache.org/jira/browse/TEZ-1968 Project: Apache Tez Issue Type: Bug Reporter: Rajesh Balamohan Assignee: Prakash Ramachandran Attachments: Screen Shot 2015-01-16 at 5.11.14 AM.png, Screen Shot 2015-01-16 at 5.11.27 AM.png, TEZ-1968.1.patch DAG view and vertices view can be a little misleading at times. I can see that Reducer 9 is part of the DAG and is listed in DAG view. However, it is not getting listed in Vertices page. I will attach the screen shots soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1942) Number of tasks show in Tez UI with auto-reduce parallelism is misleading
[
https://issues.apache.org/jira/browse/TEZ-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280317#comment-14280317
]
Hadoop QA commented on TEZ-1942:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692760/TEZ-1942.3.patch
against master revision cdf253f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 68 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/46//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/46//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/46//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/46//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/46//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/46//console
This message is automatically generated.
Number of tasks show in Tez UI with auto-reduce parallelism is misleading
-
Key: TEZ-1942
URL: https://issues.apache.org/jira/browse/TEZ-1942
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.5.2
Reporter: Rajesh Balamohan
Assignee: Prakash Ramachandran
Priority: Blocker
Attachments: Screen Shot 2015-01-14 at 9.18.21 AM.png, Screen Shot
2015-01-14 at 9.18.54 AM.png, TEZ-1942.1.patch, TEZ-1942.2.patch,
TEZ-1942.3.patch, output.json, result_with_direct_vertex.png,
result_with_primary_filter.png
Ran a simple hive query (with tez) and --hiveconf
hive.tez.auto.reducer.parallelism=true . This internally turns on tez's
auto reduce parallelism.
- Job started off with 1009 reduce tasks
- Tez reduces the number of reducers to 253
- Job completes successfully, but TEZ UI shows 1009 as the number of reducers
(and 253 tasks as successful tasks). This can be a little misleading.
I will attach the screenshots soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-1968) Tez UI - All vertices of DAG are not listed in vertices page
[
https://issues.apache.org/jira/browse/TEZ-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280424#comment-14280424
]
Hadoop QA commented on TEZ-1968:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692763/TEZ-1968.1.patch
against master revision cdf253f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 68 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/47//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/47//artifact/patchprocess/newPatchFindbugsWarningstez-runtime-internals.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/47//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/47//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/47//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/47//console
This message is automatically generated.
Tez UI - All vertices of DAG are not listed in vertices page
Key: TEZ-1968
URL: https://issues.apache.org/jira/browse/TEZ-1968
Project: Apache Tez
Issue Type: Bug
Reporter: Rajesh Balamohan
Assignee: Prakash Ramachandran
Priority: Blocker
Fix For: 0.6.0
Attachments: Screen Shot 2015-01-16 at 5.11.14 AM.png, Screen Shot
2015-01-16 at 5.11.27 AM.png, TEZ-1968.1.patch
DAG view and vertices view can be a little misleading at times. I can see
that Reducer 9 is part of the DAG and is listed in DAG view. However, it
is not getting listed in Vertices page. I will attach the screen shots soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-1879) Create local UGI instances for each task and the AM, when running in LocalMode
[ https://issues.apache.org/jira/browse/TEZ-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280439#comment-14280439 ] Hitesh Shah commented on TEZ-1879: -- +1 Create local UGI instances for each task and the AM, when running in LocalMode -- Key: TEZ-1879 URL: https://issues.apache.org/jira/browse/TEZ-1879 Project: Apache Tez Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: TEZ-1879.1.txt, TEZ-1879.2.txt, TEZ-1879.3.txt, TEZ-1879.4.txt Modifying the client UGI can cause issues when the client tries to submit another job - or has tokens already populated in the UGI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1661) LocalTaskScheduler hangs when shutdown
[
https://issues.apache.org/jira/browse/TEZ-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280507#comment-14280507
]
Oleg Zhurakousky commented on TEZ-1661:
---
Here is the code to reproduce it:
{code}
public static void main(String[] args) throws Exception {
TezClient client = TezClient.create(foo, new TezConfiguration());
client.start();
client.stop();
System.out.println(Done);
}
{code}
Make sure you run it as Java application (main) and not JUnit since it will
essentially do System.exit.
LocalTaskScheduler hangs when shutdown
--
Key: TEZ-1661
URL: https://issues.apache.org/jira/browse/TEZ-1661
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.5.0
Environment: Local Mode
Reporter: Oleg Zhurakousky
Assignee: Jeff Zhang
Fix For: 0.7.0
Attachments: TEZ-1661-1.patch, TEZ-1661-2.patch
LocalTaskScheduler hangs on 'take' from the 'taskRequestQueue ' when
TezClient shuts down (e.g., TezClient.stop).
Below is jstack output observed when running in Tez local mode:
{code}
Thread-53 prio=5 tid=0x7fc876d8f800 nid=0xac07 runnable
[0x00011df9]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
- locked 0x0007b6ce60a0 (a java.lang.InterruptedException)
at java.lang.Throwable.init(Throwable.java:250)
at java.lang.Exception.init(Exception.java:54)
at java.lang.InterruptedException.init(InterruptedException.java:57)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at
java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535)
at
org.apache.tez.dag.app.rm.LocalTaskSchedulerService$AsyncDelegateRequestHandler.processRequest(LocalTaskSchedulerService.java:310)
at
org.apache.tez.dag.app.rm.LocalTaskSchedulerService$AsyncDelegateRequestHandler.run(LocalTaskSchedulerService.java:304)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (TEZ-1661) LocalTaskScheduler hangs when shutdown
[
https://issues.apache.org/jira/browse/TEZ-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280507#comment-14280507
]
Oleg Zhurakousky edited comment on TEZ-1661 at 1/16/15 5:11 PM:
Here is the code and jstack trace to reproduce it:
{code}
public static void main(String[] args) throws Exception {
TezClient client = TezClient.create(foo, new TezConfiguration());
client.start();
client.stop();
System.out.println(Done);
}
{code}
{code}
Thread-27 prio=5 tid=0x7f7f8e99b800 nid=0x5d07 runnable
[0x000118b1e000]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
- locked 0x0007ac3c8380 (a java.lang.InterruptedException)
at java.lang.Throwable.init(Throwable.java:250)
at java.lang.Exception.init(Exception.java:54)
at java.lang.InterruptedException.init(InterruptedException.java:57)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at
java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535)
at
org.apache.tez.dag.app.rm.LocalTaskSchedulerService$AsyncDelegateRequestHandler.processRequest(LocalTaskSchedulerService.java:317)
at
org.apache.tez.dag.app.rm.LocalTaskSchedulerService$AsyncDelegateRequestHandler.run(LocalTaskSchedulerService.java:311)
at java.lang.Thread.run(Thread.java:745)
{code}
Make sure you run it as Java application (main) and not JUnit since it will
essentially do System.exit.
was (Author: ozhurakousky):
Here is the code to reproduce it:
{code}
public static void main(String[] args) throws Exception {
TezClient client = TezClient.create(foo, new TezConfiguration());
client.start();
client.stop();
System.out.println(Done);
}
{code}
Make sure you run it as Java application (main) and not JUnit since it will
essentially do System.exit.
LocalTaskScheduler hangs when shutdown
--
Key: TEZ-1661
URL: https://issues.apache.org/jira/browse/TEZ-1661
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.5.0
Environment: Local Mode
Reporter: Oleg Zhurakousky
Assignee: Jeff Zhang
Fix For: 0.7.0
Attachments: TEZ-1661-1.patch, TEZ-1661-2.patch
LocalTaskScheduler hangs on 'take' from the 'taskRequestQueue ' when
TezClient shuts down (e.g., TezClient.stop).
Below is jstack output observed when running in Tez local mode:
{code}
Thread-53 prio=5 tid=0x7fc876d8f800 nid=0xac07 runnable
[0x00011df9]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
- locked 0x0007b6ce60a0 (a java.lang.InterruptedException)
at java.lang.Throwable.init(Throwable.java:250)
at java.lang.Exception.init(Exception.java:54)
at java.lang.InterruptedException.init(InterruptedException.java:57)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at
java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535)
at
org.apache.tez.dag.app.rm.LocalTaskSchedulerService$AsyncDelegateRequestHandler.processRequest(LocalTaskSchedulerService.java:310)
at
org.apache.tez.dag.app.rm.LocalTaskSchedulerService$AsyncDelegateRequestHandler.run(LocalTaskSchedulerService.java:304)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (TEZ-1661) LocalTaskScheduler hangs when shutdown
[
https://issues.apache.org/jira/browse/TEZ-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280507#comment-14280507
]
Oleg Zhurakousky edited comment on TEZ-1661 at 1/16/15 5:38 PM:
Here is the code and jstack trace to reproduce it:
{code}
public static void main(String[] args) throws Exception {
TezClient client = TezClient.create(foo, new TezConfiguration());
client.start();
client.stop();
}
{code}
{code}
Thread-27 prio=5 tid=0x7f7f8e99b800 nid=0x5d07 runnable
[0x000118b1e000]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
- locked 0x0007ac3c8380 (a java.lang.InterruptedException)
at java.lang.Throwable.init(Throwable.java:250)
at java.lang.Exception.init(Exception.java:54)
at java.lang.InterruptedException.init(InterruptedException.java:57)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at
java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535)
at
org.apache.tez.dag.app.rm.LocalTaskSchedulerService$AsyncDelegateRequestHandler.processRequest(LocalTaskSchedulerService.java:317)
at
org.apache.tez.dag.app.rm.LocalTaskSchedulerService$AsyncDelegateRequestHandler.run(LocalTaskSchedulerService.java:311)
at java.lang.Thread.run(Thread.java:745)
{code}
Make sure you run it as Java application (main) and not JUnit since it will
essentially do System.exit.
was (Author: ozhurakousky):
Here is the code and jstack trace to reproduce it:
{code}
public static void main(String[] args) throws Exception {
TezClient client = TezClient.create(foo, new TezConfiguration());
client.start();
client.stop();
System.out.println(Done);
}
{code}
{code}
Thread-27 prio=5 tid=0x7f7f8e99b800 nid=0x5d07 runnable
[0x000118b1e000]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
- locked 0x0007ac3c8380 (a java.lang.InterruptedException)
at java.lang.Throwable.init(Throwable.java:250)
at java.lang.Exception.init(Exception.java:54)
at java.lang.InterruptedException.init(InterruptedException.java:57)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at
java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535)
at
org.apache.tez.dag.app.rm.LocalTaskSchedulerService$AsyncDelegateRequestHandler.processRequest(LocalTaskSchedulerService.java:317)
at
org.apache.tez.dag.app.rm.LocalTaskSchedulerService$AsyncDelegateRequestHandler.run(LocalTaskSchedulerService.java:311)
at java.lang.Thread.run(Thread.java:745)
{code}
Make sure you run it as Java application (main) and not JUnit since it will
essentially do System.exit.
LocalTaskScheduler hangs when shutdown
--
Key: TEZ-1661
URL: https://issues.apache.org/jira/browse/TEZ-1661
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.5.0
Environment: Local Mode
Reporter: Oleg Zhurakousky
Assignee: Jeff Zhang
Fix For: 0.7.0
Attachments: TEZ-1661-1.patch, TEZ-1661-2.patch
LocalTaskScheduler hangs on 'take' from the 'taskRequestQueue ' when
TezClient shuts down (e.g., TezClient.stop).
Below is jstack output observed when running in Tez local mode:
{code}
Thread-53 prio=5 tid=0x7fc876d8f800 nid=0xac07 runnable
[0x00011df9]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
- locked 0x0007b6ce60a0 (a java.lang.InterruptedException)
at java.lang.Throwable.init(Throwable.java:250)
at java.lang.Exception.init(Exception.java:54)
at java.lang.InterruptedException.init(InterruptedException.java:57)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
at
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
at
java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535)
at
[jira] [Commented] (TEZ-1942) Number of tasks show in Tez UI with auto-reduce parallelism is misleading
[ https://issues.apache.org/jira/browse/TEZ-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280625#comment-14280625 ] Hitesh Shah commented on TEZ-1942: -- +1. Committing shortly. Number of tasks show in Tez UI with auto-reduce parallelism is misleading - Key: TEZ-1942 URL: https://issues.apache.org/jira/browse/TEZ-1942 Project: Apache Tez Issue Type: Bug Affects Versions: 0.5.2 Reporter: Rajesh Balamohan Assignee: Prakash Ramachandran Priority: Blocker Attachments: Screen Shot 2015-01-14 at 9.18.21 AM.png, Screen Shot 2015-01-14 at 9.18.54 AM.png, TEZ-1942.1.patch, TEZ-1942.2.patch, TEZ-1942.3.patch, output.json, result_with_direct_vertex.png, result_with_primary_filter.png Ran a simple hive query (with tez) and --hiveconf hive.tez.auto.reducer.parallelism=true . This internally turns on tez's auto reduce parallelism. - Job started off with 1009 reduce tasks - Tez reduces the number of reducers to 253 - Job completes successfully, but TEZ UI shows 1009 as the number of reducers (and 253 tasks as successful tasks). This can be a little misleading. I will attach the screenshots soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1942) Number of tasks show in Tez UI with auto-reduce parallelism is misleading
[ https://issues.apache.org/jira/browse/TEZ-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280642#comment-14280642 ] Hitesh Shah commented on TEZ-1942: -- Committed to master and branch 0.6. [~pramachandran] This will need a re-based patch for branch 0.5 - would you mind creating one? Number of tasks show in Tez UI with auto-reduce parallelism is misleading - Key: TEZ-1942 URL: https://issues.apache.org/jira/browse/TEZ-1942 Project: Apache Tez Issue Type: Bug Affects Versions: 0.5.2 Reporter: Rajesh Balamohan Assignee: Prakash Ramachandran Priority: Blocker Attachments: Screen Shot 2015-01-14 at 9.18.21 AM.png, Screen Shot 2015-01-14 at 9.18.54 AM.png, TEZ-1942.1.patch, TEZ-1942.2.patch, TEZ-1942.3.patch, output.json, result_with_direct_vertex.png, result_with_primary_filter.png Ran a simple hive query (with tez) and --hiveconf hive.tez.auto.reducer.parallelism=true . This internally turns on tez's auto reduce parallelism. - Job started off with 1009 reduce tasks - Tez reduces the number of reducers to 253 - Job completes successfully, but TEZ UI shows 1009 as the number of reducers (and 253 tasks as successful tasks). This can be a little misleading. I will attach the screenshots soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1879) Create local UGI instances for each task and the AM, when running in LocalMode
[ https://issues.apache.org/jira/browse/TEZ-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-1879: Attachment: TEZ-1879.4.1.txt Thanks for the reviews. Attaching commit patch with amTokens renamed, and committing. Create local UGI instances for each task and the AM, when running in LocalMode -- Key: TEZ-1879 URL: https://issues.apache.org/jira/browse/TEZ-1879 Project: Apache Tez Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: TEZ-1879.1.txt, TEZ-1879.2.txt, TEZ-1879.3.txt, TEZ-1879.4.1.txt, TEZ-1879.4.txt Modifying the client UGI can cause issues when the client tries to submit another job - or has tokens already populated in the UGI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1941) Memory provided by *Context.getAvailableMemory needs to be setup explicitly
[ https://issues.apache.org/jira/browse/TEZ-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-1941: Attachment: TEZ-1941.2.txt The ExecutionContext is supposed to be shared by multiple tasks within a JVM. Maybe it needs a rename to be more clear. Updated the javadocs for ExecutionContext. Since that's per jvm - settting memory in it isn't an option, since it needs to be task specific. Memory provided by *Context.getAvailableMemory needs to be setup explicitly --- Key: TEZ-1941 URL: https://issues.apache.org/jira/browse/TEZ-1941 Project: Apache Tez Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: TEZ-1941.1.txt, TEZ-1941.2.txt *Contexts.getAvailableMemory rely on Runtime..getMaxMemory(). This doesn't work for memory scaling if multiple tasks are running within a JVM. Container sizes (sent over RPC) can be used for setting up this value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1969) Stop the DAGAppMaster when a local mode client is stopped
Siddharth Seth created TEZ-1969: --- Summary: Stop the DAGAppMaster when a local mode client is stopped Key: TEZ-1969 URL: https://issues.apache.org/jira/browse/TEZ-1969 Project: Apache Tez Issue Type: Sub-task Reporter: Siddharth Seth https://issues.apache.org/jira/browse/TEZ-1661?focusedCommentId=14275366page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14275366 Running multiple local clients in a single JVM will leak DAGAppMaster and related threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1931) Publish tez version info to Timeline
[ https://issues.apache.org/jira/browse/TEZ-1931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1931: - Attachment: TEZ-1931.1.branch-0.5.patch Seems like a safe enough and useful option in case anyone wants to trying running the UI against a 0.5 release. [~jeagles] [~pramachandran] review please. Publish tez version info to Timeline Key: TEZ-1931 URL: https://issues.apache.org/jira/browse/TEZ-1931 Project: Apache Tez Issue Type: Bug Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Critical Fix For: 0.6.0 Attachments: TEZ-1931.1.branch-0.5.patch, TEZ-1931.1.patch We are not publishing any version info to Timeline. This will be useful to compare different dags/apps over time and also to catch issues if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1941) Memory provided by *Context.getAvailableMemory needs to be setup explicitly
[ https://issues.apache.org/jira/browse/TEZ-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280752#comment-14280752 ] Hitesh Shah commented on TEZ-1941: -- +1 ( pending test patch results ). Memory provided by *Context.getAvailableMemory needs to be setup explicitly --- Key: TEZ-1941 URL: https://issues.apache.org/jira/browse/TEZ-1941 Project: Apache Tez Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: TEZ-1941.1.txt, TEZ-1941.2.txt *Contexts.getAvailableMemory rely on Runtime..getMaxMemory(). This doesn't work for memory scaling if multiple tasks are running within a JVM. Container sizes (sent over RPC) can be used for setting up this value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1941) Memory provided by *Context.getAvailableMemory needs to be setup explicitly
[
https://issues.apache.org/jira/browse/TEZ-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280770#comment-14280770
]
Hadoop QA commented on TEZ-1941:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692817/TEZ-1941.2.txt
against master revision ea46f45.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 64 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/48//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/48//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/48//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/48//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/48//console
This message is automatically generated.
Memory provided by *Context.getAvailableMemory needs to be setup explicitly
---
Key: TEZ-1941
URL: https://issues.apache.org/jira/browse/TEZ-1941
Project: Apache Tez
Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Attachments: TEZ-1941.1.txt, TEZ-1941.2.txt
*Contexts.getAvailableMemory rely on Runtime..getMaxMemory(). This doesn't
work for memory scaling if multiple tasks are running within a JVM.
Container sizes (sent over RPC) can be used for setting up this value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-1949) Whitelist TEZ_RUNTIME_OPTIMIZE_SHARED_FETCH for broadcast edges
[ https://issues.apache.org/jira/browse/TEZ-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated TEZ-1949: - Attachment: TEZ-1949.3.patch Added the extra test. Whitelist TEZ_RUNTIME_OPTIMIZE_SHARED_FETCH for broadcast edges --- Key: TEZ-1949 URL: https://issues.apache.org/jira/browse/TEZ-1949 Project: Apache Tez Issue Type: Bug Affects Versions: 0.6.0, 0.5.2, 0.7.0 Reporter: Gopal V Assignee: Gopal V Priority: Critical Attachments: TEZ-1949.1.patch, TEZ-1949.2.patch, TEZ-1949.3.patch Tez configuration whitelisting is missing TEZ_RUNTIME_OPTIMIZE_SHARED_FETCH for broadcast edges (UnorderedKVInput). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1941) Memory provided by *Context.getAvailableMemory needs to be setup explicitly
[ https://issues.apache.org/jira/browse/TEZ-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280784#comment-14280784 ] Siddharth Seth commented on TEZ-1941: - Thanks for the review. Committing. Memory provided by *Context.getAvailableMemory needs to be setup explicitly --- Key: TEZ-1941 URL: https://issues.apache.org/jira/browse/TEZ-1941 Project: Apache Tez Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: TEZ-1941.1.txt, TEZ-1941.2.txt *Contexts.getAvailableMemory rely on Runtime..getMaxMemory(). This doesn't work for memory scaling if multiple tasks are running within a JVM. Container sizes (sent over RPC) can be used for setting up this value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1949) Whitelist TEZ_RUNTIME_OPTIMIZE_SHARED_FETCH for broadcast edges
[
https://issues.apache.org/jira/browse/TEZ-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280958#comment-14280958
]
Hadoop QA commented on TEZ-1949:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692830/TEZ-1949.3.patch
against master revision 4974fb2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 64 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/49//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/49//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/49//artifact/patchprocess/newPatchFindbugsWarningstez-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/49//artifact/patchprocess/newPatchFindbugsWarningstez-mapreduce.html
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/49//console
This message is automatically generated.
Whitelist TEZ_RUNTIME_OPTIMIZE_SHARED_FETCH for broadcast edges
---
Key: TEZ-1949
URL: https://issues.apache.org/jira/browse/TEZ-1949
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.6.0, 0.5.2, 0.7.0
Reporter: Gopal V
Assignee: Gopal V
Priority: Critical
Fix For: 0.7.0, 0.5.4, 0.6.1
Attachments: TEZ-1949.1.patch, TEZ-1949.2.patch, TEZ-1949.3.patch
Tez configuration whitelisting is missing TEZ_RUNTIME_OPTIMIZE_SHARED_FETCH
for broadcast edges (UnorderedKVInput).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (TEZ-1970) Fix javadoc warnings in code
Hitesh Shah created TEZ-1970: Summary: Fix javadoc warnings in code Key: TEZ-1970 URL: https://issues.apache.org/jira/browse/TEZ-1970 Project: Apache Tez Issue Type: Bug Reporter: Hitesh Shah test-patch reports 3 existing javadoc warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-1970) Fix javadoc warnings in code
[ https://issues.apache.org/jira/browse/TEZ-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280981#comment-14280981 ] Hitesh Shah edited comment on TEZ-1970 at 1/16/15 11:20 PM: [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/tez-tests/src/main/java/org/apache/tez/mapreduce/examples/RPCLoadGen.java:51: warning: IOUtils is internal proprietary API and may be removed in a future release [WARNING] import sun.misc.IOUtils; [WARNING] ^ [WARNING] Javadoc Warnings [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/tez-examples/src/main/java/org/apache/tez/examples/SortMergeJoinExample.java:145: warning - @return tag has no arguments. [WARNING] Javadoc Warnings [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/utils/FastByteComparisons.java:26: warning: Unsafe is internal proprietary API and may be removed in a future release [WARNING] import sun.misc.Unsafe; [WARNING] ^ was (Author: hitesh): [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/tez-tests/src/main/java/org/apache/tez/mapreduce/examples/RPCLoadGen.java:51: warning: IOUtils is internal proprietary API and may be removed in a future release [WARNING] Javadoc Warnings [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/tez-examples/src/main/java/org/apache/tez/examples/SortMergeJoinExample.java:145: warning - @return tag has no arguments. [WARNING] Javadoc Warnings [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/utils/FastByteComparisons.java:26: warning: Unsafe is internal proprietary API and may be removed in a future release [WARNING] import sun.misc.Unsafe; [WARNING] ^ Fix javadoc warnings in code Key: TEZ-1970 URL: https://issues.apache.org/jira/browse/TEZ-1970 Project: Apache Tez Issue Type: Bug Reporter: Hitesh Shah test-patch reports 3 existing javadoc warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1970) Fix javadoc warnings in code
[ https://issues.apache.org/jira/browse/TEZ-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1970: - Target Version/s: 0.6.1 Fix javadoc warnings in code Key: TEZ-1970 URL: https://issues.apache.org/jira/browse/TEZ-1970 Project: Apache Tez Issue Type: Bug Reporter: Hitesh Shah test-patch reports 3 existing javadoc warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1970) Fix javadoc warnings in code
[ https://issues.apache.org/jira/browse/TEZ-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280981#comment-14280981 ] Hitesh Shah commented on TEZ-1970: -- [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/tez-tests/src/main/java/org/apache/tez/mapreduce/examples/RPCLoadGen.java:51: warning: IOUtils is internal proprietary API and may be removed in a future release [WARNING] Javadoc Warnings [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/tez-examples/src/main/java/org/apache/tez/examples/SortMergeJoinExample.java:145: warning - @return tag has no arguments. [WARNING] Javadoc Warnings [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/utils/FastByteComparisons.java:26: warning: Unsafe is internal proprietary API and may be removed in a future release [WARNING] import sun.misc.Unsafe; [WARNING] ^ Fix javadoc warnings in code Key: TEZ-1970 URL: https://issues.apache.org/jira/browse/TEZ-1970 Project: Apache Tez Issue Type: Bug Reporter: Hitesh Shah test-patch reports 3 existing javadoc warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-1971) test-patch script improvements to reduce email volume
Hitesh Shah created TEZ-1971: Summary: test-patch script improvements to reduce email volume Key: TEZ-1971 URL: https://issues.apache.org/jira/browse/TEZ-1971 Project: Apache Tez Issue Type: Bug Reporter: Hitesh Shah Today, there are 2 emails generated: - one by jenkins itself when the test patch job completes with the appropriate failed/success info - second by jira for the comment added by the jenkins job It will be useful to change the test patch script and build job such that the job only fails if it could not upload its comments to jira ( and/or other general runtime failures ). This will allow devs to monitor for problems in the test patch framework itself by watching out for job failures. The actual test patch output is already being published to jira and jira comments are propagated to issues@ in any case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1593) Refactor PipelinedSorter to remove all MMAP based ByteBuffer references
[
https://issues.apache.org/jira/browse/TEZ-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281032#comment-14281032
]
Gopal V commented on TEZ-1593:
--
+1.
This made the sorted data movement scale tests go from 23mins to 21mins.
Refactor PipelinedSorter to remove all MMAP based ByteBuffer references
---
Key: TEZ-1593
URL: https://issues.apache.org/jira/browse/TEZ-1593
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.6.0
Reporter: Gopal V
Assignee: Gopal V
Labels: Performance
Attachments: TEZ-1593.1.patch, TEZ-1593.2-WIP.patch, TEZ-1593.2.patch
The current implementation of PipelinedSorter has a slow section which
revolves around key comparisons - this was relevant when the implementation
used direct byte buffers to back the kvbuffer.
{code}
kvbuffer.position(istart);
kvbuffer.get(ki, 0, ilen);
kvbuffer.position(jstart);
kvbuffer.get(kj, 0, jlen);
// sort by key
final int cmp = comparator.compare(ki, 0, ilen, kj, 0, jlen);
{code}
The kvbuffer.get into the arrays ki and kj are the slowest part of the
comparator operation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-1593) Refactor PipelinedSorter to remove all MMAP based ByteBuffer references
[
https://issues.apache.org/jira/browse/TEZ-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated TEZ-1593:
-
Assignee: Rajesh Balamohan (was: Gopal V)
Refactor PipelinedSorter to remove all MMAP based ByteBuffer references
---
Key: TEZ-1593
URL: https://issues.apache.org/jira/browse/TEZ-1593
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.6.0
Reporter: Gopal V
Assignee: Rajesh Balamohan
Labels: Performance
Attachments: TEZ-1593.1.patch, TEZ-1593.2-WIP.patch, TEZ-1593.2.patch
The current implementation of PipelinedSorter has a slow section which
revolves around key comparisons - this was relevant when the implementation
used direct byte buffers to back the kvbuffer.
{code}
kvbuffer.position(istart);
kvbuffer.get(ki, 0, ilen);
kvbuffer.position(jstart);
kvbuffer.get(kj, 0, jlen);
// sort by key
final int cmp = comparator.compare(ki, 0, ilen, kj, 0, jlen);
{code}
The kvbuffer.get into the arrays ki and kj are the slowest part of the
comparator operation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
