[jira] [Updated] (TEZ-3673) Allocate smaller buffers in UnorderedPartitionedKVWriter

2017-05-03 Thread Harish Jaiprakash (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3673: --- Attachment: TEZ-3673.05.patch Using remaining memory as last buffer when leftover size > 50% of

[jira] [Resolved] (TEZ-3710) InMemoryReader has a lock per-byte

2017-05-03 Thread Gopal V (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V resolved TEZ-3710. -- Resolution: Duplicate > InMemoryReader has a lock per-byte > -- > >

[jira] [Assigned] (TEZ-3710) InMemoryReader has a lock per-byte

2017-05-03 Thread Gopal V (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned TEZ-3710: Assignee: Gopal V > InMemoryReader has a lock per-byte > -- > >

[jira] [Created] (TEZ-3710) InMemoryReader has a lock per-byte

2017-05-03 Thread Gopal V (JIRA)
Gopal V created TEZ-3710: Summary: InMemoryReader has a lock per-byte Key: TEZ-3710 URL: https://issues.apache.org/jira/browse/TEZ-3710 Project: Apache Tez Issue Type: Bug Reporter: Gopal

[jira] [Updated] (TEZ-3710) InMemoryReader has a lock per-byte

2017-05-03 Thread Gopal V (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated TEZ-3710: - Description: In ByteArrayDataInput - the super read() call is synchronized. {code} @Override public byte re

[jira] [Commented] (TEZ-3691) Setup fetchers to use shared executor.

2017-05-03 Thread TezQA (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996172#comment-15996172 ] TezQA commented on TEZ-3691: {color:red}-1 overall{color}. Here are the results of testing the

Failed: TEZ-3691 PreCommit Build #2414

2017-05-03 Thread Apache Jenkins Server
Jira: https://issues.apache.org/jira/browse/TEZ-3691 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2414/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated

[jira] [Updated] (TEZ-3691) Setup fetchers to use shared executor.

2017-05-03 Thread Harish Jaiprakash (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3691: --- Attachment: TEZ-3691.04.patch Fixed merge conflicts - rebased to master. > Setup fetchers to us

[jira] [Updated] (TEZ-3691) Setup fetchers to use shared executor.

2017-05-03 Thread Harish Jaiprakash (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3691: --- Attachment: TEZ-3691.03.patch * Adding configuration to enable use of shared executor service. *

[jira] [Updated] (TEZ-3691) Setup fetchers to use shared executor.

2017-05-03 Thread Harish Jaiprakash (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3691: --- Summary: Setup fetchers to use shared executor. (was: Move ShuffleScheduler to use the framewor

[jira] [Updated] (TEZ-3691) Setup fetchers to use shared executor.

2017-05-03 Thread Harish Jaiprakash (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3691: --- Description: Change fetchers in ShuffleScheduler & ShuffleManger to use the shared executor serv

[jira] [Commented] (TEZ-3673) Allocate smaller buffers in UnorderedPartitionedKVWriter

2017-05-03 Thread Harish Jaiprakash (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996124#comment-15996124 ] Harish Jaiprakash commented on TEZ-3673: I did this because if we are going to use r

[jira] [Commented] (TEZ-3697) Adding #output_record in vertex manager event payload

2017-05-03 Thread TezQA (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996061#comment-15996061 ] TezQA commented on TEZ-3697: {color:green}+1 overall{color}. Here are the results of testing th

Success: TEZ-3697 PreCommit Build #2413

2017-05-03 Thread Apache Jenkins Server
Jira: https://issues.apache.org/jira/browse/TEZ-3697 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2413/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated

[jira] [Comment Edited] (TEZ-3709) TezMerger is slow for high number of segments

2017-05-03 Thread Jonathan Eagles (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995850#comment-15995850 ] Jonathan Eagles edited comment on TEZ-3709 at 5/4/17 1:32 AM: --

[jira] [Updated] (TEZ-3697) Adding #output_record in vertex manager event payload

2017-05-03 Thread Zhiyuan Yang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-3697: -- Attachment: TEZ-3697.2.patch Upload new patch that adds test and fixes missing large record count. Please

[jira] [Commented] (TEZ-3709) TezMerger is slow for high number of segments

2017-05-03 Thread TezQA (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995942#comment-15995942 ] TezQA commented on TEZ-3709: {color:red}-1 overall{color}. Here are the results of testing the

Failed: TEZ-3709 PreCommit Build #2412

2017-05-03 Thread Apache Jenkins Server
Jira: https://issues.apache.org/jira/browse/TEZ-3709 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2412/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated

[jira] [Commented] (TEZ-3709) TezMerger is slow for high number of segments

2017-05-03 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995920#comment-15995920 ] Rajesh Balamohan commented on TEZ-3709: --- Thanks for sharing the patch [~jeagles]. Patc

[jira] [Commented] (TEZ-3696) Jobs can hang when both concurrency and speculation are enabled

2017-05-03 Thread TezQA (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995911#comment-15995911 ] TezQA commented on TEZ-3696: {color:red}-1 overall{color}. Here are the results of testing the

Failed: TEZ-3696 PreCommit Build #2411

2017-05-03 Thread Apache Jenkins Server
Jira: https://issues.apache.org/jira/browse/TEZ-3696 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2411/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated

[jira] [Commented] (TEZ-3709) TezMerger is slow for high number of segments

2017-05-03 Thread Jonathan Eagles (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995850#comment-15995850 ] Jonathan Eagles commented on TEZ-3709: -- Example performance gains | task attempt | dur

[jira] [Updated] (TEZ-3709) TezMerger is slow for high number of segments

2017-05-03 Thread Jonathan Eagles (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-3709: - Attachment: TEZ-3709.1.patch \\cc [~gopalv], [~rajesh.balamohan], [~jlowe] > TezMerger is slow for

[jira] [Updated] (TEZ-3709) TezMerger is slow for high number of segments

2017-05-03 Thread Jonathan Eagles (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-3709: - Labels: performance (was: ) > TezMerger is slow for high number of segments > -

[jira] [Created] (TEZ-3709) TezMerger is slow for high number of segments

2017-05-03 Thread Jonathan Eagles (JIRA)
Jonathan Eagles created TEZ-3709: Summary: TezMerger is slow for high number of segments Key: TEZ-3709 URL: https://issues.apache.org/jira/browse/TEZ-3709 Project: Apache Tez Issue Type: Bug

[jira] [Updated] (TEZ-3696) Jobs can hang when both concurrency and speculation are enabled

2017-05-03 Thread Eric Badger (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated TEZ-3696: - Attachment: TEZ-3696.004.patch bq. pendingAttempt can use TaskAttemptId as a key instead of TaskAttempt. Up

[jira] [Commented] (TEZ-3708) Improve parallelism and auto grouping of unpartitioned cartesian product

2017-05-03 Thread Zhiyuan Yang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995212#comment-15995212 ] Zhiyuan Yang commented on TEZ-3708: --- Build failure is caused by lack of TEZ-3697. > Impro

[jira] [Commented] (TEZ-3708) Improve parallelism and auto grouping of unpartitioned cartesian product

2017-05-03 Thread TezQA (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995137#comment-15995137 ] TezQA commented on TEZ-3708: {color:red}-1 overall{color}. Here are the results of testing the

Failed: TEZ-3708 PreCommit Build #2410

2017-05-03 Thread Apache Jenkins Server
Jira: https://issues.apache.org/jira/browse/TEZ-3708 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2410/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated

[jira] [Updated] (TEZ-3708) Improve parallelism and auto grouping of unpartitioned cartesian product

2017-05-03 Thread Zhiyuan Yang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-3708: -- Attachment: TEZ-3708.1.patch Upload the patch of improved unpartitioned cartesian product edge. Key things

[jira] [Updated] (TEZ-3708) Improve parallelism and auto grouping of unpartitioned cartesian product

2017-05-03 Thread Zhiyuan Yang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-3708: -- Summary: Improve parallelism and auto grouping of unpartitioned cartesian product (was: Arbitrary paralle

[jira] [Created] (TEZ-3708) Arbitrary parallelism for unpartitioned cartesian product regardless of # src tasks

2017-05-03 Thread Zhiyuan Yang (JIRA)
Zhiyuan Yang created TEZ-3708: - Summary: Arbitrary parallelism for unpartitioned cartesian product regardless of # src tasks Key: TEZ-3708 URL: https://issues.apache.org/jira/browse/TEZ-3708 Project: Apac

[jira] [Updated] (TEZ-3705) Modify DeletionTracker and deletion threads to be initialized only if enabled for tez_shuffle

2017-05-03 Thread Kuhu Shukla (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated TEZ-3705: - Attachment: TEZ-3705.006.patch Thanks [~sseth]. You are right, I have taken out the conf check from dagComp

[jira] [Updated] (TEZ-3705) Modify DeletionTracker and deletion threads to be initialized only if enabled for tez_shuffle

2017-05-03 Thread Kuhu Shukla (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated TEZ-3705: - Attachment: TEZ-3705.005.patch > Modify DeletionTracker and deletion threads to be initialized only if enabl