[jira] [Commented] (TEZ-2716) DefaultSorter.isRleNeeded not thread safe

2015-09-30 Thread Rajesh Balamohan (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938936#comment-14938936
 ] 

Rajesh Balamohan commented on TEZ-2716:
---

Thanks [~jeagles]. Committed to branch-0.6.
>>>
commit 3875a04093999463262ed003716c9e5c826bd8df
>>>

> DefaultSorter.isRleNeeded not thread safe
> -
>
> Key: TEZ-2716
> URL: https://issues.apache.org/jira/browse/TEZ-2716
> Project: Apache Tez
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Siddharth Seth
>Assignee: Rajesh Balamohan
> Fix For: 0.7.1, 0.6.3, 0.8.1
>
> Attachments: TEZ-2716.1.patch, TEZ-2716.2.patch, 
> TEZ-2716.branch-0.6-and-0.5.patch
>
>
> TEZ-1997.
> Should be targeted at the same set of versions that TEZ-1997 goes into.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2716) DefaultSorter.isRleNeeded not thread safe

2015-09-30 Thread Jonathan Eagles (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938845#comment-14938845
 ] 

Jonathan Eagles commented on TEZ-2716:
--

This is a good fit for 0.6 line. Thanks, [~rajesh.balamohan] and [~sseth]. Feel 
free to check this into the 0.6 branch.

> DefaultSorter.isRleNeeded not thread safe
> -
>
> Key: TEZ-2716
> URL: https://issues.apache.org/jira/browse/TEZ-2716
> Project: Apache Tez
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Siddharth Seth
>Assignee: Rajesh Balamohan
> Fix For: 0.7.1, 0.8.1
>
> Attachments: TEZ-2716.1.patch, TEZ-2716.2.patch, 
> TEZ-2716.branch-0.6-and-0.5.patch
>
>
> TEZ-1997.
> Should be targeted at the same set of versions that TEZ-1997 goes into.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2716) DefaultSorter.isRleNeeded not thread safe

2015-09-29 Thread Rajesh Balamohan (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936097#comment-14936097
 ] 

Rajesh Balamohan commented on TEZ-2716:
---

It does not cause task failures. During spills, DefaultSorter determines 
whether RLE is needed or not based on the total number of keys gathered and the 
number of same keys seen so far.  These counters are updated in main thread.  
In spill thread, these counters would not be accurately visible without proper 
synchronization. This could cause invalid computation for isRLENeeded due to 
thread safety issues. Patch tries to address this scenario.

> DefaultSorter.isRleNeeded not thread safe
> -
>
> Key: TEZ-2716
> URL: https://issues.apache.org/jira/browse/TEZ-2716
> Project: Apache Tez
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Siddharth Seth
>Assignee: Rajesh Balamohan
> Fix For: 0.7.1, 0.8.1
>
> Attachments: TEZ-2716.1.patch, TEZ-2716.2.patch, 
> TEZ-2716.branch-0.6-and-0.5.patch
>
>
> TEZ-1997.
> Should be targeted at the same set of versions that TEZ-1997 goes into.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2716) DefaultSorter.isRleNeeded not thread safe

2015-09-29 Thread Jonathan Eagles (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935678#comment-14935678
 ] 

Jonathan Eagles commented on TEZ-2716:
--

[~rajesh.balamohan], I don't have context on the severity of this issue. Does 
this cause task failures in certain conditions?

> DefaultSorter.isRleNeeded not thread safe
> -
>
> Key: TEZ-2716
> URL: https://issues.apache.org/jira/browse/TEZ-2716
> Project: Apache Tez
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Siddharth Seth
>Assignee: Rajesh Balamohan
> Fix For: 0.7.1, 0.8.1
>
> Attachments: TEZ-2716.1.patch, TEZ-2716.2.patch, 
> TEZ-2716.branch-0.6-and-0.5.patch
>
>
> TEZ-1997.
> Should be targeted at the same set of versions that TEZ-1997 goes into.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2716) DefaultSorter.isRleNeeded not thread safe

2015-09-22 Thread Siddharth Seth (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14903222#comment-14903222
 ] 

Siddharth Seth commented on TEZ-2716:
-

+1. Looks good.

> DefaultSorter.isRleNeeded not thread safe
> -
>
> Key: TEZ-2716
> URL: https://issues.apache.org/jira/browse/TEZ-2716
> Project: Apache Tez
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Siddharth Seth
> Attachments: TEZ-2716.1.patch, TEZ-2716.2.patch
>
>
> TEZ-1997.
> Should be targeted at the same set of versions that TEZ-1997 goes into.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2716) DefaultSorter.isRleNeeded not thread safe

2015-09-22 Thread TezQA (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902278#comment-14902278
 ] 

TezQA commented on TEZ-2716:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12761594/TEZ-2716.2.patch
  against master revision 0aa31af.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1165//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1165//console

This message is automatically generated.

> DefaultSorter.isRleNeeded not thread safe
> -
>
> Key: TEZ-2716
> URL: https://issues.apache.org/jira/browse/TEZ-2716
> Project: Apache Tez
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Siddharth Seth
> Attachments: TEZ-2716.1.patch, TEZ-2716.2.patch
>
>
> TEZ-1997.
> Should be targeted at the same set of versions that TEZ-1997 goes into.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2716) DefaultSorter.isRleNeeded not thread safe

2015-09-21 Thread Siddharth Seth (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902024#comment-14902024
 ] 

Siddharth Seth commented on TEZ-2716:
-

- Instead of calling sortAndSpill in a synchronzied block, it would be better 
to only read the value of totalKeys inside a sync block, and then call 
sortAndSpill outside the block.
- Does totalKeysProcessed in SpillThread need to be volatile. (Is set by the 
main thread and read in the spill thread)
- The same handling required for sameKey ? I don't think this is modified under 
any lock.

> DefaultSorter.isRleNeeded not thread safe
> -
>
> Key: TEZ-2716
> URL: https://issues.apache.org/jira/browse/TEZ-2716
> Project: Apache Tez
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Siddharth Seth
> Attachments: TEZ-2716.1.patch
>
>
> TEZ-1997.
> Should be targeted at the same set of versions that TEZ-1997 goes into.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TEZ-2716) DefaultSorter.isRleNeeded not thread safe

2015-09-21 Thread TezQA (JIRA) 

[ 
https://issues.apache.org/jira/browse/TEZ-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901806#comment-14901806
 ] 

TezQA commented on TEZ-2716:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment
  http://issues.apache.org/jira/secure/attachment/12761543/TEZ-2716.1.patch
  against master revision 0aa31af.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 3.0.1) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in :
   org.apache.tez.test.TestFaultTolerance

Test results: 
https://builds.apache.org/job/PreCommit-TEZ-Build/1162//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1162//console

This message is automatically generated.

> DefaultSorter.isRleNeeded not thread safe
> -
>
> Key: TEZ-2716
> URL: https://issues.apache.org/jira/browse/TEZ-2716
> Project: Apache Tez
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Siddharth Seth
> Attachments: TEZ-2716.1.patch
>
>
> TEZ-1997.
> Should be targeted at the same set of versions that TEZ-1997 goes into.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)