[jira] [Assigned] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path

2018-05-16 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-18866:
---

Assignee: Sergey Shelukhin

> Semijoin: Implement a Long -> Hash64 vector fast-path
> -
>
> Key: HIVE-18866
> URL: https://issues.apache.org/jira/browse/HIVE-18866
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: performance
> Attachments: 0001-hash64-WIP.patch, HIVE-18866.patch, 
> perf-hash64-long.png
>
>
> A significant amount of CPU is wasted with JMM restrictions on byte[] arrays.
> To transform from one Long -> another Long, this goes into a byte[] array, 
> which shows up as a hotspot.
> !perf-hash64-long.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path

2018-05-16 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16478454#comment-16478454
 ] 

Sergey Shelukhin commented on HIVE-18866:
-

Assigned to me for now for convenience of attaching files and pressing buttons, 
most of the patch is still [~gopalv]'s

> Semijoin: Implement a Long -> Hash64 vector fast-path
> -
>
> Key: HIVE-18866
> URL: https://issues.apache.org/jira/browse/HIVE-18866
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: performance
> Attachments: 0001-hash64-WIP.patch, HIVE-18866.patch, 
> perf-hash64-long.png
>
>
> A significant amount of CPU is wasted with JMM restrictions on byte[] arrays.
> To transform from one Long -> another Long, this goes into a byte[] array, 
> which shows up as a hotspot.
> !perf-hash64-long.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path

2018-05-16 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18866:

Status: Patch Available  (was: Open)

[~prasanth_j] can you review?

> Semijoin: Implement a Long -> Hash64 vector fast-path
> -
>
> Key: HIVE-18866
> URL: https://issues.apache.org/jira/browse/HIVE-18866
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: performance
> Attachments: 0001-hash64-WIP.patch, HIVE-18866.patch, 
> perf-hash64-long.png
>
>
> A significant amount of CPU is wasted with JMM restrictions on byte[] arrays.
> To transform from one Long -> another Long, this goes into a byte[] array, 
> which shows up as a hotspot.
> !perf-hash64-long.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path

2018-05-16 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16478452#comment-16478452
 ] 

Sergey Shelukhin commented on HIVE-18866:
-

Added a test and fixed the short impl that was wrong for some reason (even 
after fixing 32 to 16). 

> Semijoin: Implement a Long -> Hash64 vector fast-path
> -
>
> Key: HIVE-18866
> URL: https://issues.apache.org/jira/browse/HIVE-18866
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: performance
> Attachments: 0001-hash64-WIP.patch, HIVE-18866.patch, 
> perf-hash64-long.png
>
>
> A significant amount of CPU is wasted with JMM restrictions on byte[] arrays.
> To transform from one Long -> another Long, this goes into a byte[] array, 
> which shows up as a hotspot.
> !perf-hash64-long.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19578) HLL merges tempList on every add

2018-05-17 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479566#comment-16479566
 ] 

Sergey Shelukhin commented on HIVE-19578:
-

So the temp list is useless? I wonder if it would still provide perf 
improvement, esp. without merging it on getSize. How precise does getSize have 
to be.

If it's not useful, LGTM +1

> HLL merges tempList on every add
> 
>
> Key: HIVE-19578
> URL: https://issues.apache.org/jira/browse/HIVE-19578
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19578.1.patch, Screen Shot 2018-05-16 at 15.29.12 
> .png
>
>
>  See comments on HIVE-18866; this has significant perf overhead after the 
> even bigger overhead from hashing is removed.  !Screen Shot 2018-05-16 at 
> 15.29.12 .png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-19590) CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19590:
---


> CLONE - NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19590
> URL: https://issues.apache.org/jira/browse/HIVE-19590
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-19589) NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19589:
---


> NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19589
> URL: https://issues.apache.org/jira/browse/HIVE-19589
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-19594) CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19594:
---


> CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on 
> HiveQA
> 
>
> Key: HIVE-19594
> URL: https://issues.apache.org/jira/browse/HIVE-19594
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-19593) CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19593:
---


> CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19593
> URL: https://issues.apache.org/jira/browse/HIVE-19593
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-19595) CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19595:
---


> CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are 
> flaky on HiveQA
> 
>
> Key: HIVE-19595
> URL: https://issues.apache.org/jira/browse/HIVE-19595
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-19591) CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19591:
---


> CLONE - NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19591
> URL: https://issues.apache.org/jira/browse/HIVE-19591
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-19592) CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19592:
---


> CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19592
> URL: https://issues.apache.org/jira/browse/HIVE-19592
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-19596) CLONE - CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19596:
---


> CLONE - CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests 
> are flaky on HiveQA
> 
>
> Key: HIVE-19596
> URL: https://issues.apache.org/jira/browse/HIVE-19596
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19596) CLONE - CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19596:

Status: Patch Available  (was: Open)

> CLONE - CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests 
> are flaky on HiveQA
> 
>
> Key: HIVE-19596
> URL: https://issues.apache.org/jira/browse/HIVE-19596
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19596.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19595) CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19595:

Status: Patch Available  (was: Open)

> CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are 
> flaky on HiveQA
> 
>
> Key: HIVE-19595
> URL: https://issues.apache.org/jira/browse/HIVE-19595
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19595.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19593) CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19593:

Attachment: HIVE-19593.patch

> CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19593
> URL: https://issues.apache.org/jira/browse/HIVE-19593
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19593.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19594) CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19594:

Attachment: HIVE-19594.patch

> CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on 
> HiveQA
> 
>
> Key: HIVE-19594
> URL: https://issues.apache.org/jira/browse/HIVE-19594
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19594.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19595) CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19595:

Attachment: HIVE-19595.patch

> CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are 
> flaky on HiveQA
> 
>
> Key: HIVE-19595
> URL: https://issues.apache.org/jira/browse/HIVE-19595
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19595.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19596) CLONE - CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19596:

Attachment: HIVE-19596.patch

> CLONE - CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests 
> are flaky on HiveQA
> 
>
> Key: HIVE-19596
> URL: https://issues.apache.org/jira/browse/HIVE-19596
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19596.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19589) NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19589:

Attachment: HIVE-19589.patch

> NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19589
> URL: https://issues.apache.org/jira/browse/HIVE-19589
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19589.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19592) CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19592:

Attachment: HIVE-19592.patch

> CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19592
> URL: https://issues.apache.org/jira/browse/HIVE-19592
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19592.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19590) CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19590:

Attachment: HIVE-19590.patch

> CLONE - NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19590
> URL: https://issues.apache.org/jira/browse/HIVE-19590
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19590.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19591) CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19591:

Attachment: HIVE-19591.patch

> CLONE - NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19591
> URL: https://issues.apache.org/jira/browse/HIVE-19591
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19591.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19592) CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19592:

Status: Patch Available  (was: Open)

> CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19592
> URL: https://issues.apache.org/jira/browse/HIVE-19592
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19592.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19593) CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19593:

Status: Patch Available  (was: Open)

> CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19593
> URL: https://issues.apache.org/jira/browse/HIVE-19593
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19593.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19590) CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19590:

Status: Patch Available  (was: Open)

> CLONE - NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19590
> URL: https://issues.apache.org/jira/browse/HIVE-19590
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19590.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19591) CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19591:

Status: Patch Available  (was: Open)

> CLONE - NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19591
> URL: https://issues.apache.org/jira/browse/HIVE-19591
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19591.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19594) CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19594:

Status: Patch Available  (was: Open)

> CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on 
> HiveQA
> 
>
> Key: HIVE-19594
> URL: https://issues.apache.org/jira/browse/HIVE-19594
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19594.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19589) NOOP jira to see which tests are flaky on HiveQA

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19589:

Status: Patch Available  (was: Open)

> NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19589
> URL: https://issues.apache.org/jira/browse/HIVE-19589
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19589.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18866:

Attachment: HIVE-18866.01.patch

> Semijoin: Implement a Long -> Hash64 vector fast-path
> -
>
> Key: HIVE-18866
> URL: https://issues.apache.org/jira/browse/HIVE-18866
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: performance
> Attachments: 0001-hash64-WIP.patch, HIVE-18866.01.patch, 
> HIVE-18866.patch, perf-hash64-long.png
>
>
> A significant amount of CPU is wasted with JMM restrictions on byte[] arrays.
> To transform from one Long -> another Long, this goes into a byte[] array, 
> which shows up as a hotspot.
> !perf-hash64-long.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path

2018-05-17 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479618#comment-16479618
 ] 

Sergey Shelukhin commented on HIVE-18866:
-

Removed the imports. Test failures look bogus, trying again.

> Semijoin: Implement a Long -> Hash64 vector fast-path
> -
>
> Key: HIVE-18866
> URL: https://issues.apache.org/jira/browse/HIVE-18866
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: performance
> Attachments: 0001-hash64-WIP.patch, HIVE-18866.01.patch, 
> HIVE-18866.patch, perf-hash64-long.png
>
>
> A significant amount of CPU is wasted with JMM restrictions on byte[] arrays.
> To transform from one Long -> another Long, this goes into a byte[] array, 
> which shows up as a hotspot.
> !perf-hash64-long.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18866:

Attachment: (was: HIVE-18866.01.patch)

> Semijoin: Implement a Long -> Hash64 vector fast-path
> -
>
> Key: HIVE-18866
> URL: https://issues.apache.org/jira/browse/HIVE-18866
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: performance
> Attachments: 0001-hash64-WIP.patch, HIVE-18866.01.patch, 
> HIVE-18866.patch, perf-hash64-long.png
>
>
> A significant amount of CPU is wasted with JMM restrictions on byte[] arrays.
> To transform from one Long -> another Long, this goes into a byte[] array, 
> which shows up as a hotspot.
> !perf-hash64-long.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18866:

Attachment: HIVE-18866.01.patch

> Semijoin: Implement a Long -> Hash64 vector fast-path
> -
>
> Key: HIVE-18866
> URL: https://issues.apache.org/jira/browse/HIVE-18866
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: performance
> Attachments: 0001-hash64-WIP.patch, HIVE-18866.01.patch, 
> HIVE-18866.patch, perf-hash64-long.png
>
>
> A significant amount of CPU is wasted with JMM restrictions on byte[] arrays.
> To transform from one Long -> another Long, this goes into a byte[] array, 
> which shows up as a hotspot.
> !perf-hash64-long.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19418) add background stats updater similar to compactor

2018-05-17 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479621#comment-16479621
 ] 

Sergey Shelukhin commented on HIVE-19418:
-

Test failures looks bogus. Given the state of HiveQA it makes no sense to rerun 
for now, I'm assuming there will anyway be some changes based on the review.

> add background stats updater similar to compactor
> -
>
> Key: HIVE-19418
> URL: https://issues.apache.org/jira/browse/HIVE-19418
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19418.01.patch, HIVE-19418.02.patch, 
> HIVE-19418.patch
>
>
> There's a JIRA HIVE-19416 to add snapshot version to stats for MM/ACID tables 
> to make them usable in a transaction without breaking ACID (for metadata-only 
> optimization). However, stats for ACID tables can still become unusable if 
> e.g. two parallel inserts run - neither sees the data written by the other, 
> so after both finish, the snapshots on either set of stats won't match the 
> current snapshot and the stats will be unusable.
> Additionally, for ACID and non-ACID tables alike, a lot of the stats, with 
> some exceptions like numRows, cannot be aggregated (i.e. you cannot combine 
> ndvs from two inserts), and for ACID even less can be aggregated (you cannot 
> derive min/max if some rows are deleted but you don't scan the rest of the 
> dataset).
> Therefore we will add background logic to metastore (similar to, and 
> partially inside, the ACID compactor) to update stats.
> It will have 3 modes of operation.
> 1) Off.
> 2) Update only the stats that exist but are out of date (generating stats can 
> be expensive, so if the user is only analyzing a subset of tables it should 
> be able to only update that subset). We can simply look at existing stats and 
> only analyze for the relevant partitions and columns.
> 3) On: 2 + create stats for all tables and columns missing stats.
> There will also be a table parameter to skip stats update. 
> In phase 1, the process will operate outside of compactor, and run analyze 
> command on the table. The analyze command will automatically save the stats 
> with ACID snapshot information if needed, based on HIVE-19416, so we don't 
> need to do any special state management and this will work for all table 
> types. However it's also more expensive.
> In phase 2, we can explore adding stats collection during MM compaction that 
> uses a temp table. If we don't have open writers during major compaction (so 
> we overwrite all of the data), the temp table stats can simply be copied over 
> to the main table with correct snapshot information, saving us a table scan.
> In phase 3, we can add custom stats collection logic to full ACID compactor 
> that is not query based, the same way as we'd do for (2). Alternatively we 
> can wait for ACID compactor to become query based and just reuse (2).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19516) TestNegative merge_negative_5 and mm_concatenate are causing timeouts

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19516:

Attachment: HIVE-19516.03.patch

> TestNegative merge_negative_5 and mm_concatenate are causing timeouts
> -
>
> Key: HIVE-19516
> URL: https://issues.apache.org/jira/browse/HIVE-19516
> Project: Hive
>  Issue Type: Bug
>  Components: Testing Infrastructure
>Reporter: Vineet Garg
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19516.01.patch, HIVE-19516.02.patch, 
> HIVE-19516.03.patch, HIVE-19516.patch
>
>
> I haven't tried to reproduce this in isolation but it is reproducible if you 
> run in batch on local system 
> {noformat}
> mvn -B test  -Dtest.groups= -Dtest=TestNegativeCliDriver 
> -Dqfile=udf_invalid.q,authorization_uri_export.q,default_constraint_complex_default_value.q,druid_datasource2.q,check_constraint_max_length.q,view_update.q,default_partition_name.q,authorization_public_create.q,load_wrong_fileformat_rc_seq.q,default_constraint_invalid_type.q,altern1.q,describe_xpath1.q,drop_view_failure2.q,temp_table_rename.q,invalid_select_column_with_subquery.q,udf_trunc_error1.q,insert_view_failure.q,dbtxnmgr_nodbunlock.q,authorization_show_columns.q,cte_recursion.q,merge_constraint_notnull.q,clusterbyorderby.q,orc_type_promotion2.q,ctas_noperm_loc.q,udf_min.q,udf_instr_wrong_args_len.q,invalid_create_tbl2.q,part_col_complex_type.q,authorization_drop_db_empty.q,smb_mapjoin_14.q,subquery_scalar_multi_rows.q,alter_partition_coltype_2columns.q,subquery_corr_in_agg.q,insert_overwrite_notnull_constraint.q,authorization_show_grant_otheruser_wtab.q,regex_col_groupby.q,ptf_negative_DuplicateWindowAlias.q,exim_22_export_authfail.q,authorization_insert_noinspriv.q,udf_likeany_wrong1.q,groupby_key.q,ambiguous_col.q,groupby3_multi_distinct.q,authorization_alter_drop_ptn.q,invalid_cast_from_binary_5.q,show_create_table_does_not_exist.q,invalid_select_column.q,exim_20_managed_location_over_existing.q,interval_3.q,authorization_compile.q,join35.q,udf_concat_ws_wrong3.q,create_or_replace_view8.q,create_external_with_notnull_constraint.q,split_sample_out_of_range.q,materialized_view_no_transactional_rewrite.q,authorization_show_grant_otherrole.q,create_with_constraints_duplicate_name.q,invalid_stddev_samp_syntax.q,authorization_view_disable_cbo_7.q,autolocal1.q,avro_non_nullable_union.q,load_orc_negative_part.q,drop_view_failure1.q,columnstats_partlvl_invalid_values_autogather.q,exim_13_nonnative_import.q,alter_table_wrong_regex.q,udf_next_day_error_2.q,authorization_select.q,udf_trunc_error2.q,authorization_view_7.q,udf_format_number_wrong5.q,touch2.q,orc_type_promotion1.q,lateral_view_alias.q,show_tables_bad_db1.q,unset_table_property.q,alter_non_native.q,nvl_mismatch_type.q,load_orc_negative3.q,authorization_create_role_no_admin.q,invalid_distinct1.q,authorization_grant_server.q,orc_type_promotion3_acid.q,hms_using_serde_alter_table_update_columns.q,show_tables_bad1.q,macro_unused_parameter.q,drop_invalid_constraint3.q,drop_partition_filter_failure.q,char_pad_convert_fail3.q,exim_23_import_exist_authfail.q,drop_invalid_constraint4.q,authorization_create_macro1.q,archive1.q,subquery_multiple_cols_in_select.q,change_hive_hdfs_session_path.q,udf_trunc_error3.q,invalid_variance_syntax.q,authorization_truncate_2.q,invalid_avg_syntax.q,invalid_select_column_with_tablename.q,mm_truncate_cols.q,groupby_grouping_sets1.q,druid_location.q,groupby2_multi_distinct.q,authorization_sba_drop_table.q,dynamic_partitions_with_whitelist.q,compare_string_bigint_2.q,udf_greatest_error_2.q,authorization_view_6.q,show_tablestatus.q,duplicate_alias_in_transform_schema.q,create_with_fk_uk_same_tab.q,udtf_not_supported3.q,alter_table_constraint_invalid_fk_col2.q,udtf_not_supported1.q,dbtxnmgr_notableunlock.q,ptf_negative_InvalidValueBoundary.q,alter_table_constraint_duplicate_pk.q,udf_printf_wrong4.q,create_view_failure9.q,udf_elt_wrong_type.q,selectDistinctStarNeg_1.q,invalid_mapjoin1.q,load_stored_as_dirs.q,input1.q,udf_sort_array_wrong1.q,invalid_distinct2.q,invalid_select_fn.q,authorization_role_grant_otherrole.q,archive4.q,load_nonpart_authfail.q,recursive_view.q,authorization_view_disable_cbo_1.q,desc_failure4.q,create_not_acid.q,udf_sort_array_wrong3.q,char_pad_convert_fail0.q,udf_map_values_arg_type.q,alter_view_failure6_2.q,alter_partition_change_col_nonexist.q,update_non_acid_table.q,authorization_view_disable_cbo_5.q,ct_noperm_loc.q,interval_1.q,authorization_show_grant_otheruser_all.q,authorization_view_2.q,show_tables_bad2.q,groupby_rollup2.q,truncate_column_seqfile.q,create_view_failure5.q,authorization_create_view.q,ptf_window_boundaries.q,ctasnullcol.q,input_part0_neg_2.q,create_or_replace_view1.q,udf_max.q,exim_01_nonpart_over_loaded.q,

[jira] [Commented] (HIVE-19516) TestNegative merge_negative_5 and mm_concatenate are causing timeouts

2018-05-17 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479634#comment-16479634
 ] 

Sergey Shelukhin commented on HIVE-19516:
-

Unrelated failures, just rerunning. [~jcamachorodriguez] since this one only 
touched unrelated itests, all of the tests that failed above are probably good 
candidates for flaky

> TestNegative merge_negative_5 and mm_concatenate are causing timeouts
> -
>
> Key: HIVE-19516
> URL: https://issues.apache.org/jira/browse/HIVE-19516
> Project: Hive
>  Issue Type: Bug
>  Components: Testing Infrastructure
>Reporter: Vineet Garg
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19516.01.patch, HIVE-19516.02.patch, 
> HIVE-19516.03.patch, HIVE-19516.patch
>
>
> I haven't tried to reproduce this in isolation but it is reproducible if you 
> run in batch on local system 
> {noformat}
> mvn -B test  -Dtest.groups= -Dtest=TestNegativeCliDriver 
> -Dqfile=udf_invalid.q,authorization_uri_export.q,default_constraint_complex_default_value.q,druid_datasource2.q,check_constraint_max_length.q,view_update.q,default_partition_name.q,authorization_public_create.q,load_wrong_fileformat_rc_seq.q,default_constraint_invalid_type.q,altern1.q,describe_xpath1.q,drop_view_failure2.q,temp_table_rename.q,invalid_select_column_with_subquery.q,udf_trunc_error1.q,insert_view_failure.q,dbtxnmgr_nodbunlock.q,authorization_show_columns.q,cte_recursion.q,merge_constraint_notnull.q,clusterbyorderby.q,orc_type_promotion2.q,ctas_noperm_loc.q,udf_min.q,udf_instr_wrong_args_len.q,invalid_create_tbl2.q,part_col_complex_type.q,authorization_drop_db_empty.q,smb_mapjoin_14.q,subquery_scalar_multi_rows.q,alter_partition_coltype_2columns.q,subquery_corr_in_agg.q,insert_overwrite_notnull_constraint.q,authorization_show_grant_otheruser_wtab.q,regex_col_groupby.q,ptf_negative_DuplicateWindowAlias.q,exim_22_export_authfail.q,authorization_insert_noinspriv.q,udf_likeany_wrong1.q,groupby_key.q,ambiguous_col.q,groupby3_multi_distinct.q,authorization_alter_drop_ptn.q,invalid_cast_from_binary_5.q,show_create_table_does_not_exist.q,invalid_select_column.q,exim_20_managed_location_over_existing.q,interval_3.q,authorization_compile.q,join35.q,udf_concat_ws_wrong3.q,create_or_replace_view8.q,create_external_with_notnull_constraint.q,split_sample_out_of_range.q,materialized_view_no_transactional_rewrite.q,authorization_show_grant_otherrole.q,create_with_constraints_duplicate_name.q,invalid_stddev_samp_syntax.q,authorization_view_disable_cbo_7.q,autolocal1.q,avro_non_nullable_union.q,load_orc_negative_part.q,drop_view_failure1.q,columnstats_partlvl_invalid_values_autogather.q,exim_13_nonnative_import.q,alter_table_wrong_regex.q,udf_next_day_error_2.q,authorization_select.q,udf_trunc_error2.q,authorization_view_7.q,udf_format_number_wrong5.q,touch2.q,orc_type_promotion1.q,lateral_view_alias.q,show_tables_bad_db1.q,unset_table_property.q,alter_non_native.q,nvl_mismatch_type.q,load_orc_negative3.q,authorization_create_role_no_admin.q,invalid_distinct1.q,authorization_grant_server.q,orc_type_promotion3_acid.q,hms_using_serde_alter_table_update_columns.q,show_tables_bad1.q,macro_unused_parameter.q,drop_invalid_constraint3.q,drop_partition_filter_failure.q,char_pad_convert_fail3.q,exim_23_import_exist_authfail.q,drop_invalid_constraint4.q,authorization_create_macro1.q,archive1.q,subquery_multiple_cols_in_select.q,change_hive_hdfs_session_path.q,udf_trunc_error3.q,invalid_variance_syntax.q,authorization_truncate_2.q,invalid_avg_syntax.q,invalid_select_column_with_tablename.q,mm_truncate_cols.q,groupby_grouping_sets1.q,druid_location.q,groupby2_multi_distinct.q,authorization_sba_drop_table.q,dynamic_partitions_with_whitelist.q,compare_string_bigint_2.q,udf_greatest_error_2.q,authorization_view_6.q,show_tablestatus.q,duplicate_alias_in_transform_schema.q,create_with_fk_uk_same_tab.q,udtf_not_supported3.q,alter_table_constraint_invalid_fk_col2.q,udtf_not_supported1.q,dbtxnmgr_notableunlock.q,ptf_negative_InvalidValueBoundary.q,alter_table_constraint_duplicate_pk.q,udf_printf_wrong4.q,create_view_failure9.q,udf_elt_wrong_type.q,selectDistinctStarNeg_1.q,invalid_mapjoin1.q,load_stored_as_dirs.q,input1.q,udf_sort_array_wrong1.q,invalid_distinct2.q,invalid_select_fn.q,authorization_role_grant_otherrole.q,archive4.q,load_nonpart_authfail.q,recursive_view.q,authorization_view_disable_cbo_1.q,desc_failure4.q,create_not_acid.q,udf_sort_array_wrong3.q,char_pad_convert_fail0.q,udf_map_values_arg_type.q,alter_view_failure6_2.q,alter_partition_change_col_nonexist.q,update_non_acid_table.q,authorization_view_disable_cbo_5.q,ct_noperm_loc.q,interval_1.q,authorization_show_grant_otheruser_all.q,authorization_view_2.q,show_tables_bad2.q,groupby_rollup2.q,

[jira] [Assigned] (HIVE-19597) TestWorkloadManager sometimes hangs

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19597:
---


> TestWorkloadManager sometimes hangs
> ---
>
> Key: HIVE-19597
> URL: https://issues.apache.org/jira/browse/HIVE-19597
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> Seems like the tests randomly get stuck after the lines like
> {noformat}
> 2018-05-17T01:54:27,111  INFO [Workload management master] 
> tez.WorkloadManager: Processing current events
> 2018-05-17T01:54:27,603  INFO [TriggerValidator] 
> tez.PerPoolTriggerValidatorRunnable: Creating trigger validator for pool: llap
> 2018-05-17T01:54:37,090 DEBUG [Thread-28] conf.HiveConf: Found metastore URI 
> of null
> {noformat}
> Then they get killed by timeout. Happens to random tests in a few separate 
> runs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19579) remove HBase transitive dependency that drags in some snapshot

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19579:

Attachment: HIVE-19579.01.patch

> remove HBase transitive dependency that drags in some snapshot
> --
>
> Key: HIVE-19579
> URL: https://issues.apache.org/jira/browse/HIVE-19579
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19579.01.patch, HIVE-19579.patch
>
>
> This:
> {noformat}
> [INFO] org.apache.hive:hive-llap-server:jar:3.1.0-SNAPSHOT
> [INFO] \- org.apache.hbase:hbase-server:jar:2.0.0-alpha4:compile
> [INFO]\- org.glassfish.web:javax.servlet.jsp:jar:2.3.2:compile
> [INFO]   \- org.glassfish:javax.el:jar:3.0.1-b10:compile (version 
> selected from constraint [3.0.0,))
> {noformat}
> occasionally results in bizarre build failures like this:
> {noformat}
> Failed to read artifact descriptor for 
> org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Could not transfer artifact 
> org.glassfish:javax.el:pom:3.0.1-b06-SNAPSHOT from/to jvnet-nexus-snapshots 
> (https://maven.java.net/content/repositories/snapshots): Failed to transfer 
> file: 
> https://maven.java.net/content/repositories/snapshots/org/glassfish/javax.el/3.0.1-b06-SNAPSHOT/javax.el-3.0.1-b06-SNAPSHOT.pom.
>  Return code is: 402 , ReasonPhrase:Payment Required.
> {noformat}
> cc [~enis] [~stack] if you guys want to take a look at HBase side



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19258) add originals support to MM tables (and make the conversion a metadata only operation)

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19258:

Attachment: HIVE-19258.10.patch

> add originals support to MM tables (and make the conversion a metadata only 
> operation)
> --
>
> Key: HIVE-19258
> URL: https://issues.apache.org/jira/browse/HIVE-19258
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19258.01.patch, HIVE-19258.02.patch, 
> HIVE-19258.03.patch, HIVE-19258.04.patch, HIVE-19258.05.patch, 
> HIVE-19258.06.patch, HIVE-19258.07.patch, HIVE-19258.08.patch, 
> HIVE-19258.08.patch, HIVE-19258.09.patch, HIVE-19258.10.patch, 
> HIVE-19258.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19258) add originals support to MM tables (and make the conversion a metadata only operation)

2018-05-17 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479720#comment-16479720
 ] 

Sergey Shelukhin commented on HIVE-19258:
-

WM tests are flaky: HIVE-19597

> add originals support to MM tables (and make the conversion a metadata only 
> operation)
> --
>
> Key: HIVE-19258
> URL: https://issues.apache.org/jira/browse/HIVE-19258
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19258.01.patch, HIVE-19258.02.patch, 
> HIVE-19258.03.patch, HIVE-19258.04.patch, HIVE-19258.05.patch, 
> HIVE-19258.06.patch, HIVE-19258.07.patch, HIVE-19258.08.patch, 
> HIVE-19258.08.patch, HIVE-19258.09.patch, HIVE-19258.10.patch, 
> HIVE-19258.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19258) add originals support to MM tables (and make the conversion a metadata only operation)

2018-05-17 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479719#comment-16479719
 ] 

Sergey Shelukhin commented on HIVE-19258:
-

I was able to repro the failure of bucketizedhiveinputformat on the same query, 
then I realized I forgot to actually apply the patch.
 The failure, assuming it's the same one as here, is OOM in one of the stdouts:
{noformat}
[DEBUG] 2018-05-17 14:00:24.209 [IPC Client (928988232) connection to 
localhost/127.0.0.1:50278 from sergey] Client - IPC Client (928988232) 
connection to localhost/127.0.0.1:50278 from sergey: closed
[DEBUG] 2018-05-17 14:00:24.209 [IPC Client (928988232) connection to 
localhost/127.0.0.1:50278 from sergey] Client - IPC Client (928988232) 
connection to localhost/127.0.0.1:50278 from sergey: stopped, remaining 
connections 0
#
# java.lang.OutOfMemoryError: GC overhead limit exceeded
# -XX:OnOutOfMemoryError="kill %p"
#   Executing "kill 5333"...
[ERROR] 2018-05-17 14:01:39.613 [SIGTERM handler] CoarseGrainedExecutorBackend 
- RECEIVED SIGNAL TERM
[DEBUG] 2018-05-17 14:01:39.613 [Executor task launch worker for task 18] 
TaskMemoryManager - unreleased 268.0 MB memory from 
org.apache.spark.util.collection.ExternalAppendOnlyMap@141a89ea
[ERROR] 2018-05-17 14:01:39.616 [Executor task launch worker for task 18] 
Executor - Exception in task 0.0 in stage 2.2 (TID 18)
java.lang.OutOfMemoryError: GC overhead limit exceeded
at 
org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:188)
 ~[spark-core_2.11-2.3.0.jar:2.3.0]
at 
org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:185)
 ~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) 
~[spark-core_2.11-2.3.0.jar:2.3.0]
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438) 
~[scala-library-2.11.8.jar:?]
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) 
~[scala-library-2.11.8.jar:?]
at 
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32) 
~[spark-core_2.11-2.3.0.jar:2.3.0]
at 
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) 
~[spark-core_2.11-2.3.0.jar:2.3.0]
at 
org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:153)
 ~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:41) 
~[spark-core_2.11-2.3.0.jar:2.3.0]
at 
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:90)
 ~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:105) 
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) 
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) 
~[spark-core_2.11-2.3.0.jar:2.3.0]
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) 
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) 
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) 
~[spark-core_2.11-2.3.0.jar:2.3.0]
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) 
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) 
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) 
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) 
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.scheduler.Task.run(Task.scala:109) 
~[spark-core_2.11-2.3.0.jar:2.3.0]
at 
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) 
[spark-core_2.11-2.3.0.jar:2.3.0]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_45]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
[INFO ] 2018-05-17 14:01:39.617 [pool-8-thread-1] DiskBlockManager - Shutdown 
hook called
{noformat}
cc [~xuefuz] 

> add originals support to MM tables (and make the conversion a metadata only 
> operation)
> --
>
> Key: HIVE-19258
> URL: https://issues.apache.org/jira/browse/HIVE-19258
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19258.01.patch, HIVE

[jira] [Updated] (HIVE-19597) TestWorkloadManager sometimes hangs

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19597:

Description: 
Seems like the tests randomly get stuck after the lines like
{noformat}
2018-05-17T01:54:27,111  INFO [Workload management master] tez.WorkloadManager: 
Processing current events
2018-05-17T01:54:27,603  INFO [TriggerValidator] 
tez.PerPoolTriggerValidatorRunnable: Creating trigger validator for pool: llap
2018-05-17T01:54:37,090 DEBUG [Thread-28] conf.HiveConf: Found metastore URI of 
null
{noformat}

Then they get killed by timeout. Happened in the same manner, to random tests 
in a few separate runs.

  was:
Seems like the tests randomly get stuck after the lines like
{noformat}
2018-05-17T01:54:27,111  INFO [Workload management master] tez.WorkloadManager: 
Processing current events
2018-05-17T01:54:27,603  INFO [TriggerValidator] 
tez.PerPoolTriggerValidatorRunnable: Creating trigger validator for pool: llap
2018-05-17T01:54:37,090 DEBUG [Thread-28] conf.HiveConf: Found metastore URI of 
null
{noformat}

Then they get killed by timeout. Happens in the same manner, to random tests in 
a few separate runs.


> TestWorkloadManager sometimes hangs
> ---
>
> Key: HIVE-19597
> URL: https://issues.apache.org/jira/browse/HIVE-19597
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> Seems like the tests randomly get stuck after the lines like
> {noformat}
> 2018-05-17T01:54:27,111  INFO [Workload management master] 
> tez.WorkloadManager: Processing current events
> 2018-05-17T01:54:27,603  INFO [TriggerValidator] 
> tez.PerPoolTriggerValidatorRunnable: Creating trigger validator for pool: llap
> 2018-05-17T01:54:37,090 DEBUG [Thread-28] conf.HiveConf: Found metastore URI 
> of null
> {noformat}
> Then they get killed by timeout. Happened in the same manner, to random tests 
> in a few separate runs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19597) TestWorkloadManager sometimes hangs

2018-05-17 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19597:

Description: 
Seems like the tests randomly get stuck after the lines like
{noformat}
2018-05-17T01:54:27,111  INFO [Workload management master] tez.WorkloadManager: 
Processing current events
2018-05-17T01:54:27,603  INFO [TriggerValidator] 
tez.PerPoolTriggerValidatorRunnable: Creating trigger validator for pool: llap
2018-05-17T01:54:37,090 DEBUG [Thread-28] conf.HiveConf: Found metastore URI of 
null
{noformat}

Then they get killed by timeout. Happens in the same manner, to random tests in 
a few separate runs.

  was:
Seems like the tests randomly get stuck after the lines like
{noformat}
2018-05-17T01:54:27,111  INFO [Workload management master] tez.WorkloadManager: 
Processing current events
2018-05-17T01:54:27,603  INFO [TriggerValidator] 
tez.PerPoolTriggerValidatorRunnable: Creating trigger validator for pool: llap
2018-05-17T01:54:37,090 DEBUG [Thread-28] conf.HiveConf: Found metastore URI of 
null
{noformat}

Then they get killed by timeout. Happens to random tests in a few separate runs.


> TestWorkloadManager sometimes hangs
> ---
>
> Key: HIVE-19597
> URL: https://issues.apache.org/jira/browse/HIVE-19597
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> Seems like the tests randomly get stuck after the lines like
> {noformat}
> 2018-05-17T01:54:27,111  INFO [Workload management master] 
> tez.WorkloadManager: Processing current events
> 2018-05-17T01:54:27,603  INFO [TriggerValidator] 
> tez.PerPoolTriggerValidatorRunnable: Creating trigger validator for pool: llap
> 2018-05-17T01:54:37,090 DEBUG [Thread-28] conf.HiveConf: Found metastore URI 
> of null
> {noformat}
> Then they get killed by timeout. Happens in the same manner, to random tests 
> in a few separate runs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19597) TestWorkloadManager sometimes hangs

2018-05-17 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479770#comment-16479770
 ] 

Sergey Shelukhin commented on HIVE-19597:
-

The trigger thing just runs on timer, and therefore happens last after the test 
itself is probably already stuck...  appears to be a red herring.

> TestWorkloadManager sometimes hangs
> ---
>
> Key: HIVE-19597
> URL: https://issues.apache.org/jira/browse/HIVE-19597
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> Seems like the tests randomly get stuck after the lines like
> {noformat}
> 2018-05-17T01:54:27,111  INFO [Workload management master] 
> tez.WorkloadManager: Processing current events
> 2018-05-17T01:54:27,603  INFO [TriggerValidator] 
> tez.PerPoolTriggerValidatorRunnable: Creating trigger validator for pool: llap
> 2018-05-17T01:54:37,090 DEBUG [Thread-28] conf.HiveConf: Found metastore URI 
> of null
> {noformat}
> Then they get killed by timeout. Happened in the same manner, to random tests 
> in a few separate runs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19594) CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481045#comment-16481045
 ] 

Sergey Shelukhin commented on HIVE-19594:
-

{noformat}
2018-05-18T10:24:44,991  WARN [Thread-3915] mapred.LocalJobRunner: 
job_local632888732_0106
java.io.FileNotFoundException: File 
file:/tmp/hadoop/mapred/staging/hiveptest632888732/.staging/job_local632888732_0106/job.splitmetainfo
 does not exist
{noformat}
I see lots of *HCatFileStorer tests fail like this. Looks like some ptest 
issue. 
Can this test perhaps be changed to not use shared tmp directory and instead 
use the test directory?
Or we can just disable it and wait for a fix from people who care about HCat ;)
cc [~jcamachorodriguez] [~thejas]

> CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on 
> HiveQA
> 
>
> Key: HIVE-19594
> URL: https://issues.apache.org/jira/browse/HIVE-19594
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19594.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19590) CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481060#comment-16481060
 ] 

Sergey Shelukhin commented on HIVE-19590:
-

Will be handled by MASK_STATS

> CLONE - NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19590
> URL: https://issues.apache.org/jira/browse/HIVE-19590
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19590.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19592) CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481069#comment-16481069
 ] 

Sergey Shelukhin commented on HIVE-19592:
-

MiniMR timed out, there's JIRA for this.
The other test failed like others:
{noformat}
2018-05-18T08:05:47,583  WARN [Thread-2372] mapred.LocalJobRunner: 
job_local801116008_0040
java.io.FileNotFoundException: File 
file:/tmp/hadoop/mapred/staging/hiveptest801116008/.staging/job_local801116008_0040/job.splitmetainfo
 does not exist
at 
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
 ~[hadoop-common-3.1.0.jar:?]
at 
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
 ~[hadoop-common-3.1.0.jar:?]
at 
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
 ~[hadoop-common-3.1.0.jar:?]
at 
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454) 
~[hadoop-common-3.1.0.jar:?]
at 
org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51)
 ~[hadoop-mapreduce-client-core-3.1.0.jar:?]
at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:538) 
[hadoop-mapreduce-client-common-3.1.0.jar:?
{noformat}

[~jcamachorodriguez] [~vihangk1] do you know if anything is cleaning tmp dir 
during tests on ptest machines? We get a lot of random failures like this from 
various tests.

> CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19592
> URL: https://issues.apache.org/jira/browse/HIVE-19592
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19592.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19589) NOOP jira to see which tests are flaky on HiveQA

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481074#comment-16481074
 ] 

Sergey Shelukhin commented on HIVE-19589:
-

Actually it's not even a change in lineage, it's a change in ordering of lines 
for lineage it looks like. 
Masking lineage makes sense to me. Should I file a JIRA?

> NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19589
> URL: https://issues.apache.org/jira/browse/HIVE-19589
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19589.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19590) mask stats in llap_smb

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481087#comment-16481087
 ] 

Sergey Shelukhin commented on HIVE-19590:
-

+1. It cannot possibly make tests any more broken than they already are so 
maybe it should just be pushed :)

> mask stats in llap_smb
> --
>
> Key: HIVE-19590
> URL: https://issues.apache.org/jira/browse/HIVE-19590
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19590.01.patch, HIVE-19590.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-19590) mask stats in llap_smb

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19590:
---

Assignee: Jesus Camacho Rodriguez  (was: Sergey Shelukhin)

> mask stats in llap_smb
> --
>
> Key: HIVE-19590
> URL: https://issues.apache.org/jira/browse/HIVE-19590
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19590.01.patch, HIVE-19590.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19590) mask stats in llap_smb

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19590:

Summary: mask stats in llap_smb  (was: CLONE - NOOP jira to see which tests 
are flaky on HiveQA)

> mask stats in llap_smb
> --
>
> Key: HIVE-19590
> URL: https://issues.apache.org/jira/browse/HIVE-19590
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19590.01.patch, HIVE-19590.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481105#comment-16481105
 ] 

Sergey Shelukhin commented on HIVE-18866:
-

HCat test:
Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input 
path does not exist: file:/tmp/temp1540619121/tmp-2080326801

union_stats looks flaky w.r.t. these two lines. [~jcamachorodriguez] should be 
disable it?

> Semijoin: Implement a Long -> Hash64 vector fast-path
> -
>
> Key: HIVE-18866
> URL: https://issues.apache.org/jira/browse/HIVE-18866
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: performance
> Attachments: 0001-hash64-WIP.patch, HIVE-18866.01.patch, 
> HIVE-18866.patch, perf-hash64-long.png
>
>
> A significant amount of CPU is wasted with JMM restrictions on byte[] arrays.
> To transform from one Long -> another Long, this goes into a byte[] array, 
> which shows up as a hotspot.
> !perf-hash64-long.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19595) disable webui in MiniHS2

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19595:

Summary: disable webui in MiniHS2  (was: CLONE - CLONE - CLONE - CLONE - 
CLONE - NOOP jira to see which tests are flaky on HiveQA)

> disable webui in MiniHS2
> 
>
> Key: HIVE-19595
> URL: https://issues.apache.org/jira/browse/HIVE-19595
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19595.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19595) CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481115#comment-16481115
 ] 

Sergey Shelukhin commented on HIVE-19595:
-

One address in use due to WebUI... should be disabled in MiniHS2.
TestTriggersTezSessionPoolManager has some metastore communication errors, no 
idea why.

> CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are 
> flaky on HiveQA
> 
>
> Key: HIVE-19595
> URL: https://issues.apache.org/jira/browse/HIVE-19595
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19595.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19595) disable webui in MiniHS2

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19595:

Status: Open  (was: Patch Available)

> disable webui in MiniHS2
> 
>
> Key: HIVE-19595
> URL: https://issues.apache.org/jira/browse/HIVE-19595
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19595.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19595) disable webui in MiniHS2

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19595:

Attachment: HIVE-19595.01.patch

> disable webui in MiniHS2
> 
>
> Key: HIVE-19595
> URL: https://issues.apache.org/jira/browse/HIVE-19595
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19595.01.patch, HIVE-19595.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19595) disable webui in MiniHS2

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481123#comment-16481123
 ] 

Sergey Shelukhin commented on HIVE-19595:
-

[~thejas] [~jcamachorodriguez] does this make sense?

> disable webui in MiniHS2
> 
>
> Key: HIVE-19595
> URL: https://issues.apache.org/jira/browse/HIVE-19595
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19595.01.patch, HIVE-19595.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19595) disable webui in MiniHS2

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19595:

Description: Causes tests flakiness if they same port is used.

> disable webui in MiniHS2
> 
>
> Key: HIVE-19595
> URL: https://issues.apache.org/jira/browse/HIVE-19595
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19595.01.patch, HIVE-19595.patch
>
>
> Causes tests flakiness if they same port is used.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18866:

Attachment: HIVE-18866.02.patch

> Semijoin: Implement a Long -> Hash64 vector fast-path
> -
>
> Key: HIVE-18866
> URL: https://issues.apache.org/jira/browse/HIVE-18866
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: performance
> Attachments: 0001-hash64-WIP.patch, HIVE-18866.01.patch, 
> HIVE-18866.02.patch, HIVE-18866.patch, perf-hash64-long.png
>
>
> A significant amount of CPU is wasted with JMM restrictions on byte[] arrays.
> To transform from one Long -> another Long, this goes into a byte[] array, 
> which shows up as a hotspot.
> !perf-hash64-long.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-19608) disable flaky tests 2

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19608:
---


> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> union_stats
> {noformat}
> java.lang.AssertionError: 
> Client Execution succeeded but contained differences (error code = 1) after 
> executing union_stats.q 
> 362a363
> > COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19608) disable flaky tests 2

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19608:

Attachment: HIVE-19608.patch

> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError: 
> Client Execution succeeded but contained differences (error code = 1) after 
> executing union_stats.q 
> 362a363
> > COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19608) disable flaky tests 2

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19608:

Attachment: (was: HIVE-19608.patch)

> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError: 
> Client Execution succeeded but contained differences (error code = 1) after 
> executing union_stats.q 
> 362a363
> > COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19608) disable flaky tests 2

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481146#comment-16481146
 ] 

Sergey Shelukhin commented on HIVE-19608:
-

cc [~jcamachorodriguez] 
I might disable more if I see any. I think this should be ok to commit without 
HiveQA since it just removes tests.

> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError: 
> Client Execution succeeded but contained differences (error code = 1) after 
> executing union_stats.q 
> 362a363
> > COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19608) disable flaky tests 2

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19608:

Attachment: HIVE-19608.patch

> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError: 
> Client Execution succeeded but contained differences (error code = 1) after 
> executing union_stats.q 
> 362a363
> > COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19516) TestNegative merge_negative_5 and mm_concatenate are causing timeouts

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481150#comment-16481150
 ] 

Sergey Shelukhin commented on HIVE-19516:
-

[~jcamachorodriguez] should this be ok to commit? union_stats is going to be 
disabled in a separate jira. 

> TestNegative merge_negative_5 and mm_concatenate are causing timeouts
> -
>
> Key: HIVE-19516
> URL: https://issues.apache.org/jira/browse/HIVE-19516
> Project: Hive
>  Issue Type: Bug
>  Components: Testing Infrastructure
>Reporter: Vineet Garg
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19516.01.patch, HIVE-19516.02.patch, 
> HIVE-19516.03.patch, HIVE-19516.patch
>
>
> I haven't tried to reproduce this in isolation but it is reproducible if you 
> run in batch on local system 
> {noformat}
> mvn -B test  -Dtest.groups= -Dtest=TestNegativeCliDriver 
> -Dqfile=udf_invalid.q,authorization_uri_export.q,default_constraint_complex_default_value.q,druid_datasource2.q,check_constraint_max_length.q,view_update.q,default_partition_name.q,authorization_public_create.q,load_wrong_fileformat_rc_seq.q,default_constraint_invalid_type.q,altern1.q,describe_xpath1.q,drop_view_failure2.q,temp_table_rename.q,invalid_select_column_with_subquery.q,udf_trunc_error1.q,insert_view_failure.q,dbtxnmgr_nodbunlock.q,authorization_show_columns.q,cte_recursion.q,merge_constraint_notnull.q,clusterbyorderby.q,orc_type_promotion2.q,ctas_noperm_loc.q,udf_min.q,udf_instr_wrong_args_len.q,invalid_create_tbl2.q,part_col_complex_type.q,authorization_drop_db_empty.q,smb_mapjoin_14.q,subquery_scalar_multi_rows.q,alter_partition_coltype_2columns.q,subquery_corr_in_agg.q,insert_overwrite_notnull_constraint.q,authorization_show_grant_otheruser_wtab.q,regex_col_groupby.q,ptf_negative_DuplicateWindowAlias.q,exim_22_export_authfail.q,authorization_insert_noinspriv.q,udf_likeany_wrong1.q,groupby_key.q,ambiguous_col.q,groupby3_multi_distinct.q,authorization_alter_drop_ptn.q,invalid_cast_from_binary_5.q,show_create_table_does_not_exist.q,invalid_select_column.q,exim_20_managed_location_over_existing.q,interval_3.q,authorization_compile.q,join35.q,udf_concat_ws_wrong3.q,create_or_replace_view8.q,create_external_with_notnull_constraint.q,split_sample_out_of_range.q,materialized_view_no_transactional_rewrite.q,authorization_show_grant_otherrole.q,create_with_constraints_duplicate_name.q,invalid_stddev_samp_syntax.q,authorization_view_disable_cbo_7.q,autolocal1.q,avro_non_nullable_union.q,load_orc_negative_part.q,drop_view_failure1.q,columnstats_partlvl_invalid_values_autogather.q,exim_13_nonnative_import.q,alter_table_wrong_regex.q,udf_next_day_error_2.q,authorization_select.q,udf_trunc_error2.q,authorization_view_7.q,udf_format_number_wrong5.q,touch2.q,orc_type_promotion1.q,lateral_view_alias.q,show_tables_bad_db1.q,unset_table_property.q,alter_non_native.q,nvl_mismatch_type.q,load_orc_negative3.q,authorization_create_role_no_admin.q,invalid_distinct1.q,authorization_grant_server.q,orc_type_promotion3_acid.q,hms_using_serde_alter_table_update_columns.q,show_tables_bad1.q,macro_unused_parameter.q,drop_invalid_constraint3.q,drop_partition_filter_failure.q,char_pad_convert_fail3.q,exim_23_import_exist_authfail.q,drop_invalid_constraint4.q,authorization_create_macro1.q,archive1.q,subquery_multiple_cols_in_select.q,change_hive_hdfs_session_path.q,udf_trunc_error3.q,invalid_variance_syntax.q,authorization_truncate_2.q,invalid_avg_syntax.q,invalid_select_column_with_tablename.q,mm_truncate_cols.q,groupby_grouping_sets1.q,druid_location.q,groupby2_multi_distinct.q,authorization_sba_drop_table.q,dynamic_partitions_with_whitelist.q,compare_string_bigint_2.q,udf_greatest_error_2.q,authorization_view_6.q,show_tablestatus.q,duplicate_alias_in_transform_schema.q,create_with_fk_uk_same_tab.q,udtf_not_supported3.q,alter_table_constraint_invalid_fk_col2.q,udtf_not_supported1.q,dbtxnmgr_notableunlock.q,ptf_negative_InvalidValueBoundary.q,alter_table_constraint_duplicate_pk.q,udf_printf_wrong4.q,create_view_failure9.q,udf_elt_wrong_type.q,selectDistinctStarNeg_1.q,invalid_mapjoin1.q,load_stored_as_dirs.q,input1.q,udf_sort_array_wrong1.q,invalid_distinct2.q,invalid_select_fn.q,authorization_role_grant_otherrole.q,archive4.q,load_nonpart_authfail.q,recursive_view.q,authorization_view_disable_cbo_1.q,desc_failure4.q,create_not_acid.q,udf_sort_array_wrong3.q,char_pad_convert_fail0.q,udf_map_values_arg_type.q,alter_view_failure6_2.q,alter_partition_change_col_nonexist.q,update_non_acid_table.q,authorization_view_disable_cbo_5.q,ct_noperm_loc.q,interval_1.q,authorization_show_grant_otheruser_all.q,authorization_view_2.q,show_tables_bad2.q,groupby_rollup2.q,truncate_column_seqfile.q,create_view_failure5.q,authorization_create_v

[jira] [Updated] (HIVE-19463) TezTask - getting groups may fail (PartialGroupNameException in some tests)

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19463:

Attachment: HIVE-19463.05.patch

> TezTask - getting groups may fail (PartialGroupNameException in some tests)
> ---
>
> Key: HIVE-19463
> URL: https://issues.apache.org/jira/browse/HIVE-19463
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19463.01.patch, HIVE-19463.02.patch, 
> HIVE-19463.03.patch, HIVE-19463.03.patch, HIVE-19463.04.patch, 
> HIVE-19463.05.patch, HIVE-19463.patch
>
>
> {noformat}
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping$PartialGroupNameException:
>  The user name 'hive_test_user' is not found. id: hive_test_user: no such user
> id: hive_test_user: no such user
>   at 
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:294)
>  ~[hadoop-common-3.1.0.jar:?]
>   at 
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:207)
>  [hadoop-common-3.1.0.jar:?]
>   at 
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:97)
>  [hadoop-common-3.1.0.jar:?]
>   at 
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:51)
>  [hadoop-common-3.1.0.jar:?]
>   at 
> org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Groups.java:384)
>  [hadoop-common-3.1.0.jar:?]
>   at 
> org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:319) 
> [hadoop-common-3.1.0.jar:?]
>   at 
> org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:269) 
> [hadoop-common-3.1.0.jar:?]
>   at 
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542)
>  [guava-19.0.jar:?]
>   at 
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323) 
> [guava-19.0.jar:?]
>   at 
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286)
>  [guava-19.0.jar:?]
>   at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) 
> [guava-19.0.jar:?]
>   at com.google.common.cache.LocalCache.get(LocalCache.java:3953) 
> [guava-19.0.jar:?]
>   at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3957) 
> [guava-19.0.jar:?]
>   at 
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4875)
>  [guava-19.0.jar:?]
>   at org.apache.hadoop.security.Groups.getGroups(Groups.java:227) 
> [hadoop-common-3.1.0.jar:?]
>   at 
> org.apache.hadoop.security.UserGroupInformation.getGroups(UserGroupInformation.java:1540)
>  [hadoop-common-3.1.0.jar:?]
>   at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:163) 
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19596) disable bucketizedhiveinputformat for MiniSparkonYarn

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19596:

Summary: disable bucketizedhiveinputformat for MiniSparkonYarn  (was: CLONE 
- CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are 
flaky on HiveQA)

> disable bucketizedhiveinputformat for MiniSparkonYarn
> -
>
> Key: HIVE-19596
> URL: https://issues.apache.org/jira/browse/HIVE-19596
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19596.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19596) NOOP jira to see which tests are flaky on HiveQA

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19596:

Summary: NOOP jira to see which tests are flaky on HiveQA  (was: disable 
bucketizedhiveinputformat for MiniSparkonYarn)

> NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19596
> URL: https://issues.apache.org/jira/browse/HIVE-19596
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19596.patch
>
>
> Fails routinely due to some OOM



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19596) disable bucketizedhiveinputformat for MiniSparkonYarn

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19596:

Resolution: Duplicate
Status: Resolved  (was: Patch Available)

Actually I'd add it to another patch

> disable bucketizedhiveinputformat for MiniSparkonYarn
> -
>
> Key: HIVE-19596
> URL: https://issues.apache.org/jira/browse/HIVE-19596
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19596.patch
>
>
> Fails routinely due to some OOM



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19596) disable bucketizedhiveinputformat for MiniSparkonYarn

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19596:

Description: Fails routinely due to some OOM

> disable bucketizedhiveinputformat for MiniSparkonYarn
> -
>
> Key: HIVE-19596
> URL: https://issues.apache.org/jira/browse/HIVE-19596
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19596.patch
>
>
> Fails routinely due to some OOM



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19596) NOOP jira to see which tests are flaky on HiveQA

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481153#comment-16481153
 ] 

Sergey Shelukhin commented on HIVE-19596:
-

MiniMr is addressed elsehwhere, so is trigger test.
bucketizedhiveinputformat is flaky and will be disabled (seen it before)

> NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19596
> URL: https://issues.apache.org/jira/browse/HIVE-19596
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19596.patch
>
>
> Fails routinely due to some OOM



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19608) disable flaky tests 2

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19608:

Attachment: (was: HIVE-19608.patch)

> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError: 
> Client Execution succeeded but contained differences (error code = 1) after 
> executing union_stats.q 
> 362a363
> > COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs
> bucketizedinputformat for MiniSparkOnYarn; OOMs occasionally, example 
> https://issues.apache.org/jira/browse/HIVE-19596



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Reopened] (HIVE-19596) NOOP jira to see which tests are flaky on HiveQA

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reopened HIVE-19596:
-

> NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19596
> URL: https://issues.apache.org/jira/browse/HIVE-19596
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19596.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19596) NOOP jira to see which tests are flaky on HiveQA

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19596:

Description: (was: Fails routinely due to some OOM)

> NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19596
> URL: https://issues.apache.org/jira/browse/HIVE-19596
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19596.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19608) disable flaky tests 2

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19608:

Attachment: HIVE-19608.patch

> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError: 
> Client Execution succeeded but contained differences (error code = 1) after 
> executing union_stats.q 
> 362a363
> > COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs
> bucketizedinputformat for MiniSparkOnYarn; OOMs occasionally, example 
> https://issues.apache.org/jira/browse/HIVE-19596



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19608) disable flaky tests 2

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19608:

Description: 
union_stats
{noformat}
java.lang.AssertionError: 
Client Execution succeeded but contained differences (error code = 1) after 
executing union_stats.q 
362a363
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
364a366,367
>   numRows 1000
>   rawDataSize 10624
{noformat}
Every few runs
bucketizedinputformat for MiniSparkOnYarn; OOMs occasionally, example 
https://issues.apache.org/jira/browse/HIVE-19596





  was:
union_stats
{noformat}
java.lang.AssertionError: 
Client Execution succeeded but contained differences (error code = 1) after 
executing union_stats.q 
362a363
>   COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
364a366,367
>   numRows 1000
>   rawDataSize 10624
{noformat}
Every few runs


> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError: 
> Client Execution succeeded but contained differences (error code = 1) after 
> executing union_stats.q 
> 362a363
> > COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs
> bucketizedinputformat for MiniSparkOnYarn; OOMs occasionally, example 
> https://issues.apache.org/jira/browse/HIVE-19596



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Issue Comment Deleted] (HIVE-19596) NOOP jira to see which tests are flaky on HiveQA

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19596:

Comment: was deleted

(was: Actually I'd add it to another patch)

> NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19596
> URL: https://issues.apache.org/jira/browse/HIVE-19596
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19596.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19589) NOOP jira to see which tests are flaky on HiveQA

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481156#comment-16481156
 ] 

Sergey Shelukhin commented on HIVE-19589:
-

I think no stats exception is a red herring, I see it in logs all over the 
place but it happens after real failure.
I filed some jiras/patches for some other issues. Some tests need stats or 
lineage masking, 2 are flaky q files, many failures are due to tmp dir issues.

> NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19589
> URL: https://issues.apache.org/jira/browse/HIVE-19589
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19589.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19608) disable flaky tests 2

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19608:

Status: Patch Available  (was: Open)

Should be ok to commit w/o QA since this just removes tests.
I'll file follow-ups to fix. 

> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError: 
> Client Execution succeeded but contained differences (error code = 1) after 
> executing union_stats.q 
> 362a363
> > COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs
> bucketizedinputformat for MiniSparkOnYarn; OOMs occasionally, example 
> https://issues.apache.org/jira/browse/HIVE-19596



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19589) NOOP jira to see which tests are flaky on HiveQA

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481161#comment-16481161
 ] 

Sergey Shelukhin commented on HIVE-19589:
-

Yeah, checked the code, it's just logged. I will clean it up, no need to log 
the callstack

> NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19589
> URL: https://issues.apache.org/jira/browse/HIVE-19589
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19589.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-19609) pointless callstacks in the logs as usual

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19609:
---


> pointless callstacks in the logs as usual
> -
>
> Key: HIVE-19609
> URL: https://issues.apache.org/jira/browse/HIVE-19609
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19609.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19609) pointless callstacks in the logs as usual

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481163#comment-16481163
 ] 

Sergey Shelukhin commented on HIVE-19609:
-

[~thejas] can you take a look? trivial patch

> pointless callstacks in the logs as usual
> -
>
> Key: HIVE-19609
> URL: https://issues.apache.org/jira/browse/HIVE-19609
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19609.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19609) pointless callstacks in the logs as usual

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19609:

Attachment: HIVE-19609.patch

> pointless callstacks in the logs as usual
> -
>
> Key: HIVE-19609
> URL: https://issues.apache.org/jira/browse/HIVE-19609
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19609.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19609) pointless callstacks in the logs as usual

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19609:

Status: Patch Available  (was: Open)

> pointless callstacks in the logs as usual
> -
>
> Key: HIVE-19609
> URL: https://issues.apache.org/jira/browse/HIVE-19609
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19609.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19579) remove HBase transitive dependency that drags in some snapshot

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481168#comment-16481168
 ] 

Sergey Shelukhin commented on HIVE-19579:
-

After looking at flaky tests today, all of these are known flaky.. guess we'll 
keep running HiveQA and disabling tests in parallel until this can be committed 
:)

> remove HBase transitive dependency that drags in some snapshot
> --
>
> Key: HIVE-19579
> URL: https://issues.apache.org/jira/browse/HIVE-19579
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19579.01.patch, HIVE-19579.02.patch, 
> HIVE-19579.patch
>
>
> This:
> {noformat}
> [INFO] org.apache.hive:hive-llap-server:jar:3.1.0-SNAPSHOT
> [INFO] \- org.apache.hbase:hbase-server:jar:2.0.0-alpha4:compile
> [INFO]\- org.glassfish.web:javax.servlet.jsp:jar:2.3.2:compile
> [INFO]   \- org.glassfish:javax.el:jar:3.0.1-b10:compile (version 
> selected from constraint [3.0.0,))
> {noformat}
> occasionally results in bizarre build failures like this:
> {noformat}
> Failed to read artifact descriptor for 
> org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Could not transfer artifact 
> org.glassfish:javax.el:pom:3.0.1-b06-SNAPSHOT from/to jvnet-nexus-snapshots 
> (https://maven.java.net/content/repositories/snapshots): Failed to transfer 
> file: 
> https://maven.java.net/content/repositories/snapshots/org/glassfish/javax.el/3.0.1-b06-SNAPSHOT/javax.el-3.0.1-b06-SNAPSHOT.pom.
>  Return code is: 402 , ReasonPhrase:Payment Required.
> {noformat}
> cc [~enis] [~stack] if you guys want to take a look at HBase side



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19579) remove HBase transitive dependency that drags in some snapshot

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19579:

Attachment: HIVE-19579.02.patch

> remove HBase transitive dependency that drags in some snapshot
> --
>
> Key: HIVE-19579
> URL: https://issues.apache.org/jira/browse/HIVE-19579
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19579.01.patch, HIVE-19579.02.patch, 
> HIVE-19579.patch
>
>
> This:
> {noformat}
> [INFO] org.apache.hive:hive-llap-server:jar:3.1.0-SNAPSHOT
> [INFO] \- org.apache.hbase:hbase-server:jar:2.0.0-alpha4:compile
> [INFO]\- org.glassfish.web:javax.servlet.jsp:jar:2.3.2:compile
> [INFO]   \- org.glassfish:javax.el:jar:3.0.1-b10:compile (version 
> selected from constraint [3.0.0,))
> {noformat}
> occasionally results in bizarre build failures like this:
> {noformat}
> Failed to read artifact descriptor for 
> org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Could not transfer artifact 
> org.glassfish:javax.el:pom:3.0.1-b06-SNAPSHOT from/to jvnet-nexus-snapshots 
> (https://maven.java.net/content/repositories/snapshots): Failed to transfer 
> file: 
> https://maven.java.net/content/repositories/snapshots/org/glassfish/javax.el/3.0.1-b06-SNAPSHOT/javax.el-3.0.1-b06-SNAPSHOT.pom.
>  Return code is: 402 , ReasonPhrase:Payment Required.
> {noformat}
> cc [~enis] [~stack] if you guys want to take a look at HBase side



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19595) disable webui in MiniHS2

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481171#comment-16481171
 ] 

Sergey Shelukhin commented on HIVE-19595:
-

I'll wait for QA run for this one, in case some tests rely on it... will commit 
if only known flaky tests fail.

> disable webui in MiniHS2
> 
>
> Key: HIVE-19595
> URL: https://issues.apache.org/jira/browse/HIVE-19595
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19595.01.patch, HIVE-19595.patch
>
>
> Causes tests flakiness if they same port is used.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19608) disable flaky tests 2

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19608:

   Resolution: Fixed
Fix Version/s: 3.1.0
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks for the review!

> disable flaky tests 2
> -
>
> Key: HIVE-19608
> URL: https://issues.apache.org/jira/browse/HIVE-19608
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19608.patch
>
>
> union_stats
> {noformat}
> java.lang.AssertionError: 
> Client Execution succeeded but contained differences (error code = 1) after 
> executing union_stats.q 
> 362a363
> > COLUMN_STATS_ACCURATE   {\"BASIC_STATS\":\"true\"}
> 364a366,367
> > numRows 1000
> > rawDataSize 10624
> {noformat}
> Every few runs
> bucketizedinputformat for MiniSparkOnYarn; OOMs occasionally, example 
> https://issues.apache.org/jira/browse/HIVE-19596



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-19610) reeenable union_stats test if needed

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19610:
---


> reeenable union_stats test if needed
> 
>
> Key: HIVE-19610
> URL: https://issues.apache.org/jira/browse/HIVE-19610
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Ashutosh Chauhan
>Priority: Major
>
> There's some flaky stats diff. See HIVE-19608 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (HIVE-19611) reenable BHIF test on SparkonYarn if needed

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-19611:
---


> reenable BHIF test on SparkonYarn if needed
> ---
>
> Key: HIVE-19611
> URL: https://issues.apache.org/jira/browse/HIVE-19611
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sahil Takiar
>Priority: Major
>
> See HIVE-19608. Fails occasionally, looks like it's caused by an OOM in some 
> task.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19578) HLL merges tempList on every add

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481303#comment-16481303
 ] 

Sergey Shelukhin commented on HIVE-19578:
-

"if (sparseMap.containsKey" block is gone; by design?
+1

> HLL merges tempList on every add
> 
>
> Key: HIVE-19578
> URL: https://issues.apache.org/jira/browse/HIVE-19578
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19578.1.patch, HIVE-19578.2.patch, Screen Shot 
> 2018-05-16 at 15.29.12 .png
>
>
>  See comments on HIVE-18866; this has significant perf overhead after the 
> even bigger overhead from hashing is removed.  !Screen Shot 2018-05-16 at 
> 15.29.12 .png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19612) Add option to mask lineage in q files

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481326#comment-16481326
 ] 

Sergey Shelukhin commented on HIVE-19612:
-

+1

> Add option to mask lineage in q files
> -
>
> Key: HIVE-19612
> URL: https://issues.apache.org/jira/browse/HIVE-19612
> Project: Hive
>  Issue Type: Test
>  Components: Test, Testing Infrastructure
>Affects Versions: 3.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-19612.patch
>
>
> Similar to {{HIVE-19572}}, with {{-- MASK_LINEAGE}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481335#comment-16481335
 ] 

Sergey Shelukhin commented on HIVE-18629:
-

See the discussion above. [~mmccline] doesn't like it :) It may be superseded 
by the epic vectorization null fix patch.

> copyValues in BytesColumnVector may be missing null checks
> --
>
> Key: HIVE-18629
> URL: https://issues.apache.org/jira/browse/HIVE-18629
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch, 
> HIVE-18629.03.patch, HIVE-18629.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
>   at java.lang.System.arraycopy(Native Method)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133)
> {noformat}
> IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case 
> has isNull checks for v2/v3 buffers that copySelected is missing. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19592) CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481336#comment-16481336
 ] 

Sergey Shelukhin commented on HIVE-19592:
-

I've seen other directories also disappear there that are not hadoop dirs e.g. 
/tmp/testsmthorother. 

> CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19592
> URL: https://issues.apache.org/jira/browse/HIVE-19592
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19592.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (HIVE-19589) NOOP jira to see which tests are flaky on HiveQA

2018-05-18 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481364#comment-16481364
 ] 

Sergey Shelukhin commented on HIVE-19589:
-

+1 to disable

> NOOP jira to see which tests are flaky on HiveQA
> 
>
> Key: HIVE-19589
> URL: https://issues.apache.org/jira/browse/HIVE-19589
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19589.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19463) TezTask - getting groups may fail (PartialGroupNameException in some tests)

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19463:

Attachment: HIVE-19463.06.patch

> TezTask - getting groups may fail (PartialGroupNameException in some tests)
> ---
>
> Key: HIVE-19463
> URL: https://issues.apache.org/jira/browse/HIVE-19463
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19463.01.patch, HIVE-19463.02.patch, 
> HIVE-19463.03.patch, HIVE-19463.03.patch, HIVE-19463.04.patch, 
> HIVE-19463.05.patch, HIVE-19463.06.patch, HIVE-19463.patch
>
>
> {noformat}
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping$PartialGroupNameException:
>  The user name 'hive_test_user' is not found. id: hive_test_user: no such user
> id: hive_test_user: no such user
>   at 
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:294)
>  ~[hadoop-common-3.1.0.jar:?]
>   at 
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:207)
>  [hadoop-common-3.1.0.jar:?]
>   at 
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:97)
>  [hadoop-common-3.1.0.jar:?]
>   at 
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:51)
>  [hadoop-common-3.1.0.jar:?]
>   at 
> org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Groups.java:384)
>  [hadoop-common-3.1.0.jar:?]
>   at 
> org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:319) 
> [hadoop-common-3.1.0.jar:?]
>   at 
> org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:269) 
> [hadoop-common-3.1.0.jar:?]
>   at 
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542)
>  [guava-19.0.jar:?]
>   at 
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323) 
> [guava-19.0.jar:?]
>   at 
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286)
>  [guava-19.0.jar:?]
>   at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) 
> [guava-19.0.jar:?]
>   at com.google.common.cache.LocalCache.get(LocalCache.java:3953) 
> [guava-19.0.jar:?]
>   at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3957) 
> [guava-19.0.jar:?]
>   at 
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4875)
>  [guava-19.0.jar:?]
>   at org.apache.hadoop.security.Groups.getGroups(Groups.java:227) 
> [hadoop-common-3.1.0.jar:?]
>   at 
> org.apache.hadoop.security.UserGroupInformation.getGroups(UserGroupInformation.java:1540)
>  [hadoop-common-3.1.0.jar:?]
>   at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:163) 
> [hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18866:

Attachment: HIVE-18866.03.patch

> Semijoin: Implement a Long -> Hash64 vector fast-path
> -
>
> Key: HIVE-18866
> URL: https://issues.apache.org/jira/browse/HIVE-18866
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
>Priority: Major
>  Labels: performance
> Attachments: 0001-hash64-WIP.patch, HIVE-18866.01.patch, 
> HIVE-18866.02.patch, HIVE-18866.03.patch, HIVE-18866.patch, 
> perf-hash64-long.png
>
>
> A significant amount of CPU is wasted with JMM restrictions on byte[] arrays.
> To transform from one Long -> another Long, this goes into a byte[] array, 
> which shows up as a hotspot.
> !perf-hash64-long.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19595) disable webui in MiniHS2

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19595:

Attachment: HIVE-19595.02.patch

> disable webui in MiniHS2
> 
>
> Key: HIVE-19595
> URL: https://issues.apache.org/jira/browse/HIVE-19595
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19595.01.patch, HIVE-19595.02.patch, 
> HIVE-19595.patch
>
>
> Causes tests flakiness if they same port is used.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19609) pointless callstacks in the logs as usual

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19609:

Attachment: HIVE-19609.01.patch

> pointless callstacks in the logs as usual
> -
>
> Key: HIVE-19609
> URL: https://issues.apache.org/jira/browse/HIVE-19609
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19609.01.patch, HIVE-19609.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (HIVE-19516) TestNegative merge_negative_5 and mm_concatenate are causing timeouts

2018-05-18 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-19516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-19516:

   Resolution: Fixed
Fix Version/s: 3.1.0
   Status: Resolved  (was: Patch Available)

Committed to master since this is a test fix. Thanks for the review!

> TestNegative merge_negative_5 and mm_concatenate are causing timeouts
> -
>
> Key: HIVE-19516
> URL: https://issues.apache.org/jira/browse/HIVE-19516
> Project: Hive
>  Issue Type: Bug
>  Components: Testing Infrastructure
>Reporter: Vineet Garg
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19516.01.patch, HIVE-19516.02.patch, 
> HIVE-19516.03.patch, HIVE-19516.patch
>
>
> I haven't tried to reproduce this in isolation but it is reproducible if you 
> run in batch on local system 
> {noformat}
> mvn -B test  -Dtest.groups= -Dtest=TestNegativeCliDriver 
> -Dqfile=udf_invalid.q,authorization_uri_export.q,default_constraint_complex_default_value.q,druid_datasource2.q,check_constraint_max_length.q,view_update.q,default_partition_name.q,authorization_public_create.q,load_wrong_fileformat_rc_seq.q,default_constraint_invalid_type.q,altern1.q,describe_xpath1.q,drop_view_failure2.q,temp_table_rename.q,invalid_select_column_with_subquery.q,udf_trunc_error1.q,insert_view_failure.q,dbtxnmgr_nodbunlock.q,authorization_show_columns.q,cte_recursion.q,merge_constraint_notnull.q,clusterbyorderby.q,orc_type_promotion2.q,ctas_noperm_loc.q,udf_min.q,udf_instr_wrong_args_len.q,invalid_create_tbl2.q,part_col_complex_type.q,authorization_drop_db_empty.q,smb_mapjoin_14.q,subquery_scalar_multi_rows.q,alter_partition_coltype_2columns.q,subquery_corr_in_agg.q,insert_overwrite_notnull_constraint.q,authorization_show_grant_otheruser_wtab.q,regex_col_groupby.q,ptf_negative_DuplicateWindowAlias.q,exim_22_export_authfail.q,authorization_insert_noinspriv.q,udf_likeany_wrong1.q,groupby_key.q,ambiguous_col.q,groupby3_multi_distinct.q,authorization_alter_drop_ptn.q,invalid_cast_from_binary_5.q,show_create_table_does_not_exist.q,invalid_select_column.q,exim_20_managed_location_over_existing.q,interval_3.q,authorization_compile.q,join35.q,udf_concat_ws_wrong3.q,create_or_replace_view8.q,create_external_with_notnull_constraint.q,split_sample_out_of_range.q,materialized_view_no_transactional_rewrite.q,authorization_show_grant_otherrole.q,create_with_constraints_duplicate_name.q,invalid_stddev_samp_syntax.q,authorization_view_disable_cbo_7.q,autolocal1.q,avro_non_nullable_union.q,load_orc_negative_part.q,drop_view_failure1.q,columnstats_partlvl_invalid_values_autogather.q,exim_13_nonnative_import.q,alter_table_wrong_regex.q,udf_next_day_error_2.q,authorization_select.q,udf_trunc_error2.q,authorization_view_7.q,udf_format_number_wrong5.q,touch2.q,orc_type_promotion1.q,lateral_view_alias.q,show_tables_bad_db1.q,unset_table_property.q,alter_non_native.q,nvl_mismatch_type.q,load_orc_negative3.q,authorization_create_role_no_admin.q,invalid_distinct1.q,authorization_grant_server.q,orc_type_promotion3_acid.q,hms_using_serde_alter_table_update_columns.q,show_tables_bad1.q,macro_unused_parameter.q,drop_invalid_constraint3.q,drop_partition_filter_failure.q,char_pad_convert_fail3.q,exim_23_import_exist_authfail.q,drop_invalid_constraint4.q,authorization_create_macro1.q,archive1.q,subquery_multiple_cols_in_select.q,change_hive_hdfs_session_path.q,udf_trunc_error3.q,invalid_variance_syntax.q,authorization_truncate_2.q,invalid_avg_syntax.q,invalid_select_column_with_tablename.q,mm_truncate_cols.q,groupby_grouping_sets1.q,druid_location.q,groupby2_multi_distinct.q,authorization_sba_drop_table.q,dynamic_partitions_with_whitelist.q,compare_string_bigint_2.q,udf_greatest_error_2.q,authorization_view_6.q,show_tablestatus.q,duplicate_alias_in_transform_schema.q,create_with_fk_uk_same_tab.q,udtf_not_supported3.q,alter_table_constraint_invalid_fk_col2.q,udtf_not_supported1.q,dbtxnmgr_notableunlock.q,ptf_negative_InvalidValueBoundary.q,alter_table_constraint_duplicate_pk.q,udf_printf_wrong4.q,create_view_failure9.q,udf_elt_wrong_type.q,selectDistinctStarNeg_1.q,invalid_mapjoin1.q,load_stored_as_dirs.q,input1.q,udf_sort_array_wrong1.q,invalid_distinct2.q,invalid_select_fn.q,authorization_role_grant_otherrole.q,archive4.q,load_nonpart_authfail.q,recursive_view.q,authorization_view_disable_cbo_1.q,desc_failure4.q,create_not_acid.q,udf_sort_array_wrong3.q,char_pad_convert_fail0.q,udf_map_values_arg_type.q,alter_view_failure6_2.q,alter_partition_change_col_nonexist.q,update_non_acid_table.q,authorization_view_disable_cbo_5.q,ct_noperm_loc.q,interval_1.q,authorization_show_grant_otheruser_all.q,authorization_view_2.q,show_tables_bad2.q,groupby_rollup2.q,truncate_column_seqfile.q,create_v

<    1   2   3   4   5   6   7   8   9   10   >