[jira] [Commented] (HIVE-23294) Remove sync bottleneck in TezConfigurationFactory
[ https://issues.apache.org/jira/browse/HIVE-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095005#comment-17095005 ] Rajesh Balamohan commented on HIVE-23294: - Tried out .2 in the cluster. Haven't seen major CPU burn with reflection and also removes the locking issue. +1. > Remove sync bottleneck in TezConfigurationFactory > - > > Key: HIVE-23294 > URL: https://issues.apache.org/jira/browse/HIVE-23294 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23294.1.patch, HIVE-23294.2.patch, Screenshot > 2020-04-24 at 1.53.20 PM.png > > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezConfigurationFactory.java#L53] > [https://github.com/apache/hadoop/blob/master/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L1628] > It ends up locking for property names in the config. For short running > queries with concurrency, this is an issue. > > !Screenshot 2020-04-24 at 1.53.20 PM.png|width=1086,height=459! > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23294) Remove sync bottleneck in TezConfigurationFactory
[ https://issues.apache.org/jira/browse/HIVE-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17093054#comment-17093054 ] Rajesh Balamohan commented on HIVE-23294: - [~ashutoshc] : Thanks for sharing the patch. Impact with reflection approach is that, it ends up checking for member access every time it is accessed adding to the overhead. > Remove sync bottleneck in TezConfigurationFactory > - > > Key: HIVE-23294 > URL: https://issues.apache.org/jira/browse/HIVE-23294 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23294.1.patch, HIVE-23294.2.patch, Screenshot > 2020-04-24 at 1.53.20 PM.png > > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezConfigurationFactory.java#L53] > [https://github.com/apache/hadoop/blob/master/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L1628] > It ends up locking for property names in the config. For short running > queries with concurrency, this is an issue. > > !Screenshot 2020-04-24 at 1.53.20 PM.png|width=1086,height=459! > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23294) Remove sync bottleneck in TezConfigurationFactory
[
https://issues.apache.org/jira/browse/HIVE-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092744#comment-17092744
]
Hive QA commented on HIVE-23294:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001197/HIVE-23294.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17143 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21953/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21953/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21953/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001197 - PreCommit-HIVE-Build
> Remove sync bottleneck in TezConfigurationFactory
> -
>
> Key: HIVE-23294
> URL: https://issues.apache.org/jira/browse/HIVE-23294
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23294.1.patch, HIVE-23294.2.patch, Screenshot
> 2020-04-24 at 1.53.20 PM.png
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezConfigurationFactory.java#L53]
> [https://github.com/apache/hadoop/blob/master/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L1628]
> It ends up locking for property names in the config. For short running
> queries with concurrency, this is an issue.
>
> !Screenshot 2020-04-24 at 1.53.20 PM.png|width=1086,height=459!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23294) Remove sync bottleneck in TezConfigurationFactory
[
https://issues.apache.org/jira/browse/HIVE-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092735#comment-17092735
]
Hive QA commented on HIVE-23294:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 1 new + 4 unchanged - 0 fixed
= 5 total (was 4) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21953/dev-support/hive-personality.sh
|
| git revision | master / abbe72e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21953/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21953/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove sync bottleneck in TezConfigurationFactory
> -
>
> Key: HIVE-23294
> URL: https://issues.apache.org/jira/browse/HIVE-23294
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23294.1.patch, HIVE-23294.2.patch, Screenshot
> 2020-04-24 at 1.53.20 PM.png
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezConfigurationFactory.java#L53]
> [https://github.com/apache/hadoop/blob/master/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L1628]
> It ends up locking for property names in the config. For short running
> queries with concurrency, this is an issue.
>
> !Screenshot 2020-04-24 at 1.53.20 PM.png|width=1086,height=459!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23294) Remove sync bottleneck in TezConfigurationFactory
[ https://issues.apache.org/jira/browse/HIVE-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092543#comment-17092543 ] Ashutosh Chauhan commented on HIVE-23294: - [~rajesh.balamohan] what do you think of my patch. Arguably it might be faster since we never hit synchronized method nor we need to worry about memory. Though reflection access might be as fast as hashmap lookup. > Remove sync bottleneck in TezConfigurationFactory > - > > Key: HIVE-23294 > URL: https://issues.apache.org/jira/browse/HIVE-23294 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23294.1.patch, HIVE-23294.2.patch, Screenshot > 2020-04-24 at 1.53.20 PM.png > > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezConfigurationFactory.java#L53] > [https://github.com/apache/hadoop/blob/master/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L1628] > It ends up locking for property names in the config. For short running > queries with concurrency, this is an issue. > > !Screenshot 2020-04-24 at 1.53.20 PM.png|width=1086,height=459! > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23294) Remove sync bottleneck in TezConfigurationFactory
[
https://issues.apache.org/jira/browse/HIVE-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092017#comment-17092017
]
Hive QA commented on HIVE-23294:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001057/HIVE-23294.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17141 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMergeOnTezEdges
(batchId=275)
org.apache.hadoop.hive.ql.parse.TestScheduledReplicationScenarios.testAcidTablesReplLoadBootstrapIncr
(batchId=206)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21927/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21927/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21927/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001057 - PreCommit-HIVE-Build
> Remove sync bottleneck in TezConfigurationFactory
> -
>
> Key: HIVE-23294
> URL: https://issues.apache.org/jira/browse/HIVE-23294
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23294.1.patch, Screenshot 2020-04-24 at 1.53.20
> PM.png
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezConfigurationFactory.java#L53]
> [https://github.com/apache/hadoop/blob/master/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L1628]
> It ends up locking for property names in the config. For short running
> queries with concurrency, this is an issue.
>
> !Screenshot 2020-04-24 at 1.53.20 PM.png|width=1086,height=459!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23294) Remove sync bottleneck in TezConfigurationFactory
[
https://issues.apache.org/jira/browse/HIVE-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092013#comment-17092013
]
Hive QA commented on HIVE-23294:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 5m
44s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
15s{color} | {color:red} ql: The patch generated 2 new + 4 unchanged - 0 fixed
= 6 total (was 4) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21927/dev-support/hive-personality.sh
|
| git revision | master / 88053b2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21927/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21927/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove sync bottleneck in TezConfigurationFactory
> -
>
> Key: HIVE-23294
> URL: https://issues.apache.org/jira/browse/HIVE-23294
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23294.1.patch, Screenshot 2020-04-24 at 1.53.20
> PM.png
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezConfigurationFactory.java#L53]
> [https://github.com/apache/hadoop/blob/master/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L1628]
> It ends up locking for property names in the config. For short running
> queries with concurrency, this is an issue.
>
> !Screenshot 2020-04-24 at 1.53.20 PM.png|width=1086,height=459!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
