[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734549#comment-16734549
]
Hive QA commented on HIVE-20760:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12953790/HIVE-20760.13.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15723 tests
executed
*Failed tests:*
{noformat}
TestCompactor - did not produce a TEST-*.xml file (likely timed out)
(batchId=242)
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_delimited]
(batchId=274)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15504/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15504/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15504/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12953790 - PreCommit-HIVE-Build
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.10.patch, HIVE-20760.11.patch,
> HIVE-20760.12.patch, HIVE-20760.13.patch, HIVE-20760.4.patch,
> HIVE-20760.5.patch, HIVE-20760.6.patch, HIVE-20760.7.patch,
> HIVE-20760.8.patch, HIVE-20760.9.patch, HIVE-20760.patch,
> hiveconf_interned.html, hiveconf_original.html
>
>
> The issue is that every Hive task has to load its own version of
> {{HiveConf}}. When running with a large number of cores per executor (HoS),
> there is a significant (~10%) amount of memory wasted due to this
> duplication.
> I looked into the problem and found a way to reduce the overhead caused by
> the multiple HiveConf objects.
> I've created an implementation of Properties, somewhat similar to
> CopyOnFirstWriteProperties. CopyOnFirstWriteProperties can't be used to solve
> this problem, because it drops the interned Properties right after we add a
> new property.
> So my implementation looks like this:
> * When we create a new HiveConf from an existing one (copy constructor), we
> change the properties object stored by HiveConf to the new Properties
> implementation (HiveConfProperties). We have 2 possible way to do this.
> Either we change the visibility of the properties field in the ancestor class
> (Configuration which comes from hadoop) to protected, or a simpler way is to
> just change the type using reflection.
> * HiveConfProperties instantly intern the given properties. After this,
> every time we add a new property to HiveConf, we add it to an additional
> Properties object. This way if we create multiple HiveConf with the same base
> properties, they will use the same Properties object but each session/task
> can add its own unique properties.
> * Getting a property from HiveConfProperties would look like this: (I stored
> the non-interned properties in super class)
> String property=super.getProperty(key);
> if (property == null) property= interned.getProperty(key);
> return property;
> Running some tests showed that the interning works (with 50 connections to
> HiveServer2, heapdumps created after sessions are created for queries):
> Overall memory:
> original: 34,599K interned: 20,582K
> Retained memory of HiveConfs:
> original: 16,366K interned: 10,804K
> I attach the JXray reports about the heapdumps.
> What are your thoughts about this solution?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734526#comment-16734526
]
Hive QA commented on HIVE-20760:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
41s{color} | {color:red} common generated 2 new + 65 unchanged - 0 fixed = 67
total (was 65) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:common |
| | Inconsistent synchronization of
org.apache.hadoop.hive.common.HiveConfProperties.interned; locked 76% of time
Unsynchronized access at HiveConfProperties.java:76% of time Unsynchronized
access at HiveConfProperties.java:[line 106] |
| | org.apache.hadoop.hive.common.HiveConfProperties.getProperty(String,
String) is unsynchronized,
org.apache.hadoop.hive.common.HiveConfProperties.setProperty(String, String) is
synchronized At HiveConfProperties.java:String) is synchronized At
HiveConfProperties.java:[lines 121-128] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15504/dev-support/hive-personality.sh
|
| git revision | master / 16d39c6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15504/yetus/new-findbugs-common.html
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15504/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.10.patch, HIVE-20760.11.patch,
> HIVE-20760.12.patch, HIVE-20760.13.patch, HIVE-20760.4.patch,
> HIVE-20760.5.patch, HIVE-20760.6.patch, HIVE-20760.7.patch,
> HIVE-20760.8.patch, HIVE-20760.9.patch, HIVE-20760.patch,
> hiveconf_interned.html, hiveconf_original.html
>
>
> The issue is that every Hive task has to load its own version of
> {{HiveConf}}. When running with a large number of cores per executor (HoS),
> there is a significant
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723982#comment-16723982
]
Hive QA commented on HIVE-20760:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12952160/HIVE-20760.12.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 15693 tests
executed
*Failed tests:*
{noformat}
TestAlterTableMetadata - did not produce a TEST-*.xml file (likely timed out)
(batchId=250)
TestCompactor - did not produce a TEST-*.xml file (likely timed out)
(batchId=241)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=250)
TestSemanticAnalyzerHookLoading - did not produce a TEST-*.xml file (likely
timed out) (batchId=250)
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testOperationsOnCompletedTxnComponentsForMmTable
(batchId=283)
org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testOperationsOnCompletedTxnComponentsForMmTable
(batchId=303)
org.apache.hadoop.hive.ql.TestTxnConcatenate.testConcatenateMM (batchId=292)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15365/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15365/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15365/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12952160 - PreCommit-HIVE-Build
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.10.patch, HIVE-20760.11.patch,
> HIVE-20760.12.patch, HIVE-20760.4.patch, HIVE-20760.5.patch,
> HIVE-20760.6.patch, HIVE-20760.7.patch, HIVE-20760.8.patch,
> HIVE-20760.9.patch, HIVE-20760.patch, hiveconf_interned.html,
> hiveconf_original.html
>
>
> The issue is that every Hive task has to load its own version of
> {{HiveConf}}. When running with a large number of cores per executor (HoS),
> there is a significant (~10%) amount of memory wasted due to this
> duplication.
> I looked into the problem and found a way to reduce the overhead caused by
> the multiple HiveConf objects.
> I've created an implementation of Properties, somewhat similar to
> CopyOnFirstWriteProperties. CopyOnFirstWriteProperties can't be used to solve
> this problem, because it drops the interned Properties right after we add a
> new property.
> So my implementation looks like this:
> * When we create a new HiveConf from an existing one (copy constructor), we
> change the properties object stored by HiveConf to the new Properties
> implementation (HiveConfProperties). We have 2 possible way to do this.
> Either we change the visibility of the properties field in the ancestor class
> (Configuration which comes from hadoop) to protected, or a simpler way is to
> just change the type using reflection.
> * HiveConfProperties instantly intern the given properties. After this,
> every time we add a new property to HiveConf, we add it to an additional
> Properties object. This way if we create multiple HiveConf with the same base
> properties, they will use the same Properties object but each session/task
> can add its own unique properties.
> * Getting a property from HiveConfProperties would look like this: (I stored
> the non-interned properties in super class)
> String property=super.getProperty(key);
> if (property == null) property= interned.getProperty(key);
> return property;
> Running some tests showed that the interning works (with 50 connections to
> HiveServer2, heapdumps created after sessions are created for queries):
> Overall memory:
> original: 34,599K interned: 20,582K
> Retained memory of HiveConfs:
> original: 16,366K interned: 10,804K
> I attach the JXray reports about the heapdumps.
> What are your thoughts about this solution?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723933#comment-16723933
]
Hive QA commented on HIVE-20760:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} common: The patch generated 1 new + 428 unchanged - 0
fixed = 429 total (was 428) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
37s{color} | {color:red} common generated 2 new + 65 unchanged - 0 fixed = 67
total (was 65) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:common |
| | Inconsistent synchronization of
org.apache.hadoop.hive.common.HiveConfProperties.interned; locked 74% of time
Unsynchronized access at HiveConfProperties.java:74% of time Unsynchronized
access at HiveConfProperties.java:[line 108] |
| | org.apache.hadoop.hive.common.HiveConfProperties.getProperty(String,
String) is unsynchronized,
org.apache.hadoop.hive.common.HiveConfProperties.setProperty(String, String) is
synchronized At HiveConfProperties.java:String) is synchronized At
HiveConfProperties.java:[lines 123-130] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15365/dev-support/hive-personality.sh
|
| git revision | master / ef7c396 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15365/yetus/diff-checkstyle-common.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15365/yetus/new-findbugs-common.html
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15365/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.10.patch, HIVE-20760.11.patch,
> HIVE-20760.12.patch, HIVE-20760.4.patch, HIVE-20760.5.patch,
> HIVE-20760.6.patch, HIVE-20760.7.patch, HIVE-20760.8.patch,
> HIVE-20760.9.patch, HIVE-20760.patch, hiveconf_interned.html,
> hiveconf_original.html
>
>
> The issue is that every Hive
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721532#comment-16721532
]
Hive QA commented on HIVE-20760:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} common: The patch generated 1 new + 428 unchanged - 0
fixed = 429 total (was 428) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
38s{color} | {color:red} common generated 3 new + 65 unchanged - 0 fixed = 68
total (was 65) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:common |
| | org.apache.hadoop.hive.common.HiveConfProperties.clone() does not call
super.clone() At HiveConfProperties.java: At HiveConfProperties.java:[line
271] |
| | Inconsistent synchronization of
org.apache.hadoop.hive.common.HiveConfProperties.interned; locked 74% of time
Unsynchronized access at HiveConfProperties.java:74% of time Unsynchronized
access at HiveConfProperties.java:[line 106] |
| | org.apache.hadoop.hive.common.HiveConfProperties.getProperty(String,
String) is unsynchronized,
org.apache.hadoop.hive.common.HiveConfProperties.setProperty(String, String) is
synchronized At HiveConfProperties.java:String) is synchronized At
HiveConfProperties.java:[lines 121-128] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15325/dev-support/hive-personality.sh
|
| git revision | master / 687aeef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15325/yetus/diff-checkstyle-common.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15325/yetus/new-findbugs-common.html
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15325/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.10.patch, HIVE-20760.11.patch,
> HIVE-20760.4.patch, HIVE-20760.5.patch, HIVE-20760.6.patch,
> HIVE-20760.7.patch,
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717224#comment-16717224
]
Hive QA commented on HIVE-20760:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951351/HIVE-20760.10.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15621 tests
executed
*Failed tests:*
{noformat}
TestCompactor - did not produce a TEST-*.xml file (likely timed out)
(batchId=241)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15257/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15257/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15257/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951351 - PreCommit-HIVE-Build
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.10.patch, HIVE-20760.4.patch,
> HIVE-20760.5.patch, HIVE-20760.6.patch, HIVE-20760.7.patch,
> HIVE-20760.8.patch, HIVE-20760.9.patch, HIVE-20760.patch,
> hiveconf_interned.html, hiveconf_original.html
>
>
> The issue is that every Hive task has to load its own version of
> {{HiveConf}}. When running with a large number of cores per executor (HoS),
> there is a significant (~10%) amount of memory wasted due to this
> duplication.
> I looked into the problem and found a way to reduce the overhead caused by
> the multiple HiveConf objects.
> I've created an implementation of Properties, somewhat similar to
> CopyOnFirstWriteProperties. CopyOnFirstWriteProperties can't be used to solve
> this problem, because it drops the interned Properties right after we add a
> new property.
> So my implementation looks like this:
> * When we create a new HiveConf from an existing one (copy constructor), we
> change the properties object stored by HiveConf to the new Properties
> implementation (HiveConfProperties). We have 2 possible way to do this.
> Either we change the visibility of the properties field in the ancestor class
> (Configuration which comes from hadoop) to protected, or a simpler way is to
> just change the type using reflection.
> * HiveConfProperties instantly intern the given properties. After this,
> every time we add a new property to HiveConf, we add it to an additional
> Properties object. This way if we create multiple HiveConf with the same base
> properties, they will use the same Properties object but each session/task
> can add its own unique properties.
> * Getting a property from HiveConfProperties would look like this: (I stored
> the non-interned properties in super class)
> String property=super.getProperty(key);
> if (property == null) property= interned.getProperty(key);
> return property;
> Running some tests showed that the interning works (with 50 connections to
> HiveServer2, heapdumps created after sessions are created for queries):
> Overall memory:
> original: 34,599K interned: 20,582K
> Retained memory of HiveConfs:
> original: 16,366K interned: 10,804K
> I attach the JXray reports about the heapdumps.
> What are your thoughts about this solution?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717088#comment-16717088
]
Hive QA commented on HIVE-20760:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
36s{color} | {color:red} common generated 3 new + 65 unchanged - 0 fixed = 68
total (was 65) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:common |
| | org.apache.hadoop.hive.common.HiveConfProperties.clone() does not call
super.clone() At HiveConfProperties.java: At HiveConfProperties.java:[line
271] |
| | Inconsistent synchronization of
org.apache.hadoop.hive.common.HiveConfProperties.interned; locked 74% of time
Unsynchronized access at HiveConfProperties.java:74% of time Unsynchronized
access at HiveConfProperties.java:[line 106] |
| | org.apache.hadoop.hive.common.HiveConfProperties.getProperty(String,
String) is unsynchronized,
org.apache.hadoop.hive.common.HiveConfProperties.setProperty(String, String) is
synchronized At HiveConfProperties.java:String) is synchronized At
HiveConfProperties.java:[lines 121-128] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15257/dev-support/hive-personality.sh
|
| git revision | master / 9925eb1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15257/yetus/new-findbugs-common.html
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15257/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.10.patch, HIVE-20760.4.patch,
> HIVE-20760.5.patch, HIVE-20760.6.patch, HIVE-20760.7.patch,
> HIVE-20760.8.patch, HIVE-20760.9.patch, HIVE-20760.patch,
> hiveconf_interned.html, hiveconf_original.html
>
>
> The issue is that every Hive task has to load its own version of
>
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715322#comment-16715322
]
Hive QA commented on HIVE-20760:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951218/HIVE-20760.9.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15621 tests
executed
*Failed tests:*
{noformat}
TestCompactor - did not produce a TEST-*.xml file (likely timed out)
(batchId=241)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15244/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15244/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15244/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951218 - PreCommit-HIVE-Build
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.4.patch, HIVE-20760.5.patch,
> HIVE-20760.6.patch, HIVE-20760.7.patch, HIVE-20760.8.patch,
> HIVE-20760.9.patch, HIVE-20760.patch, hiveconf_interned.html,
> hiveconf_original.html
>
>
> The issue is that every Hive task has to load its own version of
> {{HiveConf}}. When running with a large number of cores per executor (HoS),
> there is a significant (~10%) amount of memory wasted due to this
> duplication.
> I looked into the problem and found a way to reduce the overhead caused by
> the multiple HiveConf objects.
> I've created an implementation of Properties, somewhat similar to
> CopyOnFirstWriteProperties. CopyOnFirstWriteProperties can't be used to solve
> this problem, because it drops the interned Properties right after we add a
> new property.
> So my implementation looks like this:
> * When we create a new HiveConf from an existing one (copy constructor), we
> change the properties object stored by HiveConf to the new Properties
> implementation (HiveConfProperties). We have 2 possible way to do this.
> Either we change the visibility of the properties field in the ancestor class
> (Configuration which comes from hadoop) to protected, or a simpler way is to
> just change the type using reflection.
> * HiveConfProperties instantly intern the given properties. After this,
> every time we add a new property to HiveConf, we add it to an additional
> Properties object. This way if we create multiple HiveConf with the same base
> properties, they will use the same Properties object but each session/task
> can add its own unique properties.
> * Getting a property from HiveConfProperties would look like this: (I stored
> the non-interned properties in super class)
> String property=super.getProperty(key);
> if (property == null) property= interned.getProperty(key);
> return property;
> Running some tests showed that the interning works (with 50 connections to
> HiveServer2, heapdumps created after sessions are created for queries):
> Overall memory:
> original: 34,599K interned: 20,582K
> Retained memory of HiveConfs:
> original: 16,366K interned: 10,804K
> I attach the JXray reports about the heapdumps.
> What are your thoughts about this solution?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715197#comment-16715197
]
Hive QA commented on HIVE-20760:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} common: The patch generated 2 new + 428 unchanged - 0
fixed = 430 total (was 428) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
40s{color} | {color:red} common generated 3 new + 65 unchanged - 0 fixed = 68
total (was 65) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:common |
| | org.apache.hadoop.hive.common.HiveConfProperties.clone() does not call
super.clone() At HiveConfProperties.java: At HiveConfProperties.java:[line
273] |
| | Inconsistent synchronization of
org.apache.hadoop.hive.common.HiveConfProperties.interned; locked 74% of time
Unsynchronized access at HiveConfProperties.java:74% of time Unsynchronized
access at HiveConfProperties.java:[line 108] |
| | org.apache.hadoop.hive.common.HiveConfProperties.getProperty(String,
String) is unsynchronized,
org.apache.hadoop.hive.common.HiveConfProperties.setProperty(String, String) is
synchronized At HiveConfProperties.java:String) is synchronized At
HiveConfProperties.java:[lines 123-130] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15244/dev-support/hive-personality.sh
|
| git revision | master / b42fdc2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15244/yetus/diff-checkstyle-common.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15244/yetus/new-findbugs-common.html
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15244/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.4.patch, HIVE-20760.5.patch,
> HIVE-20760.6.patch, HIVE-20760.7.patch, HIVE-20760.8.patch,
> HIVE-20760.9.patch,
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698778#comment-16698778
]
Hive QA commented on HIVE-20760:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949463/HIVE-20760.8.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 15500 tests
executed
*Failed tests:*
{noformat}
TestCompactor - did not produce a TEST-*.xml file (likely timed out)
(batchId=244)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testOperationsOnCompletedTxnComponentsForMmTable
(batchId=284)
org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testOperationsOnCompletedTxnComponentsForMmTable
(batchId=306)
org.apache.hadoop.hive.ql.TestTxnConcatenate.testConcatenateMM (batchId=293)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15055/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15055/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15055/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949463 - PreCommit-HIVE-Build
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.4.patch, HIVE-20760.5.patch,
> HIVE-20760.6.patch, HIVE-20760.7.patch, HIVE-20760.8.patch, HIVE-20760.patch,
> hiveconf_interned.html, hiveconf_original.html
>
>
> The issue is that every Hive task has to load its own version of
> {{HiveConf}}. When running with a large number of cores per executor (HoS),
> there is a significant (~10%) amount of memory wasted due to this
> duplication.
> I looked into the problem and found a way to reduce the overhead caused by
> the multiple HiveConf objects.
> I've created an implementation of Properties, somewhat similar to
> CopyOnFirstWriteProperties. CopyOnFirstWriteProperties can't be used to solve
> this problem, because it drops the interned Properties right after we add a
> new property.
> So my implementation looks like this:
> * When we create a new HiveConf from an existing one (copy constructor), we
> change the properties object stored by HiveConf to the new Properties
> implementation (HiveConfProperties). We have 2 possible way to do this.
> Either we change the visibility of the properties field in the ancestor class
> (Configuration which comes from hadoop) to protected, or a simpler way is to
> just change the type using reflection.
> * HiveConfProperties instantly intern the given properties. After this,
> every time we add a new property to HiveConf, we add it to an additional
> Properties object. This way if we create multiple HiveConf with the same base
> properties, they will use the same Properties object but each session/task
> can add its own unique properties.
> * Getting a property from HiveConfProperties would look like this: (I stored
> the non-interned properties in super class)
> String property=super.getProperty(key);
> if (property == null) property= interned.getProperty(key);
> return property;
> Running some tests showed that the interning works (with 50 connections to
> HiveServer2, heapdumps created after sessions are created for queries):
> Overall memory:
> original: 34,599K interned: 20,582K
> Retained memory of HiveConfs:
> original: 16,366K interned: 10,804K
> I attach the JXray reports about the heapdumps.
> What are your thoughts about this solution?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698693#comment-16698693
]
Hive QA commented on HIVE-20760:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} common: The patch generated 3 new + 426 unchanged - 0
fixed = 429 total (was 426) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
38s{color} | {color:red} common generated 3 new + 65 unchanged - 0 fixed = 68
total (was 65) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:common |
| | org.apache.hadoop.hive.common.HiveConfProperties.clone() does not call
super.clone() At HiveConfProperties.java: At HiveConfProperties.java:[line
260] |
| | Inconsistent synchronization of
org.apache.hadoop.hive.common.HiveConfProperties.interned; locked 70% of time
Unsynchronized access at HiveConfProperties.java:70% of time Unsynchronized
access at HiveConfProperties.java:[line 108] |
| | org.apache.hadoop.hive.common.HiveConfProperties.getProperty(String,
String) is unsynchronized,
org.apache.hadoop.hive.common.HiveConfProperties.setProperty(String, String) is
synchronized At HiveConfProperties.java:String) is synchronized At
HiveConfProperties.java:[lines 123-130] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15055/dev-support/hive-personality.sh
|
| git revision | master / 0fee288 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15055/yetus/diff-checkstyle-common.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15055/yetus/new-findbugs-common.html
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15055/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.4.patch, HIVE-20760.5.patch,
> HIVE-20760.6.patch, HIVE-20760.7.patch, HIVE-20760.8.patch, HIVE-20760.patch,
>
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692104#comment-16692104
]
Hive QA commented on HIVE-20760:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12948737/HIVE-20760.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15551 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.hooks.TestHiveProtoLoggingHook.testRolloverFiles
(batchId=319)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14989/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14989/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14989/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12948737 - PreCommit-HIVE-Build
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.4.patch, HIVE-20760.5.patch,
> HIVE-20760.6.patch, HIVE-20760.7.patch, HIVE-20760.patch,
> hiveconf_interned.html, hiveconf_original.html
>
>
> The issue is that every Hive task has to load its own version of
> {{HiveConf}}. When running with a large number of cores per executor (HoS),
> there is a significant (~10%) amount of memory wasted due to this
> duplication.
> I looked into the problem and found a way to reduce the overhead caused by
> the multiple HiveConf objects.
> I've created an implementation of Properties, somewhat similar to
> CopyOnFirstWriteProperties. CopyOnFirstWriteProperties can't be used to solve
> this problem, because it drops the interned Properties right after we add a
> new property.
> So my implementation looks like this:
> * When we create a new HiveConf from an existing one (copy constructor), we
> change the properties object stored by HiveConf to the new Properties
> implementation (HiveConfProperties). We have 2 possible way to do this.
> Either we change the visibility of the properties field in the ancestor class
> (Configuration which comes from hadoop) to protected, or a simpler way is to
> just change the type using reflection.
> * HiveConfProperties instantly intern the given properties. After this,
> every time we add a new property to HiveConf, we add it to an additional
> Properties object. This way if we create multiple HiveConf with the same base
> properties, they will use the same Properties object but each session/task
> can add its own unique properties.
> * Getting a property from HiveConfProperties would look like this: (I stored
> the non-interned properties in super class)
> String property=super.getProperty(key);
> if (property == null) property= interned.getProperty(key);
> return property;
> Running some tests showed that the interning works (with 50 connections to
> HiveServer2, heapdumps created after sessions are created for queries):
> Overall memory:
> original: 34,599K interned: 20,582K
> Retained memory of HiveConfs:
> original: 16,366K interned: 10,804K
> I attach the JXray reports about the heapdumps.
> What are your thoughts about this solution?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692056#comment-16692056
]
Hive QA commented on HIVE-20760:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} common: The patch generated 3 new + 426 unchanged - 0
fixed = 429 total (was 426) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
39s{color} | {color:red} common generated 3 new + 65 unchanged - 0 fixed = 68
total (was 65) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:common |
| | org.apache.hadoop.hive.common.HiveConfProperties.clone() does not call
super.clone() At HiveConfProperties.java: At HiveConfProperties.java:[line
258] |
| | Inconsistent synchronization of
org.apache.hadoop.hive.common.HiveConfProperties.interned; locked 70% of time
Unsynchronized access at HiveConfProperties.java:70% of time Unsynchronized
access at HiveConfProperties.java:[line 106] |
| | org.apache.hadoop.hive.common.HiveConfProperties.getProperty(String,
String) is unsynchronized,
org.apache.hadoop.hive.common.HiveConfProperties.setProperty(String, String) is
synchronized At HiveConfProperties.java:String) is synchronized At
HiveConfProperties.java:[lines 121-128] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14989/dev-support/hive-personality.sh
|
| git revision | master / ea1173a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14989/yetus/diff-checkstyle-common.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14989/yetus/new-findbugs-common.html
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14989/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.4.patch, HIVE-20760.5.patch,
> HIVE-20760.6.patch, HIVE-20760.7.patch, HIVE-20760.patch,
> hiveconf_interned.html,
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691655#comment-16691655
]
Hive QA commented on HIVE-20760:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12948687/HIVE-20760.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 15526 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=197)
[druidmini_masking.q,druidmini_joins.q,druid_timestamptz.q]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_limit]
(batchId=171)
org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocks (batchId=243)
org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocksWithMaterializedView
(batchId=243)
org.apache.hadoop.hive.ql.hooks.TestHiveProtoLoggingHook.testRolloverFiles
(batchId=319)
org.apache.hive.beeline.cli.TestHiveCli.testSetHeaderValue (batchId=203)
org.apache.hive.jdbc.TestJdbcGenericUDTFGetSplits.org.apache.hive.jdbc.TestJdbcGenericUDTFGetSplits
(batchId=257)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.org.apache.hive.jdbc.TestTriggersMoveWorkloadManager
(batchId=259)
org.apache.hive.jdbc.TestTriggersNoTezSessionPool.org.apache.hive.jdbc.TestTriggersNoTezSessionPool
(batchId=259)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.org.apache.hive.jdbc.TestTriggersTezSessionPoolManager
(batchId=259)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14984/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14984/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14984/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12948687 - PreCommit-HIVE-Build
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.4.patch, HIVE-20760.5.patch,
> HIVE-20760.6.patch, HIVE-20760.patch, hiveconf_interned.html,
> hiveconf_original.html
>
>
> The issue is that every Hive task has to load its own version of
> {{HiveConf}}. When running with a large number of cores per executor (HoS),
> there is a significant (~10%) amount of memory wasted due to this
> duplication.
> I looked into the problem and found a way to reduce the overhead caused by
> the multiple HiveConf objects.
> I've created an implementation of Properties, somewhat similar to
> CopyOnFirstWriteProperties. CopyOnFirstWriteProperties can't be used to solve
> this problem, because it drops the interned Properties right after we add a
> new property.
> So my implementation looks like this:
> * When we create a new HiveConf from an existing one (copy constructor), we
> change the properties object stored by HiveConf to the new Properties
> implementation (HiveConfProperties). We have 2 possible way to do this.
> Either we change the visibility of the properties field in the ancestor class
> (Configuration which comes from hadoop) to protected, or a simpler way is to
> just change the type using reflection.
> * HiveConfProperties instantly intern the given properties. After this,
> every time we add a new property to HiveConf, we add it to an additional
> Properties object. This way if we create multiple HiveConf with the same base
> properties, they will use the same Properties object but each session/task
> can add its own unique properties.
> * Getting a property from HiveConfProperties would look like this: (I stored
> the non-interned properties in super class)
> String property=super.getProperty(key);
> if (property == null) property= interned.getProperty(key);
> return property;
> Running some tests showed that the interning works (with 50 connections to
> HiveServer2, heapdumps created after sessions are created for queries):
> Overall memory:
> original: 34,599K interned: 20,582K
> Retained memory of HiveConfs:
> original: 16,366K interned: 10,804K
> I attach the
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691598#comment-16691598
]
Hive QA commented on HIVE-20760:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
43s{color} | {color:red} common generated 3 new + 65 unchanged - 0 fixed = 68
total (was 65) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:common |
| | org.apache.hadoop.hive.common.HiveConfProperties.clone() does not call
super.clone() At HiveConfProperties.java: At HiveConfProperties.java:[lines
247-262] |
| | Inconsistent synchronization of
org.apache.hadoop.hive.common.HiveConfProperties.interned; locked 61% of time
Unsynchronized access at HiveConfProperties.java:61% of time Unsynchronized
access at HiveConfProperties.java:[line 83] |
| | org.apache.hadoop.hive.common.HiveConfProperties.getProperty(String,
String) is unsynchronized,
org.apache.hadoop.hive.common.HiveConfProperties.setProperty(String, String) is
synchronized At HiveConfProperties.java:String) is synchronized At
HiveConfProperties.java:[lines 98-105] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14984/dev-support/hive-personality.sh
|
| git revision | master / fd5f34f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14984/yetus/new-findbugs-common.html
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14984/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.4.patch, HIVE-20760.5.patch,
> HIVE-20760.6.patch, HIVE-20760.patch, hiveconf_interned.html,
> hiveconf_original.html
>
>
> The issue is that every Hive task has to load its own version of
> {{HiveConf}}. When running with a large number of cores per executor (HoS),
> there is a
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685397#comment-16685397
]
Hive QA commented on HIVE-20760:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947965/HIVE-20760.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 392 failed/errored test(s), 15434 tests
executed
*Failed tests:*
{noformat}
TestBeeLineWithArgs - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestBeelinePasswordOption - did not produce a TEST-*.xml file (likely timed
out) (batchId=253)
TestEmbeddedThriftBinaryCLIService - did not produce a TEST-*.xml file (likely
timed out) (batchId=254)
TestHiveSessionImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=254)
TestSchemaTool - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestThriftCLIServiceWithBinary - did not produce a TEST-*.xml file (likely
timed out) (batchId=254)
TestThriftCLIServiceWithHttp - did not produce a TEST-*.xml file (likely timed
out) (batchId=254)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[explain_outputs]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[mapjoin2]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[select_dummy_source]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_10]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_13]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_16]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_2]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_3]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_7]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[udf_unix_timestamp]
(batchId=272)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[auto_sortmerge_join_16]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucket4]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucket5]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucket6]
(batchId=189)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketmapjoin6]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketmapjoin7]
(batchId=189)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_partitioner]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_semijoin]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[disable_merge_for_bucketing]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[dynamic_rdd_cache]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[empty_dir_in_table]
(batchId=189)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[external_table_with_space_in_location_path]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[file_with_header_footer]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[gen_udf_example_add10]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[import_exported_table]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_bucketed_table]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_map_operators]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_merge]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[input16_cc]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[insert_overwrite_directory2]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[leftsemijoin_mr]
(batchId=189)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[list_bucket_dml_10]
(batchId=189)
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685315#comment-16685315
]
Hive QA commented on HIVE-20760:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
38s{color} | {color:red} common generated 3 new + 65 unchanged - 0 fixed = 68
total (was 65) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:common |
| | org.apache.hadoop.hive.common.HiveConfProperties.clone() does not call
super.clone() At HiveConfProperties.java: At HiveConfProperties.java:[lines
247-258] |
| | Inconsistent synchronization of
org.apache.hadoop.hive.common.HiveConfProperties.interned; locked 61% of time
Unsynchronized access at HiveConfProperties.java:61% of time Unsynchronized
access at HiveConfProperties.java:[line 83] |
| | org.apache.hadoop.hive.common.HiveConfProperties.getProperty(String,
String) is unsynchronized,
org.apache.hadoop.hive.common.HiveConfProperties.setProperty(String, String) is
synchronized At HiveConfProperties.java:String) is synchronized At
HiveConfProperties.java:[lines 98-105] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14912/dev-support/hive-personality.sh
|
| git revision | master / 99d25f0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14912/yetus/new-findbugs-common.html
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14912/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.4.patch, HIVE-20760.5.patch, HIVE-20760.patch,
> hiveconf_interned.html, hiveconf_original.html
>
>
> The issue is that every Hive task has to load its own version of
> {{HiveConf}}. When running with a large number of cores per executor (HoS),
> there is a significant (~10%)
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677457#comment-16677457
]
Hive QA commented on HIVE-20760:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12947054/HIVE-20760.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1404 failed/errored test(s), 10569 tests
executed
*Failed tests:*
{noformat}
TestBeeLineWithArgs - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestBeelinePasswordOption - did not produce a TEST-*.xml file (likely timed
out) (batchId=253)
TestContext - did not produce a TEST-*.xml file (likely timed out) (batchId=307)
TestEmbeddedThriftBinaryCLIService - did not produce a TEST-*.xml file (likely
timed out) (batchId=254)
TestHiveCredentialProviders - did not produce a TEST-*.xml file (likely timed
out) (batchId=307)
TestHiveLockObject - did not produce a TEST-*.xml file (likely timed out)
(batchId=323)
TestHiveSessionImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=254)
TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=259)
TestKeyWrapperFactory - did not produce a TEST-*.xml file (likely timed out)
(batchId=307)
TestLog4j2Appenders - did not produce a TEST-*.xml file (likely timed out)
(batchId=307)
TestOperators - did not produce a TEST-*.xml file (likely timed out)
(batchId=307)
TestSSL - did not produce a TEST-*.xml file (likely timed out) (batchId=257)
TestSchemaTool - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestSlidingFilenameRolloverStrategy - did not produce a TEST-*.xml file (likely
timed out) (batchId=307)
TestThriftCLIServiceWithBinary - did not produce a TEST-*.xml file (likely
timed out) (batchId=254)
TestThriftCLIServiceWithHttp - did not produce a TEST-*.xml file (likely timed
out) (batchId=254)
TestTxnCommands - did not produce a TEST-*.xml file (likely timed out)
(batchId=323)
TestTxnCommands2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=307)
TestTxnCommands2WithSplitUpdateAndVectorization - did not produce a TEST-*.xml
file (likely timed out) (batchId=319)
TestTxnCommandsWithSplitUpdateAndVectorization - did not produce a TEST-*.xml
file (likely timed out) (batchId=307)
TestTxnNoBuckets - did not produce a TEST-*.xml file (likely timed out)
(batchId=307)
TestTxnNoBucketsVectorized - did not produce a TEST-*.xml file (likely timed
out) (batchId=307)
TestZookeeperLockManager - did not produce a TEST-*.xml file (likely timed out)
(batchId=323)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.org.apache.hadoop.hive.cli.TestAccumuloCliDriver
(batchId=264)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[colstats_all_nulls]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[explain_outputs]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[mapjoin2]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[select_dummy_source]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_10]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_13]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_16]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_2]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_3]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_7]
(batchId=272)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[udf_unix_timestamp]
(batchId=272)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[buckets]
(batchId=275)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_database]
(batchId=275)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_like]
(batchId=275)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_blobstore_to_blobstore]
(batchId=275)
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677422#comment-16677422
]
Hive QA commented on HIVE-20760:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
37s{color} | {color:red} common generated 2 new + 65 unchanged - 0 fixed = 67
total (was 65) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:common |
| | org.apache.hadoop.hive.common.HiveConfProperties.clone() does not call
super.clone() At HiveConfProperties.java: At HiveConfProperties.java:[lines
215-218] |
| | org.apache.hadoop.hive.common.HiveConfProperties.getProperty(String,
String) is unsynchronized,
org.apache.hadoop.hive.common.HiveConfProperties.setProperty(String, String) is
synchronized At HiveConfProperties.java:String) is synchronized At
HiveConfProperties.java:[lines 86-90] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14777/dev-support/hive-personality.sh
|
| git revision | master / 6d713b6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14777/yetus/new-findbugs-common.html
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14777/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760-3.patch, HIVE-20760.4.patch, HIVE-20760.patch,
> hiveconf_interned.html, hiveconf_original.html
>
>
> The issue is that every Hive task has to load its own version of
> {{HiveConf}}. When running with a large number of cores per executor (HoS),
> there is a significant (~10%) amount of memory wasted due to this
> duplication.
> I looked into the problem and found a way to reduce the overhead caused by
> the multiple HiveConf objects.
> I've created an implementation of Properties, somewhat similar to
> CopyOnFirstWriteProperties.
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663433#comment-16663433
]
Barnabas Maidics commented on HIVE-20760:
-
Created review request: https://reviews.apache.org/r/69155/
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760-1.patch, HIVE-20760-2.patch,
> HIVE-20760.patch, hiveconf_interned.html, hiveconf_original.html
>
>
> The issue is that every Hive task has to load its own version of
> {{HiveConf}}. When running with a large number of cores per executor (HoS),
> there is a significant (~10%) amount of memory wasted due to this
> duplication.
> I looked into the problem and found a way to reduce the overhead caused by
> the multiple HiveConf objects.
> I've created an implementation of Properties, somewhat similar to
> CopyOnFirstWriteProperties. CopyOnFirstWriteProperties can't be used to solve
> this problem, because it drops the interned Properties right after we add a
> new property.
> So my implementation looks like this:
> * When we create a new HiveConf from an existing one (copy constructor), we
> change the properties object stored by HiveConf to the new Properties
> implementation (HiveConfProperties). We have 2 possible way to do this.
> Either we change the visibility of the properties field in the ancestor class
> (Configuration which comes from hadoop) to protected, or a simpler way is to
> just change the type using reflection.
> * HiveConfProperties instantly intern the given properties. After this,
> every time we add a new property to HiveConf, we add it to an additional
> Properties object. This way if we create multiple HiveConf with the same base
> properties, they will use the same Properties object but each session/task
> can add its own unique properties.
> * Getting a property from HiveConfProperties would look like this: (I stored
> the non-interned properties in super class)
> String property=super.getProperty(key);
> if (property == null) property= interned.getProperty(key);
> return property;
> Running some tests showed that the interning works (with 50 connections to
> HiveServer2, heapdumps created after sessions are created for queries):
> Overall memory:
> original: 34,599K interned: 20,582K
> Retained memory of HiveConfs:
> original: 16,366K interned: 10,804K
> I attach the JXray reports about the heapdumps.
> What are your thoughts about this solution?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20760) Reducing memory overhead due to multiple HiveConfs
[
https://issues.apache.org/jira/browse/HIVE-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654502#comment-16654502
]
Hive QA commented on HIVE-20760:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 15s{color}
| {color:red} common generated 26 new + 0 unchanged - 0 fixed = 26 total (was
0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} common: The patch generated 51 new + 424 unchanged - 0
fixed = 475 total (was 424) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
36s{color} | {color:red} common generated 3 new + 65 unchanged - 0 fixed = 68
total (was 65) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:common |
| | org.apache.hadoop.hive.common.HiveConfProperties.clone() does not call
super.clone() At HiveConfProperties.java: At HiveConfProperties.java:[lines
164-167] |
| | org.apache.hadoop.hive.common.HiveConfProperties defines equals but not
hashCode At HiveConfProperties.java:hashCode At
HiveConfProperties.java:[lines 201-282] |
| | org.apache.hadoop.hive.common.HiveConfProperties.getProperty(String,
String) is unsynchronized,
org.apache.hadoop.hive.common.HiveConfProperties.setProperty(String, String) is
synchronized At HiveConfProperties.java:String) is synchronized At
HiveConfProperties.java:[lines 44-46] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14531/dev-support/hive-personality.sh
|
| git revision | master / 3c70302 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| javac |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14531/yetus/diff-compile-javac-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14531/yetus/diff-checkstyle-common.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14531/yetus/new-findbugs-common.html
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14531/yetus/patch-asflicense-problems.txt
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14531/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reducing memory overhead due to multiple HiveConfs
> --
>
> Key: HIVE-20760
> URL: https://issues.apache.org/jira/browse/HIVE-20760
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Reporter: Barnabas Maidics
>Assignee: Barnabas Maidics
>Priority: Major
> Attachments: HIVE-20760.patch,
