[jira] [Updated] (HIVE-21035) Race condition in SparkUtilities#getSparkSession
[ https://issues.apache.org/jira/browse/HIVE-21035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-21035: --- Status: Patch Available (was: In Progress) > Race condition in SparkUtilities#getSparkSession > > > Key: HIVE-21035 > URL: https://issues.apache.org/jira/browse/HIVE-21035 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-21035.01.patch > > > It can happen, that when in one given session, multiple queries are executed, > that due to a race condition, multiple spark application master gets kicked > off. > In this case, the one that started earlier, will not be killed, when the hive > session closes, consuming resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21035) Race condition in SparkUtilities#getSparkSession
[ https://issues.apache.org/jira/browse/HIVE-21035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-21035: -- > Race condition in SparkUtilities#getSparkSession > > > Key: HIVE-21035 > URL: https://issues.apache.org/jira/browse/HIVE-21035 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > It can happen, that when in one given session, multiple queries are executed, > that due to a race condition, multiple spark application master gets kicked > off. > In this case, the one that started earlier, will not be killed, when the hive > session closes, consuming resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-21035) Race condition in SparkUtilities#getSparkSession
[ https://issues.apache.org/jira/browse/HIVE-21035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21035 started by Antal Sinkovits. -- > Race condition in SparkUtilities#getSparkSession > > > Key: HIVE-21035 > URL: https://issues.apache.org/jira/browse/HIVE-21035 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > It can happen, that when in one given session, multiple queries are executed, > that due to a race condition, multiple spark application master gets kicked > off. > In this case, the one that started earlier, will not be killed, when the hive > session closes, consuming resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700472#comment-16700472 ] Antal Sinkovits commented on HIVE-20440: Finally a green one. [~stakiar] I've added the e2e test. Can you please check this? > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch, HIVE-20440.08.patch, > HIVE-20440.09.patch, HIVE-20440.10.patch, HIVE-20440.11.patch, > HIVE-20440.12.patch, HIVE-20440.13.patch, HIVE-20440.14.patch.txt, > HIVE-20440.15.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.15.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch, HIVE-20440.08.patch, > HIVE-20440.09.patch, HIVE-20440.10.patch, HIVE-20440.11.patch, > HIVE-20440.12.patch, HIVE-20440.13.patch, HIVE-20440.14.patch.txt, > HIVE-20440.15.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.14.patch.txt > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch, HIVE-20440.08.patch, > HIVE-20440.09.patch, HIVE-20440.10.patch, HIVE-20440.11.patch, > HIVE-20440.12.patch, HIVE-20440.13.patch, HIVE-20440.14.patch.txt > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.13.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch, HIVE-20440.08.patch, > HIVE-20440.09.patch, HIVE-20440.10.patch, HIVE-20440.11.patch, > HIVE-20440.12.patch, HIVE-20440.13.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699154#comment-16699154 ] Antal Sinkovits commented on HIVE-20440: Test failiure not related. Uploading again. > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch, HIVE-20440.08.patch, > HIVE-20440.09.patch, HIVE-20440.10.patch, HIVE-20440.11.patch, > HIVE-20440.12.patch, HIVE-20440.13.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.12.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch, HIVE-20440.08.patch, > HIVE-20440.09.patch, HIVE-20440.10.patch, HIVE-20440.11.patch, > HIVE-20440.12.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698914#comment-16698914 ] Antal Sinkovits commented on HIVE-20440: Rebase > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch, HIVE-20440.08.patch, > HIVE-20440.09.patch, HIVE-20440.10.patch, HIVE-20440.11.patch, > HIVE-20440.12.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20907) TestGetPartitionsUsingProjectionAndFilterSpecs is flaky
[ https://issues.apache.org/jira/browse/HIVE-20907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684421#comment-16684421 ] Antal Sinkovits commented on HIVE-20907: [~vihangk1] I've faced the issue in https://issues.apache.org/jira/browse/HIVE-20904 > TestGetPartitionsUsingProjectionAndFilterSpecs is flaky > --- > > Key: HIVE-20907 > URL: https://issues.apache.org/jira/browse/HIVE-20907 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20907.01.patch, HIVE-20907.02.patch > > > private void verifyLocations(List origPartitions, > StorageDescriptor sharedSD, > List partitionWithoutSDS) > method expects, that the order of the two list are the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20907) TestGetPartitionsUsingProjectionAndFilterSpecs is flaky
[ https://issues.apache.org/jira/browse/HIVE-20907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20907: --- Attachment: HIVE-20907.02.patch > TestGetPartitionsUsingProjectionAndFilterSpecs is flaky > --- > > Key: HIVE-20907 > URL: https://issues.apache.org/jira/browse/HIVE-20907 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20907.01.patch, HIVE-20907.02.patch > > > private void verifyLocations(List origPartitions, > StorageDescriptor sharedSD, > List partitionWithoutSDS) > method expects, that the order of the two list are the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20907) TestGetPartitionsUsingProjectionAndFilterSpecs is flaky
[ https://issues.apache.org/jira/browse/HIVE-20907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684061#comment-16684061 ] Antal Sinkovits commented on HIVE-20907: rebase > TestGetPartitionsUsingProjectionAndFilterSpecs is flaky > --- > > Key: HIVE-20907 > URL: https://issues.apache.org/jira/browse/HIVE-20907 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20907.01.patch, HIVE-20907.02.patch > > > private void verifyLocations(List origPartitions, > StorageDescriptor sharedSD, > List partitionWithoutSDS) > method expects, that the order of the two list are the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20907) TestGetPartitionsUsingProjectionAndFilterSpecs is flaky
[ https://issues.apache.org/jira/browse/HIVE-20907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20907: --- Assignee: Antal Sinkovits Status: Patch Available (was: Open) > TestGetPartitionsUsingProjectionAndFilterSpecs is flaky > --- > > Key: HIVE-20907 > URL: https://issues.apache.org/jira/browse/HIVE-20907 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20907.01.patch > > > private void verifyLocations(List origPartitions, > StorageDescriptor sharedSD, > List partitionWithoutSDS) > method expects, that the order of the two list are the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20907) TestGetPartitionsUsingProjectionAndFilterSpecs is flaky
[ https://issues.apache.org/jira/browse/HIVE-20907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20907: --- Attachment: HIVE-20907.01.patch > TestGetPartitionsUsingProjectionAndFilterSpecs is flaky > --- > > Key: HIVE-20907 > URL: https://issues.apache.org/jira/browse/HIVE-20907 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Priority: Major > Attachments: HIVE-20907.01.patch > > > private void verifyLocations(List origPartitions, > StorageDescriptor sharedSD, > List partitionWithoutSDS) > method expects, that the order of the two list are the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20904) Yetus fails to resolve module dependencies due to usage of exec plugin in metastore-server
[ https://issues.apache.org/jira/browse/HIVE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683877#comment-16683877 ] Antal Sinkovits commented on HIVE-20904: Tests run fine locally. Re applying patch. > Yetus fails to resolve module dependencies due to usage of exec plugin in > metastore-server > -- > > Key: HIVE-20904 > URL: https://issues.apache.org/jira/browse/HIVE-20904 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20904.01.patch, HIVE-20904.02.patch > > > metastore-server uses exec-maven-plugin to generate > metastore-site.xml.template with ConfTemplatePrinter. > It expects some arguments. > Because yetus also uses the exec-maven-plugin to determine the order of the > modules to be built, but with zero params, the execution fails. > https://github.com/apache/yetus/blob/6ebaa1119e611db14f219e289e33ab8ac5c254a7/precommit/src/main/shell/test-patch.d/maven.sh#L658 > Steps to reproduce the issue: > mvn -q exec:exec -Dexec.executable=pwd -Dexec.args='' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20904) Yetus fails to resolve module dependencies due to usage of exec plugin in metastore-server
[ https://issues.apache.org/jira/browse/HIVE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20904: --- Attachment: HIVE-20904.02.patch > Yetus fails to resolve module dependencies due to usage of exec plugin in > metastore-server > -- > > Key: HIVE-20904 > URL: https://issues.apache.org/jira/browse/HIVE-20904 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20904.01.patch, HIVE-20904.02.patch > > > metastore-server uses exec-maven-plugin to generate > metastore-site.xml.template with ConfTemplatePrinter. > It expects some arguments. > Because yetus also uses the exec-maven-plugin to determine the order of the > modules to be built, but with zero params, the execution fails. > https://github.com/apache/yetus/blob/6ebaa1119e611db14f219e289e33ab8ac5c254a7/precommit/src/main/shell/test-patch.d/maven.sh#L658 > Steps to reproduce the issue: > mvn -q exec:exec -Dexec.executable=pwd -Dexec.args='' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20904) Yetus fails to resolve module dependencies due to usage of exec plugin in metastore-server
[ https://issues.apache.org/jira/browse/HIVE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683582#comment-16683582 ] Antal Sinkovits commented on HIVE-20904: [~alangates] [~thejas] could you please check this. > Yetus fails to resolve module dependencies due to usage of exec plugin in > metastore-server > -- > > Key: HIVE-20904 > URL: https://issues.apache.org/jira/browse/HIVE-20904 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20904.01.patch > > > metastore-server uses exec-maven-plugin to generate > metastore-site.xml.template with ConfTemplatePrinter. > It expects some arguments. > Because yetus also uses the exec-maven-plugin to determine the order of the > modules to be built, but with zero params, the execution fails. > https://github.com/apache/yetus/blob/6ebaa1119e611db14f219e289e33ab8ac5c254a7/precommit/src/main/shell/test-patch.d/maven.sh#L658 > Steps to reproduce the issue: > mvn -q exec:exec -Dexec.executable=pwd -Dexec.args='' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20904) Yetus fails to resolve module dependencies due to usage of exec plugin in metastore-server
[ https://issues.apache.org/jira/browse/HIVE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-20904: -- > Yetus fails to resolve module dependencies due to usage of exec plugin in > metastore-server > -- > > Key: HIVE-20904 > URL: https://issues.apache.org/jira/browse/HIVE-20904 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > metastore-server uses exec-maven-plugin to generate > metastore-site.xml.template with ConfTemplatePrinter. > It expects some arguments. > Because yetus also uses the exec-maven-plugin to determine the order of the > modules to be built, but with zero params, the execution fails. > https://github.com/apache/yetus/blob/6ebaa1119e611db14f219e289e33ab8ac5c254a7/precommit/src/main/shell/test-patch.d/maven.sh#L658 > Steps to reproduce the issue: > mvn -q exec:exec -Dexec.executable=pwd -Dexec.args='' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20904) Yetus fails to resolve module dependencies due to usage of exec plugin in metastore-server
[ https://issues.apache.org/jira/browse/HIVE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20904: --- Status: Patch Available (was: Open) > Yetus fails to resolve module dependencies due to usage of exec plugin in > metastore-server > -- > > Key: HIVE-20904 > URL: https://issues.apache.org/jira/browse/HIVE-20904 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20904.01.patch > > > metastore-server uses exec-maven-plugin to generate > metastore-site.xml.template with ConfTemplatePrinter. > It expects some arguments. > Because yetus also uses the exec-maven-plugin to determine the order of the > modules to be built, but with zero params, the execution fails. > https://github.com/apache/yetus/blob/6ebaa1119e611db14f219e289e33ab8ac5c254a7/precommit/src/main/shell/test-patch.d/maven.sh#L658 > Steps to reproduce the issue: > mvn -q exec:exec -Dexec.executable=pwd -Dexec.args='' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20904) Yetus fails to resolve module dependencies due to usage of exec plugin in metastore-server
[ https://issues.apache.org/jira/browse/HIVE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20904: --- Attachment: HIVE-20904.01.patch > Yetus fails to resolve module dependencies due to usage of exec plugin in > metastore-server > -- > > Key: HIVE-20904 > URL: https://issues.apache.org/jira/browse/HIVE-20904 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20904.01.patch > > > metastore-server uses exec-maven-plugin to generate > metastore-site.xml.template with ConfTemplatePrinter. > It expects some arguments. > Because yetus also uses the exec-maven-plugin to determine the order of the > modules to be built, but with zero params, the execution fails. > https://github.com/apache/yetus/blob/6ebaa1119e611db14f219e289e33ab8ac5c254a7/precommit/src/main/shell/test-patch.d/maven.sh#L658 > Steps to reproduce the issue: > mvn -q exec:exec -Dexec.executable=pwd -Dexec.args='' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.11.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch, HIVE-20440.08.patch, > HIVE-20440.09.patch, HIVE-20440.10.patch, HIVE-20440.11.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.10.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch, HIVE-20440.08.patch, > HIVE-20440.09.patch, HIVE-20440.10.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.09.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch, HIVE-20440.08.patch, > HIVE-20440.09.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670892#comment-16670892
]
Antal Sinkovits commented on HIVE-20512:
LGTM
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch,
> HIVE-20512.3.patch, HIVE-20512.4.patch, HIVE-20512.5.patch, HIVE-20512.6.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20742) SparkSessionManagerImpl maintenance thread only cleans up session once
[ https://issues.apache.org/jira/browse/HIVE-20742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20742: --- Attachment: HIVE-20742.2.patch > SparkSessionManagerImpl maintenance thread only cleans up session once > -- > > Key: HIVE-20742 > URL: https://issues.apache.org/jira/browse/HIVE-20742 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20742.1.patch, HIVE-20742.2.patch > > > If there is a reconnect at the client session, the SparkSessionManagerImpl > doesn't puts it back in the created sessions, so it will not time out the > second time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20742) SparkSessionManagerImpl maintenance thread only cleans up session once
[ https://issues.apache.org/jira/browse/HIVE-20742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649930#comment-16649930 ] Antal Sinkovits commented on HIVE-20742: Test failures are not related. Reapplying the same patch. > SparkSessionManagerImpl maintenance thread only cleans up session once > -- > > Key: HIVE-20742 > URL: https://issues.apache.org/jira/browse/HIVE-20742 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20742.1.patch > > > If there is a reconnect at the client session, the SparkSessionManagerImpl > doesn't puts it back in the created sessions, so it will not time out the > second time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20742) SparkSessionManagerImpl maintenance thread only cleans up session once
[ https://issues.apache.org/jira/browse/HIVE-20742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648844#comment-16648844 ] Antal Sinkovits commented on HIVE-20742: [~stakiar] could you please take a look. Thanks. > SparkSessionManagerImpl maintenance thread only cleans up session once > -- > > Key: HIVE-20742 > URL: https://issues.apache.org/jira/browse/HIVE-20742 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20742.1.patch > > > If there is a reconnect at the client session, the SparkSessionManagerImpl > doesn't puts it back in the created sessions, so it will not time out the > second time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20742) SparkSessionManagerImpl maintenance thread only cleans up session once
[ https://issues.apache.org/jira/browse/HIVE-20742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20742: --- Status: Patch Available (was: Open) > SparkSessionManagerImpl maintenance thread only cleans up session once > -- > > Key: HIVE-20742 > URL: https://issues.apache.org/jira/browse/HIVE-20742 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20742.1.patch > > > If there is a reconnect at the client session, the SparkSessionManagerImpl > doesn't puts it back in the created sessions, so it will not time out the > second time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20742) SparkSessionManagerImpl maintenance thread only cleans up session once
[ https://issues.apache.org/jira/browse/HIVE-20742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20742: --- Attachment: HIVE-20742.1.patch > SparkSessionManagerImpl maintenance thread only cleans up session once > -- > > Key: HIVE-20742 > URL: https://issues.apache.org/jira/browse/HIVE-20742 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20742.1.patch > > > If there is a reconnect at the client session, the SparkSessionManagerImpl > doesn't puts it back in the created sessions, so it will not time out the > second time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20742) SparkSessionManagerImpl maintenance thread only cleans up session once
[ https://issues.apache.org/jira/browse/HIVE-20742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-20742: -- > SparkSessionManagerImpl maintenance thread only cleans up session once > -- > > Key: HIVE-20742 > URL: https://issues.apache.org/jira/browse/HIVE-20742 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > If there is a reconnect at the client session, the SparkSessionManagerImpl > doesn't puts it back in the created sessions, so it will not time out the > second time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644900#comment-16644900 ] Antal Sinkovits commented on HIVE-20440: Test failures are not related. I will re-upload the patch. > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch, HIVE-20440.08.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.08.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch, HIVE-20440.08.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.07.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: (was: HIVE-20440.07.patch) > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20711) Race Condition when Multi-Threading in SessionState.createRootHDFSDir
[ https://issues.apache.org/jira/browse/HIVE-20711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643208#comment-16643208 ] Antal Sinkovits commented on HIVE-20711: +1 LGTM > Race Condition when Multi-Threading in SessionState.createRootHDFSDir > - > > Key: HIVE-20711 > URL: https://issues.apache.org/jira/browse/HIVE-20711 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-20711.1.patch > > > java.util.concurrent.ExecutionException: java.lang.RuntimeException: The root > scratch dir: > /home/hiveptest/hive-ptest-cloudera-slaves-17e5-13.gce.cloudera.com-hiveptest-0/cdh-source/itests/hive-unit/target/tmp/scratchdir > on HDFS should be writable. Current permissions are: rwxr-xr-x at > org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:714) > at > org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:637) > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:567) > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:532) > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:512) at -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.07.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.06.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.05.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[
https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613731#comment-16613731
]
Antal Sinkovits commented on HIVE-20440:
Yes, this is a good idea.
But I would not bother with the eviction handler, instead fill both caches.
Here is the idea:
There are two caches. L1 is a time based one (with a maintenance thread
cleaning it), and L2 is a soft referenced (I don't think I need a cleanup
thread here).
{noformat}
At first I check L1.
If its a miss, i will check L2.
If I miss again, I load the data in both caches.
If its a hit, I would take it from L2 to return (I think it would
make sense to put it back into L1 as well)
If its a hit in L1, I return it.
{noformat}
If the data is in L1, I can be sure that its in L2 as well, because L1 contains
strong references to the object.
Since these are references, its cheap to store them in two places.
Do you see any flaw in this approach?
> Create better cache eviction policy for SmallTableCache
> ---
>
> Key: HIVE-20440
> URL: https://issues.apache.org/jira/browse/HIVE-20440
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch,
> HIVE-20440.03.patch, HIVE-20440.04.patch
>
>
> Enhance the SmallTableCache, to use guava cache with soft references, so that
> we evict when there is memory pressure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609444#comment-16609444 ] Antal Sinkovits commented on HIVE-20440: (1) I think because of the soft values, the GC decides when to clean up them. (When it needs more memory) (2) AFAIK we can use them together, but it won't stop the GC from clearing the values. > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.04.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.03.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.02.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Status: Patch Available (was: Open) > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-20440: --- Attachment: HIVE-20440.01.patch > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-20440: -- > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19486) Discrepancy in HikariCP config naming
[ https://issues.apache.org/jira/browse/HIVE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-19486: --- Attachment: HIVE-19486.2.patch > Discrepancy in HikariCP config naming > - > > Key: HIVE-19486 > URL: https://issues.apache.org/jira/browse/HIVE-19486 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-19486.1.patch, HIVE-19486.2.patch > > > HiveConf hive.conf.restricted.list contains "hikari." instead of "hikaricp." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19486) Discrepancy in HikariCP config naming
[ https://issues.apache.org/jira/browse/HIVE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-19486: --- Description: HiveConf hive.conf.restricted.list contains "hikari." instead of "hikaricp." (was: MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" while the code checks for "Hikari".) > Discrepancy in HikariCP config naming > - > > Key: HIVE-19486 > URL: https://issues.apache.org/jira/browse/HIVE-19486 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-19486.1.patch > > > HiveConf hive.conf.restricted.list contains "hikari." instead of "hikaricp." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19486) Discrepancy in HikariCP config naming
[ https://issues.apache.org/jira/browse/HIVE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-19486: --- Summary: Discrepancy in HikariCP config naming (was: Discrepancy between the config and the code in Hikari connectionPoolingType) > Discrepancy in HikariCP config naming > - > > Key: HIVE-19486 > URL: https://issues.apache.org/jira/browse/HIVE-19486 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-19486.1.patch > > > MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" while the > code checks for "Hikari". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492332#comment-16492332 ] Antal Sinkovits commented on HIVE-17317: [~pvary]: I've fixed the checkstyle/findbugs issues. > Make Dbcp configurable using hive properties in hive-site.xml > - > > Key: HIVE-17317 > URL: https://issues.apache.org/jira/browse/HIVE-17317 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-17317.01.patch, HIVE-17317.02.patch, > HIVE-17317.03.patch, HIVE-17317.04.patch, HIVE-17317.05.patch, > HIVE-17317.06.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-17317: --- Attachment: HIVE-17317.06.patch > Make Dbcp configurable using hive properties in hive-site.xml > - > > Key: HIVE-17317 > URL: https://issues.apache.org/jira/browse/HIVE-17317 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-17317.01.patch, HIVE-17317.02.patch, > HIVE-17317.03.patch, HIVE-17317.04.patch, HIVE-17317.05.patch, > HIVE-17317.06.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-17317: --- Attachment: HIVE-17317.05.patch > Make Dbcp configurable using hive properties in hive-site.xml > - > > Key: HIVE-17317 > URL: https://issues.apache.org/jira/browse/HIVE-17317 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-17317.01.patch, HIVE-17317.02.patch, > HIVE-17317.03.patch, HIVE-17317.04.patch, HIVE-17317.05.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-17317: -- Assignee: Antal Sinkovits (was: Barna Zsombor Klara) > Make Dbcp configurable using hive properties in hive-site.xml > - > > Key: HIVE-17317 > URL: https://issues.apache.org/jira/browse/HIVE-17317 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-17317.01.patch, HIVE-17317.02.patch, > HIVE-17317.03.patch, HIVE-17317.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19486) Discrepancy between the config and the code in Hikari connectionPoolingType
[ https://issues.apache.org/jira/browse/HIVE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-19486: --- Attachment: HIVE-19486.1.patch > Discrepancy between the config and the code in Hikari connectionPoolingType > --- > > Key: HIVE-19486 > URL: https://issues.apache.org/jira/browse/HIVE-19486 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-19486.1.patch > > > MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" while the > code checks for "Hikari". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-19486) Discrepancy between the config and the code in Hikari connectionPoolingType
[ https://issues.apache.org/jira/browse/HIVE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-19486 started by Antal Sinkovits. -- > Discrepancy between the config and the code in Hikari connectionPoolingType > --- > > Key: HIVE-19486 > URL: https://issues.apache.org/jira/browse/HIVE-19486 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-19486.1.patch > > > MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" while the > code checks for "Hikari". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19486) Discrepancy between the config and the code in Hikari connectionPoolingType
[ https://issues.apache.org/jira/browse/HIVE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-19486: --- Status: Patch Available (was: In Progress) > Discrepancy between the config and the code in Hikari connectionPoolingType > --- > > Key: HIVE-19486 > URL: https://issues.apache.org/jira/browse/HIVE-19486 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-19486.1.patch > > > MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" while the > code checks for "Hikari". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19486) Discrepancy between the config and the code in Hikari connectionPoolingType
[ https://issues.apache.org/jira/browse/HIVE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits updated HIVE-19486: --- Description: MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" while the code checks for "Hikari". (was: MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" not "Hikari".) > Discrepancy between the config and the code in Hikari connectionPoolingType > --- > > Key: HIVE-19486 > URL: https://issues.apache.org/jira/browse/HIVE-19486 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" while the > code checks for "Hikari". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17318) Make Hikari CP configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470368#comment-16470368 ] Antal Sinkovits commented on HIVE-17318: Yes you are right [~pvary] Apart from the MetaStoreConf change, the [https://github.com/apache/hive/blob/master/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L4169] needs to be updated as well, to use the constant defined in HikariCPDataSourceProvider, instead of the hardcoded value. I've created a jira for it: https://issues.apache.org/jira/browse/HIVE-19486 > Make Hikari CP configurable using hive properties in hive-site.xml > -- > > Key: HIVE-17318 > URL: https://issues.apache.org/jira/browse/HIVE-17318 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-17318.01.patch, HIVE-17318.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19486) Discrepancy between the config and the code in Hikari connectionPoolingType
[ https://issues.apache.org/jira/browse/HIVE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-19486: -- > Discrepancy between the config and the code in Hikari connectionPoolingType > --- > > Key: HIVE-19486 > URL: https://issues.apache.org/jira/browse/HIVE-19486 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > > MetaStoreConf contains datanucleus.connectionPoolingType "HikariCP" not > "Hikari". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467006#comment-16467006
]
Antal Sinkovits commented on HIVE-18906:
The test failures are not related.
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18906.1.patch
>
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181
> {code}
> 2018-02-26 14:32:50,620 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:09,566 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:43,170 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:46,301 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:16,966 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:18,072 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:25,064 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:51,781 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:04,685 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:26,689 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> {code}
> I wouldn't mind knowing this information when the service starts up and
> detects that there is a Direct SQL compatible database, but I don't need to
> know about this every time.
> Please lower to _debug_ level logging and parameterize with SLF4J. Direct
> SQL is a performance thing and not something I care about during the normal
> course of operation (unless there is an error which would presumably be
> covered by an _error_ logging message).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-18906:
---
Status: Patch Available (was: Open)
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18906.1.patch
>
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181
> {code}
> 2018-02-26 14:32:50,620 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:09,566 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:43,170 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:46,301 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:16,966 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:18,072 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:25,064 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:51,781 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:04,685 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:26,689 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> {code}
> I wouldn't mind knowing this information when the service starts up and
> detects that there is a Direct SQL compatible database, but I don't need to
> know about this every time.
> Please lower to _debug_ level logging and parameterize with SLF4J. Direct
> SQL is a performance thing and not something I care about during the normal
> course of operation (unless there is an error which would presumably be
> covered by an _error_ logging message).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-18906:
---
Attachment: HIVE-18906.1.patch
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18906.1.patch
>
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181
> {code}
> 2018-02-26 14:32:50,620 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:09,566 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:43,170 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:46,301 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:16,966 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:18,072 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:25,064 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:51,781 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:04,685 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:26,689 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> {code}
> I wouldn't mind knowing this information when the service starts up and
> detects that there is a Direct SQL compatible database, but I don't need to
> know about this every time.
> Please lower to _debug_ level logging and parameterize with SLF4J. Direct
> SQL is a performance thing and not something I care about during the normal
> course of operation (unless there is an error which would presumably be
> covered by an _error_ logging message).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462273#comment-16462273
]
Antal Sinkovits commented on HIVE-18906:
Thanks for the answers [~belugabehr]
1. The current implementation executes a test query, to determine whether it
supports direct sql or not. See:
https://github.com/apache/hive/blob/master/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L182
Not sure why it was implemented that way, it might be a good idea, to use the
database type instead, but I think we shouldn't mix the two, as it creates
ambiguity.
As this enhancement might be a good idea (even enhance performance as well), I
do feel, that its over the scope of this jira.
2. It seems so to me. This is what the doc says, and the related jira. I've
also did a bit of a smoke test, by running two beeline, changing the config in
one of them and checking the value in both.
I will use the described approach.
Docs/jira/commands:
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.metastore.try.direct.sql
https://issues.apache.org/jira/browse/HIVE-7532
set metaconf:hive.metastore.try.direct.sql=false;
set metaconf:hive.metastore.try.direct.sql;
3. Yes this makes sense, but in this case, we have the same issue as in 1.
4. Ok, this is also fine then.
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181
> {code}
> 2018-02-26 14:32:50,620 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:09,566 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:43,170 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:46,301 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:16,966 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:18,072 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:25,064 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:51,781 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:04,685 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:26,689 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> {code}
> I wouldn't mind knowing this information when the service starts up and
> detects that there is a Direct SQL compatible database, but I don't need to
> know about this every time.
> Please lower to _debug_ level logging and parameterize with SLF4J. Direct
> SQL is a performance thing and not something I care about during the normal
> course of operation (unless there is an error which would presumably be
> covered by an _error_ logging message).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461358#comment-16461358
]
Antal Sinkovits commented on HIVE-18906:
"An INFO message at startup which tells the administrator if the back-end
database is supported for the direct SQL optimization"
It seems to me, that by design, there are no db connection at startup, so I
think this would introduce an overhead just for logging.
Is this really necessary?
"An INFO message at startup which tells the administrator if the direct SQL
optimization is enabled/disabled"
This is fine, and since this param can be overridden at session level, I would
suggest that we should print it there as well.
"A WARN message if the SQL direct optimization is disabled for a database that
is supported"
We can do this, for each query executed, although I do feel that its a bit
verbose. Do we want this?
"A DEBUG message for each query if it is using (or not using) direct SQL
optimization"
The concrete sql (when used) is logged at trace level. Do we need this?
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181
> {code}
> 2018-02-26 14:32:50,620 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:09,566 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:43,170 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:46,301 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:16,966 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:18,072 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:25,064 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:51,781 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:04,685 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:26,689 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> {code}
> I wouldn't mind knowing this information when the service starts up and
> detects that there is a Direct SQL compatible database, but I don't need to
> know about this every time.
> Please lower to _debug_ level logging and parameterize with SLF4J. Direct
> SQL is a performance thing and not something I care about during the normal
> course of operation (unless there is an error which would presumably be
> covered by an _error_ logging message).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18903) Lower Logging Level for ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-18903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456631#comment-16456631
]
Antal Sinkovits commented on HIVE-18903:
The test failures are not related.
> Lower Logging Level for ObjectStore
> ---
>
> Key: HIVE-18903
> URL: https://issues.apache.org/jira/browse/HIVE-18903
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Minor
> Labels: noob
> Attachments: HIVE-18903.2.patch, HIVE-18903.patch
>
>
> [https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java]
>
> {code:java}
> 2018-03-01 06:51:58,051 INFO org.apache.hadoop.hive.metastore.ObjectStore:
> [pool-4-thread-13]: ObjectStore, initialize called
> 2018-03-01 06:51:58,052 INFO org.apache.hadoop.hive.metastore.ObjectStore:
> [pool-4-thread-13]: Initialized ObjectStore
> {code}
> Noting actionable or all that useful here. Please lower to _debug_ or
> _trace_ level logging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18880) Change Log to Debug in CombineHiveInputFormat
[
https://issues.apache.org/jira/browse/HIVE-18880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456617#comment-16456617
]
Antal Sinkovits commented on HIVE-18880:
The test failures are not related.
> Change Log to Debug in CombineHiveInputFormat
> -
>
> Key: HIVE-18880
> URL: https://issues.apache.org/jira/browse/HIVE-18880
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18880.2.patch, HIVE-18880.patch
>
>
> [https://github.com/apache/hive/blob/1e74aca8d09ea2ef636311d2168b4d34198f7194/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L467]
> {code:java}
> private InputSplit[] getCombineSplits(JobConf job, int numSplits,
> Map pathToPartitionInfo) {
> ...
> LOG.info("number of splits " + result.size());
> ...
> }
> {code}
> [https://github.com/apache/hive/blob/1e74aca8d09ea2ef636311d2168b4d34198f7194/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L587]
> {code:java}
> public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
> ...
> LOG.info("Number of all splits " + result.size());
> ...
> }
> {code}
> # Capitalize "N"umber in the first logging to be consistent across all
> logging statements
> # Change the first logging message to be _debug_ level seeing as it's in a
> private method.
> It's an implementation logging and the entire total (most useful for a
> client) is captured in _info_ level at the end of the public method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18881) Lower Logging for FSStatsAggregator
[
https://issues.apache.org/jira/browse/HIVE-18881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456613#comment-16456613
]
Antal Sinkovits commented on HIVE-18881:
The test failures are not related.
> Lower Logging for FSStatsAggregator
> ---
>
> Key: HIVE-18881
> URL: https://issues.apache.org/jira/browse/HIVE-18881
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18881.2.patch, HIVE-18881.patch
>
>
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java#L101]
> {code:java}
> LOG.info("Read stats for : " + partID + "\t" + statType + "\t" + counter);
> {code}
> # All the other logging in this class is _debug_ or _error_ level logging.
> This should be _debug_ as well
> # Remove tab characters to allow splitting on tabs in any kind of
> tab-separated file of log lines
> # Use SLF4J parameterized logging
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18881) Lower Logging for FSStatsAggregator
[
https://issues.apache.org/jira/browse/HIVE-18881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-18881:
---
Attachment: HIVE-18881.2.patch
> Lower Logging for FSStatsAggregator
> ---
>
> Key: HIVE-18881
> URL: https://issues.apache.org/jira/browse/HIVE-18881
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18881.2.patch, HIVE-18881.patch
>
>
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java#L101]
> {code:java}
> LOG.info("Read stats for : " + partID + "\t" + statType + "\t" + counter);
> {code}
> # All the other logging in this class is _debug_ or _error_ level logging.
> This should be _debug_ as well
> # Remove tab characters to allow splitting on tabs in any kind of
> tab-separated file of log lines
> # Use SLF4J parameterized logging
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18903) Lower Logging Level for ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-18903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-18903:
---
Attachment: HIVE-18903.2.patch
> Lower Logging Level for ObjectStore
> ---
>
> Key: HIVE-18903
> URL: https://issues.apache.org/jira/browse/HIVE-18903
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Minor
> Labels: noob
> Attachments: HIVE-18903.2.patch, HIVE-18903.patch
>
>
> [https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java]
>
> {code:java}
> 2018-03-01 06:51:58,051 INFO org.apache.hadoop.hive.metastore.ObjectStore:
> [pool-4-thread-13]: ObjectStore, initialize called
> 2018-03-01 06:51:58,052 INFO org.apache.hadoop.hive.metastore.ObjectStore:
> [pool-4-thread-13]: Initialized ObjectStore
> {code}
> Noting actionable or all that useful here. Please lower to _debug_ or
> _trace_ level logging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18880) Change Log to Debug in CombineHiveInputFormat
[
https://issues.apache.org/jira/browse/HIVE-18880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-18880:
---
Attachment: HIVE-18880.2.patch
> Change Log to Debug in CombineHiveInputFormat
> -
>
> Key: HIVE-18880
> URL: https://issues.apache.org/jira/browse/HIVE-18880
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18880.2.patch, HIVE-18880.patch
>
>
> [https://github.com/apache/hive/blob/1e74aca8d09ea2ef636311d2168b4d34198f7194/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L467]
> {code:java}
> private InputSplit[] getCombineSplits(JobConf job, int numSplits,
> Map pathToPartitionInfo) {
> ...
> LOG.info("number of splits " + result.size());
> ...
> }
> {code}
> [https://github.com/apache/hive/blob/1e74aca8d09ea2ef636311d2168b4d34198f7194/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L587]
> {code:java}
> public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
> ...
> LOG.info("Number of all splits " + result.size());
> ...
> }
> {code}
> # Capitalize "N"umber in the first logging to be consistent across all
> logging statements
> # Change the first logging message to be _debug_ level seeing as it's in a
> private method.
> It's an implementation logging and the entire total (most useful for a
> client) is captured in _info_ level at the end of the public method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18903) Lower Logging Level for ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-18903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-18903:
---
Status: Patch Available (was: Open)
> Lower Logging Level for ObjectStore
> ---
>
> Key: HIVE-18903
> URL: https://issues.apache.org/jira/browse/HIVE-18903
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Minor
> Labels: noob
> Attachments: HIVE-18903.patch
>
>
> [https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java]
>
> {code:java}
> 2018-03-01 06:51:58,051 INFO org.apache.hadoop.hive.metastore.ObjectStore:
> [pool-4-thread-13]: ObjectStore, initialize called
> 2018-03-01 06:51:58,052 INFO org.apache.hadoop.hive.metastore.ObjectStore:
> [pool-4-thread-13]: Initialized ObjectStore
> {code}
> Noting actionable or all that useful here. Please lower to _debug_ or
> _trace_ level logging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18903) Lower Logging Level for ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-18903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-18903:
---
Attachment: HIVE-18903.patch
> Lower Logging Level for ObjectStore
> ---
>
> Key: HIVE-18903
> URL: https://issues.apache.org/jira/browse/HIVE-18903
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Minor
> Labels: noob
> Attachments: HIVE-18903.patch
>
>
> [https://github.com/apache/hive/blob/7c22d74c8d0eb0650adf6e84e0536127c103e46c/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java]
>
> {code:java}
> 2018-03-01 06:51:58,051 INFO org.apache.hadoop.hive.metastore.ObjectStore:
> [pool-4-thread-13]: ObjectStore, initialize called
> 2018-03-01 06:51:58,052 INFO org.apache.hadoop.hive.metastore.ObjectStore:
> [pool-4-thread-13]: Initialized ObjectStore
> {code}
> Noting actionable or all that useful here. Please lower to _debug_ or
> _trace_ level logging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18881) Lower Logging for FSStatsAggregator
[
https://issues.apache.org/jira/browse/HIVE-18881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-18881:
---
Status: Patch Available (was: Open)
> Lower Logging for FSStatsAggregator
> ---
>
> Key: HIVE-18881
> URL: https://issues.apache.org/jira/browse/HIVE-18881
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18881.patch
>
>
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java#L101]
> {code:java}
> LOG.info("Read stats for : " + partID + "\t" + statType + "\t" + counter);
> {code}
> # All the other logging in this class is _debug_ or _error_ level logging.
> This should be _debug_ as well
> # Remove tab characters to allow splitting on tabs in any kind of
> tab-separated file of log lines
> # Use SLF4J parameterized logging
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18881) Lower Logging for FSStatsAggregator
[
https://issues.apache.org/jira/browse/HIVE-18881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454225#comment-16454225
]
Antal Sinkovits commented on HIVE-18881:
Patch contains fix for 2. and 3.
1. is skipped as the information logged is potentially useful.
> Lower Logging for FSStatsAggregator
> ---
>
> Key: HIVE-18881
> URL: https://issues.apache.org/jira/browse/HIVE-18881
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18881.patch
>
>
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java#L101]
> {code:java}
> LOG.info("Read stats for : " + partID + "\t" + statType + "\t" + counter);
> {code}
> # All the other logging in this class is _debug_ or _error_ level logging.
> This should be _debug_ as well
> # Remove tab characters to allow splitting on tabs in any kind of
> tab-separated file of log lines
> # Use SLF4J parameterized logging
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18881) Lower Logging for FSStatsAggregator
[
https://issues.apache.org/jira/browse/HIVE-18881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-18881:
---
Attachment: HIVE-18881.patch
> Lower Logging for FSStatsAggregator
> ---
>
> Key: HIVE-18881
> URL: https://issues.apache.org/jira/browse/HIVE-18881
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18881.patch
>
>
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java#L101]
> {code:java}
> LOG.info("Read stats for : " + partID + "\t" + statType + "\t" + counter);
> {code}
> # All the other logging in this class is _debug_ or _error_ level logging.
> This should be _debug_ as well
> # Remove tab characters to allow splitting on tabs in any kind of
> tab-separated file of log lines
> # Use SLF4J parameterized logging
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18880) Change Log to Debug in CombineHiveInputFormat
[
https://issues.apache.org/jira/browse/HIVE-18880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-18880:
---
Status: Patch Available (was: Open)
> Change Log to Debug in CombineHiveInputFormat
> -
>
> Key: HIVE-18880
> URL: https://issues.apache.org/jira/browse/HIVE-18880
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18880.patch
>
>
> [https://github.com/apache/hive/blob/1e74aca8d09ea2ef636311d2168b4d34198f7194/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L467]
> {code:java}
> private InputSplit[] getCombineSplits(JobConf job, int numSplits,
> Map pathToPartitionInfo) {
> ...
> LOG.info("number of splits " + result.size());
> ...
> }
> {code}
> [https://github.com/apache/hive/blob/1e74aca8d09ea2ef636311d2168b4d34198f7194/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L587]
> {code:java}
> public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
> ...
> LOG.info("Number of all splits " + result.size());
> ...
> }
> {code}
> # Capitalize "N"umber in the first logging to be consistent across all
> logging statements
> # Change the first logging message to be _debug_ level seeing as it's in a
> private method.
> It's an implementation logging and the entire total (most useful for a
> client) is captured in _info_ level at the end of the public method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18880) Change Log to Debug in CombineHiveInputFormat
[
https://issues.apache.org/jira/browse/HIVE-18880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-18880:
---
Attachment: HIVE-18880.patch
> Change Log to Debug in CombineHiveInputFormat
> -
>
> Key: HIVE-18880
> URL: https://issues.apache.org/jira/browse/HIVE-18880
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18880.patch
>
>
> [https://github.com/apache/hive/blob/1e74aca8d09ea2ef636311d2168b4d34198f7194/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L467]
> {code:java}
> private InputSplit[] getCombineSplits(JobConf job, int numSplits,
> Map pathToPartitionInfo) {
> ...
> LOG.info("number of splits " + result.size());
> ...
> }
> {code}
> [https://github.com/apache/hive/blob/1e74aca8d09ea2ef636311d2168b4d34198f7194/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java#L587]
> {code:java}
> public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException {
> ...
> LOG.info("Number of all splits " + result.size());
> ...
> }
> {code}
> # Capitalize "N"umber in the first logging to be consistent across all
> logging statements
> # Change the first logging message to be _debug_ level seeing as it's in a
> private method.
> It's an implementation logging and the entire total (most useful for a
> client) is captured in _info_ level at the end of the public method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits reassigned HIVE-18906:
--
Assignee: Antal Sinkovits
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181
> {code}
> 2018-02-26 14:32:50,620 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:09,566 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:43,170 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:46,301 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:16,966 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:18,072 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:25,064 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:51,781 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:04,685 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:26,689 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> {code}
> I wouldn't mind knowing this information when the service starts up and
> detects that there is a Direct SQL compatible database, but I don't need to
> know about this every time.
> Please lower to _debug_ level logging and parameterize with SLF4J. Direct
> SQL is a performance thing and not something I care about during the normal
> course of operation (unless there is an error which would presumably be
> covered by an _error_ logging message).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
