[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722389#comment-16722389 ] Hive QA commented on HIVE-20992: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12951915/HIVE-20992.6.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 15728 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15340/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15340/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15340/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12951915 - PreCommit-HIVE-Build > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch, > HIVE-20992.4.patch, HIVE-20992.5.patch, HIVE-20992.6.patch, HIVE-20992.patch > > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the > following properties: > * *_hive.metastore.dbaccess.ssl.use.SSL (metastore.dbaccess.ssl.use.SSL)_* > ** Set this to true to for using SSL/TLS encryption from the HMS Service to > the HMS backend store > ** Default: false > * _*hive.metastore.dbaccess.ssl.truststore.path > (metastore.dbaccess.ssl.truststore.path)*_ > ** Truststore location > ** Directly maps to _javax.net.ssl.trustStore_ System property > ** Default: None > ** E.g. _/tmp/truststore_ > * *_hive.metastore.dbaccess.ssl.truststore.password > (metastore.dbaccess.ssl.truststore.password)_* > ** Truststore password > ** Directly maps to _javax.net.ssl.trustStorePassword_ System property > ** Default: None > ** E.g. _password_ > * *_hive.metastore.dbaccess.ssl.truststore.type > (metastore.dbaccess.ssl.truststore.type)_* > ** Truststore type > ** Directly maps to _javax.net.ssl.trustStoreType_ System property > ** Default: JKS > ** E.g. _pkcs12_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is si
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722386#comment-16722386 ] Hive QA commented on HIVE-20992: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 49s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 44s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 16s{color} | {color:blue} standalone-metastore/metastore-common in master has 29 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 3s{color} | {color:blue} standalone-metastore/metastore-server in master has 188 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 21m 46s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15340/dev-support/hive-personality.sh | | git revision | master / 4e41560 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: standalone-metastore/metastore-common standalone-metastore/metastore-server U: standalone-metastore | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15340/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch, > HIVE-20992.4.patch, HIVE-20992.5.patch, HIVE-20992.6.patch, HIVE-20992.patch > > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _
[jira] [Commented] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722379#comment-16722379 ] Hive QA commented on HIVE-18661: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12951914/HIVE-18661.09.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 15727 tests executed *Failed tests:* {noformat} TestLocationQueries - did not produce a TEST-*.xml file (likely timed out) (batchId=250) org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=229) org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=229) org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup (batchId=229) org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession (batchId=229) org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics (batchId=229) org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps (batchId=229) org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse (batchId=229) org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=229) org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError (batchId=229) org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=229) org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=229) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions (batchId=261) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll (batchId=261) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomNonExistent (batchId=261) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesRead (batchId=261) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes (batchId=261) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryElapsedTime (batchId=261) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryExecutionTime (batchId=261) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15339/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15339/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15339/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 19 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12951914 - PreCommit-HIVE-Build > CachedStore: Use metastore notification log events to update cache > -- > > Key: HIVE-18661 > URL: https://issues.apache.org/jira/browse/HIVE-18661 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18661.02.patch, HIVE-18661.03.patch, > HIVE-18661.04.patch, HIVE-18661.05.patch, HIVE-18661.06.patch, > HIVE-18661.07.patch, HIVE-18661.08.patch, HIVE-18661.09.patch > > > Currently, a background thread updates the entire cache which is pretty > inefficient. We capture the updates to metadata in NOTIFICATION_LOG table > which is getting used in the Replication work. We should have the background > thread apply these notifications to incrementally update the cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Morio Ramdenbourg updated HIVE-20992: - Attachment: HIVE-20992.6.patch Status: Patch Available (was: In Progress) > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch, > HIVE-20992.4.patch, HIVE-20992.5.patch, HIVE-20992.6.patch, HIVE-20992.patch > > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the > following properties: > * *_hive.metastore.dbaccess.ssl.use.SSL (metastore.dbaccess.ssl.use.SSL)_* > ** Set this to true to for using SSL/TLS encryption from the HMS Service to > the HMS backend store > ** Default: false > * _*hive.metastore.dbaccess.ssl.truststore.path > (metastore.dbaccess.ssl.truststore.path)*_ > ** Truststore location > ** Directly maps to _javax.net.ssl.trustStore_ System property > ** Default: None > ** E.g. _/tmp/truststore_ > * *_hive.metastore.dbaccess.ssl.truststore.password > (metastore.dbaccess.ssl.truststore.password)_* > ** Truststore password > ** Directly maps to _javax.net.ssl.trustStorePassword_ System property > ** Default: None > ** E.g. _password_ > * *_hive.metastore.dbaccess.ssl.truststore.type > (metastore.dbaccess.ssl.truststore.type)_* > ** Truststore type > ** Directly maps to _javax.net.ssl.trustStoreType_ System property > ** Default: JKS > ** E.g. _pkcs12_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722372#comment-16722372 ] Hive QA commented on HIVE-18661: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 4s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 21s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 25s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 14s{color} | {color:blue} standalone-metastore/metastore-common in master has 29 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 4s{color} | {color:blue} standalone-metastore/metastore-server in master has 188 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 37s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 25s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 35s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 43s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 5s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 37s{color} | {color:red} ql: The patch generated 4 new + 3 unchanged - 0 fixed = 7 total (was 3) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 16s{color} | {color:red} itests/hive-unit: The patch generated 20 new + 86 unchanged - 0 fixed = 106 total (was 86) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 14s{color} | {color:red} standalone-metastore/metastore-server generated 1 new + 188 unchanged - 0 fixed = 189 total (was 188) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 17s{color} | {color:red} standalone-metastore_metastore-server generated 1 new + 48 unchanged - 0 fixed = 49 total (was 48) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 44m 47s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:standalone-metastore/metastore-server | | | Load of known null value in org.apache.hadoop.hive.metastore.cache.CachedStore.get_aggr_stats_for(String, String, String, List, List, String) At CachedStore.java:in org.apache.hadoop.hive.metastore.cache.CachedStore.get_aggr_stats_for(String, String, String, List, List, String) At CachedStore.java:[line ] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15339/dev-support/hive-personality.sh | | git revision | master / 4e41560 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 |
[jira] [Updated] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Morio Ramdenbourg updated HIVE-20992: - Status: In Progress (was: Patch Available) > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch, > HIVE-20992.4.patch, HIVE-20992.5.patch, HIVE-20992.patch > > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the > following properties: > * *_hive.metastore.dbaccess.ssl.use.SSL (metastore.dbaccess.ssl.use.SSL)_* > ** Set this to true to for using SSL/TLS encryption from the HMS Service to > the HMS backend store > ** Default: false > * _*hive.metastore.dbaccess.ssl.truststore.path > (metastore.dbaccess.ssl.truststore.path)*_ > ** Truststore location > ** Directly maps to _javax.net.ssl.trustStore_ System property > ** Default: None > ** E.g. _/tmp/truststore_ > * *_hive.metastore.dbaccess.ssl.truststore.password > (metastore.dbaccess.ssl.truststore.password)_* > ** Truststore password > ** Directly maps to _javax.net.ssl.trustStorePassword_ System property > ** Default: None > ** E.g. _password_ > * *_hive.metastore.dbaccess.ssl.truststore.type > (metastore.dbaccess.ssl.truststore.type)_* > ** Truststore type > ** Directly maps to _javax.net.ssl.trustStoreType_ System property > ** Default: JKS > ** E.g. _pkcs12_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-18661: --- Attachment: (was: HIVE-18661.09.patch) > CachedStore: Use metastore notification log events to update cache > -- > > Key: HIVE-18661 > URL: https://issues.apache.org/jira/browse/HIVE-18661 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18661.02.patch, HIVE-18661.03.patch, > HIVE-18661.04.patch, HIVE-18661.05.patch, HIVE-18661.06.patch, > HIVE-18661.07.patch, HIVE-18661.08.patch > > > Currently, a background thread updates the entire cache which is pretty > inefficient. We capture the updates to metadata in NOTIFICATION_LOG table > which is getting used in the Replication work. We should have the background > thread apply these notifications to incrementally update the cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-18661: --- Attachment: HIVE-18661.09.patch > CachedStore: Use metastore notification log events to update cache > -- > > Key: HIVE-18661 > URL: https://issues.apache.org/jira/browse/HIVE-18661 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18661.02.patch, HIVE-18661.03.patch, > HIVE-18661.04.patch, HIVE-18661.05.patch, HIVE-18661.06.patch, > HIVE-18661.07.patch, HIVE-18661.08.patch, HIVE-18661.09.patch > > > Currently, a background thread updates the entire cache which is pretty > inefficient. We capture the updates to metadata in NOTIFICATION_LOG table > which is getting used in the Replication work. We should have the background > thread apply these notifications to incrementally update the cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-18661: --- Status: Patch Available (was: Open) > CachedStore: Use metastore notification log events to update cache > -- > > Key: HIVE-18661 > URL: https://issues.apache.org/jira/browse/HIVE-18661 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18661.02.patch, HIVE-18661.03.patch, > HIVE-18661.04.patch, HIVE-18661.05.patch, HIVE-18661.06.patch, > HIVE-18661.07.patch, HIVE-18661.08.patch, HIVE-18661.09.patch > > > Currently, a background thread updates the entire cache which is pretty > inefficient. We capture the updates to metadata in NOTIFICATION_LOG table > which is getting used in the Replication work. We should have the background > thread apply these notifications to incrementally update the cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18661) CachedStore: Use metastore notification log events to update cache
[ https://issues.apache.org/jira/browse/HIVE-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-18661: --- Status: Open (was: Patch Available) > CachedStore: Use metastore notification log events to update cache > -- > > Key: HIVE-18661 > URL: https://issues.apache.org/jira/browse/HIVE-18661 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-18661.02.patch, HIVE-18661.03.patch, > HIVE-18661.04.patch, HIVE-18661.05.patch, HIVE-18661.06.patch, > HIVE-18661.07.patch, HIVE-18661.08.patch > > > Currently, a background thread updates the entire cache which is pretty > inefficient. We capture the updates to metadata in NOTIFICATION_LOG table > which is getting used in the Replication work. We should have the background > thread apply these notifications to incrementally update the cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21047) Read the HMS backend database password and truststore password during PersistenceManagerFactory initialization time
[ https://issues.apache.org/jira/browse/HIVE-21047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Morio Ramdenbourg updated HIVE-21047: - Description: This was pointed out by [~vihangk1] as part of the review for [HIVE-20992|https://issues.apache.org/jira/browse/HIVE-20992]. As part of the redaction of the _javax.jdo.option.ConnectionPassword_ and _metastore.dbaccess.ssl.truststore.password_ properties, they both use the [Hadoop Credential Provider API|https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html] to prevent the passwords from being stored in plain text. However, these are both being read in [setConf()|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L198-L247] in ObjectStore, thereby calling the expensive decrypt during every new database connection initialization despite these values almost never changing. We should instead move these reads into the PersistenceManagerFactory [initPMF()|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java#L227-L273] method and cache their values so they are only read once when the HMS starts. was: This was pointed out by [~vihangk1] as part of the review for [HIVE-20992|https://issues.apache.org/jira/browse/HIVE-20992]. As part of the redaction of the _javax.jdo.option.ConnectionPassword_ and _metastore.dbaccess.ssl.truststore.password_ properties, they both use the [Hadoop Credential Provider API|https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html] to prevent the passwords from being stored in plain text. However, these are both being read in during every new database connection initialization in [setConf()|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L198-L247] in ObjectStore, thereby calling the expensive decrypt every time despite these values almost never changing. We should instead move these reads into the PersistenceManagerFactory [initPMF()|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java#L227-L273] method and cache their values so they are only read once when the HMS starts. > Read the HMS backend database password and truststore password during > PersistenceManagerFactory initialization time > --- > > Key: HIVE-21047 > URL: https://issues.apache.org/jira/browse/HIVE-21047 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Priority: Major > > This was pointed out by [~vihangk1] as part of the review for > [HIVE-20992|https://issues.apache.org/jira/browse/HIVE-20992]. > As part of the redaction of the _javax.jdo.option.ConnectionPassword_ and > _metastore.dbaccess.ssl.truststore.password_ properties, they both use the > [Hadoop Credential Provider > API|https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html] > to prevent the passwords from being stored in plain text. > However, these are both being read in > [setConf()|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L198-L247] > in ObjectStore, thereby calling the expensive decrypt during every new > database connection initialization despite these values almost never changing. > We should instead move these reads into the PersistenceManagerFactory > [initPMF()|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java#L227-L273] > method and cache their values so they are only read once when the HMS starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16255) Support percentile_cont / percentile_disc
[ https://issues.apache.org/jira/browse/HIVE-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722341#comment-16722341 ] Hive QA commented on HIVE-16255: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12951912/HIVE-16255.02.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15754 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parallel_orderby] (batchId=58) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_functions] (batchId=79) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15338/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15338/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15338/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12951912 - PreCommit-HIVE-Build > Support percentile_cont / percentile_disc > - > > Key: HIVE-16255 > URL: https://issues.apache.org/jira/browse/HIVE-16255 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-16255.01.patch, HIVE-16255.02.patch > > > Way back in HIVE-259, a percentile function was added that provides a subset > of the standard percentile_cont aggregate function. > The SQL standard provides some additional options and also a percentile_disc > aggregate function with different rules. In the standard you specify an > ordering with arbitrary value expression and the results are drawn from this > value expression. This aggregate functions should be usable as analytic > functions as well (i.e. support the over clause). The current percentile > function is able to be used with an over clause. > The rough outline of how this works is: > percentile_cont(number) within group (order by expression) [ over(window > spec) ] > percentile_disc(number) within group (order by expression) [ over(window > spec) ] > The value of number should be between 0 and 1. The value expression is > evaluated for each row of the group, nulls are discarded, and the remaining > rows are ordered. > — If PERCENTILE_CONT is specified, by considering the pair of consecutive > rows that are indicated by the argument, treated as a fraction of the total > number of rows in the group, and interpolating the value of the value > expression evaluated for these rows. > — If PERCENTILE_DISC is specified, by treating the group as a window > partition of the CUME_DIST window function, using the specified ordering of > the value expression as the window ordering, and returning the first value > expression whose cumulative distribution value is greater than or equal to > the argument. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16255) Support percentile_cont / percentile_disc
[ https://issues.apache.org/jira/browse/HIVE-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722333#comment-16722333 ] Hive QA commented on HIVE-16255: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 38s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 51s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s{color} | {color:red} ql: The patch generated 48 new + 72 unchanged - 0 fixed = 120 total (was 72) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 1s{color} | {color:red} ql generated 3 new + 2310 unchanged - 0 fixed = 2313 total (was 2310) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 23m 40s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$DoubleComparator implements Comparator but not Serializable At GenericUDAFPercentileCont.java:Serializable At GenericUDAFPercentileCont.java:[lines 109-114] | | | org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$LongComparator implements Comparator but not Serializable At GenericUDAFPercentileCont.java:Serializable At GenericUDAFPercentileCont.java:[lines 101-105] | | | org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$PercentileContEvaluator.terminatePartial(GenericUDAFEvaluator$AggregationBuffer) may expose internal representation by returning GenericUDAFPercentileCont$PercentileContEvaluator.partialResult At GenericUDAFPercentileCont.java:by returning GenericUDAFPercentileCont$PercentileContEvaluator.partialResult At GenericUDAFPercentileCont.java:[line 186] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15338/dev-support/hive-personality.sh | | git revision | master / 4e41560 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-15338/yetus/diff-checkstyle-ql.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-15338/yetus/new-findbugs-ql.html | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-15338/yetus/patch-asflicense-problems.txt | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15338/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Support percentile_cont / percentile_disc > - > > Key: HIVE-16255 > URL: https://issues.apache.org/jira/browse/HIVE-16255 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Laszlo Bodor >
[jira] [Updated] (HIVE-16255) Support percentile_cont / percentile_disc
[ https://issues.apache.org/jira/browse/HIVE-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-16255: Attachment: HIVE-16255.02.patch > Support percentile_cont / percentile_disc > - > > Key: HIVE-16255 > URL: https://issues.apache.org/jira/browse/HIVE-16255 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-16255.01.patch, HIVE-16255.02.patch > > > Way back in HIVE-259, a percentile function was added that provides a subset > of the standard percentile_cont aggregate function. > The SQL standard provides some additional options and also a percentile_disc > aggregate function with different rules. In the standard you specify an > ordering with arbitrary value expression and the results are drawn from this > value expression. This aggregate functions should be usable as analytic > functions as well (i.e. support the over clause). The current percentile > function is able to be used with an over clause. > The rough outline of how this works is: > percentile_cont(number) within group (order by expression) [ over(window > spec) ] > percentile_disc(number) within group (order by expression) [ over(window > spec) ] > The value of number should be between 0 and 1. The value expression is > evaluated for each row of the group, nulls are discarded, and the remaining > rows are ordered. > — If PERCENTILE_CONT is specified, by considering the pair of consecutive > rows that are indicated by the argument, treated as a fraction of the total > number of rows in the group, and interpolating the value of the value > expression evaluated for these rows. > — If PERCENTILE_DISC is specified, by treating the group as a window > partition of the CUME_DIST window function, using the specified ordering of > the value expression as the window ordering, and returning the first value > expression whose cumulative distribution value is greater than or equal to > the argument. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16255) Support percentile_cont / percentile_disc
[ https://issues.apache.org/jira/browse/HIVE-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-16255: Status: Patch Available (was: Open) > Support percentile_cont / percentile_disc > - > > Key: HIVE-16255 > URL: https://issues.apache.org/jira/browse/HIVE-16255 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-16255.01.patch, HIVE-16255.02.patch > > > Way back in HIVE-259, a percentile function was added that provides a subset > of the standard percentile_cont aggregate function. > The SQL standard provides some additional options and also a percentile_disc > aggregate function with different rules. In the standard you specify an > ordering with arbitrary value expression and the results are drawn from this > value expression. This aggregate functions should be usable as analytic > functions as well (i.e. support the over clause). The current percentile > function is able to be used with an over clause. > The rough outline of how this works is: > percentile_cont(number) within group (order by expression) [ over(window > spec) ] > percentile_disc(number) within group (order by expression) [ over(window > spec) ] > The value of number should be between 0 and 1. The value expression is > evaluated for each row of the group, nulls are discarded, and the remaining > rows are ordered. > — If PERCENTILE_CONT is specified, by considering the pair of consecutive > rows that are indicated by the argument, treated as a fraction of the total > number of rows in the group, and interpolating the value of the value > expression evaluated for these rows. > — If PERCENTILE_DISC is specified, by treating the group as a window > partition of the CUME_DIST window function, using the specified ordering of > the value expression as the window ordering, and returning the first value > expression whose cumulative distribution value is greater than or equal to > the argument. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19968) UDF exception is not throw out
[ https://issues.apache.org/jira/browse/HIVE-19968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722173#comment-16722173 ] Laszlo Bodor commented on HIVE-19968: - [~kgyrtkirk] : 04.patch is okay from my side, and it achieved a green run > UDF exception is not throw out > -- > > Key: HIVE-19968 > URL: https://issues.apache.org/jira/browse/HIVE-19968 > Project: Hive > Issue Type: Bug >Reporter: sandflee >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-19968.01.patch, HIVE-19968.02.patch, > HIVE-19968.03.patch, HIVE-19968.04.patch, hive-udf.png > > > udf init failed, and throw a exception, but hive catch it and do nothing, > leading to app succ, but no data is generated. > {code} > GenericUDFReflect.java#evaluate() > try { > o = null; > o = ReflectionUtils.newInstance(c, null); > } catch (Exception e) { > // ignored > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20079) Populate more accurate rawDataSize for parquet format
[ https://issues.apache.org/jira/browse/HIVE-20079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722137#comment-16722137 ] Antal Sinkovits commented on HIVE-20079: Hi, I'm planing to work on this next week. Let me know, if there are any concerns regarding it. Thanks. > Populate more accurate rawDataSize for parquet format > - > > Key: HIVE-20079 > URL: https://issues.apache.org/jira/browse/HIVE-20079 > Project: Hive > Issue Type: Improvement > Components: File Formats >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Priority: Major > Attachments: HIVE-20079.1.patch, HIVE-20079.2.patch > > > Run the following queries and you will see the raw data for the table is 4 > (that is the number of fields) incorrectly. We need to populate correct data > size so data can be split properly. > {noformat} > SET hive.stats.autogather=true; > CREATE TABLE parquet_stats (id int,str string) STORED AS PARQUET; > INSERT INTO parquet_stats values(0, 'this is string 0'), (1, 'string 1'); > DESC FORMATTED parquet_stats; > {noformat} > {noformat} > Table Parameters: > COLUMN_STATS_ACCURATE true > numFiles1 > numRows 2 > rawDataSize 4 > totalSize 373 > transient_lastDdlTime 1530660523 > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)