[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16723415#comment-16723415 ] Vihang Karajgaonkar commented on HIVE-20992: Patch merged into master. Thanks for your contribution [~mramdenbourg] > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch, > HIVE-20992.4.patch, HIVE-20992.5.patch, HIVE-20992.6.patch, > HIVE-20992.7.patch, HIVE-20992.patch > > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the > following properties: > * *_hive.metastore.dbaccess.ssl.use.SSL (metastore.dbaccess.ssl.use.SSL)_* > ** Set this to true to for using SSL/TLS encryption from the HMS Service to > the HMS backend store > ** Default: false > * _*hive.metastore.dbaccess.ssl.truststore.path > (metastore.dbaccess.ssl.truststore.path)*_ > ** Truststore location > ** Directly maps to _javax.net.ssl.trustStore_ System property > ** Default: None > ** E.g. _/tmp/truststore_ > * *_hive.metastore.dbaccess.ssl.truststore.password > (metastore.dbaccess.ssl.truststore.password)_* > ** Truststore password > ** Directly maps to _javax.net.ssl.trustStorePassword_ System property > ** Default: None > ** E.g. _password_ > * *_hive.metastore.dbaccess.ssl.truststore.type > (metastore.dbaccess.ssl.truststore.type)_* > ** Truststore type > ** Directly maps to _javax.net.ssl.trustStoreType_ System property > ** Default: JKS > ** E.g. _pkcs12_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16723349#comment-16723349 ] Vihang Karajgaonkar commented on HIVE-20992: Hi [~mramdenbourg] Left couple of minor edits to the patch on the review board. Rest looks good. I don't think we need to wait for full tests given most of the changes suggested are trivial. Just make sure that you run the newly added test locally. +1 > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch, > HIVE-20992.4.patch, HIVE-20992.5.patch, HIVE-20992.6.patch, HIVE-20992.patch > > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the > following properties: > * *_hive.metastore.dbaccess.ssl.use.SSL (metastore.dbaccess.ssl.use.SSL)_* > ** Set this to true to for using SSL/TLS encryption from the HMS Service to > the HMS backend store > ** Default: false > * _*hive.metastore.dbaccess.ssl.truststore.path > (metastore.dbaccess.ssl.truststore.path)*_ > ** Truststore location > ** Directly maps to _javax.net.ssl.trustStore_ System property > ** Default: None > ** E.g. _/tmp/truststore_ > * *_hive.metastore.dbaccess.ssl.truststore.password > (metastore.dbaccess.ssl.truststore.password)_* > ** Truststore password > ** Directly maps to _javax.net.ssl.trustStorePassword_ System property > ** Default: None > ** E.g. _password_ > * *_hive.metastore.dbaccess.ssl.truststore.type > (metastore.dbaccess.ssl.truststore.type)_* > ** Truststore type > ** Directly maps to _javax.net.ssl.trustStoreType_ System property > ** Default: JKS > ** E.g. _pkcs12_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[
https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722389#comment-16722389
]
Hive QA commented on HIVE-20992:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951915/HIVE-20992.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15728 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15340/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15340/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15340/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951915 - PreCommit-HIVE-Build
> Split the config "hive.metastore.dbaccess.ssl.properties" into more
> meaningful configs
> --
>
> Key: HIVE-20992
> URL: https://issues.apache.org/jira/browse/HIVE-20992
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Security, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Minor
> Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch,
> HIVE-20992.4.patch, HIVE-20992.5.patch, HIVE-20992.6.patch, HIVE-20992.patch
>
>
> HIVE-13044 brought in the ability to enable TLS encryption from the HMS
> Service to the HMSDB by configuring the following two properties:
> # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC
> metastore. To use SSL to encrypt/authenticate the connection, provide
> database-specific SSL flag in the connection URL. (E.g.
> "jdbc:postgresql://myhost/db?ssl=true")
> # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties
> for metastore to access database when JDO connection URL. (E.g.
> javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd)
> However, the latter configuration option is opaque and poses some problems.
> The most glaring of which is it takes in _any_
> [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html]
> system property, whether it is
> [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization]
> or not. This can cause some unintended side-effects for other components of
> the HMS, especially if it overrides an already-set system property. If the
> user truly wishes to add an unrelated Java property, setting it statically
> using the "-D" option of the _java_ command is more appropriate. Secondly,
> the truststore password is stored in plain text. We should add Hadoop Shims
> back to the HMS to prevent exposing these passwords, but this effort can be
> done after this ticket.
> I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the
> following properties:
> * *_hive.metastore.dbaccess.ssl.use.SSL (metastore.dbaccess.ssl.use.SSL)_*
> ** Set this to true to for using SSL/TLS encryption from the HMS Service to
> the HMS backend store
> ** Default: false
> * _*hive.metastore.dbaccess.ssl.truststore.path
> (metastore.dbaccess.ssl.truststore.path)*_
> ** Truststore location
> ** Directly maps to _javax.net.ssl.trustStore_ System property
> ** Default: None
> ** E.g. _/tmp/truststore_
> * *_hive.metastore.dbaccess.ssl.truststore.password
> (metastore.dbaccess.ssl.truststore.password)_*
> ** Truststore password
> ** Directly maps to _javax.net.ssl.trustStorePassword_ System property
> ** Default: None
> ** E.g. _password_
> * *_hive.metastore.dbaccess.ssl.truststore.type
> (metastore.dbaccess.ssl.truststore.type)_*
> ** Truststore type
> ** Directly maps to _javax.net.ssl.trustStoreType_ System property
> ** Default: JKS
> ** E.g. _pkcs12_
> We should guide the user towards an easier TLS configuration experience. This
> is the minimum configuration necessary to configure TLS to the HMSDB. If we
> need other options, such as the keystore location/password for
> dual-authentication, then we can add those on afterwards.
> Also, document these changes -
> [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL]
> does not have up-to-date documentation, and these new parameters will need
> documentation as well.
> Note "TLS" refers to both SSL and TLS. TLS is si
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[
https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722386#comment-16722386
]
Hive QA commented on HIVE-20992:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
16s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
3s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15340/dev-support/hive-personality.sh
|
| git revision | master / 4e41560 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15340/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Split the config "hive.metastore.dbaccess.ssl.properties" into more
> meaningful configs
> --
>
> Key: HIVE-20992
> URL: https://issues.apache.org/jira/browse/HIVE-20992
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Security, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Minor
> Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch,
> HIVE-20992.4.patch, HIVE-20992.5.patch, HIVE-20992.6.patch, HIVE-20992.patch
>
>
> HIVE-13044 brought in the ability to enable TLS encryption from the HMS
> Service to the HMSDB by configuring the following two properties:
> # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC
> metastore. To use SSL to encrypt/authenticate the connection, provide
> database-specific SSL flag in the connection URL. (E.g.
> "jdbc:postgresql://myhost/db?ssl=true")
> # _
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[
https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16721290#comment-16721290
]
Hive QA commented on HIVE-20992:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951783/HIVE-20992.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15665 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15320/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15320/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15320/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951783 - PreCommit-HIVE-Build
> Split the config "hive.metastore.dbaccess.ssl.properties" into more
> meaningful configs
> --
>
> Key: HIVE-20992
> URL: https://issues.apache.org/jira/browse/HIVE-20992
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Security, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Minor
> Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch,
> HIVE-20992.4.patch, HIVE-20992.5.patch, HIVE-20992.patch
>
>
> HIVE-13044 brought in the ability to enable TLS encryption from the HMS
> Service to the HMSDB by configuring the following two properties:
> # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC
> metastore. To use SSL to encrypt/authenticate the connection, provide
> database-specific SSL flag in the connection URL. (E.g.
> "jdbc:postgresql://myhost/db?ssl=true")
> # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties
> for metastore to access database when JDO connection URL. (E.g.
> javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd)
> However, the latter configuration option is opaque and poses some problems.
> The most glaring of which is it takes in _any_
> [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html]
> system property, whether it is
> [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization]
> or not. This can cause some unintended side-effects for other components of
> the HMS, especially if it overrides an already-set system property. If the
> user truly wishes to add an unrelated Java property, setting it statically
> using the "-D" option of the _java_ command is more appropriate. Secondly,
> the truststore password is stored in plain text. We should add Hadoop Shims
> back to the HMS to prevent exposing these passwords, but this effort can be
> done after this ticket.
> I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the
> following properties:
> * *_hive.metastore.dbaccess.ssl.use.SSL (metastore.dbaccess.ssl.use.SSL)_*
> ** Set this to true to for using SSL/TLS encryption from the HMS Service to
> the HMS backend store
> ** Default: false
> * _*hive.metastore.dbaccess.ssl.truststore.path
> (metastore.dbaccess.ssl.truststore.path)*_
> ** Truststore location
> ** Directly maps to _javax.net.ssl.trustStore_ System property
> ** Default: None
> ** E.g. _/tmp/truststore_
> * *_hive.metastore.dbaccess.ssl.truststore.password
> (metastore.dbaccess.ssl.truststore.password)_*
> ** Truststore password
> ** Directly maps to _javax.net.ssl.trustStorePassword_ System property
> ** Default: None
> ** E.g. _password_
> * *_hive.metastore.dbaccess.ssl.truststore.type
> (metastore.dbaccess.ssl.truststore.type)_*
> ** Truststore type
> ** Directly maps to _javax.net.ssl.trustStoreType_ System property
> ** Default: JKS
> ** E.g. _pkcs12_
> We should guide the user towards an easier TLS configuration experience. This
> is the minimum configuration necessary to configure TLS to the HMSDB. If we
> need other options, such as the keystore location/password for
> dual-authentication, then we can add those on afterwards.
> Also, document these changes -
> [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL]
> does not have up-to-date documentation, and these new parameters will need
> documentation as well.
> Note "TLS" refers to both SSL and TLS. TLS is simply the successor o
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[
https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16721238#comment-16721238
]
Hive QA commented on HIVE-20992:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
14s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
1s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15320/dev-support/hive-personality.sh
|
| git revision | master / e8e0396 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15320/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Split the config "hive.metastore.dbaccess.ssl.properties" into more
> meaningful configs
> --
>
> Key: HIVE-20992
> URL: https://issues.apache.org/jira/browse/HIVE-20992
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Security, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Minor
> Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch,
> HIVE-20992.4.patch, HIVE-20992.5.patch, HIVE-20992.patch
>
>
> HIVE-13044 brought in the ability to enable TLS encryption from the HMS
> Service to the HMSDB by configuring the following two properties:
> # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC
> metastore. To use SSL to encrypt/authenticate the connection, provide
> database-specific SSL flag in the connection URL. (E.g.
> "jdbc:postgresql://myhost/db?ssl=true")
> # _hive.metastore.dbacc
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[
https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16721008#comment-16721008
]
Hive QA commented on HIVE-20992:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951748/HIVE-20992.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15317/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15317/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15317/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-12-14 07:09:07.167
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15317/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-12-14 07:09:07.171
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at e8e0396 HIVE-21020: log which table/partition is being processed
by a txn in Worker (Eugene Koifman, reviewed by Jason Dere, Igor Kryvenko)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at e8e0396 HIVE-21020: log which table/partition is being processed
by a txn in Worker (Eugene Koifman, reviewed by Jason Dere, Igor Kryvenko)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-12-14 07:09:08.283
+ rm -rf ../yetus_PreCommit-HIVE-Build-15317
+ mkdir ../yetus_PreCommit-HIVE-Build-15317
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15317
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15317/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc8193302175318580749.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc8193302175318580749.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc276867551589549715.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
log4j:WARN No appenders could be found for logger (DataNucleus.Persistence).
log4j:WARN Please initialize the log4j system properly.
DataNucleus Enhancer (version 4.1.17) for API "JDO"
DataNucleus Enhancer completed with success for 41 classes.
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(process-resource-bundles) on project hive-spark-client: Execution
process-resource-bundles of goal
org.apache.maven.plugins:maven-remote-reso
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[
https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16715867#comment-16715867
]
Hive QA commented on HIVE-20992:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951261/HIVE-20992.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15648 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=154)
[intersect_all.q,unionDistinct_1.q,table_nonprintable.q,orc_llap_counters1.q,mm_cttas.q,whroot_external1.q,global_limit.q,cte_2.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,results_cache_diff_fs.q,cttl.q,parallel_colstats.q,load_hdfs_file_with_space_in_the_name.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15250/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15250/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15250/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951261 - PreCommit-HIVE-Build
> Split the config "hive.metastore.dbaccess.ssl.properties" into more
> meaningful configs
> --
>
> Key: HIVE-20992
> URL: https://issues.apache.org/jira/browse/HIVE-20992
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Security, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Minor
> Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch, HIVE-20992.patch
>
>
> HIVE-13044 brought in the ability to enable TLS encryption from the HMS
> Service to the HMSDB by configuring the following two properties:
> # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC
> metastore. To use SSL to encrypt/authenticate the connection, provide
> database-specific SSL flag in the connection URL. (E.g.
> "jdbc:postgresql://myhost/db?ssl=true")
> # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties
> for metastore to access database when JDO connection URL. (E.g.
> javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd)
> However, the latter configuration option is opaque and poses some problems.
> The most glaring of which is it takes in _any_
> [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html]
> system property, whether it is
> [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization]
> or not. This can cause some unintended side-effects for other components of
> the HMS, especially if it overrides an already-set system property. If the
> user truly wishes to add an unrelated Java property, setting it statically
> using the "-D" option of the _java_ command is more appropriate. Secondly,
> the truststore password is stored in plain text. We should add Hadoop Shims
> back to the HMS to prevent exposing these passwords, but this effort can be
> done after this ticket.
> I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the
> following properties:
> * *_hive.metastore.dbaccess.ssl.use.SSL (metastore.dbaccess.ssl.use.SSL)_*
> ** Set this to true to for using SSL/TLS encryption from the HMS Service to
> the HMS backend store
> ** Default: false
> * _*hive.metastore.dbaccess.ssl.truststore.path
> (metastore.dbaccess.ssl.truststore.path)*_
> ** Truststore location
> ** Directly maps to _javax.net.ssl.trustStore_ System property
> ** Default: None
> ** E.g. _/tmp/truststore_
> * *_hive.metastore.dbaccess.ssl.truststore.password
> (metastore.dbaccess.ssl.truststore.password)_*
> ** Truststore password
> ** Directly maps to _javax.net.ssl.trustStorePassword_ System property
> ** Default: None
> ** E.g. _password_
> * *_hive.metastore.dbaccess.ssl.truststore.type
> (metastore.dbaccess.ssl.truststore.type)_*
> ** Truststore type
> ** Directly maps to _javax.net.ssl.trustStoreType_ System property
> ** Default: JKS
> ** E.g. _pkcs12_
> We should guide the user towards an easier TLS configuration experience. This
> is the minimum configuration necessary to configure TLS to the HMSDB. If we
> need other options, such as th
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[
https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16715813#comment-16715813
]
Hive QA commented on HIVE-20992:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
13s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
7s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15250/dev-support/hive-personality.sh
|
| git revision | master / a6091c3 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15250/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Split the config "hive.metastore.dbaccess.ssl.properties" into more
> meaningful configs
> --
>
> Key: HIVE-20992
> URL: https://issues.apache.org/jira/browse/HIVE-20992
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Security, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Minor
> Attachments: HIVE-20992.2.patch, HIVE-20992.3.patch, HIVE-20992.patch
>
>
> HIVE-13044 brought in the ability to enable TLS encryption from the HMS
> Service to the HMSDB by configuring the following two properties:
> # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC
> metastore. To use SSL to encrypt/authenticate the connection, provide
> database-specific SSL flag in the connection URL. (E.g.
> "jdbc:postgresql://myhost/db?ssl=true")
> # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL pr
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16715092#comment-16715092 ] Morio Ramdenbourg commented on HIVE-20992: -- After further feedback, I've modified the property names to better reflect that they are parameters for configuring SSL encryption to the database store. > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Attachments: HIVE-20992.patch > > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the > following properties: > * *_hive.metastore.dbaccess.ssl.use.SSL (metastore.dbaccess.ssl.use.SSL)_* > ** Set this to true to for using SSL/TLS encryption from the HMS Service to > the HMS backend store > ** Default: false > * _*hive.metastore.dbaccess.ssl.truststore.path > (metastore.dbaccess.ssl.truststore.path)*_ > ** Truststore location > ** Directly maps to _javax.net.ssl.trustStore_ System property > ** Default: None > ** E.g. _/tmp/truststore_ > * *_hive.metastore.dbaccess.ssl.truststore.password > (metastore.dbaccess.ssl.truststore.password)_* > ** Truststore password > ** Directly maps to _javax.net.ssl.trustStorePassword_ System property > ** Default: None > ** E.g. _password_ > * *_hive.metastore.dbaccess.ssl.truststore.type > (metastore.dbaccess.ssl.truststore.type)_* > ** Truststore type > ** Directly maps to _javax.net.ssl.trustStoreType_ System property > ** Default: JKS > ** E.g. _pkcs12_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[
https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713589#comment-16713589
]
Hive QA commented on HIVE-20992:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12951088/HIVE-20992.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15662 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15225/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15225/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15225/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12951088 - PreCommit-HIVE-Build
> Split the config "hive.metastore.dbaccess.ssl.properties" into more
> meaningful configs
> --
>
> Key: HIVE-20992
> URL: https://issues.apache.org/jira/browse/HIVE-20992
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Security, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Minor
> Attachments: HIVE-20992.patch
>
>
> HIVE-13044 brought in the ability to enable TLS encryption from the HMS
> Service to the HMSDB by configuring the following two properties:
> # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC
> metastore. To use SSL to encrypt/authenticate the connection, provide
> database-specific SSL flag in the connection URL. (E.g.
> "jdbc:postgresql://myhost/db?ssl=true")
> # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties
> for metastore to access database when JDO connection URL. (E.g.
> javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd)
> However, the latter configuration option is opaque and poses some problems.
> The most glaring of which is it takes in _any_
> [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html]
> system property, whether it is
> [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization]
> or not. This can cause some unintended side-effects for other components of
> the HMS, especially if it overrides an already-set system property. If the
> user truly wishes to add an unrelated Java property, setting it statically
> using the "-D" option of the _java_ command is more appropriate. Secondly,
> the truststore password is stored in plain text. We should add Hadoop Shims
> back to the HMS to prevent exposing these passwords, but this effort can be
> done after this ticket.
> I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the
> following properties:
> * *_hive.metastore.dbaccess.use.SSL_*
> ** Set this to true to for using SSL/TLS encryption from the HMS Service to
> the HMS backend store
> ** Default: false
> * *_javax.net.ssl.trustStore_*
> ** Truststore location
> ** Default: None
> ** E.g. _/tmp/truststore_
> * *_javax.net.ssl.trustStorePassword_*
> ** Truststore password
> ** Default: None
> ** E.g. _password_
> * *_javax.net.ssl.trustStoreType_*
> ** Truststore type
> ** Default: JKS
> ** E.g. _pkcs12_
> We should guide the user towards an easier TLS configuration experience. This
> is the minimum configuration necessary to configure TLS to the HMSDB. If we
> need other options, such as the keystore location/password for
> dual-authentication, then we can add those on afterwards.
> Also, document these changes -
> [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL]
> does not have up-to-date documentation, and these new parameters will need
> documentation as well.
> Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[
https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713584#comment-16713584
]
Hive QA commented on HIVE-20992:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
11s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
2s{color} | {color:blue} standalone-metastore/metastore-server in master has
184 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15225/dev-support/hive-personality.sh
|
| git revision | master / d8c500b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15225/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Split the config "hive.metastore.dbaccess.ssl.properties" into more
> meaningful configs
> --
>
> Key: HIVE-20992
> URL: https://issues.apache.org/jira/browse/HIVE-20992
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Security, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Minor
> Attachments: HIVE-20992.patch
>
>
> HIVE-13044 brought in the ability to enable TLS encryption from the HMS
> Service to the HMSDB by configuring the following two properties:
> # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC
> metastore. To use SSL to encrypt/authenticate the connection, provide
> database-specific SSL flag in the connection URL. (E.g.
> "jdbc:postgresql://myhost/db?ssl=true")
> # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties
> for metastore to access data
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713566#comment-16713566 ] Morio Ramdenbourg commented on HIVE-20992: -- Also, adding Shims back is not necessary. The method [MetastoreConf.getPassword()|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java#L1695-L1711] takes care of this for us. It already handles secure password access, and automatically checks from the CredentialsProvider. > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Attachments: HIVE-20992.patch > > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we deprecate _hive.metastore.dbaccess.ssl.properties_, and add the > following properties: > * *_hive.metastore.dbaccess.use.SSL_* > ** Set this to true to for using SSL/TLS encryption from the HMS Service to > the HMS backend store > ** Default: false > * *_javax.net.ssl.trustStore_* > ** Truststore location > ** Default: None > ** E.g. _/tmp/truststore_ > * *_javax.net.ssl.trustStorePassword_* > ** Truststore password > ** Default: None > ** E.g. _password_ > * *_javax.net.ssl.trustStoreType_* > ** Truststore type > ** Default: JKS > ** E.g. _pkcs12_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16705507#comment-16705507 ] Morio Ramdenbourg commented on HIVE-20992: -- Thanks for the feedback everyone. I'll keep the existing property deprecated, while having the new properties take precedence over it. [~vihangk1], it was mainly intended for consistency purposes, since there is already a property [hive.metastore.use.SSL|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java#L982-L983] on the HMS client to HMS Service side. My intent for it was to simply use it as a toggle for whether these new properties are set or not, similar to the logic [here|https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L9265-L9284]. It won't modify the JDO connectionURL - the _ssl=true_ part will still need to be inputted manually on the JDO connection string. > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we split _hive.metastore.dbaccess.ssl.properties_ into the > following properties: > * *_hive.metastore.dbaccess.ssl.use.SSL_* > ** Set this to true to use TLS encryption from the HMS Service to the HMSDB > * *_hive.metastore.dbaccess.ssl.truststore.path_* > ** TLS truststore file location > ** Java property: _javax.net.ssl.trustStore_ > ** E.g. _/tmp/truststore_ > * *_hive.metastore.dbaccess.ssl.truststore.password_* > ** Password of the truststore file > ** Java property: _javax.net.ssl.trustStorePassword_ > ** E.g. _pwd_ > * _*hive.metastore.dbaccess.ssl.truststore.type*_ > ** Type of the truststore file > ** Java property: _javax.net.ssl.trustStoreType_ > ** E.g. _JKS_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[
https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16705477#comment-16705477
]
Vihang Karajgaonkar commented on HIVE-20992:
Okay, I take that back. Looks like {{StringSetValidator}} does not apply to a
comma separated key=value pairs. So I think its okay having multiple
configurations since I see a similar pattern with other related configurations
(kerberos principal, keytab for example).
Does {{hive.metastore.dbaccess.ssl.use.SSL}} control the {{ssl=true}} part of
the JDO connectionURL?
> Split the config "hive.metastore.dbaccess.ssl.properties" into more
> meaningful configs
> --
>
> Key: HIVE-20992
> URL: https://issues.apache.org/jira/browse/HIVE-20992
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Security, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Minor
>
> HIVE-13044 brought in the ability to enable TLS encryption from the HMS
> Service to the HMSDB by configuring the following two properties:
> # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC
> metastore. To use SSL to encrypt/authenticate the connection, provide
> database-specific SSL flag in the connection URL. (E.g.
> "jdbc:postgresql://myhost/db?ssl=true")
> # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties
> for metastore to access database when JDO connection URL. (E.g.
> javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd)
> However, the latter configuration option is opaque and poses some problems.
> The most glaring of which is it takes in _any_
> [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html]
> system property, whether it is
> [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization]
> or not. This can cause some unintended side-effects for other components of
> the HMS, especially if it overrides an already-set system property. If the
> user truly wishes to add an unrelated Java property, setting it statically
> using the "-D" option of the _java_ command is more appropriate. Secondly,
> the truststore password is stored in plain text. We should add Hadoop Shims
> back to the HMS to prevent exposing these passwords, but this effort can be
> done after this ticket.
> I propose we split _hive.metastore.dbaccess.ssl.properties_ into the
> following properties:
> * *_hive.metastore.dbaccess.ssl.use.SSL_*
> ** Set this to true to use TLS encryption from the HMS Service to the HMSDB
> * *_hive.metastore.dbaccess.ssl.truststore.path_*
> ** TLS truststore file location
> ** Java property: _javax.net.ssl.trustStore_
> ** E.g. _/tmp/truststore_
> * *_hive.metastore.dbaccess.ssl.truststore.password_*
> ** Password of the truststore file
> ** Java property: _javax.net.ssl.trustStorePassword_
> ** E.g. _pwd_
> * _*hive.metastore.dbaccess.ssl.truststore.type*_
> ** Type of the truststore file
> ** Java property: _javax.net.ssl.trustStoreType_
> ** E.g. _JKS_
> We should guide the user towards an easier TLS configuration experience. This
> is the minimum configuration necessary to configure TLS to the HMSDB. If we
> need other options, such as the keystore location/password for
> dual-authentication, then we can add those on afterwards.
> Also, document these changes -
> [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL]
> does not have up-to-date documentation, and these new parameters will need
> documentation as well.
> Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[
https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16705469#comment-16705469
]
Vihang Karajgaonkar commented on HIVE-20992:
Removing old config would be backwards incompatible and break existing setups
which rely on that property. However, you can {{Deprecate}} it. The advantage
of having one property which takes in comma separated value v/s many individual
properties, is it is easy to configure. Setting many related configurations
with different names properties could be seen as tedious and easy to
misconfigure. Is the concern mainly because this configuration doesn't validate
the strings which are allowed? This can be easily fixed using the
{{StringSetValidator}} in the {{MetastoreConf}} where it is defined.
> Split the config "hive.metastore.dbaccess.ssl.properties" into more
> meaningful configs
> --
>
> Key: HIVE-20992
> URL: https://issues.apache.org/jira/browse/HIVE-20992
> Project: Hive
> Issue Type: Improvement
> Components: Metastore, Security, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Morio Ramdenbourg
>Assignee: Morio Ramdenbourg
>Priority: Minor
>
> HIVE-13044 brought in the ability to enable TLS encryption from the HMS
> Service to the HMSDB by configuring the following two properties:
> # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC
> metastore. To use SSL to encrypt/authenticate the connection, provide
> database-specific SSL flag in the connection URL. (E.g.
> "jdbc:postgresql://myhost/db?ssl=true")
> # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties
> for metastore to access database when JDO connection URL. (E.g.
> javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd)
> However, the latter configuration option is opaque and poses some problems.
> The most glaring of which is it takes in _any_
> [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html]
> system property, whether it is
> [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization]
> or not. This can cause some unintended side-effects for other components of
> the HMS, especially if it overrides an already-set system property. If the
> user truly wishes to add an unrelated Java property, setting it statically
> using the "-D" option of the _java_ command is more appropriate. Secondly,
> the truststore password is stored in plain text. We should add Hadoop Shims
> back to the HMS to prevent exposing these passwords, but this effort can be
> done after this ticket.
> I propose we split _hive.metastore.dbaccess.ssl.properties_ into the
> following properties:
> * *_hive.metastore.dbaccess.ssl.use.SSL_*
> ** Set this to true to use TLS encryption from the HMS Service to the HMSDB
> * *_hive.metastore.dbaccess.ssl.truststore.path_*
> ** TLS truststore file location
> ** Java property: _javax.net.ssl.trustStore_
> ** E.g. _/tmp/truststore_
> * *_hive.metastore.dbaccess.ssl.truststore.password_*
> ** Password of the truststore file
> ** Java property: _javax.net.ssl.trustStorePassword_
> ** E.g. _pwd_
> * _*hive.metastore.dbaccess.ssl.truststore.type*_
> ** Type of the truststore file
> ** Java property: _javax.net.ssl.trustStoreType_
> ** E.g. _JKS_
> We should guide the user towards an easier TLS configuration experience. This
> is the minimum configuration necessary to configure TLS to the HMSDB. If we
> need other options, such as the keystore location/password for
> dual-authentication, then we can add those on afterwards.
> Also, document these changes -
> [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL]
> does not have up-to-date documentation, and these new parameters will need
> documentation as well.
> Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16705437#comment-16705437 ] Mike Yoder commented on HIVE-20992: --- You'll need to keep the existing option, and use it if it exists, but deprecate it. Otherwise look straightforward to me. > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we split _hive.metastore.dbaccess.ssl.properties_ into the > following properties: > * *_hive.metastore.dbaccess.ssl.use.SSL_* > ** Set this to true to use TLS encryption from the HMS Service to the HMSDB > * *_hive.metastore.dbaccess.ssl.truststore.path_* > ** TLS truststore file location > ** Java property: _javax.net.ssl.trustStore_ > ** E.g. _/tmp/truststore_ > * *_hive.metastore.dbaccess.ssl.truststore.password_* > ** Password of the truststore file > ** Java property: _javax.net.ssl.trustStorePassword_ > ** E.g. _pwd_ > * _*hive.metastore.dbaccess.ssl.truststore.type*_ > ** Type of the truststore file > ** Java property: _javax.net.ssl.trustStoreType_ > ** E.g. _JKS_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20992) Split the config "hive.metastore.dbaccess.ssl.properties" into more meaningful configs
[ https://issues.apache.org/jira/browse/HIVE-20992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16705404#comment-16705404 ] Karthik Manamcheri commented on HIVE-20992: --- Would this break current use cases where they have some other custom SSL properties? Should we keep the existing ssl.properties as a deprecated option? Thoughts [~vihangk1] [~pvary] > Split the config "hive.metastore.dbaccess.ssl.properties" into more > meaningful configs > -- > > Key: HIVE-20992 > URL: https://issues.apache.org/jira/browse/HIVE-20992 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Morio Ramdenbourg >Assignee: Morio Ramdenbourg >Priority: Minor > Original Estimate: 168h > Remaining Estimate: 168h > > HIVE-13044 brought in the ability to enable TLS encryption from the HMS > Service to the HMSDB by configuring the following two properties: > # _javax.jdo.option.ConnectionURL_: JDBC connect string for a JDBC > metastore. To use SSL to encrypt/authenticate the connection, provide > database-specific SSL flag in the connection URL. (E.g. > "jdbc:postgresql://myhost/db?ssl=true") > # _hive.metastore.dbaccess.ssl.properties_: Comma-separated SSL properties > for metastore to access database when JDO connection URL. (E.g. > javax.net.ssl.trustStore=/tmp/truststore,javax.net.ssl.trustStorePassword=pwd) > However, the latter configuration option is opaque and poses some problems. > The most glaring of which is it takes in _any_ > [java.lang.System|https://docs.oracle.com/javase/7/docs/api/java/lang/System.html] > system property, whether it is > [TLS-related|https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization] > or not. This can cause some unintended side-effects for other components of > the HMS, especially if it overrides an already-set system property. If the > user truly wishes to add an unrelated Java property, setting it statically > using the "-D" option of the _java_ command is more appropriate. Secondly, > the truststore password is stored in plain text. We should add Hadoop Shims > back to the HMS to prevent exposing these passwords, but this effort can be > done after this ticket. > I propose we split _hive.metastore.dbaccess.ssl.properties_ into the > following properties: > * *_hive.metastore.dbaccess.ssl.use.SSL_* > ** Set this to true to use TLS encryption from the HMS Service to the HMSDB > * *_hive.metastore.dbaccess.ssl.truststore.path_* > ** TLS truststore file location > ** Java property: _javax.net.ssl.trustStore_ > ** E.g. _/tmp/truststore_ > * *_hive.metastore.dbaccess.ssl.truststore.password_* > ** Password of the truststore file > ** Java property: _javax.net.ssl.trustStorePassword_ > ** E.g. _pwd_ > * _*hive.metastore.dbaccess.ssl.truststore.type*_ > ** Type of the truststore file > ** Java property: _javax.net.ssl.trustStoreType_ > ** E.g. _JKS_ > We should guide the user towards an easier TLS configuration experience. This > is the minimum configuration necessary to configure TLS to the HMSDB. If we > need other options, such as the keystore location/password for > dual-authentication, then we can add those on afterwards. > Also, document these changes - > [javax.jdo.option.ConnectionURL|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-javax.jdo.option.ConnectionURL] > does not have up-to-date documentation, and these new parameters will need > documentation as well. > Note "TLS" refers to both SSL and TLS. TLS is simply the successor of SSL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
