[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425017#comment-15425017 ] ASF GitHub Bot commented on PHOENIX-3189: - GitHub user joshelser opened a pull request: https://github.com/apache/phoenix/pull/191 PHOENIX-3189 Perform Kerberos login before ConnectionInfo is constructed Now that ConnectionInfo has the current User/UGI stored inside, we must make sure that any automatic Kerberos login occurs before the ConnectionInfo object is constructed. Otherwise, we will have multiple instances of ConnectionInfo that differ only by the User, which will leak HBase/ZK connections in the connectionQueryServicesMap. You can merge this pull request into a Git repository by running: $ git pull https://github.com/joshelser/phoenix 3189-secure-cnxninfo Alternatively you can review and apply these changes as the patch at: https://github.com/apache/phoenix/pull/191.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #191 commit ac4184b9ec52ab261fc4d9f73edc6676eb421d79 Author: Josh Elser Date: 2016-08-17T17:34:59Z PHOENIX-3189 Perform Kerberos login before ConnectionInfo is constructed Now that ConnectionInfo has the current User/UGI stored inside, we must make sure that any automatic Kerberos login occurs before the ConnectionInfo object is constructed. Otherwise, we will have multiple instances of ConnectionInfo that differ only by the User, which will leak HBase/ZK connections in the connectionQueryServicesMap. > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15426835#comment-15426835 ] Josh Elser commented on PHOENIX-3189: - I've been testing out some changes using a little code snippet at https://github.com/joshelser/phoenix-test Essentially, the test alternates between creating a PhoenixConnection with one of two users, using the principal and keytab URL properties. Even with these changes I have staged, this still leaks connections because we have no good way to disambiguate multi UGI instances that refer to the same end-user. I need to think about how we can try to avoid users shooting themselves in the foot with this.. > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433847#comment-15433847 ] Josh Elser commented on PHOENIX-3189: - Alright, just pushed an update to the pull request ([linked|https://github.com/apache/phoenix/pull/191]) which has some tests to try to catch a regression like this in the future. I wasted a bunch of time trying to reproduce this end to end, and ultimately gave up. Instead, I'm testing functionality by putting ConnectionInfo objects into a HashSet which should mimic what the ConcurrentHashMap is doing in PhoenixDriver. Feedback welcome. > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437732#comment-15437732 ] Josh Elser commented on PHOENIX-3189: - Hey [~devaraj], do you have a moment to look at the pull request attached here? Maybe [~jamestaylor] too? > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437844#comment-15437844 ] ASF GitHub Bot commented on PHOENIX-3189: - Github user JamesRTaylor commented on a diff in the pull request: https://github.com/apache/phoenix/pull/191#discussion_r76336767 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixEmbeddedDriver.java --- @@ -333,6 +337,55 @@ public ConnectionInfo normalize(ReadOnlyProps props) throws SQLException { keytab = props.get(QueryServices.HBASE_CLIENT_KEYTAB); } } +if (!isConnectionless()) { +boolean credsProvidedInUrl = null != principal && null != keytab; +boolean credsProvidedInProps = info.containsKey(QueryServices.HBASE_CLIENT_PRINCIPAL) && info.containsKey(QueryServices.HBASE_CLIENT_KEYTAB); +if (credsProvidedInUrl || credsProvidedInProps) { +// PHOENIX-3189 Because ConnectionInfo is immutable, we must make sure all parts of it are correct before +// construction; this also requires the Kerberos user credentials object (since they are compared by reference +// and not by value. If the user provided a principal and keytab via the JDBC url, we must make sure that the +// Kerberos login happens *before* we construct the ConnectionInfo object. Otherwise, the use of ConnectionInfo +// to determine when ConnectionQueryServices impl's should be reused will be broken. +Configuration config = HBaseFactoryProvider.getConfigurationFactory().getConfiguration(); +// Add QueryServices properties +for (Entry entry : props) { +config.set(entry.getKey(), entry.getValue()); +} +// Add any user-provided properties (via DriverManager) +if (info != null) { +for (Object key : info.keySet()) { +config.set((String) key, info.getProperty((String) key)); +} +} +// Set the principal and keytab if provided from the URL (overriding those provided in Properties) +if (null != principal) { +config.set(QueryServices.HBASE_CLIENT_PRINCIPAL, principal); +} +if (null != keytab) { +config.set(QueryServices.HBASE_CLIENT_KEYTAB, keytab); +} +try { +// Check if we need to authenticate with kerberos so that we cache the correct ConnectionInfo +UserGroupInformation currentUser = UserGroupInformation.getCurrentUser(); +if (!currentUser.hasKerberosCredentials() || !currentUser.getUserName().equals(principal)) { --- End diff -- Looks like we only use config inside of this block. Can we create it here only instead? > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437852#comment-15437852 ] ASF GitHub Bot commented on PHOENIX-3189: - Github user JamesRTaylor commented on the issue: https://github.com/apache/phoenix/pull/191 Would an alternative be to not rely on the User.equals method, but instead modify ConnectionInfo.equals to do some other kind of equality check on User? > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439778#comment-15439778 ] ASF GitHub Bot commented on PHOENIX-3189: - Github user joshelser commented on the issue: https://github.com/apache/phoenix/pull/191 > Would an alternative be to not rely on the User.equals method, but instead modify ConnectionInfo.equals to do some other kind of equality check on User? I think I initially had this idea too (wondering why User doesn't implement a "normal" value-based equals/hashCode implementation). Devaraj didn't remember exactly why, but he recalled that there was a reason that UGI (and thus User also) did this. I could try to dig into the Hadoop archives to see if I can understand why this was done. Mostly, I think this approach keeps us in line with what HDFS and HBase does now which has some value. > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439795#comment-15439795 ] ASF GitHub Bot commented on PHOENIX-3189: - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/191#discussion_r76484922 --- Diff: phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixEmbeddedDriver.java --- @@ -333,6 +337,55 @@ public ConnectionInfo normalize(ReadOnlyProps props) throws SQLException { keytab = props.get(QueryServices.HBASE_CLIENT_KEYTAB); } } +if (!isConnectionless()) { +boolean credsProvidedInUrl = null != principal && null != keytab; +boolean credsProvidedInProps = info.containsKey(QueryServices.HBASE_CLIENT_PRINCIPAL) && info.containsKey(QueryServices.HBASE_CLIENT_KEYTAB); +if (credsProvidedInUrl || credsProvidedInProps) { +// PHOENIX-3189 Because ConnectionInfo is immutable, we must make sure all parts of it are correct before +// construction; this also requires the Kerberos user credentials object (since they are compared by reference +// and not by value. If the user provided a principal and keytab via the JDBC url, we must make sure that the +// Kerberos login happens *before* we construct the ConnectionInfo object. Otherwise, the use of ConnectionInfo +// to determine when ConnectionQueryServices impl's should be reused will be broken. +Configuration config = HBaseFactoryProvider.getConfigurationFactory().getConfiguration(); +// Add QueryServices properties +for (Entry entry : props) { +config.set(entry.getKey(), entry.getValue()); +} +// Add any user-provided properties (via DriverManager) +if (info != null) { +for (Object key : info.keySet()) { +config.set((String) key, info.getProperty((String) key)); +} +} +// Set the principal and keytab if provided from the URL (overriding those provided in Properties) +if (null != principal) { +config.set(QueryServices.HBASE_CLIENT_PRINCIPAL, principal); +} +if (null != keytab) { +config.set(QueryServices.HBASE_CLIENT_KEYTAB, keytab); +} +try { +// Check if we need to authenticate with kerberos so that we cache the correct ConnectionInfo +UserGroupInformation currentUser = UserGroupInformation.getCurrentUser(); +if (!currentUser.hasKerberosCredentials() || !currentUser.getUserName().equals(principal)) { --- End diff -- Yes, good catch. Can definitely do this only when necessary. > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439832#comment-15439832 ] ASF GitHub Bot commented on PHOENIX-3189: - Github user joshelser commented on the issue: https://github.com/apache/phoenix/pull/191 Just pushed d17a8d8 to address the issue you mentioned, @JamesRTaylor > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15446848#comment-15446848 ] ASF GitHub Bot commented on PHOENIX-3189: - Github user JamesRTaylor commented on the issue: https://github.com/apache/phoenix/pull/191 Thanks for pursuing this tricky issue, @joshelser. I think what you have here is definitely an improvement and should be pulled in for 4.8.1, but I do think we should look at doing a value-based equality check for User instead. There's a fair amount of overhead in what you're doing and Phoenix does not do connection pooling but relies on being able to quickly/cheaply get a connection. Would you have some cycles to investigate that a bit first? > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15446895#comment-15446895 ] ASF GitHub Bot commented on PHOENIX-3189: - Github user dbahir commented on the issue: https://github.com/apache/phoenix/pull/191 This solution is not thread safe and will not allow to safely create multiple instances of a driver on different threads in the JVM. This area should be protected, https://github.com/joshelser/phoenix/blob/d17a8d855dc4a2c8cff578dd26e14c6c2c13cc3a/phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixEmbeddedDriver.java#L351. With that said I am not sure that you can support multiple users and support renewals with the way the UGI works. If in the same JVM a driver is instantiated for User A and then another driver is instantiated for User B the last call to loginUserFromKeytab will set the the user information in the UGI. loginUserFromKeytabAndReturnUGI can be used which will preserve the original user info in the UGI but I think will not work correctly with renewing. Do we want the Phoenix driver to allow multiple instances instantiated with a different logged in user for each in the same JVM ? > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447291#comment-15447291 ] ASF GitHub Bot commented on PHOENIX-3189: - Github user joshelser commented on the issue: https://github.com/apache/phoenix/pull/191 > This solution is not thread safe and will not allow to safely create multiple instances of a driver on different threads in the JVM. Yes, that's why I directed you over here, bud. That wasn't an initial goal of these changes. > With that said I am not sure that you can support multiple users and support renewals with the way the UGI works. Right.. you're catching on to what I was pointing out. This is something that you should be managing inside of Storm. We cannot do this effectively inside of Phoenix. We can only put a bandaid on top. > Do we want the Phoenix driver to allow multiple instances instantiated with a different logged in user for each in the same JVM ? The only change I think we can do here is to prevent multiple clients from doing what you're suggesting and hope they don't shoot themselves in the foot. > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447350#comment-15447350 ] ASF GitHub Bot commented on PHOENIX-3189: - Github user joshelser commented on the issue: https://github.com/apache/phoenix/pull/191 @dbahir I believe b5be8d8 would address your concerns. WDYT? > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15451068#comment-15451068 ] ASF GitHub Bot commented on PHOENIX-3189: - Github user joshelser commented on the issue: https://github.com/apache/phoenix/pull/191 > Did you get to look at the comment regarding the user login ? Assuming you mean the below.. > With that said I am not sure that you can support multiple users and support renewals with the way the UGI works. > If in the same JVM a driver is instantiated for User A and then another driver is instantiated for User B the last call to loginUserFromKeytab will set the the user information in the UGI. Yes, the best way to approach this is that Phoenix's automatic Kerberos login should _only_ be used when there a single user accessing Phoenix at one time. I was thinking that Phoenix could likely use its own docs page on the interactions with Kerberos. That would be a good place to warn people and instruct them how they need to properly support multiple concurrent users on their own. > Can you allow another to login with a different principal? Would that cause a security issue? If we create one driver(One) with user A and then create another driver(Two) with user B the info in the UGI now is that of user B. So there can be a situation where driver One will be using credentials of user B. My thinking is this (but I should probably test it): says userA uses phoenix, then userB starts using Phoenix. This implies that userA would be logged out and I would hope that subsequent interactions with Phoenix by the "human" userA would fail because its credentials are no longer there (the RPCs themselves would fail). In other words, my thinking is that the RPC authentication would protect us in Phoenix from this being an issue. > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15451082#comment-15451082 ] ASF GitHub Bot commented on PHOENIX-3189: - Github user joshelser commented on the issue: https://github.com/apache/phoenix/pull/191 > Thanks for pursuing this tricky issue, @joshelser. I think what you have here is definitely an improvement and should be pulled in for 4.8.1 Thanks as always for your careful eye. > but I do think we should look at doing a value-based equality check for User instead. There's a fair amount of overhead in what you're doing and Phoenix does not do connection pooling but relies on being able to quickly/cheaply get a connection. Would you have some cycles to investigate that a bit first? To my point in the last comment, I believe it's become apparent that people are misusing this functionality. So, I believe there is some education that needs to happen about how clients of Phoenix who need to support multiple Kerberos users concurrently should interact with the system. I think this documentation is point 1. It flat-out shouldn't be used to concurrently support multiple Kerberos users -- developers must write their own code to do this correctly. It's difficult to handle this for them because of the required doAs() call. Can investigate this further too, maybe I'm overlooking something obvious. I do need to make good on my promise to dig through Hadoop/UGI git-log and try to piece together why exactly UGI isn't doing value-based checks already. This is point 2. I'd like to move ahead and push then (rather then let it rot on the vine), but will try to address both of these points this week if that's OK with you. > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453179#comment-15453179 ] ASF GitHub Bot commented on PHOENIX-3189: - Github user joshelser commented on the issue: https://github.com/apache/phoenix/pull/191 > My thinking is this (but I should probably test it): says userA uses phoenix, then userB starts using Phoenix. This implies that userA would be logged out and I would hope that subsequent interactions with Phoenix by the "human" userA would fail because its credentials are no longer there (the RPCs themselves would fail). In other words, my thinking is that the RPC authentication would protect us in Phoenix from this being an issue. No dice. This is still a hole (although not a new hole as I understand it). Will track this separately. > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453297#comment-15453297 ] Josh Elser commented on PHOENIX-3189: - PHOENIX-3232 to track this issue. > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453905#comment-15453905 ] ASF GitHub Bot commented on PHOENIX-3189: - Github user asfgit closed the pull request at: https://github.com/apache/phoenix/pull/191 > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453928#comment-15453928 ] Hudson commented on PHOENIX-3189: - FAILURE: Integrated in Jenkins build Phoenix-4.8-HBase-1.2 #3 (See [https://builds.apache.org/job/Phoenix-4.8-HBase-1.2/3/]) PHOENIX-3189 Perform Kerberos login before ConnectionInfo is constructed (elserj: rev 6b26389a4416fa2ac7157f6468a1bf223c938024) * (edit) pom.xml * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/util/InstanceResolver.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixEmbeddedDriver.java * (add) phoenix-core/src/test/java/org/apache/phoenix/jdbc/SecureUserConnectionsTest.java * (edit) phoenix-core/src/test/resources/log4j.properties * (edit) phoenix-core/pom.xml * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixDriver.java > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453938#comment-15453938 ] Hudson commented on PHOENIX-3189: - FAILURE: Integrated in Jenkins build Phoenix-master #1377 (See [https://builds.apache.org/job/Phoenix-master/1377/]) PHOENIX-3189 Perform Kerberos login before ConnectionInfo is constructed (elserj: rev 2dc6be4e9ffb205aaa5b0f6a1f2746eed8d23426) * (add) phoenix-core/src/test/java/org/apache/phoenix/jdbc/SecureUserConnectionsTest.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixDriver.java * (edit) phoenix-core/pom.xml * (edit) phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java * (edit) phoenix-core/src/test/resources/log4j.properties * (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixEmbeddedDriver.java * (edit) phoenix-core/src/main/java/org/apache/phoenix/util/InstanceResolver.java * (edit) pom.xml > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454029#comment-15454029 ] Josh Elser commented on PHOENIX-3189: - Alright, something to do with the local /etc/krb5.conf I have on my machine. Digging in... > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454057#comment-15454057 ] Josh Elser commented on PHOENIX-3189: - Ok, {{o.a.h.s.a.KerberosName#defaultRealm}} is somehow getting cached before the testing KDC is started. Just need to make sure this is updated correctly before this test runs. > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454093#comment-15454093 ] Josh Elser commented on PHOENIX-3189: - Pushed a fix to make sure that {{KerberosName#defaultRealm}} is accurate after the KDC is set up. Should fix the issue, but will wait for Jenkins to confirm. > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454241#comment-15454241 ] Hudson commented on PHOENIX-3189: - FAILURE: Integrated in Jenkins build Phoenix-master #1378 (See [https://builds.apache.org/job/Phoenix-master/1378/]) PHOENIX-3189 Ensure that there is no invalid cached state in (elserj: rev 1ce9845b8f6c2b456fbab913cd1bd647c6c4d281) * (edit) phoenix-core/src/test/java/org/apache/phoenix/jdbc/SecureUserConnectionsTest.java > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[ https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454279#comment-15454279 ] Hudson commented on PHOENIX-3189: - SUCCESS: Integrated in Jenkins build Phoenix-4.8-HBase-1.2 #4 (See [https://builds.apache.org/job/Phoenix-4.8-HBase-1.2/4/]) PHOENIX-3189 Ensure that there is no invalid cached state in (elserj: rev 1c448ccde7a3f2b370cd0c386e2a2dd425caaeb0) * (edit) phoenix-core/src/test/java/org/apache/phoenix/jdbc/SecureUserConnectionsTest.java > HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url > > > Key: PHOENIX-3189 > URL: https://issues.apache.org/jira/browse/PHOENIX-3189 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Blocker > Fix For: 4.9.0, 4.8.1 > > > We've been doing some more testing after PHOENIX-3126 and, with the help of > [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm > and Phoenix. > Storm was configured to create a JDBC Bolt, specifying the principal and > keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for > them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, > and we observed that there were over 140 active ZK threads in the JVM. > This results in a subtle change where every time the client tries to get a > new Connection, we end up getting a new UGI instance (because the > {{ConnectionQueryServicesImpl#openConnection()}} always does a new login). > If users are correctly caching Connections, there isn't an issue (best as I > can presently tell). However, if users rely on the getting the same > connection every time (the pre-PHOENIX-3126), they will saturate their local > JVM with connections and crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)