[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425017#comment-15425017
]
ASF GitHub Bot commented on PHOENIX-3189:
-
GitHub user joshelser opened a pull request:
https://github.com/apache/phoenix/pull/191
PHOENIX-3189 Perform Kerberos login before ConnectionInfo is constructed
Now that ConnectionInfo has the current User/UGI stored inside, we must
make sure that any automatic Kerberos login occurs before the ConnectionInfo
object is constructed. Otherwise, we will have multiple instances of
ConnectionInfo that differ only by the User, which will leak HBase/ZK
connections in the connectionQueryServicesMap.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/joshelser/phoenix 3189-secure-cnxninfo
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/phoenix/pull/191.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #191
commit ac4184b9ec52ab261fc4d9f73edc6676eb421d79
Author: Josh Elser
Date: 2016-08-17T17:34:59Z
PHOENIX-3189 Perform Kerberos login before ConnectionInfo is constructed
Now that ConnectionInfo has the current User/UGI stored inside, we must
make sure that any automatic Kerberos login occurs before the ConnectionInfo
object is constructed. Otherwise, we will have multiple instances of
ConnectionInfo that differ only by the User, which will leak HBase/ZK
connections in the connectionQueryServicesMap.
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15426835#comment-15426835
]
Josh Elser commented on PHOENIX-3189:
-
I've been testing out some changes using a little code snippet at
https://github.com/joshelser/phoenix-test
Essentially, the test alternates between creating a PhoenixConnection with one
of two users, using the principal and keytab URL properties. Even with these
changes I have staged, this still leaks connections because we have no good way
to disambiguate multi UGI instances that refer to the same end-user.
I need to think about how we can try to avoid users shooting themselves in the
foot with this..
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433847#comment-15433847
]
Josh Elser commented on PHOENIX-3189:
-
Alright, just pushed an update to the pull request
([linked|https://github.com/apache/phoenix/pull/191]) which has some tests to
try to catch a regression like this in the future.
I wasted a bunch of time trying to reproduce this end to end, and ultimately
gave up. Instead, I'm testing functionality by putting ConnectionInfo objects
into a HashSet which should mimic what the ConcurrentHashMap is doing in
PhoenixDriver.
Feedback welcome.
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437732#comment-15437732
]
Josh Elser commented on PHOENIX-3189:
-
Hey [~devaraj], do you have a moment to look at the pull request attached here?
Maybe [~jamestaylor] too?
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437844#comment-15437844
]
ASF GitHub Bot commented on PHOENIX-3189:
-
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/191#discussion_r76336767
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixEmbeddedDriver.java
---
@@ -333,6 +337,55 @@ public ConnectionInfo normalize(ReadOnlyProps props)
throws SQLException {
keytab =
props.get(QueryServices.HBASE_CLIENT_KEYTAB);
}
}
+if (!isConnectionless()) {
+boolean credsProvidedInUrl = null != principal && null !=
keytab;
+boolean credsProvidedInProps =
info.containsKey(QueryServices.HBASE_CLIENT_PRINCIPAL) &&
info.containsKey(QueryServices.HBASE_CLIENT_KEYTAB);
+if (credsProvidedInUrl || credsProvidedInProps) {
+// PHOENIX-3189 Because ConnectionInfo is immutable,
we must make sure all parts of it are correct before
+// construction; this also requires the Kerberos user
credentials object (since they are compared by reference
+// and not by value. If the user provided a principal
and keytab via the JDBC url, we must make sure that the
+// Kerberos login happens *before* we construct the
ConnectionInfo object. Otherwise, the use of ConnectionInfo
+// to determine when ConnectionQueryServices impl's
should be reused will be broken.
+Configuration config =
HBaseFactoryProvider.getConfigurationFactory().getConfiguration();
+// Add QueryServices properties
+for (Entry entry : props) {
+config.set(entry.getKey(), entry.getValue());
+}
+// Add any user-provided properties (via DriverManager)
+if (info != null) {
+for (Object key : info.keySet()) {
+config.set((String) key,
info.getProperty((String) key));
+}
+}
+// Set the principal and keytab if provided from the
URL (overriding those provided in Properties)
+if (null != principal) {
+config.set(QueryServices.HBASE_CLIENT_PRINCIPAL,
principal);
+}
+if (null != keytab) {
+config.set(QueryServices.HBASE_CLIENT_KEYTAB,
keytab);
+}
+try {
+// Check if we need to authenticate with kerberos
so that we cache the correct ConnectionInfo
+UserGroupInformation currentUser =
UserGroupInformation.getCurrentUser();
+if (!currentUser.hasKerberosCredentials() ||
!currentUser.getUserName().equals(principal)) {
--- End diff --
Looks like we only use config inside of this block. Can we create it here
only instead?
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437852#comment-15437852
]
ASF GitHub Bot commented on PHOENIX-3189:
-
Github user JamesRTaylor commented on the issue:
https://github.com/apache/phoenix/pull/191
Would an alternative be to not rely on the User.equals method, but instead
modify ConnectionInfo.equals to do some other kind of equality check on User?
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439778#comment-15439778
]
ASF GitHub Bot commented on PHOENIX-3189:
-
Github user joshelser commented on the issue:
https://github.com/apache/phoenix/pull/191
> Would an alternative be to not rely on the User.equals method, but
instead modify ConnectionInfo.equals to do some other kind of equality check on
User?
I think I initially had this idea too (wondering why User doesn't implement
a "normal" value-based equals/hashCode implementation). Devaraj didn't remember
exactly why, but he recalled that there was a reason that UGI (and thus User
also) did this. I could try to dig into the Hadoop archives to see if I can
understand why this was done.
Mostly, I think this approach keeps us in line with what HDFS and HBase
does now which has some value.
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439795#comment-15439795
]
ASF GitHub Bot commented on PHOENIX-3189:
-
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/191#discussion_r76484922
--- Diff:
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixEmbeddedDriver.java
---
@@ -333,6 +337,55 @@ public ConnectionInfo normalize(ReadOnlyProps props)
throws SQLException {
keytab =
props.get(QueryServices.HBASE_CLIENT_KEYTAB);
}
}
+if (!isConnectionless()) {
+boolean credsProvidedInUrl = null != principal && null !=
keytab;
+boolean credsProvidedInProps =
info.containsKey(QueryServices.HBASE_CLIENT_PRINCIPAL) &&
info.containsKey(QueryServices.HBASE_CLIENT_KEYTAB);
+if (credsProvidedInUrl || credsProvidedInProps) {
+// PHOENIX-3189 Because ConnectionInfo is immutable,
we must make sure all parts of it are correct before
+// construction; this also requires the Kerberos user
credentials object (since they are compared by reference
+// and not by value. If the user provided a principal
and keytab via the JDBC url, we must make sure that the
+// Kerberos login happens *before* we construct the
ConnectionInfo object. Otherwise, the use of ConnectionInfo
+// to determine when ConnectionQueryServices impl's
should be reused will be broken.
+Configuration config =
HBaseFactoryProvider.getConfigurationFactory().getConfiguration();
+// Add QueryServices properties
+for (Entry entry : props) {
+config.set(entry.getKey(), entry.getValue());
+}
+// Add any user-provided properties (via DriverManager)
+if (info != null) {
+for (Object key : info.keySet()) {
+config.set((String) key,
info.getProperty((String) key));
+}
+}
+// Set the principal and keytab if provided from the
URL (overriding those provided in Properties)
+if (null != principal) {
+config.set(QueryServices.HBASE_CLIENT_PRINCIPAL,
principal);
+}
+if (null != keytab) {
+config.set(QueryServices.HBASE_CLIENT_KEYTAB,
keytab);
+}
+try {
+// Check if we need to authenticate with kerberos
so that we cache the correct ConnectionInfo
+UserGroupInformation currentUser =
UserGroupInformation.getCurrentUser();
+if (!currentUser.hasKerberosCredentials() ||
!currentUser.getUserName().equals(principal)) {
--- End diff --
Yes, good catch. Can definitely do this only when necessary.
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439832#comment-15439832
]
ASF GitHub Bot commented on PHOENIX-3189:
-
Github user joshelser commented on the issue:
https://github.com/apache/phoenix/pull/191
Just pushed d17a8d8 to address the issue you mentioned, @JamesRTaylor
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15446848#comment-15446848
]
ASF GitHub Bot commented on PHOENIX-3189:
-
Github user JamesRTaylor commented on the issue:
https://github.com/apache/phoenix/pull/191
Thanks for pursuing this tricky issue, @joshelser. I think what you have
here is definitely an improvement and should be pulled in for 4.8.1, but I do
think we should look at doing a value-based equality check for User instead.
There's a fair amount of overhead in what you're doing and Phoenix does not do
connection pooling but relies on being able to quickly/cheaply get a
connection. Would you have some cycles to investigate that a bit first?
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15446895#comment-15446895
]
ASF GitHub Bot commented on PHOENIX-3189:
-
Github user dbahir commented on the issue:
https://github.com/apache/phoenix/pull/191
This solution is not thread safe and will not allow to safely create
multiple instances of a driver on different threads in the JVM.
This area should be protected,
https://github.com/joshelser/phoenix/blob/d17a8d855dc4a2c8cff578dd26e14c6c2c13cc3a/phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixEmbeddedDriver.java#L351.
With that said I am not sure that you can support multiple users and
support renewals with the way the UGI works.
If in the same JVM a driver is instantiated for User A and then another
driver is instantiated for User B the last call to loginUserFromKeytab will set
the the user information in the UGI.
loginUserFromKeytabAndReturnUGI can be used which will preserve the
original user info in the UGI but I think will not work correctly with renewing.
Do we want the Phoenix driver to allow multiple instances instantiated with
a different logged in user for each in the same JVM ?
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447291#comment-15447291
]
ASF GitHub Bot commented on PHOENIX-3189:
-
Github user joshelser commented on the issue:
https://github.com/apache/phoenix/pull/191
> This solution is not thread safe and will not allow to safely create
multiple instances of a driver on different threads in the JVM.
Yes, that's why I directed you over here, bud. That wasn't an initial goal
of these changes.
> With that said I am not sure that you can support multiple users and
support renewals with the way the UGI works.
Right.. you're catching on to what I was pointing out. This is something
that you should be managing inside of Storm. We cannot do this effectively
inside of Phoenix. We can only put a bandaid on top.
> Do we want the Phoenix driver to allow multiple instances instantiated
with a different logged in user for each in the same JVM ?
The only change I think we can do here is to prevent multiple clients from
doing what you're suggesting and hope they don't shoot themselves in the foot.
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447350#comment-15447350
]
ASF GitHub Bot commented on PHOENIX-3189:
-
Github user joshelser commented on the issue:
https://github.com/apache/phoenix/pull/191
@dbahir I believe b5be8d8 would address your concerns. WDYT?
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15451068#comment-15451068
]
ASF GitHub Bot commented on PHOENIX-3189:
-
Github user joshelser commented on the issue:
https://github.com/apache/phoenix/pull/191
> Did you get to look at the comment regarding the user login ?
Assuming you mean the below..
> With that said I am not sure that you can support multiple users and
support renewals with the way the UGI works.
> If in the same JVM a driver is instantiated for User A and then another
driver is instantiated for User B the last call to loginUserFromKeytab will set
the the user information in the UGI.
Yes, the best way to approach this is that Phoenix's automatic Kerberos
login should _only_ be used when there a single user accessing Phoenix at one
time. I was thinking that Phoenix could likely use its own docs page on the
interactions with Kerberos. That would be a good place to warn people and
instruct them how they need to properly support multiple concurrent users on
their own.
> Can you allow another to login with a different principal? Would that
cause a security issue?
If we create one driver(One) with user A and then create another
driver(Two) with user B the info in the UGI now is that of user B. So there can
be a situation where driver One will be using credentials of user B.
My thinking is this (but I should probably test it): says userA uses
phoenix, then userB starts using Phoenix. This implies that userA would be
logged out and I would hope that subsequent interactions with Phoenix by the
"human" userA would fail because its credentials are no longer there (the RPCs
themselves would fail). In other words, my thinking is that the RPC
authentication would protect us in Phoenix from this being an issue.
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15451082#comment-15451082
]
ASF GitHub Bot commented on PHOENIX-3189:
-
Github user joshelser commented on the issue:
https://github.com/apache/phoenix/pull/191
> Thanks for pursuing this tricky issue, @joshelser. I think what you have
here is definitely an improvement and should be pulled in for 4.8.1
Thanks as always for your careful eye.
> but I do think we should look at doing a value-based equality check for
User instead. There's a fair amount of overhead in what you're doing and
Phoenix does not do connection pooling but relies on being able to
quickly/cheaply get a connection. Would you have some cycles to investigate
that a bit first?
To my point in the last comment, I believe it's become apparent that people
are misusing this functionality. So, I believe there is some education that
needs to happen about how clients of Phoenix who need to support multiple
Kerberos users concurrently should interact with the system. I think this
documentation is point 1. It flat-out shouldn't be used to concurrently support
multiple Kerberos users -- developers must write their own code to do this
correctly. It's difficult to handle this for them because of the required
doAs() call. Can investigate this further too, maybe I'm overlooking something
obvious.
I do need to make good on my promise to dig through Hadoop/UGI git-log and
try to piece together why exactly UGI isn't doing value-based checks already.
This is point 2.
I'd like to move ahead and push then (rather then let it rot on the vine),
but will try to address both of these points this week if that's OK with you.
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453179#comment-15453179
]
ASF GitHub Bot commented on PHOENIX-3189:
-
Github user joshelser commented on the issue:
https://github.com/apache/phoenix/pull/191
> My thinking is this (but I should probably test it): says userA uses
phoenix, then userB starts using Phoenix. This implies that userA would be
logged out and I would hope that subsequent interactions with Phoenix by the
"human" userA would fail because its credentials are no longer there (the RPCs
themselves would fail). In other words, my thinking is that the RPC
authentication would protect us in Phoenix from this being an issue.
No dice. This is still a hole (although not a new hole as I understand it).
Will track this separately.
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453297#comment-15453297
]
Josh Elser commented on PHOENIX-3189:
-
PHOENIX-3232 to track this issue.
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453905#comment-15453905
]
ASF GitHub Bot commented on PHOENIX-3189:
-
Github user asfgit closed the pull request at:
https://github.com/apache/phoenix/pull/191
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453928#comment-15453928
]
Hudson commented on PHOENIX-3189:
-
FAILURE: Integrated in Jenkins build Phoenix-4.8-HBase-1.2 #3 (See
[https://builds.apache.org/job/Phoenix-4.8-HBase-1.2/3/])
PHOENIX-3189 Perform Kerberos login before ConnectionInfo is constructed
(elserj: rev 6b26389a4416fa2ac7157f6468a1bf223c938024)
* (edit) pom.xml
* (edit)
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
* (edit)
phoenix-core/src/main/java/org/apache/phoenix/util/InstanceResolver.java
* (edit)
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixEmbeddedDriver.java
* (add)
phoenix-core/src/test/java/org/apache/phoenix/jdbc/SecureUserConnectionsTest.java
* (edit) phoenix-core/src/test/resources/log4j.properties
* (edit) phoenix-core/pom.xml
* (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixDriver.java
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453938#comment-15453938
]
Hudson commented on PHOENIX-3189:
-
FAILURE: Integrated in Jenkins build Phoenix-master #1377 (See
[https://builds.apache.org/job/Phoenix-master/1377/])
PHOENIX-3189 Perform Kerberos login before ConnectionInfo is constructed
(elserj: rev 2dc6be4e9ffb205aaa5b0f6a1f2746eed8d23426)
* (add)
phoenix-core/src/test/java/org/apache/phoenix/jdbc/SecureUserConnectionsTest.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixDriver.java
* (edit) phoenix-core/pom.xml
* (edit)
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
* (edit) phoenix-core/src/test/resources/log4j.properties
* (edit)
phoenix-core/src/main/java/org/apache/phoenix/jdbc/PhoenixEmbeddedDriver.java
* (edit)
phoenix-core/src/main/java/org/apache/phoenix/util/InstanceResolver.java
* (edit) pom.xml
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454029#comment-15454029
]
Josh Elser commented on PHOENIX-3189:
-
Alright, something to do with the local /etc/krb5.conf I have on my machine.
Digging in...
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454057#comment-15454057
]
Josh Elser commented on PHOENIX-3189:
-
Ok, {{o.a.h.s.a.KerberosName#defaultRealm}} is somehow getting cached before
the testing KDC is started. Just need to make sure this is updated correctly
before this test runs.
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454093#comment-15454093
]
Josh Elser commented on PHOENIX-3189:
-
Pushed a fix to make sure that {{KerberosName#defaultRealm}} is accurate after
the KDC is set up. Should fix the issue, but will wait for Jenkins to confirm.
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454241#comment-15454241
]
Hudson commented on PHOENIX-3189:
-
FAILURE: Integrated in Jenkins build Phoenix-master #1378 (See
[https://builds.apache.org/job/Phoenix-master/1378/])
PHOENIX-3189 Ensure that there is no invalid cached state in (elserj: rev
1ce9845b8f6c2b456fbab913cd1bd647c6c4d281)
* (edit)
phoenix-core/src/test/java/org/apache/phoenix/jdbc/SecureUserConnectionsTest.java
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3189) HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
[
https://issues.apache.org/jira/browse/PHOENIX-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454279#comment-15454279
]
Hudson commented on PHOENIX-3189:
-
SUCCESS: Integrated in Jenkins build Phoenix-4.8-HBase-1.2 #4 (See
[https://builds.apache.org/job/Phoenix-4.8-HBase-1.2/4/])
PHOENIX-3189 Ensure that there is no invalid cached state in (elserj: rev
1c448ccde7a3f2b370cd0c386e2a2dd425caaeb0)
* (edit)
phoenix-core/src/test/java/org/apache/phoenix/jdbc/SecureUserConnectionsTest.java
> HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url
>
>
> Key: PHOENIX-3189
> URL: https://issues.apache.org/jira/browse/PHOENIX-3189
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 4.9.0, 4.8.1
>
>
> We've been doing some more testing after PHOENIX-3126 and, with the help of
> [~arpitgupta] and [~harsha_ch], we've found an issue in a test between Storm
> and Phoenix.
> Storm was configured to create a JDBC Bolt, specifying the principal and
> keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for
> them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt,
> and we observed that there were over 140 active ZK threads in the JVM.
> This results in a subtle change where every time the client tries to get a
> new Connection, we end up getting a new UGI instance (because the
> {{ConnectionQueryServicesImpl#openConnection()}} always does a new login).
> If users are correctly caching Connections, there isn't an issue (best as I
> can presently tell). However, if users rely on the getting the same
> connection every time (the pre-PHOENIX-3126), they will saturate their local
> JVM with connections and crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
