[jira] [Commented] (HBASE-18243) HBase Thrift server lacks logic for renewing kerberos tickets
[
https://issues.apache.org/jira/browse/HBASE-18243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066014#comment-16066014
]
Steen Manniche commented on HBASE-18243:
Starting the thrift server with a {{jaas}} configuration file which has
instructions to not use a ticket cache ({{useTicketCache=false}}, but instead
use a keytab ({{useKeyTab=true}} and values for {{keyTab}} and {{principal}})
seems to have solved the problem for us. However, I do not see any log-files
from the {{UserGroupInformation#reloginFromTicketCache}}, which allegedly
should be handling the kerberos ticket renewal for the thrift server?
> HBase Thrift server lacks logic for renewing kerberos tickets
> -
>
> Key: HBASE-18243
> URL: https://issues.apache.org/jira/browse/HBASE-18243
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Affects Versions: 2.0.0, 1.1.2
>Reporter: Steen Manniche
>Priority: Minor
> Labels: security
>
> I have been looking through the hbase-thrift code looking for where
> the server performs renewals of kerberos tickets for the provided
> principal/keytab. There seems to be no logic in place for renewing tickets.
> The hadoop-common provides the class
> UserGroupInformation, which exposes the method
> {{checkTGTAndReloginFromKeytab}}. I can see that the {{ThriftServerRunner}}
> class
> has a handle to the class
> (https://github.com/apache/hbase/blob/master/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java#L205),
> but I do not see the ticket renewal logic being called anywhere.
> A possible workaround is to renew the ticket outside the java process.
> The documentation on the {{checkTGTAndReloginFromKeytab}} states that if the
> ticket is still valid, a call to the method is essentially a no-op.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-18243) HBase Thrift server lacks logic for renewing kerberos tickets
Steen Manniche created HBASE-18243:
--
Summary: HBase Thrift server lacks logic for renewing kerberos
tickets
Key: HBASE-18243
URL: https://issues.apache.org/jira/browse/HBASE-18243
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 1.1.2, 2.0.0
Reporter: Steen Manniche
Priority: Minor
I have been looking through the hbase-thrift code looking for where
the server performs renewals of kerberos tickets for the provided
principal/keytab. There seems to be no logic in place for renewing tickets.
The hadoop-common provides the class
UserGroupInformation, which exposes the method
{{checkTGTAndReloginFromKeytab}}. I can see that the {{ThriftServerRunner}}
class
has a handle to the class
(https://github.com/apache/hbase/blob/master/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java#L205),
but I do not see the ticket renewal logic being called anywhere.
A possible workaround is to renew the ticket outside the java process.
The documentation on the {{checkTGTAndReloginFromKeytab}} states that if the
ticket is still valid, a call to the method is essentially a no-op.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steen Manniche updated HBASE-17817:
---
Hadoop Flags: Reviewed
Release Note: Add table name to exception logging when a coprocessor is
removed from a table by the region server
Status: Patch Available (was: Open)
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
> Attachments: HBASE-17817.master.001.patch,
> HBASE-17817.master.002.patch, HBASE-17817.master.003.patch,
> HBASE-17817.master.004.patch
>
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> If the coprocessor was loaded on a specific table, the output in the logs
> give no indication as to which table the coprocessor was removed from (or
> which version, or jarfile is the culprit). This causes longer debugging and
> recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steen Manniche updated HBASE-17817:
---
Attachment: HBASE-17817.master.004.patch
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
> Attachments: HBASE-17817.master.001.patch,
> HBASE-17817.master.002.patch, HBASE-17817.master.003.patch,
> HBASE-17817.master.004.patch
>
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> If the coprocessor was loaded on a specific table, the output in the logs
> give no indication as to which table the coprocessor was removed from (or
> which version, or jarfile is the culprit). This causes longer debugging and
> recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steen Manniche updated HBASE-17817:
---
Attachment: HBASE-17817.master.003.patch
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
> Attachments: HBASE-17817.master.001.patch,
> HBASE-17817.master.002.patch, HBASE-17817.master.003.patch
>
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> If the coprocessor was loaded on a specific table, the output in the logs
> give no indication as to which table the coprocessor was removed from (or
> which version, or jarfile is the culprit). This causes longer debugging and
> recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964643#comment-15964643
]
Steen Manniche commented on HBASE-17817:
Thank you for the elaborate comments. As per your suggestion on hbase-user, I
created a review-board issue and continued the development of the patch there.
With regards to the {{isAssignableFrom}} comment: you are absolutely right, and
I will make that part of the patch.
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
> Attachments: HBASE-17817.master.001.patch,
> HBASE-17817.master.002.patch
>
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> If the coprocessor was loaded on a specific table, the output in the logs
> give no indication as to which table the coprocessor was removed from (or
> which version, or jarfile is the culprit). This causes longer debugging and
> recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steen Manniche updated HBASE-17817:
---
Attachment: HBASE-17817.master.001.patch
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
> Attachments: HBASE-17817.master.001.patch,
> HBASE-17817.master.002.patch
>
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> If the coprocessor was loaded on a specific table, the output in the logs
> give no indication as to which table the coprocessor was removed from (or
> which version, or jarfile is the culprit). This causes longer debugging and
> recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steen Manniche updated HBASE-17817:
---
Attachment: HBASE-17817.master.002.patch
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
> Attachments: HBASE-17817.master.002.patch
>
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> If the coprocessor was loaded on a specific table, the output in the logs
> give no indication as to which table the coprocessor was removed from (or
> which version, or jarfile is the culprit). This causes longer debugging and
> recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steen Manniche updated HBASE-17817:
---
Attachment: (was: HBASE-17817.master.001.patch)
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> If the coprocessor was loaded on a specific table, the output in the logs
> give no indication as to which table the coprocessor was removed from (or
> which version, or jarfile is the culprit). This causes longer debugging and
> recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steen Manniche updated HBASE-17817:
---
Attachment: (was: HBASE-17817.master.002.patch)
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
> Attachments: HBASE-17817.master.001.patch
>
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> If the coprocessor was loaded on a specific table, the output in the logs
> give no indication as to which table the coprocessor was removed from (or
> which version, or jarfile is the culprit). This causes longer debugging and
> recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steen Manniche updated HBASE-17817:
---
Attachment: HBASE-17817.master.002.patch
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
> Attachments: HBASE-17817.master.001.patch,
> HBASE-17817.master.002.patch
>
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> If the coprocessor was loaded on a specific table, the output in the logs
> give no indication as to which table the coprocessor was removed from (or
> which version, or jarfile is the culprit). This causes longer debugging and
> recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steen Manniche updated HBASE-17817:
---
Attachment: HBASE-17817.master.001.patch
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
> Attachments: HBASE-17817.master.001.patch
>
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> If the coprocessor was loaded on a specific table, the output in the logs
> give no indication as to which table the coprocessor was removed from (or
> which version, or jarfile is the culprit). This causes longer debugging and
> recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943057#comment-15943057
]
Steen Manniche commented on HBASE-17817:
Great, that was what I was hoping for: some mechanism to supply the most
detailed information available at the time the exception is thrown. Thanks!
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> If the coprocessor was loaded on a specific table, the output in the logs
> give no indication as to which table the coprocessor was removed from (or
> which version, or jarfile is the culprit). This causes longer debugging and
> recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942756#comment-15942756
]
Steen Manniche commented on HBASE-17817:
Got access to the log files again so I can give a bit more detail on the issue.
The hbase exception log message originates from here:
https://github.com/apache/hbase/blob/ee1549cc9778af7124e3c7c6b187a0b124385a90/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java#L604
In this method, we have access to
[{{CoprocessorEnvironment}}|https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/CoprocessorEnvironment.html],
but I'm not sure if we can reliably extract the tablename from that?
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> If the coprocessor was loaded on a specific table, the output in the logs
> give no indication as to which table the coprocessor was removed from (or
> which version, or jarfile is the culprit). This causes longer debugging and
> recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steen Manniche updated HBASE-17817:
---
Description:
When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
handles this according to {{hbase.coprocessor.abortonerror}}.
If the coprocessor was loaded on a specific table, the output in the logs give
no indication as to which table the coprocessor was removed from (or which
version, or jarfile is the culprit). This causes longer debugging and recovery
times.
was:
When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
handles this according to {{hbase.coprocessor.abortonerror}}.
The output in the logs give no indication as to which table the coprocessor was
removed from (or which version, or jarfile is the culprit). This causes longer
debugging and recovery times.
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> If the coprocessor was loaded on a specific table, the output in the logs
> give no indication as to which table the coprocessor was removed from (or
> which version, or jarfile is the culprit). This causes longer debugging and
> recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
[
https://issues.apache.org/jira/browse/HBASE-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937292#comment-15937292
]
Steen Manniche commented on HBASE-17817:
I don't have the relevant loglines with me right now. But the case is, when
looking in the hbase-regionserver log, the exception shows the qualified name
of the coprocessors as well as the line where the exception occurred. There is
no trace of which table the coprocessor was loaded on.
> Make Regionservers log which tables it removed coprocessors from when aborting
> --
>
> Key: HBASE-17817
> URL: https://issues.apache.org/jira/browse/HBASE-17817
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, regionserver
>Affects Versions: 1.1.2
>Reporter: Steen Manniche
> Labels: logging
>
> When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
> handles this according to {{hbase.coprocessor.abortonerror}}.
> The output in the logs give no indication as to which table the coprocessor
> was removed from (or which version, or jarfile is the culprit). This causes
> longer debugging and recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-17817) Make Regionservers log which tables it removed coprocessors from when aborting
Steen Manniche created HBASE-17817:
--
Summary: Make Regionservers log which tables it removed
coprocessors from when aborting
Key: HBASE-17817
URL: https://issues.apache.org/jira/browse/HBASE-17817
Project: HBase
Issue Type: Improvement
Components: Coprocessors, regionserver
Affects Versions: 1.1.2
Reporter: Steen Manniche
When a coprocessor throws a runtime exception (e.g. NPE), the regionserver
handles this according to {{hbase.coprocessor.abortonerror}}.
The output in the logs give no indication as to which table the coprocessor was
removed from (or which version, or jarfile is the culprit). This causes longer
debugging and recovery times.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-14205) RegionCoprocessorHost System.nanoTime() performance bottleneck
[ https://issues.apache.org/jira/browse/HBASE-14205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842877#comment-15842877 ] Steen Manniche commented on HBASE-14205: The official HBase documentation still lists this functionality as available, which cost me some time looking through configuration files before I found this issue. The part in question is this: http://hbase.apache.org/book.html#_monitor_time_spent_in_coprocessors > RegionCoprocessorHost System.nanoTime() performance bottleneck > -- > > Key: HBASE-14205 > URL: https://issues.apache.org/jira/browse/HBASE-14205 > Project: HBase > Issue Type: Bug > Components: Coprocessors, Performance, regionserver >Reporter: Jan Van Besien >Assignee: Andrew Purtell >Priority: Critical > Labels: needs_releasenote > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14205.patch > > > The tracking of execution time of coprocessor methods introduced in > HBASE-11516 introduces 2 calls to System.nanoTime() per coprocessor method > per coprocessor. This is resulting in a serious performance bottleneck in > certain scenarios. > For example consider the scenario where many rows are being ingested (PUT) in > a table which has multiple coprocessors (we have up to 20 coprocessors). This > results in 8 extra calls to System.nanoTime() per coprocessor (prePut, > postPut, postStartRegionOperation and postCloseRegionOperation) which has in > total (i.e. times 20) been seen to result in a 50% increase of execution time. > I think it is generally considered bad practice to measure execution times on > such a small scale (per single operation). Also note that measurements are > taken even for coprocessors that do not even have an actual implementation > for certain operations, making the problem worse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
