[
https://issues.apache.org/jira/browse/HBASE-10379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879416#comment-13879416
]
takeshi.miao commented on HBASE-10379:
--------------------------------------
I did some test on +new version of Krb5-server (1.10.3), and seems this issue
is mitigated+. I have been using 20-threads of HBase Client to put 7 records
into one HTable with corresponding 7 regions scattered on three regionservers,
with three round of tests; this simple test can simply reproduce this 'Request
is a replay' issue on the first environment (HBase-0.94.16 with
krb5-server-1.6.1 on CentOS-5.3); But in second environment (HBase-0.94.16 with
krb5-server-1.10.3 on RHEL-6.4_GA-x86_64-10-Hourly2 on AWS), I did not
reproduce this issue. It seems that this issue was fixed in the later of
krb5-server release.
I did this test further on new version of krb5-server is due to I googled two
tickets ([#5924|http://krbdev.mit.edu/rt/Ticket/Display.html?id=5924] &
[#1201|http://krbdev.mit.edu/rt/Ticket/Display.html?id=1201]) whom were talking
about '_KRB5KRB_AP_ERR_REPEAT_' bug in krb5 community.
> Turn the msg "Request is a replay (34) - PROCESS_TGS" from logging level
> ERROR to WARN
> --------------------------------------------------------------------------------------
>
> Key: HBASE-10379
> URL: https://issues.apache.org/jira/browse/HBASE-10379
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.16
> Reporter: takeshi.miao
> Assignee: takeshi.miao
> Priority: Minor
>
> Hi All,
> Recently we got the error msg "Request is a replay (34) - PROCESS_TGS" while
> we are using the HBase client API to put data into HBase-0.94.16 with
> krb5-1.6.1 enabled. The related msg as follows...
> {code}
> [2014-01-15
> 09:40:38,452][hbase-tablepool-1-thread-3][ERROR][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1124)):
> PriviledgedActionException as:takeshi_miao@LAB
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Request is a
> replay (34) - PROCESS_TGS)]
> [2014-01-15
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.logPriviledgedAction(UserGroupInformation.java:1143)):
> PriviledgedAction as:takeshi_miao@LAB
> from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> [2014-01-15
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.ipc.SecureClient](org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:213)):
> Exception encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Request is a
> replay (34) - PROCESS_TGS)]
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:657)):
> Initiating logout for takeshi_miao@LAB
> [2014-01-15
> 09:40:38,454][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.logout(UserGroupInformation.java:154)):
> hadoop logout
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:667)):
> Initiating re-login for takeshi_miao@LAB
> [2014-01-15
> 09:40:38,455][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.login(UserGroupInformation.java:146)):
> hadoop login
> [2014-01-15
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:95)):
> hadoop login commit
> [2014-01-15
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:100)):
> using existing subject:[takeshi_miao@LAB, UnixPrincipal: takeshi_miao,
> UnixNumericUserPrincipal: 501, UnixNumericGroupPrincipal [Primary Group]:
> 501, UnixNumericGroupPrincipal [Supplementary Group]: 502, takeshi_miao@LAB]
> {code}
> At the beginning, we were worry about the "data loss" occurring while we
> found the "Request is a replay (34) - PROCESS_TGS" (especially it is the
> ERROR level) in log, but after code study, this is basically *NOT* a data
> loss issue due to HBase client API would try 5 times internally
> (o.a.h.hbase.ipc.SecureClient, L#296, a one thread) and also 10 times of
> retry externally (o.a.h.hbase.client.HConnectionManager, L#1661, for all
> failed thread), The HTable API would also throw IOEcxeption to client code if
> any thread still fail after these retries.
> Based on HBase users' viewpoint as us, we think this is better to change the
> logging level from 'ERROR' to 'WARN', due to the 'ERROR' level had been
> confused us for a while...But this code change may need to change boht HBase
> code and Hadoop code as well; so I am wondering how community think about
> this small thing but may be important to the pure HBase users.
> mailing list
> http://mail-archives.apache.org/mod_mbox/hbase-user/201401.mbox/%3CCADcMMgGiEyho0HGwgbfOUS78ymDpCo5Q0PStWAPUk40W%3DPfcFQ%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
