Hi Gary, This is exactly what I missed. Without those settings in my hbase-site.xml, I saw these messages in regional server log file:
2011-09-09 17:28:14,172 WARN org.apache.hadoop.ipc.Client: Couldn't setup connection for [email protected] tonn/[email protected] 2011-09-09 17:28:14,172 WARN org.apache.hadoop.hdfs.DFSClient: Problem renewing lease for DFSClient_hb_rs_hrt8n38.cc1.ygridcore.net,60020,1315520835373_1315520835752 java.io.IOException: Call to hrt8n37.cc1.ygridcore.net/98.137.233.217:8020 failed on local exception: java.io.IOException: Couldn't setup connection [email protected] to nn/[email protected] at org.apache.hadoop.ipc.Client.wrapException(Client.java:1104) at org.apache.hadoop.ipc.Client.call(Client.java:1072) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) at $Proxy9.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at $Proxy9.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.renew(DFSClient.java:1176) at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1188) at java.lang.Thread.run(Thread.java:619) The region server can respond query but data is not committed to HDFS. I manually run kinit on the command line, and the log shows: 2011-09-09 17:28:50,466 INFO org.apache.hadoop.security.UserGroupInformation: Initiating logout for [email protected] 2011-09-09 17:28:50,466 INFO org.apache.hadoop.security.UserGroupInformation: Initiating re-login for [email protected] I will put the keytab and kerberos.principal in the configuration file to verify token renewal in HBase. Thank you. regards, Eric On Sep 11, 2011, at 3:57 PM, Gary Helmling wrote: > Hi Eric, > > If you configure > > hbase.master.keytab.file > hbase.master.kerberos.principal > hbase.regionserver.keytab.file > hbase.regionserver.kerberos.principal > > in your hbase-site.xml, then the master and region server processes should > login from the keytab files on startup, as Todd mentions. It's also my > understanding that they don't need a renewal thread in that case. The RPC > client just tries a relogin from the keytab in the case of a connection > error. > > Can you describe a bit more what you're seeing so that we can understand the > context? > > Gary > > > On Sun, Sep 11, 2011 at 3:13 PM, Todd Lipcon <[email protected]> wrote: > >> Hi Eric, >> >> Could you please explain more fully what you mean by this? The daemons >> generally run using keytabs, not user credentials, and thus shouldn't >> need the explicit TGT Renewer, right? >> >> -Todd >> >> On Sun, Sep 11, 2011 at 11:04 AM, Eric Yang <[email protected]> wrote: >>> Hi all, >>> >>> Hortonworks has a patch for secure append for Apache Hadoop 0.20.205 to >> work with HBase 0.90.x. However, secure Hadoop and HBase would work until >> kerberos token expires. There is currently no code that renews kerberos >> token in HBase. Hence, it is possible to add a cron job to periodically >> renew the HBase user token to keep the system running. What does the >> community think about having a setup script for cron job as part of HBase >> upcoming minor release, and fix the token renewal in HBase code for the next >> major version. On the other hand, would the community accept the token >> renewal code in HBase as part of the upcoming 0.90.5 release? If yes, what >> is the time line for 0.90.5? >>> >>> regards, >>> Eric >> >> >> >> -- >> Todd Lipcon >> Software Engineer, Cloudera >>
