[ https://issues.apache.org/jira/browse/HBASE-27658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17750589#comment-17750589 ]
Zhen Wang commented on HBASE-27658: ----------------------------------- [~reidchan] Thanks, I will open a pr later. > HBase token expired because the connection was not released > ----------------------------------------------------------- > > Key: HBASE-27658 > URL: https://issues.apache.org/jira/browse/HBASE-27658 > Project: HBase > Issue Type: Bug > Components: spark > Environment: spark: 3.1.1 > hbase-connector: 1.0.1-SNAPSHOT > Reporter: Zhen Wang > Priority: Major > > I used hbase-spark to read hbase data in spark streaming application, and the > token expired error occurred after 7 days. > Error: > {code:java} > 23/02/20 19:29:07 INFO AsyncRequestFutureImpl: id=38643, table=XXX, > attempt=11/16, failureCount=1ops, last exception=java.io.IOException: Call to > XXX.XXX/XXX.XXX.XXX.XXX:60020 failed on local exception: java.io.IOException: > org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): > Token has expired on XXX.XXX,60020,1675839912342, tracking started null, > retrying after=10009ms, operationsToReplay=1 {code} > HBase RpcClient has expired connections that have not been cleaned up. > arthas command: > {code:java} > watch org.apache.hadoop.hbase.ipc.AbstractRpcClient getConnection > "target.connections.values.{token.toString}" -x 2 -n 2 {code} > result: > {code:java} > @ArrayList[ > @String[Kind: HBASE_AUTH_TOKEN, Service: > a88deda4-d74c-483e-af8b-0fb9de7c7594, Ident: ((username=XXX, keyId=575, > issueDate=1676858059133, expirationDate=1677462859133, > sequenceNumber=772469))], > ... > @String[Kind: HBASE_AUTH_TOKEN, Service: > a88deda4-d74c-483e-af8b-0fb9de7c7594, Ident: ((username=XXX, keyId=573, > issueDate=1676656459093, expirationDate=1677261259093, > sequenceNumber=587447))], > ... > @String[Kind: HBASE_AUTH_TOKEN, Service: > a88deda4-d74c-483e-af8b-0fb9de7c7594, Ident: ((username=XXX, keyId=571, > issueDate=1676454859019, expirationDate=1677059659019, > sequenceNumber=430605))], > ... > ]{code} > The hbase connection seems to be cleaned up only after being idle for a > period of time. > {code:java} > org.apache.hadoop.hbase.spark.HBaseConnectionCache#connectionMap > org.apache.hadoop.hbase.ipc.AbstractRpcClient#connections {code} > -- This message was sent by Atlassian Jira (v8.20.10#820010)