[
https://issues.apache.org/jira/browse/HBASE-27658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17750588#comment-17750588
]
Reid Chan commented on HBASE-27658:
-----------------------------------
could submit a pull request
> HBase token expired because the connection was not released
> -----------------------------------------------------------
>
> Key: HBASE-27658
> URL: https://issues.apache.org/jira/browse/HBASE-27658
> Project: HBase
> Issue Type: Bug
> Components: spark
> Environment: spark: 3.1.1
> hbase-connector: 1.0.1-SNAPSHOT
> Reporter: Zhen Wang
> Priority: Major
>
> I used hbase-spark to read hbase data in spark streaming application, and the
> token expired error occurred after 7 days.
> Error:
> {code:java}
> 23/02/20 19:29:07 INFO AsyncRequestFutureImpl: id=38643, table=XXX,
> attempt=11/16, failureCount=1ops, last exception=java.io.IOException: Call to
> XXX.XXX/XXX.XXX.XXX.XXX:60020 failed on local exception: java.io.IOException:
> org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Token has expired on XXX.XXX,60020,1675839912342, tracking started null,
> retrying after=10009ms, operationsToReplay=1 {code}
> HBase RpcClient has expired connections that have not been cleaned up.
> arthas command:
> {code:java}
> watch org.apache.hadoop.hbase.ipc.AbstractRpcClient getConnection
> "target.connections.values.{token.toString}" -x 2 -n 2 {code}
> result:
> {code:java}
> @ArrayList[
> @String[Kind: HBASE_AUTH_TOKEN, Service:
> a88deda4-d74c-483e-af8b-0fb9de7c7594, Ident: ((username=XXX, keyId=575,
> issueDate=1676858059133, expirationDate=1677462859133,
> sequenceNumber=772469))],
> ...
> @String[Kind: HBASE_AUTH_TOKEN, Service:
> a88deda4-d74c-483e-af8b-0fb9de7c7594, Ident: ((username=XXX, keyId=573,
> issueDate=1676656459093, expirationDate=1677261259093,
> sequenceNumber=587447))],
> ...
> @String[Kind: HBASE_AUTH_TOKEN, Service:
> a88deda4-d74c-483e-af8b-0fb9de7c7594, Ident: ((username=XXX, keyId=571,
> issueDate=1676454859019, expirationDate=1677059659019,
> sequenceNumber=430605))],
> ...
> ]{code}
> The hbase connection seems to be cleaned up only after being idle for a
> period of time.
> {code:java}
> org.apache.hadoop.hbase.spark.HBaseConnectionCache#connectionMap
> org.apache.hadoop.hbase.ipc.AbstractRpcClient#connections {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
