Charles Connell created HBASE-28931:
---------------------------------------
Summary: RPC TLS certificate is not reloaded when in Kubernetes
Secret directory
Key: HBASE-28931
URL: https://issues.apache.org/jira/browse/HBASE-28931
Project: HBase
Issue Type: Bug
Reporter: Charles Connell
Assignee: Charles Connell
At my company we have an issue with our HBase servers not reloading TLS
certificate files after they change on disk. We run our HMasters inside
Kubernetes Pods, and define our certificate contents as Kubernetes Secrets.
Then, the Secrets are projected into the HMaster containers as files. When the
value of a Secret changes, the file changes automatically. However, Kubernetes
does some complicated indirection, and does not change the files directly. It
swaps a new directory in with new files in it.
HBase sets up a WatchService on the directory containing the TLS cert. For
example, at my company, the cert is at
{{{}/etc/hadoop/conf/ssl/cert/server-chain.pem{}}}. Then, events from that
WatchService are delivered to a [handler
method|https://github.com/apache/hbase/blob/836630422df2776287a860eff9d7104c3eca0582/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/tls/X509Util.java#L530]
which contains this check:
{code:java}
Path eventFilePath = dirPath.resolve((Path) event.context());
if (filePath.equals(eventFilePath)) {
shouldResetContext = true;
}{code}
Debug logs show why this conditional is never true:
2024-10-21T17:48:13,659 [FileChangeWatcher-server-chain.pem] DEBUG
org.apache.hadoop.hbase.io.FileChangeWatcher: Got file changed event:
ENTRY_CREATE with context: ..2024_10_21_17_48_13.2471317370
2024-10-21T17:48:13,659 [FileChangeWatcher-server-chain.pem] DEBUG
org.apache.hadoop.hbase.io.FileChangeWatcher: Got file changed event:
ENTRY_CREATE with context: ..2024_10_21_17_48_13.2471317370
2024-10-21T17:48:13,660 [FileChangeWatcher-server-chain.pem] DEBUG
org.apache.hadoop.hbase.io.crypto.tls.X509Util: Ignoring watch event and
keeping previous default SSL context. Event kind: ENTRY_CREATE with context:
..2024_10_21_17_48_13.2471317370
....
The watch events have a variety of files attached to them, but none of them are
{{{}server-chain.pem{}}}, so HBase thinks they are not relevant.
I propose that we simply remove the condition inspecting the file name that was
changed, and always reload the SSL context if a watcher event fires.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
