Bryan Bende created NIFI-16319:
----------------------------------
Summary: nifi.security.autoreload.enabled does not reload
keystore/truststore when files are Kubernetes Secret mounts (..data atomic
symlink swap)
Key: NIFI-16319
URL: https://issues.apache.org/jira/browse/NIFI-16319
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 2.11.0, 2.10.0, 2.9.0, 2.7.2, 2.7.1, 2.8.0, 2.7.0, 2.6.0,
2.5.0, 2.4.0, 2.3.0, 2.2.0, 2.1.0, 2.0.0
Reporter: Bryan Bende
With {{{}nifi.security.autoreload.enabled=true{}}}, NiFi is supposed to pick up
keystore and truststore changes without a restart. That fails when the stores
are Kubernetes Secret or projected-volume mounts, which is the normal way to
inject certs on Kubernetes.
Kubelet does not overwrite {{keystore.jks}} / {{truststore.jks}} in place. It
writes a new timestamped directory, atomically retargets the {{..data}}
symlink, and deletes the old directory:
{{/nifi/certs/truststore.jks -> ..data/truststore.jks ->
..2026_07_21_12_00_00.123/truststore.jks}}
after Secret update:
{{/nifi/certs/truststore.jks -> ..data/truststore.jks ->
..2026_08_30_12_00_00.456/truststore.jks}}
The stable names never change. Only the directory behind {{..data}} changes.
h3. What 2.x does
NIFI-12125 replaced the Jetty {{StoreScanner}} with
{{java.nio.file.WatchService}} in {{{}SslContextConfiguration{}}}. On startup
it:
* registers {{ENTRY_CREATE}} / {{ENTRY_MODIFY}} on the store parent directory
_and_ on {{storePath.toRealPath()}} (the timestamped kubelet dir)
* reloads only when an event filename is {{keystore.jks}} or
{{truststore.jks}} ({{{}SecurityStoreChangedPathListener{}}})
* polls one {{WatchKey}} per {{nifi.security.autoreload.interval}}
({{{}WatchServiceMonitorCommand{}}})
The timestamped directory is never modified; it is replaced. The watch key on
that directory becomes invalid ({{{}Watch Key reset failed: Watch Service no
longer valid{}}}). Events on the parent are for {{..data}} or
{{{}..<timestamp>{}}}, which do not match the filename filter. After the first
Secret update, autoreload never fires again for the life of the JVM.
The wrapping {{StandardX509ExtendedKeyManager}} /
{{StandardX509ExtendedTrustManager}} from NIFI-12125 are fine: new sockets
(including cluster-protocol heartbeats) would use reloaded material _if_ a
reload happened. The bug is that the watcher never re-reads the files.
h3. Regression
This is NIFI-10425 again. That ticket described the same kubelet layout. The
1.x fix was Jetty {{reportRealPaths=false}} so the scanner tracked the
{_}symlink{_}, not the real file. NIFI-12125 dropped that scanner.
{{toRealPath()}} in {{SslContextConfiguration.addStorePath}} /
{{addStoreFileName}} is the opposite of that fix.
h3. Expected
Updating the Secret that backs {{nifi.security.keystore}} /
{{nifi.security.truststore}} causes NiFi, within about one autoreload interval,
to log that the Key Manager and Trust Manager were reloaded and to use the new
material on the next TLS handshake (Jetty and cluster protocol).
h3. Actual
The process keeps the Key Manager / Trust Manager loaded at startup. Cluster
protocol and HTTPS keep working on the old in-memory trust anchors. A later pod
restart loads the on-disk stores and can fail (for example PKIX path validation
if the CA rotated and the leaf did not).
h3. How to reproduce
* Run NiFi 2.x in Kubernetes with {{nifi.security.autoreload.enabled=true}}
and keystore/truststore coming from a Secret or projected volume (not a copy
into a regular directory).
* Confirm startup log: {{{}Scheduled Security Store Monitor with Duration
[...]{}}}.
* Rotate the Secret (new {{truststore.jks}} / {{keystore.jks}} bytes)
* Wait several autoreload intervals.
Expected: {{{}Key Manager and Trust Manager Reloaded from Changed Path{}}}.
Actual: no reload; often {{Watch Key reset failed: Watch Service no longer
valid}} when the old timestamped dir is removed. {{openssl s_client}} / a new
cluster handshake still presents or trusts the original material until the pod
restarts.
A local stand-in without a cluster: point the stores at symlinks whose targets
are replaced by swapping a {{{}..data{}}}-style link the way kubelet does,
rather than overwriting the file in place.
h3. Suggested fix
Do not pin {{WatchService}} to {{toRealPath()}} of a kubelet timestamped
directory.
Reload from the _configured_ paths in {{nifi.properties}} (follow the symlink
at read time). The most reliable approach is to poll those paths every
{{{}nifi.security.autoreload.interval{}}}: compare digest or
{{{}lastModified{}}}, then rebuild with {{FrameworkKeyManagerBuilder}} /
{{FrameworkTrustManagerBuilder}} and {{setKeyManager}} / {{setTrustManager}} as
today.
Watching the parent for {{..data}} changes can be an extra trigger, but polling
is what actually works with projected Secrets. In-place file writes on a normal
filesystem should keep working.
h3. Related
* NIFI-10425 — same kubelet symlink layout; Jetty scanner
* NIFI-11536 — autoreload broken after 1.17.0
* NIFI-12125 — WatchService + shared SSLContext / cluster protocol; introduced
this watcher
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
