Sandor Molnar created KNOX-3579:
-----------------------------------
Summary: DefaultKeystoreService reloads keystores from disk under
a global lock on every request, capping SSO login and cookie verification
throughput
Key: KNOX-3579
URL: https://issues.apache.org/jira/browse/KNOX-3579
Project: Apache Knox
Issue Type: Improvement
Components: Server
Affects Versions: 2.1.1, 3.0.0, 2.1.0, 1.6.1, 1.6.0, 2.0.0
Reporter: Sandor Molnar
Assignee: Sandor Molnar
Fix For: 3.1.0
h3. Problem
Under load, Knox serves WebSSO logins and SSO cookie verifications only up to a
fixed rate, about 25 logins/s and 12 cookie verifications/s on one gateway
instance, while the gateway still has CPU free. Requests beyond that rate queue
up and eventually time out. With 1,500 or more users holding sessions and
polling every 10 s, 64-96% of requests timed out at 60 s.
Environment: BCFKS keystores (FIPS), one gateway instance, LDAP-backed WebSSO.
Thread dumps taken under load show almost all Jetty request threads (245 of
254) BLOCKED on the single DefaultKeystoreService monitor, waiting in:
* {{DefaultKeystoreService.getSigningKeystore}} (SSO cookie / JWT verification)
* {{KnoxLdapContextFactory.getSystemPassword}} (LDAP bind)
* {{WebSSOResource.init}} -> {{TokenUtils.getSignatureAlgorithm}} (WebSSO login)
In every dump, the thread holding the monitor is in
{{DefaultKeystoreService.loadKeyStore}} -> {{KeyStore.load}} -> BCFKS
{{engineLoad}} -> {{PKCS5S2ParametersGenerator}} (PBKDF2-HMAC-SHA512).
h3. Root cause
# *Loaded keystores are never cached.* Every {{getSigningKeystore()}} and
{{getCredentialStoreForCluster()}} call reads the file and calls
{{KeyStore.load()}}. For BCFKS, that runs PBKDF2 for the integrity check, which
costs tens of milliseconds of CPU per load. Only credential values (alias ->
password) are cached, and only for aliases that exist. A lookup for an alias
that doesn't exist reloads the whole credential store every time.
# *One global lock serializes all loads.* {{getKeystore}}, {{loadKeyStore}},
{{getCredentialForCluster}} and the write methods are all synchronized on the
service instance. The gateway therefore runs one PBKDF2 at a time, however many
cores it has.
Hot paths that reload a store on every request:
||Path||Store(s) reloaded on every request||
|SSO cookie / JWT verification
({{DefaultTokenAuthorityService.selectVerificationKey}})|The signing keystore,
always. Also the gateway-level credential store when the signing keystore
password alias (default: gateway-identity-keystore-password) is not defined.|
|LDAP bind ({{KnoxLdapContextFactory.getSystemPassword}})|The topology
credential store, when the system password alias is defined only at gateway
level.|
|WebSSO login ({{WebSSOResource.init}} ->
{{TokenUtils.getSignatureAlgorithm}})|The gateway-level credential store, when
gateway.signing.hmac.secret is not defined.|
Throughput is therefore capped at roughly 1 / (PBKDF2 time x loads per request).
h3. Proposed fix
# Cache the loaded signing keystore and credential stores in memory, keyed by
path and type. On every access, check the cached copy against a cheap file
stamp (last-modified time, size, file key) and reload only when the file has
changed. External updates, such as {{knoxcli create-alias}} or re-mounted
secrets, are still picked up. Keep a digest of the store password with each
entry, so a wrong password still fails as it does today. Re-check once any
entry that was loaded within the mtime granularity window of its file's last
change.
# Replace the global monitor with a {{ReentrantReadWriteLock}}. Reads share the
lock. Writes (add/remove alias, create store, add self-signed cert) take it
exclusively. Loads of the same file run one at a time. Within one JVM this is
required anyway: concurrent {{FileChannel.lock()}} calls on the same file throw
{{OverlappingFileLockException}}. The credential-cache hit path becomes
lock-free.
# Writers ({{addCredentialsForCluster}}, {{removeCredentialsForCluster}},
{{addCertForGateway}}) load a private copy under the write lock and drop the
cached entry after writing, so a shared cached instance is never modified.
# Private-key lookups ({{getSigningKey}}, {{getKeyForGateway}}) and the gateway
identity keystore, truststore and HTTP-client keystore keep loading a fresh
copy, but no longer under a global lock. Sharing BCFKS instances for
private-key access is unsafe: {{BcFKSKeyStoreSpi.engineGetKey}} fills an
unsynchronized {{privateKeyCache}} HashMap, and once a key is cached it is
returned whatever password is supplied.
# Add {{gateway.keystore.file.cache.enabled}} (default: true) to
gateway-site.xml, so the cache can be turned off without a rebuild.
h3. Expected result
On the hot paths, each request costs an in-memory lookup plus a stat instead of
1-2 PBKDF2 runs under a global lock. Login and cookie verification throughput
grows with the CPU available, and no request thread waits on
DefaultKeystoreService for a store that is already loaded.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)