[
https://issues.apache.org/jira/browse/CXF-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17416063#comment-17416063
]
Don Steinberg commented on CXF-7576:
------------------------------------
We have been countering this issue randomly at various sites over the last
several months. Our web services are defined with ws-security using SAML2 with
a UsernameToken policy. We are running our services under a JBoss EAP 7
application server with wss4j version 2.1.8 and cxf version 3.1.10. In our
situation, the system runs fine for several days, and then randomly starts
receiving the error noted above. The only resolution for the issue is to
restart of application server, which does resolve the issue until the next time
it happens days later.
Previously encountered a similar issue with ehcache due to this [CXF
Issue|https://issues.apache.org/jira/browse/CXF-7890?jql=text%20~%20%22net.sf.ehcache.CacheException%3A%20java.io.FileNotFoundException%22],
which is supposed to be resolved once we upgrade to CXF 3.4 or higher. We
have not upgraded this particular release of our application to CXF 3.4+, but
have been able to work-around this issue by using
'-Dnet.sf.ehcache.disable=true' on the command line when starting up the
application server. We know this is working correctly, because attempts to add
to the TokenStore cache are logged as follows:
[2021-09-10 07:19:03,207] WARNING [net.sf.ehcache.Cache] (default task-336)
Cache: org.apache.cxf.ws.security.tokenstore.TokenStore-925578529 is disabled
because the net.sf.ehcache.disabled property was set to true. No elements will
be added to the cache.
However, this setting does not appear to impact this particular issue in any
way. As I understand it, there are 3 different ehcaches used with ws-security
(UsernameToken nonces, Signed Timestamps, SAML 2.0 OneTimeUse Assertions).
From what I have read, the only way to disable these is through configuration
by setting the following to false in the WS configuration files (
ws-security.enable.nonce.cache, ws-security.enable.timestamp.cache,
ws-security.enable.saml.cache). We have not tried this because it is very
difficult, if not impossible, for us to reproduce this issue outside of our
production sites, so we don't want to disable these within the application and
deploy it without knowing whether or not it resolves the issue.
I have included our latest stack trace for this issue below. What I would like
to know is, is there a way through the environment or command-line to disable
these usages of ehcache so that we can do it without configuration/deployment.
We would also like to know if perhaps this issue has been discovered
internally and fixed since this issue was closed.
Any help would be appreciated. This is becoming a real concern.
Thanks.
[2021-09-10 07:19:03,522] WARNING [org.apache.cxf.phase.PhaseInterceptorChain]
(default task-336) Interceptor for
\{http://site.com/services/external/ConfigurationService}ConfigurationService
has thrown exception, unwinding now: net.sf.ehcache.CacheException:
java.io.FileNotFoundException:
/tmp/cxf145687351/ws-security%002esaml%002ecache%002einstance-925578529.data
(No such file or directory)[2021-09-10 07:19:03,522] WARNING
[org.apache.cxf.phase.PhaseInterceptorChain] (default task-336) Interceptor for
\{http://site.com/services/external/ConfigurationService}ConfigurationService
has thrown exception, unwinding now: net.sf.ehcache.CacheException:
java.io.FileNotFoundException:
/tmp/cxf145687351/ws-security%002esaml%002ecache%002einstance-925578529.data
(No such file or directory) at
net.sf.ehcache.store.disk.DiskStorageFactory.<init>(DiskStorageFactory.java:146)
[ehcache-2.10.3.jar:2.10.3] at
net.sf.ehcache.store.disk.DiskStore.create(DiskStore.java:154)
[ehcache-2.10.3.jar:2.10.3] at
net.sf.ehcache.store.disk.DiskStore.createCacheStore(DiskStore.java:182)
[ehcache-2.10.3.jar:2.10.3] at net.sf.ehcache.Cache.initialise(Cache.java:1216)
[ehcache-2.10.3.jar:2.10.3] at
net.sf.ehcache.CacheManager.initializeEhcache(CacheManager.java:1357)
[ehcache-2.10.3.jar:2.10.3] at
net.sf.ehcache.CacheManager.addCacheNoCheck(CacheManager.java:1423)
[ehcache-2.10.3.jar:2.10.3] at
net.sf.ehcache.CacheManager.addCacheIfAbsent(CacheManager.java:1927)
[ehcache-2.10.3.jar:2.10.3] at
org.apache.wss4j.common.cache.EHCacheReplayCache.<init>(EHCacheReplayCache.java:54)
[wss4j-ws-security-common-2.1.8.jar:2.1.8] at
org.apache.cxf.ws.security.cache.CXFEHCacheReplayCache.<init>(CXFEHCacheReplayCache.java:37)
[cxf-rt-ws-security-3.1.10.jar:3.1.10] at
org.apache.cxf.ws.security.wss4j.WSS4JUtils.getReplayCache(WSS4JUtils.java:141)
[cxf-rt-ws-security-3.1.10.jar:3.1.10] at
org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.getReplayCache(WSS4JInInterceptor.java:644)
[cxf-rt-ws-security-3.1.10.jar:3.1.10] at
org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.configureReplayCaches(WSS4JInInterceptor.java:436)
[cxf-rt-ws-security-3.1.10.jar:3.1.10] at
org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessageInternal(WSS4JInInterceptor.java:237)
[cxf-rt-ws-security-3.1.10.jar:3.1.10] at
org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:171)
[cxf-rt-ws-security-3.1.10.jar:3.1.10] at
org.apache.cxf.ws.security.wss4j.PolicyBasedWSS4JInInterceptor.handleMessage(PolicyBasedWSS4JInInterceptor.java:80)
[cxf-rt-ws-security-3.1.10.jar:3.1.10] at
org.apache.cxf.ws.security.wss4j.PolicyBasedWSS4JInInterceptor.handleMessage(PolicyBasedWSS4JInInterceptor.java:66)
[cxf-rt-ws-security-3.1.10.jar:3.1.10] at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
[cxf-core-3.1.10.jar:3.1.10] at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
[cxf-core-3.1.10.jar:3.1.10] at
org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:262)
[cxf-rt-transports-http-3.1.10.jar:3.1.10] at
org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
[cxf-rt-transports-http-3.1.10.jar:3.1.10] at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
[cxf-rt-transports-http-3.1.10.jar:3.1.10] at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
[cxf-rt-transports-http-3.1.10.jar:3.1.10] at
org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:180)
[cxf-rt-transports-http-3.1.10.jar:3.1.10] at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:299)
[cxf-rt-transports-http-3.1.10.jar:3.1.10] at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:218)
[cxf-rt-transports-http-3.1.10.jar:3.1.10] at
javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
[jboss-servlet-api_3.1_spec-1.0.0.Final-redhat-1.jar:1.0.0.Final-redhat-1] at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:274)
[cxf-rt-transports-http-3.1.10.jar:3.1.10] at
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
[undertow-servlet-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
[undertow-servlet-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
[undertow-servlet-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
[undertow-core-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
[undertow-servlet-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
[undertow-servlet-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
[undertow-core-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51)
[undertow-core-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
[undertow-core-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
[undertow-servlet-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)
[undertow-servlet-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
[undertow-core-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
[undertow-servlet-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
[undertow-core-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
[undertow-core-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
[undertow-core-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
[undertow-core-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
[undertow-core-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:285)
[undertow-servlet-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:264)
[undertow-servlet-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[undertow-servlet-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:175)
[undertow-servlet-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.server.Connectors.executeRootHandler(Connectors.java:324)
[undertow-core-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:803)
[undertow-core-1.3.31.Final-redhat-3.jar:1.3.31.Final-redhat-3] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_162] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_162] at java.lang.Thread.run(Thread.java:748)
[rt.jar:1.8.0_162]Caused by: java.io.FileNotFoundException:
/tmp/cxf145687351/ws-security%002esaml%002ecache%002einstance-925578529.data
(No such file or directory) at java.io.RandomAccessFile.open0(Native Method)
[rt.jar:1.8.0_162] at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
[rt.jar:1.8.0_162] at
java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) [rt.jar:1.8.0_162]
at
net.sf.ehcache.store.disk.DiskStorageFactory.allocateRandomAccessFiles(DiskStorageFactory.java:207)
[ehcache-2.10.3.jar:2.10.3] at
net.sf.ehcache.store.disk.DiskStorageFactory.<init>(DiskStorageFactory.java:144)
[ehcache-2.10.3.jar:2.10.3] ... 55 more
> ws-security enonce ecache einstance data not found
> --------------------------------------------------
>
> Key: CXF-7576
> URL: https://issues.apache.org/jira/browse/CXF-7576
> Project: CXF
> Issue Type: Bug
> Components: JAX-WS Runtime
> Affects Versions: 3.1.6
> Reporter: Soumya Panda
> Assignee: Colm O hEigeartaigh
> Priority: Major
>
> _Hi Team,
> We are seeing a weird issue. It was working fine and nothing change but now
> it started failing with below error :
> _
> _{color:red}net.sf.ehcache.CacheException: java.io.FileNotFoundException:
> /tmp/cxf745201861/ws-security%002enonce%002ecache%002einstance-
> {color}_1663277296.data (No such file or directory)
> at
> net.sf.ehcache.store.disk.DiskStorageFactory.<init>(DiskStorageFactory.java:146)
> at net.sf.ehcache.store.disk.DiskStore.create(DiskStore.java:154)
> at
> net.sf.ehcache.store.disk.DiskStore.createCacheStore(DiskStore.java:182)
> at net.sf.ehcache.Cache.initialise(Cache.java:1199)
> at
> net.sf.ehcache.CacheManager.initializeEhcache(CacheManager.java:1370)
> at net.sf.ehcache.CacheManager.addCacheNoCheck(CacheManager.java:1436)
> at
> net.sf.ehcache.CacheManager.addCacheIfAbsent(CacheManager.java:1936)
> at
> org.apache.wss4j.common.cache.EHCacheReplayCache.<init>(EHCacheReplayCache.java:54)
> at
> org.apache.cxf.ws.security.cache.CXFEHCacheReplayCache.<init>(CXFEHCacheReplayCache.java:37)
> at
> org.apache.cxf.ws.security.wss4j.WSS4JUtils.getReplayCache(WSS4JUtils.java:124)
> at
> org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.getReplayCache(WSS4JInInterceptor.java:776)
> at
> org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.configureReplayCaches(WSS4JInInterceptor.java:413)
> at
> org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessageInternal(WSS4JInInterceptor.java:249)
> at
> org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:184)
> at
> com.sbc.cc.clarify.api.util.cxf.interceptors.security.WebInInterceptor.handleMessage(WebInInterceptor.java:60)
> at
> com.sbc.cc.clarify.api.util.cxf.interceptors.security.WebInInterceptor.handleMessage(WebInInterceptor.java:19)
> at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
> at
> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
> at
> org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:251)
> at
> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:234)
> at
> org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:70)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1129)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1065)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:798)
> _{color:red}Caused by:
> java.io.FileNotFoundException:
> /tmp/cxf745201861/ws-security%002enonce%002ecache%002einstance-1663277296.data
> (No such file or directory){color}_
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:253)
> at
> net.sf.ehcache.store.disk.DiskStorageFactory.allocateRandomAccessFiles(DiskStorageFactory.java:207)
> at
> net.sf.ehcache.store.disk.DiskStorageFactory.<init>(DiskStorageFactory.java:144)
> ... 32 more
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
