Re: WS-Security concurrency issue with CXF 2.2.12 + WSS4J 1.5.10

[Alessio Soldano]

OK, here I am with some data.
I've been able to reproduce the problem locally (had to tune the load 
test application to run on my laptop and still reproduce the problem 
;-)) and it seems CXF 2.4.1-SNAPSHOT + WSS4J 1.6 is not affected.
So I had a look at the commits that went in to the 1.5.x fix branch of 
WSS4J and I think I've found the commit that introduces the problem, 
http://svn.apache.org/viewvc?view=revision&revision=947604 . I've not 
enough knowledge of the project (at least atm) to really understand the 
reason, however the binary obtained by building the 947603 revision is 
not showing the concurrency issue below, while 947604 reproduces the 
problem (both versions tested with CXF 2.2.12).
At first sight the only way I see the WSDocInfo could be used 
concurrently is because of some issues with getting it from the 
WSDocInfoStore... but again, I'm not an expert at all here. Dan, could 
this perhaps be related to the hashing issue you mentioned in the other 
reply? I'm indeed running a 64bit JDK.

Thanks
Alessio


On 06/01/2011 11:47 AM, Alessio Soldano wrote:
Hi Colm,
yes, I'll try reproducing this locally (the load test is actually run 
on a bench environment that can't be moved quickly to cxf 2.4.0 / 
wss4j 1.6) and let you know asap.
Thanks
Alessio

On 06/01/2011 11:26 AM, Colm O hEigeartaigh wrote:
Hi Alessio,

Very interesting, I haven't seen this problem before. It looks like a
problem in WSS4J - I'm not sure how multiple threads are altering the
same WSDocInfo object.

Would it be possible to check whether it exists with CXF 2.4.0 and
WSS4J 1.6.0? I want to get a WSS4J 1.6.1 release out asap, so it would
be cool to fix this problem if it's there.

Colm.

On Wed, Jun 1, 2011 at 9:52 AM, Alessio Soldano<asold...@redhat.com>  
wrote:
Hi,
we're running some WS-Security load tests with Apache CXF 2.2.12 + 
WSS4J
1.5.10 and might have found a concurrency issue.
We're getting the following exception:

[runTest(bash)] OUT>    Caused by: 
java.util.ConcurrentModificationException
[runTest(bash)] OUT>      at
java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372) 

[runTest(bash)] OUT>      at
java.util.AbstractList$Itr.next(AbstractList.java:343)
[runTest(bash)] OUT>      at
org.apache.ws.security.WSDocInfo.getSecurityTokenReference(WSDocInfo.java:86) 

[runTest(bash)] OUT>      at
org.apache.ws.security.message.EnvelopeIdResolver.engineResolve(EnvelopeIdResolver.java:114) 

[runTest(bash)] OUT>      at
org.apache.xml.security.utils.resolver.ResourceResolver.resolve(Unknown
Source)
[runTest(bash)] OUT>      at
org.apache.xml.security.signature.Reference.getContentsBeforeTransformation(Unknown 

Source)
[runTest(bash)] OUT>      at
org.apache.xml.security.signature.Reference.dereferenceURIandPerformTransforms(Unknown 

Source)
[runTest(bash)] OUT>      at
org.apache.xml.security.signature.Reference.calculateDigest(Unknown 
Source)
[runTest(bash)] OUT>      at
org.apache.xml.security.signature.Reference.verify(Unknown Source)
[runTest(bash)] OUT>      at
org.apache.xml.security.signature.Manifest.verifyReferences(Unknown 
Source)
[runTest(bash)] OUT>      at
org.apache.xml.security.signature.SignedInfo.verify(Unknown Source)
[runTest(bash)] OUT>      at
org.apache.xml.security.signature.XMLSignature.checkSignatureValue(Unknown 

Source)
[runTest(bash)] OUT>      at
org.apache.xml.security.signature.XMLSignature.checkSignatureValue(Unknown 

Source)
[runTest(bash)] OUT>      at
org.apache.ws.security.processor.SignatureProcessor.verifyXMLSignature(SignatureProcessor.java:470) 

[runTest(bash)] OUT>      at
org.apache.ws.security.processor.SignatureProcessor.handleToken(SignatureProcessor.java:114) 

[runTest(bash)] OUT>      at
org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:328) 

[runTest(bash)] OUT>      at
org.apache.ws.security.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:245) 

[runTest(bash)] OUT>      at
org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:208) 

[runTest(bash)] OUT>      at
org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor.handleMessage(WSS4JInInterceptor.java:78) 

[runTest(bash)] OUT>      at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243) 

[runTest(bash)] OUT>      at
org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:755)
[runTest(bash)] OUT>      at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:2408) 

[runTest(bash)] OUT>      at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:2278) 

[runTest(bash)] OUT>      at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:2121) 

[runTest(bash)] OUT>      at
org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:66)
[runTest(bash)] OUT>      at
org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:695)
[runTest(bash)] OUT>      at
org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) 

[runTest(bash)] OUT>      at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:243) 

[runTest(bash)] OUT>      at
org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:516)
[runTest(bash)] OUT>      at
org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:313)
[runTest(bash)] OUT>      at
org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:265)
[runTest(bash)] OUT>      at
org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
[runTest(bash)] OUT>      at
org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)
[runTest(bash)] OUT>      ... 9 more


Interestingly, forcing the use of WSS4J 1.5.8 instead (we discovered 
this by
accident ;-) ) the problem goes away.
The application basically builds up a JAXWS client using WS-Security 
(Sign
only) through programmatically configured WSS4J interceptors:
http://fpaste.org/pYHM/
Once the proxy is ready on client side, it's called concurrently using
multiple threads (performIteration() method in the code).
Does the problem/exception above ring any bell? I'm going to 
investigate
this a bit more in any case.
Thanks
Alessio

--
Alessio Soldano
Web Service Lead, JBoss








--
Alessio Soldano
Web Service Lead, JBoss