Hi Vincent, Thanks for the report. This does appear to be thread-unsafe code, and it hasn't changed significantly since 3.2 so I expect this to be an issue with 3.5 as well. I've created this issue to track it:
https://jira.duraspace.org/browse/FCREPO-1075 At first glance, it seems that changing the type of m_currentResults to a ConcurrentHashMap would prevent this from happening. That is, in FieldSearchSQLImpl.java, change the following: import java.util.HashMap; // ... private final HashMap<String, FieldSearchResultSQLImpl> m_currentResults = new HashMap<String, FieldSearchResultSQLImpl>(); to this: import java.util.Map; import java.util.concurrent.ConcurrentHashMap; // ... private final Map<String, FieldSearchResultSQLImpl> m_currentResults = new ConcurrentHashMap<String, FieldSearchResultSQLImpl>(); I have compiled an alternate version of the Fedora 3.2-released revision of that file, which includes the above changes, and have attached the .class file to the JIRA issue above. Can you try re-running your test after putting the new .class file into your tomcat/webapps/fedora/WEB-INF/classes/fedora/server/search/ directory and restarting Fedora? Thanks, Chris On Wed, Mar 7, 2012 at 9:25 AM, Nguyen, Vincent (CDC/OD/OADS) (CTR) <[email protected]> wrote: > Hi, I’m seeing this error in the log when we try to run JMeter tests on our > instance of fedora with about 500 concurrent users. Has anyone else > encountered this? > > > > We’re running fedora version 3.2 on Windows Server 2008 box with Tomcat 6. > We currently use Muradora as our front end. > > > > I initially thought the problem might be because “m_currentResults” was > being modified by one thread as another thread was trying to access it in > this method: > > > > // erase and cleanup expired stuff > > private void closeAndForgetOldResults() { > > Iterator<FieldSearchResultSQLImpl> iter = > > m_currentResults.values().iterator(); > > ArrayList<String> toRemove = new ArrayList<String>(); > > while (iter.hasNext()) { > > FieldSearchResultSQLImpl r = iter.next(); > > if (r.isExpired()) { > > LOG.debug("listSession " + r.getToken() > > + " expired; will forget it."); > > toRemove.add(r.getToken()); > > } > > } > > for (int i = 0; i < toRemove.size(); i++) { > > String token = toRemove.get(i); > > m_currentResults.remove(token); > > } > > } > > > > > > > > java.util.ConcurrentModificationException > > at java.util.HashMap$HashIterator.nextEntry(Unknown Source) > > at java.util.HashMap$ValueIterator.next(Unknown Source) > > at > fedora.server.search.FieldSearchSQLImpl.closeAndForgetOldResults(FieldSearchSQLImpl.java:387) > > at > fedora.server.search.FieldSearchSQLImpl.findObjects(FieldSearchSQLImpl.java:337) > > at > fedora.server.search.FieldSearchSQLModule.findObjects(FieldSearchSQLModule.java:184) > > at > fedora.server.storage.DefaultDOManager.findObjects(DefaultDOManager.java:1882) > > at > fedora.server.access.DefaultAccess.findObjects(DefaultAccess.java:694) > > at > fedora.server.access.FedoraAPIABindingSOAPHTTPImpl.findObjects(FedoraAPIABindingSOAPHTTPImpl.java:214) > > at > fedora.server.access.FedoraAPIABindingSOAPHTTPSkeleton.findObjects(FedoraAPIABindingSOAPHTTPSkeleton.java:198) > > at sun.reflect.GeneratedMethodAccessor203.invoke(Unknown Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at > org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397) > > at > org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186) > > at > org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323) > > at > org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32) > > at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118) > > at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83) > > at > org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:453) > > at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281) > > at > org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:637) > > at > org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:270) > > at > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:269) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAsPrivileged(Unknown Source) > > at > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:302) > > at > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:163) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:283) > > at > org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:56) > > at > org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189) > > at java.security.AccessController.doPrivileged(Native Method) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185) > > at > fedora.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:256) > > at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:270) > > at > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:269) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAsPrivileged(Unknown Source) > > at > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:302) > > at > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:243) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) > > at > org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:56) > > at > org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189) > > at java.security.AccessController.doPrivileged(Native Method) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185) > > at > fedora.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:256) > > at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:270) > > at > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:269) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAsPrivileged(Unknown Source) > > at > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:302) > > at > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:243) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) > > at > org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:56) > > at > org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189) > > at java.security.AccessController.doPrivileged(Native Method) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185) > > at > fedora.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:256) > > at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:270) > > at > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:269) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAsPrivileged(Unknown Source) > > at > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:302) > > at > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:243) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) > > at > org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:56) > > at > org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189) > > at java.security.AccessController.doPrivileged(Native Method) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185) > > at > fedora.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:256) > > at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:270) > > at > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:269) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAsPrivileged(Unknown Source) > > at > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:302) > > at > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:243) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) > > at > org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:56) > > at > org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189) > > at java.security.AccessController.doPrivileged(Native Method) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185) > > at > fedora.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:256) > > at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:270) > > at > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:269) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAsPrivileged(Unknown Source) > > at > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:302) > > at > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:243) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) > > at > org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:56) > > at > org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189) > > at java.security.AccessController.doPrivileged(Native Method) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185) > > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563) > > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615) > > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) > > at > org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:877) > > at > org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:594) > > at > org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1675) > > at java.lang.Thread.run(Unknown Source) > > > > > > > > Vincent Vu Nguyen > > > > > ------------------------------------------------------------------------------ > Virtualization & Cloud Management Using Capacity Planning > Cloud computing makes use of virtualization - but cloud computing > also focuses on allowing computing to be delivered as a service. > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > _______________________________________________ > Fedora-commons-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users > ------------------------------------------------------------------------------ Virtualization & Cloud Management Using Capacity Planning Cloud computing makes use of virtualization - but cloud computing also focuses on allowing computing to be delivered as a service. http://www.accelacomm.com/jaw/sfnl/114/51521223/ _______________________________________________ Fedora-commons-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
