So the current status is that David has got neo4j REST running on Jetty with all tests passing. We've also searched through the code, and found that there are no interrupt() calls in the jersey source, while there are a few on the grizzly side. There is one in particular that we have been looking at, related to keep-alive timeouts, that may be the culprit. If that was the problem, we've got a fix for it.
We have, however, been unable to recreate the problem so far, so we can't tell if we've solved it or not :) Brock: could you give us an idea of what types of requests you were throwing at the server, and a rough estimate of how many? /Jacob On Wed, Aug 11, 2010 at 2:35 PM, Jacob Hansson <ja...@voltvoodoo.com> wrote: > Hi all! > > Johan took a look at the stack trace, and explained the problem. What > happens is that something, either the Grizzly server or the jersey wrapper > calls Thread.interrupt() on one of the neo4j threads (which should be > considered a bug in whichever one of them does that). This triggers an > IOError deep down in neo4j, which in turn causes the rest of the problems. > > I'm working on recreating the situation, and David is working on switching > the REST system over to run on Jetty instead of Grizzly. We'll keep you > posted on the progress. > > /Jacob > > > On Wed, Aug 11, 2010 at 1:51 PM, Peter Neubauer < > peter.neuba...@neotechnology.com> wrote: > >> Nice, >> will try that out Jim! Grinder seems cool. >> >> Cheers, >> >> /peter neubauer >> >> COO and Sales, Neo Technology >> >> GTalk: neubauer.peter >> Skype peter.neubauer >> Phone +46 704 106975 >> LinkedIn http://www.linkedin.com/in/neubauer >> Twitter http://twitter.com/peterneubauer >> >> http://www.neo4j.org - Your high performance graph >> database. >> http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. >> >> >> >> On Wed, Aug 11, 2010 at 12:52 PM, Jim Webber <j...@webber.name> wrote: >> > Perhaps something as simple as a Grinder script might help? >> > >> > Jim >> > >> > >> > On 11 Aug 2010, at 17:57, Brock Rousseau wrote: >> > >> >> Thanks Peter. Let us know if there is anything else we can provide in >> the >> >> way of logs or diagnosis from our server. >> >> >> >> -Brock >> >> >> >> On Tue, Aug 10, 2010 at 11:51 PM, Peter Neubauer < >> >> peter.neuba...@neotechnology.com> wrote: >> >> >> >>> Mmh, >> >>> seems we should stresstest the server and Grizzly with e.g. >> >>> http://www.soapui.org and see if we can reproduce the scenario, if >> >>> there is no obvious hint to this. Will try to set it up ... >> >>> >> >>> Cheers, >> >>> >> >>> /peter neubauer >> >>> >> >>> COO and Sales, Neo Technology >> >>> >> >>> GTalk: neubauer.peter >> >>> Skype peter.neubauer >> >>> Phone +46 704 106975 >> >>> LinkedIn http://www.linkedin.com/in/neubauer >> >>> Twitter http://twitter.com/peterneubauer >> >>> >> >>> http://www.neo4j.org - Your high performance graph >> database. >> >>> http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing >> party. >> >>> >> >>> >> >>> >> >>> On Wed, Aug 11, 2010 at 4:14 AM, Brock Rousseau <bro...@gmail.com> >> wrote: >> >>>> The patch worked perfectly for increasing the concurrent transaction >> cap, >> >>>> but unfortunately exposed another issue. >> >>>> >> >>>> After increasing the load hitting our rest server, it performs >> smoothly >> >>> for >> >>>> 10-15 minutes then begins issuing 500 responses on all transactions. >> When >> >>> it >> >>>> happens, the number of open transactions freezes in JMX and the heap >> size >> >>>> essentially remains static. Below are the two stack traces we see in >> the >> >>>> wrapper.log. Here are what i think to be the relevant configuration >> >>> lines: >> >>>> >> >>>> wrapper.conf: >> >>>> wrapper.java.additional.1=-d64 >> >>>> wrapper.java.additional.2=-server >> >>>> wrapper.java.additional.4=-Xmx8192m >> >>>> wrapper.java.additional.3=-XX:+UseConcMarkSweepGC >> >>>> wrapper.java.additional.4=-Dcom.sun.management.jmxremote >> >>>> >> >>>> neo4j.properties: >> >>>> >> >>>> rest_min_grizzly_threads=4 >> >>>> rest_max_grizzly_threads=128 >> >>>> >> >>>> neostore.nodestore.db.mapped_memory=4000M >> >>>> neostore.relationshipstore.db.mapped_memory=40000M >> >>>> neostore.propertystore.db.mapped_memory=1800M >> >>>> neostore.propertystore.db.index.mapped_memory=100M >> >>>> neostore.propertystore.db.index.keys.mapped_memory=100M >> >>>> neostore.propertystore.db.strings.mapped_memory=3G >> >>>> neostore.propertystore.db.arrays.mapped_memory=0M >> >>>> >> >>>> The server has 64Gb of total RAM so there should be a little over 6 >> left >> >>> for >> >>>> the system. >> >>>> >> >>>> >> >>>> At the initial time of failure there are several of this error: >> >>>> >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | Aug 10, 2010 1:00:33 PM >> >>>> com.sun.grizzly.http.KeepAliveThreadAttachment timedOut >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | WARNING: Interrupting idle >> >>> Thread: >> >>>> Grizzly-9555-WorkerThread(1) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | Aug 10, 2010 1:00:33 PM >> >>>> com.sun.jersey.spi.container.ContainerResponse >> >>> mapMappableContainerException >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | SEVERE: The >> RuntimeException >> >>> could >> >>>> not be mapped to a response, re-throwing to the HTTP container >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | >> >>>> org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException: Unable >> to >> >>>> load position[7280476] @[968303308] >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> org.neo4j.kernel.impl.nioneo.store.PersistenceRow.readPosition(PersistenceRow.java:101) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:152) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:474) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> org.neo4j.kernel.impl.nioneo.store.AbstractDynamicStore.getLightRecords(AbstractDynamicStore.java:375) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> org.neo4j.kernel.impl.nioneo.store.PropertyStore.getRecord(PropertyStore.java:324) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.propertyGetValue(ReadTransaction.java:237) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> org.neo4j.kernel.impl.nioneo.xa.NioNeoDbPersistenceSource$ReadOnlyResourceConnection.loadPropertyValue(NioNeoDbPersistenceSource.java:216) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> org.neo4j.kernel.impl.persistence.PersistenceManager.loadPropertyValue(PersistenceManager.java:79) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> org.neo4j.kernel.impl.core.NodeManager.loadPropertyValue(NodeManager.java:579) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> org.neo4j.kernel.impl.core.Primitive.getPropertyValue(Primitive.java:546) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> org.neo4j.kernel.impl.core.Primitive.getProperty(Primitive.java:167) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> org.neo4j.kernel.impl.core.NodeProxy.getProperty(NodeProxy.java:134) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> org.neo4j.rest.domain.PropertiesMap.<init>(PropertiesMap.java:20) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> org.neo4j.rest.domain.NodeRepresentation.<init>(NodeRepresentation.java:20) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> org.neo4j.rest.domain.StorageActions$TraverserReturnType$1.toRepresentation(StorageActions.java:421) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> org.neo4j.rest.domain.StorageActions.traverseAndCollect(StorageActions.java:403) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> org.neo4j.rest.web.GenericWebService.traverse(GenericWebService.java:725) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> org.neo4j.rest.web.JsonAndHtmlWebService.jsonTraverse(JsonAndHtmlWebService.java:324) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> java.lang.reflect.Method.invoke(Method.java:616) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:184) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:276) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:83) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:71) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1171) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1103) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1053) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1043) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:406) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:477) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:662) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.grizzly.http.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:188) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.grizzly.http.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:137) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.grizzly.http.servlet.ServletAdapter.doService(ServletAdapter.java:376) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.grizzly.http.servlet.ServletAdapter.service(ServletAdapter.java:324) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.grizzly.tcp.http11.GrizzlyAdapterChain.service(GrizzlyAdapterChain.java:180) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:753) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:661) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:914) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:166) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:166) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> com.sun.grizzly.ContextTask.run(ContextTask.java:69) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.grizzly.util.FixedThreadPool$BasicWorker.dowork(FixedThreadPool.java:379) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> com.sun.grizzly.util.FixedThreadPool$BasicWorker.run(FixedThreadPool.java:360) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> java.lang.Thread.run(Thread.java:636) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | Caused by: >> >>>> java.nio.channels.ClosedByInterruptException >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:201) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:335) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | at >> >>>> >> >>> >> org.neo4j.kernel.impl.nioneo.store.PersistenceRow.readPosition(PersistenceRow.java:80) >> >>>> INFO | jvm 1 | 2010/08/10 13:00:33 | ... 56 more >> >>>> >> >>>> Then this to this for every subsequent request: >> >>>> >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | Aug 10, 2010 1:02:13 PM >> >>>> com.sun.jersey.spi.container.ContainerResponse >> >>> mapMappableContainerException >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | SEVERE: The >> RuntimeException >> >>> could >> >>>> not be mapped to a response, re-throwing to the HTTP container >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | >> >>>> org.neo4j.graphdb.TransactionFailureException: Unable to begin >> >>> transaction >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> org.neo4j.kernel.EmbeddedGraphDbImpl.beginTx(EmbeddedGraphDbImpl.java:334) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> org.neo4j.kernel.EmbeddedGraphDatabase.beginTx(EmbeddedGraphDatabase.java:142) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> org.neo4j.rest.domain.StorageActions.createNode(StorageActions.java:46) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> org.neo4j.rest.web.GenericWebService.createNode(GenericWebService.java:147) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> org.neo4j.rest.web.JsonAndHtmlWebService.jsonCreateNode(JsonAndHtmlWebService.java:90) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> java.lang.reflect.Method.invoke(Method.java:616) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:184) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:276) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:83) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:133) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:71) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1171) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1103) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1053) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1043) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:406) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:477) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:662) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.grizzly.http.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:188) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.grizzly.http.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:137) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.grizzly.http.servlet.ServletAdapter.doService(ServletAdapter.java:376) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.grizzly.http.servlet.ServletAdapter.service(ServletAdapter.java:324) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.grizzly.tcp.http11.GrizzlyAdapterChain.service(GrizzlyAdapterChain.java:180) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:166) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:753) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:661) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:914) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:166) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> com.sun.grizzly.ContextTask.run(ContextTask.java:69) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.grizzly.util.FixedThreadPool$BasicWorker.dowork(FixedThreadPool.java:379) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> com.sun.grizzly.util.FixedThreadPool$BasicWorker.run(FixedThreadPool.java:360) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> java.lang.Thread.run(Thread.java:636) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | Caused by: >> >>>> javax.transaction.SystemException: TM has encountered some problem, >> >>> please >> >>>> perform neccesary action (tx recovery/restart) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> org.neo4j.kernel.impl.transaction.TxManager.begin(TxManager.java:497) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | at >> >>>> >> >>> >> org.neo4j.kernel.EmbeddedGraphDbImpl.beginTx(EmbeddedGraphDbImpl.java:329) >> >>>> INFO | jvm 1 | 2010/08/10 13:02:13 | ... 43 more >> >>>> >> >>>> Do those look like they point to anything in particular? As always >> your >> >>>> insight is greatly appreciated! >> >>>> >> >>>> Thanks, >> >>>> Brock >> >>>> >> >>>> On Tue, Aug 10, 2010 at 7:53 AM, Jacob Hansson <ja...@voltvoodoo.com >> > >> >>> wrote: >> >>>> >> >>>>> The problem with the resource starvation turned out to be fairly >> simple, >> >>>>> and >> >>>>> was due to a missing "connection:close" header in the neo4j REST >> client >> >>> I >> >>>>> was using. >> >>>>> >> >>>>> It would be interesting to look into the "connection:keep-alive" >> >>> behaviour >> >>>>> of Grizzly, using HTTP 1.1 pipelining would be an excellent way to >> >>> increase >> >>>>> performance in the clients, but the few things I've read so far >> suggests >> >>>>> Grizzly does not yet support that. >> >>>>> >> >>>>> /Jacob >> >>>>> >> >>>>> On Tue, Aug 10, 2010 at 11:19 AM, Brock Rousseau <bro...@gmail.com> >> >>> wrote: >> >>>>> >> >>>>>> The patched trunk code is working fine after dropping it over our >> >>>>> existing >> >>>>>> deployment. I'm going to wait until we have the support of our >> >>> site-speed >> >>>>>> engineers in the morning before testing the transaction limit under >> >>>>>> full production load, but I'll post the results as soon as we have >> >>> them. >> >>>>>> >> >>>>>> Thanks, >> >>>>>> Brock >> >>>>>> >> >>>>>> >> >>>>>> On Tue, Aug 10, 2010 at 1:47 AM, Brock Rousseau <bro...@gmail.com> >> >>>>> wrote: >> >>>>>> >> >>>>>>> I got the patch Jacob, thanks! >> >>>>>>> >> >>>>>>> We're running Centos 5.5, Java 1.6.0, and Maven 2.2.1 >> >>>>>>> >> >>>>>>> If it's fine for you guys then as you said, likely some >> >>> configuration >> >>>>>>> difference with our system, so i'll just run it >> >>>>>> with"-Dmaven.test.skip=true" >> >>>>>>> >> >>>>>>> Applying the patch now, i'll let you know how it goes. Thanks >> again >> >>> for >> >>>>>> all >> >>>>>>> the support. >> >>>>>>> >> >>>>>>> -Brock >> >>>>>>> >> >>>>>>> On Tue, Aug 10, 2010 at 1:37 AM, Anders Nawroth < >> >>>>>>> and...@neotechnology.com> wrote: >> >>>>>>> >> >>>>>>>> hi! >> >>>>>>>> >> >>>>>>>> On 08/10/2010 10:06 AM, Brock Rousseau wrote: >> >>>>>>>>> The standalone project built fine for me, but we're getting >> these >> >>>>> test >> >>>>>>>>> failures for mvn install on the rest component: >> >>>>>>>> >> >>>>>>>> Try using Java 6 or skip the tests. >> >>>>>>>> >> >>>>>>>> From the pom.xml: >> >>>>>>>> <!-- NOTICE: Tests will not run on standard Java 5! >> >>>>>>>> This is due to the fact that there is no easily >> >>>>>> available >> >>>>>>>> implementation of the javax.script API for Java 5 >> >>> in >> >>>>>> any >> >>>>>>>> maven repositories. For building on Java 5, >> either >> >>>>> skip >> >>>>>>>> running the tests, or include the javax.script js >> >>>>>>>> reference >> >>>>>>>> implementation from >> >>>>>> http://jcp.org/en/jsr/detail?id=223 >> >>>>>>>> in the classpath of the JVM. >> >>>>>>>> --> >> >>>>>>>> >> >>>>>>>> /anders >> >>>>>>>> _______________________________________________ >> >>>>>>>> Neo4j mailing list >> >>>>>>>> User@lists.neo4j.org >> >>>>>>>> https://lists.neo4j.org/mailman/listinfo/user >> >>>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>> _______________________________________________ >> >>>>>> Neo4j mailing list >> >>>>>> User@lists.neo4j.org >> >>>>>> https://lists.neo4j.org/mailman/listinfo/user >> >>>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> Jacob Hansson >> >>>>> Phone: +46 (0) 763503395 >> >>>>> Twitter: @jakewins >> >>>>> _______________________________________________ >> >>>>> Neo4j mailing list >> >>>>> User@lists.neo4j.org >> >>>>> https://lists.neo4j.org/mailman/listinfo/user >> >>>>> >> >>>> _______________________________________________ >> >>>> Neo4j mailing list >> >>>> User@lists.neo4j.org >> >>>> https://lists.neo4j.org/mailman/listinfo/user >> >>>> >> >>> _______________________________________________ >> >>> Neo4j mailing list >> >>> User@lists.neo4j.org >> >>> https://lists.neo4j.org/mailman/listinfo/user >> >>> >> >> _______________________________________________ >> >> Neo4j mailing list >> >> User@lists.neo4j.org >> >> https://lists.neo4j.org/mailman/listinfo/user >> > >> > _______________________________________________ >> > Neo4j mailing list >> > User@lists.neo4j.org >> > https://lists.neo4j.org/mailman/listinfo/user >> > >> _______________________________________________ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> > > > > -- > Jacob Hansson > Phone: +46 (0) 763503395 > Twitter: @jakewins > -- Jacob Hansson Phone: +46 (0) 763503395 Twitter: @jakewins _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
