Hi Enrico, Yes, we use transactions. In each transaction, we create up-to a couple thousand vertexes and have light weight edges created in the process too.
On Mon, Jan 5, 2015 at 9:30 AM, Enrico Risa <enrico.r...@gmail.com> wrote: > Hi Jing > > i Use Yourkit java profiler > > are you using Transactions? > > how many records in insert? > > 2015-01-05 18:27 GMT+01:00 Jing Chen <jingjing...@gmail.com>: > >> Hi Enrico, >> >> Yes, we are executing massive insert using the Java API directly. >> >> What profiling tool do you use? >> >> Thanks, >> Jing >> >> On Tuesday, December 30, 2014 10:47:23 AM UTC-8, Enrico Risa wrote: >>> >>> Hi Jing >>> >>> are you executing massive insert behind >>> AdminService.processTasks >>> >>> Can you send me some profiling session? >>> >>> 2014-12-30 19:05 GMT+01:00 Jing Chen <jingj...@gmail.com>: >>> >>>> Thanks for the quick response Enrico. Our application is a tomcat web >>>> application running on a Amazon Redhat 7 instance which has 3.5G memory. We >>>> have 3G allocated to tomcat JVM. Here are the error message from the tomcat >>>> log: >>>> >>>> 2014-12-21 05:32:19:694 INFO Deleting partial network >>>> 31fdf743-88f5-11e4-a88d-0aa4c1de39d1 in order to rollback in response >>>> to error [NdexPersistenceService]Exception in thread "Lucene Merge >>>> Thread #11" org.apache.lucene.index.MergePolicy$MergeException: >>>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>>> >>>> at org.apache.lucene.index.ConcurrentMergeScheduler. >>>> handleMergeException(ConcurrentMergeScheduler.java:545) >>>> >>>> at org.apache.lucene.index.ConcurrentMergeScheduler$ >>>> MergeThread.run(ConcurrentMergeScheduler.java:518) >>>> >>>> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded >>>> >>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: >>>> GC overhead limit exceeded >>>> >>>> at java.util.concurrent.FutureTask.report(Unknown Source) >>>> >>>> at java.util.concurrent.FutureTask.get(Unknown Source) >>>> >>>> at org.ndexbio.task.NdexTaskExecutor.call( >>>> NdexTaskExecutor.java:69) >>>> >>>> at org.ndexbio.task.NdexTaskExecutor.call( >>>> NdexTaskExecutor.java:32) >>>> >>>> at java.util.concurrent.FutureTask.run(Unknown Source) >>>> >>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >>>> Source) >>>> >>>> at java.util.concurrent.FutureTask.run(Unknown Source) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown >>>> Source) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >>>> Source) >>>> >>>> at java.lang.Thread.run(Unknown Source) >>>> >>>> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded >>>> >>>> java.util.concurrent.ExecutionException: java.lang.Exception: Error >>>> occured when executing task. java.lang.OutOfMemoryError: GC overhead limit >>>> exceeded >>>> >>>> at java.util.concurrent.FutureTask.report(Unknown Source) >>>> >>>> at java.util.concurrent.FutureTask.get(Unknown Source) >>>> >>>> at org.ndexbio.task.NdexQueuedTaskProcessor.processQueuedTasks( >>>> NdexQueuedTaskProcessor.java:101) >>>> >>>> at org.ndexbio.task.NdexQueuedTaskProcessor.processAll( >>>> NdexQueuedTaskProcessor.java:176) >>>> >>>> at org.ndexbio.rest.services.AdminService.processTasks( >>>> AdminService.java:103) >>>> >>>> at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) >>>> >>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >>>> Source) >>>> >>>> at java.lang.reflect.Method.invoke(Unknown Source) >>>> >>>> at org.jboss.resteasy.core.MethodInjectorImpl.invoke( >>>> MethodInjectorImpl.java:137) >>>> >>>> at org.jboss.resteasy.core.ResourceMethodInvoker. >>>> invokeOnTarget(ResourceMethodInvoker.java:296) >>>> >>>> at org.jboss.resteasy.core.ResourceMethodInvoker.invoke( >>>> ResourceMethodInvoker.java:250) >>>> >>>> at org.jboss.resteasy.core.ResourceMethodInvoker.invoke( >>>> ResourceMethodInvoker.java:237) >>>> >>>> at org.jboss.resteasy.core.SynchronousDispatcher.invoke( >>>> SynchronousDispatcher.java:356) >>>> >>>> at org.jboss.resteasy.core.SynchronousDispatcher.invoke( >>>> SynchronousDispatcher.java:179) >>>> >>>> at org.jboss.resteasy.plugins.server.servlet. >>>> ServletContainerDispatcher.service(ServletContainerDispatcher.java:220) >>>> >>>> at org.jboss.resteasy.plugins.server.servlet. >>>> HttpServletDispatcher.service(HttpServletDispatcher.java:56) >>>> >>>> at org.jboss.resteasy.plugins.server.servlet. >>>> HttpServletDispatcher.service(HttpServletDispatcher.java:51) >>>> >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) >>>> >>>> at org.apache.catalina.core.ApplicationFilterChain. >>>> internalDoFilter(ApplicationFilterChain.java:303) >>>> >>>> at org.apache.catalina.core.ApplicationFilterChain.doFilter( >>>> ApplicationFilterChain.java:208) >>>> >>>> at org.apache.tomcat.websocket.server.WsFilter.doFilter( >>>> WsFilter.java:52) >>>> >>>> at org.apache.catalina.core.ApplicationFilterChain. >>>> internalDoFilter(ApplicationFilterChain.java:241) >>>> >>>> at org.apache.catalina.core.ApplicationFilterChain.doFilter( >>>> ApplicationFilterChain.java:208) >>>> >>>> at org.apache.catalina.core.StandardWrapperValve.invoke( >>>> StandardWrapperValve.java:220) >>>> >>>> at org.apache.catalina.core.StandardContextValve.invoke( >>>> StandardContextValve.java:122) >>>> >>>> at org.apache.catalina.authenticator.AuthenticatorBase.invoke( >>>> AuthenticatorBase.java:501) >>>> >>>> at org.apache.catalina.core.StandardHostValve.invoke( >>>> StandardHostValve.java:171) >>>> >>>> at org.apache.catalina.valves.ErrorReportValve.invoke( >>>> ErrorReportValve.java:103) >>>> >>>> at org.apache.catalina.valves.AccessLogValve.invoke( >>>> AccessLogValve.java:950) >>>> >>>> at org.apache.catalina.core.StandardEngineValve.invoke( >>>> StandardEngineValve.java:116) >>>> >>>> at org.apache.catalina.connector.CoyoteAdapter.service( >>>> CoyoteAdapter.java:408) >>>> >>>> at org.apache.coyote.http11.AbstractHttp11Processor.process( >>>> AbstractHttp11Processor.java:1070) >>>> >>>> at org.apache.coyote.AbstractProtocol$ >>>> AbstractConnectionHandler.process(AbstractProtocol.java:611) >>>> >>>> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor. >>>> run(JIoEndpoint.java:314) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown >>>> Source) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >>>> Source) >>>> >>>> ... 1 more >>>> >>>> java.lang.IllegalStateException: this writer hit an OutOfMemoryError; >>>> cannot commit >>>> >>>> at org.apache.lucene.index.IndexWriter.prepareCommitInternal( >>>> IndexWriter.java:2842) >>>> >>>> at org.apache.lucene.index.IndexWriter.commitInternal( >>>> IndexWriter.java:3022) >>>> >>>> at org.apache.lucene.index.IndexWriter.commit( >>>> IndexWriter.java:2989) >>>> >>>> at com.orientechnologies.lucene.manager. >>>> OLuceneIndexManagerAbstract.commit(OLuceneIndexManagerAbstract. >>>> java:118) >>>> >>>> at com.orientechnologies.lucene.OLuceneIndexEngine.closeDb( >>>> OLuceneIndexEngine.java:139) >>>> >>>> at com.orientechnologies.orient.core.index.OIndexAbstract. >>>> onClose(OIndexAbstract.java:802) >>>> >>>> at com.orientechnologies.orient.core.db.record. >>>> ODatabaseRecordAbstract.callOnCloseListeners( >>>> ODatabaseRecordAbstract.java:1536) >>>> >>>> at com.orientechnologies.orient.core.db.document. >>>> ODatabaseDocumentTxPooled.close(ODatabaseDocumentTxPooled.java:112) >>>> >>>> at org.ndexbio.common.access.NdexDatabase.close( >>>> NdexDatabase.java:82) >>>> >>>> at org.ndexbio.rest.services.AdminService.processTasks( >>>> AdminService.java:109) >>>> >>>> at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) >>>> >>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >>>> Source) >>>> >>>> at java.lang.reflect.Method.invoke(Unknown Source) >>>> >>>> at org.jboss.resteasy.core.MethodInjectorImpl.invoke( >>>> MethodInjectorImpl.java:137) >>>> >>>> at org.jboss.resteasy.core.ResourceMethodInvoker. >>>> invokeOnTarget(ResourceMethodInvoker.java:296) >>>> >>>> at org.jboss.resteasy.core.ResourceMethodInvoker.invoke( >>>> ResourceMethodInvoker.java:250) >>>> >>>> at org.jboss.resteasy.core.ResourceMethodInvoker.invoke( >>>> ResourceMethodInvoker.java:237) >>>> >>>> at org.jboss.resteasy.core.SynchronousDispatcher.invoke( >>>> SynchronousDispatcher.java:356) >>>> >>>> at org.jboss.resteasy.core.SynchronousDispatcher.invoke( >>>> SynchronousDispatcher.java:179) >>>> >>>> at org.jboss.resteasy.plugins.server.servlet. >>>> ServletContainerDispatcher.service(ServletContainerDispatcher.java:220) >>>> >>>> at org.jboss.resteasy.plugins.server.servlet. >>>> HttpServletDispatcher.service(HttpServletDispatcher.java:56) >>>> >>>> at org.jboss.resteasy.plugins.server.servlet. >>>> HttpServletDispatcher.service(HttpServletDispatcher.java:51) >>>> >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) >>>> >>>> at org.apache.catalina.core.ApplicationFilterChain. >>>> internalDoFilter(ApplicationFilterChain.java:303) >>>> >>>> at org.apache.catalina.core.ApplicationFilterChain.doFilter( >>>> ApplicationFilterChain.java:208) >>>> >>>> at org.apache.tomcat.websocket.server.WsFilter.doFilter( >>>> WsFilter.java:52) >>>> >>>> at org.apache.catalina.core.ApplicationFilterChain. >>>> internalDoFilter(ApplicationFilterChain.java:241) >>>> >>>> at org.apache.catalina.core.ApplicationFilterChain.doFilter( >>>> ApplicationFilterChain.java:208) >>>> >>>> at org.apache.catalina.core.StandardWrapperValve.invoke( >>>> StandardWrapperValve.java:220) >>>> >>>> at org.apache.catalina.core.StandardContextValve.invoke( >>>> StandardContextValve.java:122) >>>> >>>> at org.apache.catalina.authenticator.AuthenticatorBase.invoke( >>>> AuthenticatorBase.java:501) >>>> >>>> at org.apache.catalina.core.StandardHostValve.invoke( >>>> StandardHostValve.java:171) >>>> >>>> at org.apache.catalina.valves.ErrorReportValve.invoke( >>>> ErrorReportValve.java:103) >>>> >>>> at org.apache.catalina.valves.AccessLogValve.invoke( >>>> AccessLogValve.java:950) >>>> >>>> at org.apache.catalina.core.StandardEngineValve.invoke( >>>> StandardEngineValve.java:116) >>>> >>>> at org.apache.catalina.connector.CoyoteAdapter.service( >>>> CoyoteAdapter.java:408) >>>> >>>> at org.apache.coyote.http11.AbstractHttp11Processor.process( >>>> AbstractHttp11Processor.java:1070) >>>> >>>> at org.apache.coyote.AbstractProtocol$ >>>> AbstractConnectionHandler.process(AbstractProtocol.java:611) >>>> >>>> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor. >>>> run(JIoEndpoint.java:314) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown >>>> Source) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >>>> Source) >>>> >>>> at org.apache.tomcat.util.threads.TaskThread$ >>>> WrappingRunnable.run(TaskThread.java:61) >>>> >>>> at java.lang.Thread.run(Unknown Source) >>>> >>>> java.lang.IllegalStateException: this writer hit an OutOfMemoryError; >>>> cannot commit >>>> >>>> at org.apache.lucene.index.IndexWriter.prepareCommitInternal( >>>> IndexWriter.java:2842) >>>> >>>> at org.apache.lucene.index.IndexWriter.commitInternal( >>>> IndexWriter.java:3022) >>>> >>>> at org.apache.lucene.index.IndexWriter.commit( >>>> IndexWriter.java:2989) >>>> >>>> at com.orientechnologies.lucene.manager. >>>> OLuceneIndexManagerAbstract.commit(OLuceneIndexManagerAbstract. >>>> java:118) >>>> >>>> at com.orientechnologies.lucene.OLuceneIndexEngine.closeDb( >>>> OLuceneIndexEngine.java:139) >>>> >>>> at com.orientechnologies.orient.core.index.OIndexAbstract. >>>> onClose(OIndexAbstract.java:802) >>>> >>>> at com.orientechnologies.orient.core.db.record. >>>> ODatabaseRecordAbstract.callOnCloseListeners( >>>> ODatabaseRecordAbstract.java:1536) >>>> >>>> at com.orientechnologies.orient.core.db.document. >>>> ODatabaseDocumentTxPooled.close(ODatabaseDocumentTxPooled.java:112) >>>> >>>> at org.ndexbio.rest.filters.BasicAuthenticationFilter.filter( >>>> BasicAuthenticationFilter.java:75) >>>> >>>> at org.jboss.resteasy.core.ResourceMethodInvoker. >>>> invokeOnTarget(ResourceMethodInvoker.java:264) >>>> >>>> at org.jboss.resteasy.core.ResourceMethodInvoker.invoke( >>>> ResourceMethodInvoker.java:250) >>>> >>>> at org.jboss.resteasy.core.ResourceMethodInvoker.invoke( >>>> ResourceMethodInvoker.java:237) >>>> >>>> at org.jboss.resteasy.core.SynchronousDispatcher.invoke( >>>> SynchronousDispatcher.java:356) >>>> >>>> at org.jboss.resteasy.core.SynchronousDispatcher.invoke( >>>> SynchronousDispatcher.java:179) >>>> >>>> at org.jboss.resteasy.plugins.server.servlet. >>>> ServletContainerDispatcher.service(ServletContainerDispatcher.java:220) >>>> >>>> at org.jboss.resteasy.plugins.server.servlet. >>>> HttpServletDispatcher.service(HttpServletDispatcher.java:56) >>>> >>>> at org.jboss.resteasy.plugins.server.servlet. >>>> HttpServletDispatcher.service(HttpServletDispatcher.java:51) >>>> >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) >>>> >>>> at org.apache.catalina.core.ApplicationFilterChain. >>>> internalDoFilter(ApplicationFilterChain.java:303) >>>> >>>> at org.apache.catalina.core.ApplicationFilterChain.doFilter( >>>> ApplicationFilterChain.java:208) >>>> >>>> at org.apache.tomcat.websocket.server.WsFilter.doFilter( >>>> WsFilter.java:52) >>>> >>>> at org.apache.catalina.core.ApplicationFilterChain. >>>> internalDoFilter(ApplicationFilterChain.java:241) >>>> >>>> at org.apache.catalina.core.ApplicationFilterChain.doFilter( >>>> ApplicationFilterChain.java:208) >>>> >>>> at org.apache.catalina.core.StandardWrapperValve.invoke( >>>> StandardWrapperValve.java:220) >>>> >>>> at org.apache.catalina.core.StandardContextValve.invoke( >>>> StandardContextValve.java:122) >>>> >>>> at org.apache.catalina.authenticator.AuthenticatorBase.invoke( >>>> AuthenticatorBase.java:501) >>>> >>>> at org.apache.catalina.core.StandardHostValve.invoke( >>>> StandardHostValve.java:171) >>>> >>>> at org.apache.catalina.valves.ErrorReportValve.invoke( >>>> ErrorReportValve.java:103) >>>> >>>> at org.apache.catalina.valves.AccessLogValve.invoke( >>>> AccessLogValve.java:950) >>>> >>>> at org.apache.catalina.core.StandardEngineValve.invoke( >>>> StandardEngineValve.java:116) >>>> >>>> at org.apache.catalina.connector.CoyoteAdapter.service( >>>> CoyoteAdapter.java:408) >>>> >>>> at org.apache.coyote.http11.AbstractHttp11Processor.process( >>>> AbstractHttp11Processor.java:1070) >>>> >>>> at org.apache.coyote.AbstractProtocol$ >>>> AbstractConnectionHandler.process(AbstractProtocol.java:611) >>>> >>>> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor. >>>> run(JIoEndpoint.java:314) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown >>>> Source) >>>> >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >>>> Source) >>>> >>>> at org.apache.tomcat.util.threads.TaskThread$ >>>> WrappingRunnable.run(TaskThread.java:61) >>>> >>>> at java.lang.Thread.run(Unknown Source) >>>> >>>> java.lang.IllegalStateException: this writer hit an OutOfMemoryError; >>>> cannot commit >>>> >>>> at org.apache.lucene.index.IndexWriter.prepareCommitInternal( >>>> IndexWriter.java:2842) >>>> >>>> at org.apache.lucene.index.IndexWriter.commitInternal( >>>> IndexWriter.java:3022) >>>> ... >>>> >>>> >>>> >>>> On Tuesday, December 30, 2014 12:52:50 AM UTC-7, Enrico Risa wrote: >>>>> >>>>> Hi Jin Chen >>>>> >>>>> can you tell me the configuration of your application? >>>>> >>>>> Do you have some log or profiling session? >>>>> >>>>> >>>>> >>>>> 2014-12-30 5:49 GMT+01:00 Jing Chen <jingj...@gmail.com>: >>>>> >>>>>> Hi there, >>>>>> >>>>>> We are using OrientDb 1.7.10 with Orient-Lucene index. We only use >>>>>> lucene text index. We use Orientdb as an embedded server in our web >>>>>> application. The application ran out of memory a few times recently. Our >>>>>> log shows that it was because Lucene can't allocate more memory. Is >>>>>> there a >>>>>> way to estimate the memory footprint of Orientdb-lucene index? When our >>>>>> database grows, does the memory requirement of Lucene index grow as well? >>>>>> >>>>>> Thanks advance for your answers. >>>>>> >>>>>> Jing Chen >>>>>> >>>>>> -- >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "OrientDB" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to orient-databa...@googlegroups.com. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "OrientDB" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to orient-databa...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "OrientDB" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to orient-database+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > > --- > You received this message because you are subscribed to the Google Groups > "OrientDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to orient-database+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to orient-database+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
