Nolan, what memory settings are you running the code with, and do you have any neo4j.properties tuned somehwere?
Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Fri, May 13, 2011 at 8:25 AM, Nolan Darilek <no...@thewordnerd.info>wrote: > I'm importing the dataset for Texas. My version is a few weeks old, but > you can find the newest here: > > > http://downloads.cloudmade.com/americas/northern_america/united_states/texas/texas.osm.bz2 > > My import code, more or less, let me know if you need more > implementation details: > > class Neo4jImport(filename:String, layer:String = "map") extends Import { > > val importer = new OSMImporter(layer) > > private var processed = 0 > > def processedEntities = processed > > private def ds = dataset.asInstanceOf[Neo4JDataSet] > private def database = ds.database > > class MyBatchInserter extends BatchInserterImpl(database.getStoreDir) { > > override def createNode(properties:JMap[String, Object]) = { > processed += 1 > super.createNode(properties) > } > > override def createNode(id:Long, properties:JMap[String, Object]){ > super.createNode(id, properties) > processed += 1 > } > > override def createRelationship(n1:Long, n2:Long, > rt:RelationshipType, properties:JMap[String, Object]) = { > processed += 1 > super.createRelationship(n1, n2, rt, properties) > } > > } > > def performImport() { > database.shutdown() > val batchInserter = new MyBatchInserter > importer.importFile(batchInserter, filename) > batchInserter.shutdown() > ds.init(true) > importer.reIndex(database, 1000) > } > > } > > Console output: > > Fri May 13 10:22:20 CDT 2011: Saving node 6525309 > (13713.904715468341 node/second) > Fri May 13 10:22:21 CDT 2011: Saving node 6539916 > (13703.333682556313 node/second) > java.lang.OutOfMemoryError: Java heap space > Dumping heap to java_pid13506.hprof ... > Heap dump file created [1426787760 bytes in 30.001 secs] > scala.actors.Actor$$anon$1@764e2837: caught > java.lang.IllegalStateException: this writer hit an OutOfMemoryError; > cannot flush > java.lang.IllegalStateException: this writer hit an OutOfMemoryError; > cannot flush > at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3307) > at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3296) > at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2376) > at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2352) > at > > org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.closeWriter(LuceneBatchInserterIndex.java:279) > > at > > org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.shutdown(LuceneBatchInserterIndex.java:354) > > at > > org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.shutdown(LuceneBatchInserterIndexProvider.java:145) > > at > > org.neo4j.gis.spatial.osm.OSMImporter$OSMBatchWriter.finish(OSMImporter.java:1144) > > at > org.neo4j.gis.spatial.osm.OSMImporter.importFile(OSMImporter.java:1320) > at > org.neo4j.gis.spatial.osm.OSMImporter.importFile(OSMImporter.java:1219) > at > org.neo4j.gis.spatial.osm.OSMImporter.importFile(OSMImporter.java:1215) > at > > info.hermesnav.core.model.data.impl.neo4j.Neo4jImport.performImport(neo4j.scala:54) > > at > > info.hermesnav.core.model.data.Import$$anonfun$start$1.apply$mcV$sp(data.scala:25) > > at scala.actors.Actor$$anon$1.act(Actor.scala:135) > at scala.actors.Reactor$$anonfun$dostart$1.apply(Reactor.scala:222) > at scala.actors.Reactor$$anonfun$dostart$1.apply(Reactor.scala:222) > at scala.actors.ReactorTask.run(ReactorTask.scala:36) > at > > scala.concurrent.forkjoin.ForkJoinPool$AdaptedRunnable.exec(ForkJoinPool.java:611) > > at > scala.concurrent.forkjoin.ForkJoinTask.quietlyExec(ForkJoinTask.java:422) > at > > scala.concurrent.forkjoin.ForkJoinWorkerThread.mainLoop(ForkJoinWorkerThread.java:340) > > at > > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:325) > > > > On 05/13/2011 09:34 AM, Peter Neubauer wrote: > > Nolan, > > do you have the importing code and what dataset are you importing? Also, > do > > you have any console output? It could be very big transactions or other > > database settings not adjusted to the size of your import ... > > > > Cheers, > > > > /peter neubauer > > > > GTalk: neubauer.peter > > Skype peter.neubauer > > Phone +46 704 106975 > > LinkedIn http://www.linkedin.com/in/neubauer > > Twitter http://twitter.com/peterneubauer > > > > http://www.neo4j.org - Your high performance graph > database. > > http://startupbootcamp.org/ - Öresund - Innovation happens HERE. > > http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. > > > > > > On Fri, May 13, 2011 at 4:13 AM, Nolan Darilek<no...@thewordnerd.info > >wrote: > > > >> Picking up my slow port to Neo4j Spatial again, and am hitting an > >> out-of-memory error when trying to import large datasets. Given that > >> this code works fine if I use a different database and swap out the > >> implementations, I suspect Neo4j as the issue. This is Neo4j > >> 1.4-SNAPSHOT and Spatial 0.6-SNAPSHOT. > >> > >> Not sure if this is enough to diagnose the issue, but I have a heap > dump: > >> > >> http://dl.dropbox.com/u/147071/java_pid7405.hprof.bz2 > >> > >> It's currently uploading to Dropbox, so maybe grab it in an hour or two. > >> It's something like 185M compressed, 1.5G uncompressed. > >> > >> Thanks, please let me know if I might provide any more details. > >> The exact error was "this writer hit an OutOfMemoryError; cannot flush", > >> at org.apache.lucene.IndexWriter.doFlush():3307. Not sure how relevant > >> the stacktrace is, as my experience with OutOfMemoryErrors is that the > >> code just fails wherever. > >> _______________________________________________ > >> Neo4j mailing list > >> User@lists.neo4j.org > >> https://lists.neo4j.org/mailman/listinfo/user > >> > > _______________________________________________ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
