Sorry for the lack of details. I wrote the email late at night, as I am again.
Anyway, the relevant code in github is OSMImporter.java<https://github.com/neo4j/neo4j-spatial/blob/master/src/main/java/org/neo4j/gis/spatial/osm/OSMImporter.java>. When adding nodes to the graph, it also adds the osm-id to a lucene index. There is no index#removal call, only multiple index#add calls within the same transaction. In fact we call index.add and index.get for one index (osm changesets), while calling index.add on another (osm-nodes). The relevant lines of code are 812 for adding new OSM nodes to the graph, and 914 for finding changesets in a different index. I have not investigated for which version of neo4j the slowdown started, or if there is somehow some other cause. I will try find time to do that later this week. But I thought I should ask on the list anyway in case anyone else has a similar problem, or if there are some obvious answers. On Sun, Jun 26, 2011 at 1:45 PM, Mattias Persson <matt...@neotechnology.com>wrote: > Please elaborate on how you are using your index. Are you using > Index#remove(entity,key) or Index#remove(entity) followed by get/query in > the same tx? There was a recent change in transactional state > implementation, where a full representation (in-memory lucene index) was > needed for it to be able to return accurate results in some corner cases. > That change could slow things down, but not that much though. I'll give > some > different scenarios a go and see if I can find some culprit for this. > > But again, a little more information would be useful, as always. > > 2011/6/26 Craig Taverner <cr...@amanzi.com> > > > Hi, > > > > Has anyone noticed a slowdown of imports into neo4j with recent > snapshots? > > Neo4j-spatial importing OSM data (which uses lucene to find matching > nodes > > for ways) is suddenly running much slower than usual on non-batch > imports. > > For most of my medium sized test cases, I normally have surprisingly > > similar > > import times for batch inserter and non-batch inserter > > (EmbeddedGraphDatabase) versions of the OSM import, but in recent runs > the > > normal API is now more than 10 times slower. Down to 70 nodes per second, > > which is insanely slow. > > > > Any idea if there is something in the recent snapshots for me to look > into? > > Reproducing the problem requires simply running the TestOSMImport test > > cases > > in neo4j-spatial. I have only tried this on my laptop, so I have not > ruled > > out that there is something local going on. > > > > Regards, Craig > > _______________________________________________ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > -- > Mattias Persson, [matt...@neotechnology.com] > Hacker, Neo Technology > www.neotechnology.com > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
