Sorry for the lack of details. I wrote the email late at night, as I am
again.

Anyway, the relevant code in github is
OSMImporter.java<https://github.com/neo4j/neo4j-spatial/blob/master/src/main/java/org/neo4j/gis/spatial/osm/OSMImporter.java>.
When adding nodes to the graph, it also adds the osm-id to a lucene index.
There is no index#removal call, only multiple index#add calls within the
same transaction. In fact we call index.add and index.get for one index (osm
changesets), while calling index.add on another (osm-nodes). The relevant
lines of code are 812 for adding new OSM nodes to the graph, and 914 for
finding changesets in a different index.

I have not investigated for which version of neo4j the slowdown started, or
if there is somehow some other cause. I will try find time to do that later
this week. But I thought I should ask on the list anyway in case anyone else
has a similar problem, or if there are some obvious answers.

On Sun, Jun 26, 2011 at 1:45 PM, Mattias Persson
<matt...@neotechnology.com>wrote:

> Please elaborate on how you are using your index. Are you using
> Index#remove(entity,key) or Index#remove(entity) followed by get/query in
> the same tx? There was a recent change in transactional state
> implementation, where a full representation (in-memory lucene index) was
> needed for it to be able to return accurate results in some corner cases.
> That change could slow things down, but not that much though. I'll give
> some
> different scenarios a go and see if I can find some culprit for this.
>
> But again, a little more information would be useful, as always.
>
> 2011/6/26 Craig Taverner <cr...@amanzi.com>
>
> > Hi,
> >
> > Has anyone noticed a slowdown of imports into neo4j with recent
> snapshots?
> > Neo4j-spatial importing OSM data (which uses lucene to find matching
> nodes
> > for ways) is suddenly running much slower than usual on non-batch
> imports.
> > For most of my medium sized test cases, I normally have surprisingly
> > similar
> > import times for batch inserter and non-batch inserter
> > (EmbeddedGraphDatabase) versions of the OSM import, but in recent runs
> the
> > normal API is now more than 10 times slower. Down to 70 nodes per second,
> > which is insanely slow.
> >
> > Any idea if there is something in the recent snapshots for me to look
> into?
> > Reproducing the problem requires simply running the TestOSMImport test
> > cases
> > in neo4j-spatial. I have only tried this on my laptop, so I have not
> ruled
> > out that there is something local going on.
> >
> > Regards, Craig
> > _______________________________________________
> > Neo4j mailing list
> > User@lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
> >
>
>
>
> --
> Mattias Persson, [matt...@neotechnology.com]
> Hacker, Neo Technology
> www.neotechnology.com
> _______________________________________________
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
_______________________________________________
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Reply via email to