Hi Mattias, I have already done it 10 minutes ago. If you need an example to see the format of the 4 csv files, I can send it to you. Thanks again,
Núria. 2009/12/9 Mattias Persson <matt...@neotechnology.com> > Oh ok, It could be our attachments filter / security or something... > could you try to mail them to me directly at matt...@neotechnology.com > ? > > 2009/12/9 Núria Trench <nuriatre...@gmail.com>: > > Hi Mattias, > > > > In my last e-mail I have attached the sample code, haven't you received > it? > > I will try to attach it again. > > > > Núria. > > > > 2009/12/9 Mattias Persson <matt...@neotechnology.com> > > > >> Hi again, Núria (it was I, Mattias who asked for the sample code). > >> Well... the fact that you parse 4 csv files doesn't really help me > >> setup a test for this... I mean how can I know that my test will be > >> similar to yours? Would it be ok to attach your code/csv files as > >> well? > >> > >> / Mattias > >> > >> 2009/12/9 Núria Trench <nuriatre...@gmail.com>: > >> > Hi Todd, > >> > > >> > The sample code creates nodes and relationships by parsing 4 csv > files. > >> > Thank you for trying to trigger this behaviour with this sample. > >> > > >> > Núria > >> > > >> > 2009/12/9 Mattias Persson <matt...@neotechnology.com> > >> > > >> >> Could you provide me with some sample code which can trigger this > >> >> behaviour with the latest index-util-0.9-SNAPSHOT Núria? > >> >> > >> >> 2009/12/9 Núria Trench <nuriatre...@gmail.com>: > >> >> > Todd, > >> >> > > >> >> > I haven't the same problem. In my case, after indexing all the > >> >> > attributes/properties of each node, the application creates all the > >> edges > >> >> by > >> >> > looking up the tail node and the head node. So, it calls the method > >> >> > "org.neo4j.util.index. > >> >> > LuceneIndexBatchInserterImpl.getSingleNode" which returns -1 (no > found > >> >> node) > >> >> > in many occasions. > >> >> > > >> >> > Any one has an alternative to get a node with indexex > >> >> attributes/properties? > >> >> > > >> >> > Thank you, > >> >> > > >> >> > Núria. > >> >> > > >> >> > > >> >> > 2009/12/7 Mattias Persson <matt...@neotechnology.com> > >> >> > > >> >> >> Todd, are you sure you have the latest index-util 0.9-SNAPSHOT? > This > >> >> >> is a bug that we fixed yesterday... (assuming it's the same bug). > >> >> >> > >> >> >> 2009/12/7 Todd Stavish <toddstav...@gmail.com>: > >> >> >> > Hi Mattias, Núria. > >> >> >> > > >> >> >> > I am also running into scalability problems with the Lucene > batch > >> >> >> > inserter at much smaller numbers, 30,000 indexed nodes. I tried > >> >> >> > calling optimize more. Increasing ulimit didn't help. > >> >> >> > > >> >> >> > INFO] Exception in thread "main" java.lang.RuntimeException: > >> >> >> > java.io.FileNotFoundException: > >> >> >> > > >> >> >> > >> >> > >> > /Users/todd/Code/neo4Jprototype/target/classes/data/graph/lucene/name/_0.cfx > >> >> >> > (Too many open files) > >> >> >> > [INFO] at > >> >> >> > >> >> > >> > org.neo4j.util.index.LuceneIndexBatchInserterImpl.getNodes(LuceneIndexBatchInserterImpl.java:186) > >> >> >> > [INFO] at > >> >> >> > >> >> > >> > org.neo4j.util.index.LuceneIndexBatchInserterImpl.getSingleNode(LuceneIndexBatchInserterImpl.java:238) > >> >> >> > [INFO] at > >> >> >> > >> com.collectiveintelligence.QueryNeo.loadDataToGraph(QueryNeo.java:277) > >> >> >> > [INFO] at > >> com.collectiveintelligence.QueryNeo.main(QueryNeo.java:57) > >> >> >> > [INFO] Caused by: java.io.FileNotFoundException: > >> >> >> > > >> >> >> > >> >> > >> > /Users/todd/Code/neo4Jprototype/target/classes/data/graph/lucene/name/_0.cfx > >> >> >> > (Too many open files) > >> >> >> > > >> >> >> > I tried breaking up to separate batchinserter instances, and it > >> hangs > >> >> >> > now. Can I create more than one batch inserter per process if > they > >> run > >> >> >> > sequentially and non-threaded? > >> >> >> > > >> >> >> > Thanks, > >> >> >> > Todd > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > On Mon, Dec 7, 2009 at 7:28 AM, Núria Trench < > >> nuriatre...@gmail.com> > >> >> >> wrote: > >> >> >> >> Hi again Mattias, > >> >> >> >> > >> >> >> >> I have tried to execute my application with the last version > >> >> available > >> >> >> in > >> >> >> >> the maven repository and I still have the same problem. After > >> >> creating > >> >> >> and > >> >> >> >> indexing all the nodes, the application calls the "optimize" > >> method > >> >> and, > >> >> >> >> then, it creates all the edges by calling the method "getNodes" > in > >> >> order > >> >> >> to > >> >> >> >> select the tail and head node of the edge, but it doesn't work > >> >> because > >> >> >> many > >> >> >> >> nodes are not found. > >> >> >> >> > >> >> >> >> I have tried to create only 30 nodes and 15 edges and it works > >> >> properly, > >> >> >> but > >> >> >> >> if I try to create a big graph (180 million edges + 20 million > >> nodes) > >> >> it > >> >> >> >> doesn't. > >> >> >> >> > >> >> >> >> I have also tried to call the "optimize" method every time the > >> >> >> application > >> >> >> >> has been created 1 million nodes but it doesn't work. > >> >> >> >> > >> >> >> >> Have you tried to create as many nodes as I have said with the > >> newer > >> >> >> >> index-util version? > >> >> >> >> > >> >> >> >> Thank you, > >> >> >> >> > >> >> >> >> Núria. > >> >> >> >> > >> >> >> >> 2009/12/4 Núria Trench <nuriatre...@gmail.com> > >> >> >> >> > >> >> >> >>> Hi Mattias, > >> >> >> >>> > >> >> >> >>> Thank you very much for fixing the problem so fast. I will try > it > >> as > >> >> >> soon > >> >> >> >>> as the new changes will be available in the maven repository. > >> >> >> >>> > >> >> >> >>> Núria. > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> 2009/12/4 Mattias Persson <matt...@neotechnology.com> > >> >> >> >>> > >> >> >> >>>> I fixed the problem and also added a cache per key for faster > >> >> >> >>>> getNodes/getSingleNode lookup during the insert process. > However > >> >> the > >> >> >> >>>> cache assumes that there's nothing in the index when the > process > >> >> >> >>>> starts (which almost always will be true) to speed things up > >> even > >> >> >> >>>> further. > >> >> >> >>>> > >> >> >> >>>> You can control the cache size and if it should be used by > >> >> overriding > >> >> >> >>>> the (this is also documented in the Javadoc): > >> >> >> >>>> > >> >> >> >>>> boolean useCache() > >> >> >> >>>> int getMaxCacheSizePerKey() > >> >> >> >>>> > >> >> >> >>>> methods in your LuceneIndexBatchInserterImpl instance. The > new > >> >> changes > >> >> >> >>>> should be available in the maven repository within an hour. > >> >> >> >>>> > >> >> >> >>>> 2009/12/4 Mattias Persson <matt...@neotechnology.com>: > >> >> >> >>>> > I think I found the problem... it's indexing as it should, > but > >> it > >> >> >> >>>> > isn't reflected in getNodes/getSingleNode properly until > you > >> >> >> >>>> > flush/optimize/shutdown the index. I'll try to fix it > today! > >> >> >> >>>> > > >> >> >> >>>> > 2009/12/3 Núria Trench <nuriatre...@gmail.com>: > >> >> >> >>>> >> Thank you very much for your response. > >> >> >> >>>> >> If you need more information, you only have to send an > e-mail > >> >> and I > >> >> >> >>>> will try > >> >> >> >>>> >> to explain it better. > >> >> >> >>>> >> > >> >> >> >>>> >> Núria. > >> >> >> >>>> >> > >> >> >> >>>> >> 2009/12/3 Mattias Persson <matt...@neotechnology.com> > >> >> >> >>>> >> > >> >> >> >>>> >>> This is something I'd like to reproduce and I'll do some > >> >> testing > >> >> >> on > >> >> >> >>>> >>> this tomorrow > >> >> >> >>>> >>> > >> >> >> >>>> >>> 2009/12/3 Núria Trench <nuriatre...@gmail.com>: > >> >> >> >>>> >>> > Hello, > >> >> >> >>>> >>> > > >> >> >> >>>> >>> > Last week, I decided to download your graph database > core > >> in > >> >> >> order > >> >> >> >>>> to use > >> >> >> >>>> >>> > it. First, I created a new project to parse my CSV > files > >> and > >> >> >> create > >> >> >> >>>> a new > >> >> >> >>>> >>> > graph database with Neo4j. This CSV files contain 150 > >> milion > >> >> >> edges > >> >> >> >>>> and 20 > >> >> >> >>>> >>> > milion nodes. > >> >> >> >>>> >>> > > >> >> >> >>>> >>> > When I finished to write the code which will create the > >> graph > >> >> >> >>>> database, I > >> >> >> >>>> >>> > executed it and, after six hours of execution, the > program > >> >> >> crashes > >> >> >> >>>> >>> because > >> >> >> >>>> >>> > of a Lucene exception. The exception is related to the > >> index > >> >> >> merging > >> >> >> >>>> and > >> >> >> >>>> >>> it > >> >> >> >>>> >>> > has the following message: > >> >> >> >>>> >>> > "mergeFields produced an invalid result: docCount is > >> >> 385282378 > >> >> >> but > >> >> >> >>>> fdx > >> >> >> >>>> >>> file > >> >> >> >>>> >>> > size is 3082259028; now aborting this merge to prevent > >> index > >> >> >> >>>> corruption" > >> >> >> >>>> >>> > > >> >> >> >>>> >>> > I have searched on the net and I found that it is a > lucene > >> >> bug. > >> >> >> The > >> >> >> >>>> >>> > libraries used for executing my project were: > >> >> >> >>>> >>> > neo-1.0-b10 > >> >> >> >>>> >>> > index-util-0.7 > >> >> >> >>>> >>> > lucene-core-2.4.0 > >> >> >> >>>> >>> > > >> >> >> >>>> >>> > So, I decided to use a newer Lucene version. I found > that > >> you > >> >> >> have a > >> >> >> >>>> >>> newer > >> >> >> >>>> >>> > index-util version so I updated the libraries: > >> >> >> >>>> >>> > neo-1.0-b10 > >> >> >> >>>> >>> > index-util-0.9 > >> >> >> >>>> >>> > lucene-core-2.9.1 > >> >> >> >>>> >>> > > >> >> >> >>>> >>> > When I had updated those libraries, I tried to execute > my > >> >> >> project > >> >> >> >>>> again > >> >> >> >>>> >>> and > >> >> >> >>>> >>> > I found that, in many occassions, it was not indexing > >> >> properly. > >> >> >> So, > >> >> >> >>>> I > >> >> >> >>>> >>> tried > >> >> >> >>>> >>> > to optimize the index after every time I indexed > >> something. > >> >> This > >> >> >> was > >> >> >> >>>> a > >> >> >> >>>> >>> > solution because, after that, it was indexing properly > but > >> >> the > >> >> >> time > >> >> >> >>>> >>> > execution increased a lot. > >> >> >> >>>> >>> > > >> >> >> >>>> >>> > I am not using transactions, instead of this, I am > using > >> the > >> >> >> Batch > >> >> >> >>>> >>> Inserter > >> >> >> >>>> >>> > with the LuceneIndexBatchInserter. > >> >> >> >>>> >>> > > >> >> >> >>>> >>> > So, my question is: What can I do to solve this > problem? > >> If > >> >> use > >> >> >> >>>> >>> > index-util-0.7 I cannot finish the execution of > creating > >> the > >> >> >> graph > >> >> >> >>>> >>> database > >> >> >> >>>> >>> > and I use index-util-0.9 I have to optimize the index > in > >> >> every > >> >> >> >>>> insertion > >> >> >> >>>> >>> and > >> >> >> >>>> >>> > the execution never ever ends. > >> >> >> >>>> >>> > > >> >> >> >>>> >>> > Thank you very much in advance, > >> >> >> >>>> >>> > > >> >> >> >>>> >>> > Núria. > >> >> >> >>>> >>> > _______________________________________________ > >> >> >> >>>> >>> > Neo mailing list > >> >> >> >>>> >>> > User@lists.neo4j.org > >> >> >> >>>> >>> > https://lists.neo4j.org/mailman/listinfo/user > >> >> >> >>>> >>> > > >> >> >> >>>> >>> > >> >> >> >>>> >>> > >> >> >> >>>> >>> > >> >> >> >>>> >>> -- > >> >> >> >>>> >>> Mattias Persson, [matt...@neotechnology.com] > >> >> >> >>>> >>> Neo Technology, www.neotechnology.com > >> >> >> >>>> >>> _______________________________________________ > >> >> >> >>>> >>> Neo mailing list > >> >> >> >>>> >>> User@lists.neo4j.org > >> >> >> >>>> >>> https://lists.neo4j.org/mailman/listinfo/user > >> >> >> >>>> >>> > >> >> >> >>>> >> _______________________________________________ > >> >> >> >>>> >> Neo mailing list > >> >> >> >>>> >> User@lists.neo4j.org > >> >> >> >>>> >> https://lists.neo4j.org/mailman/listinfo/user > >> >> >> >>>> >> > >> >> >> >>>> > > >> >> >> >>>> > > >> >> >> >>>> > > >> >> >> >>>> > -- > >> >> >> >>>> > Mattias Persson, [matt...@neotechnology.com] > >> >> >> >>>> > Neo Technology, www.neotechnology.com > >> >> >> >>>> > > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> > >> >> >> >>>> -- > >> >> >> >>>> Mattias Persson, [matt...@neotechnology.com] > >> >> >> >>>> Neo Technology, www.neotechnology.com > >> >> >> >>>> _______________________________________________ > >> >> >> >>>> Neo mailing list > >> >> >> >>>> User@lists.neo4j.org > >> >> >> >>>> https://lists.neo4j.org/mailman/listinfo/user > >> >> >> >>>> > >> >> >> >>> > >> >> >> >>> > >> >> >> >> _______________________________________________ > >> >> >> >> Neo mailing list > >> >> >> >> User@lists.neo4j.org > >> >> >> >> https://lists.neo4j.org/mailman/listinfo/user > >> >> >> >> > >> >> >> > _______________________________________________ > >> >> >> > Neo mailing list > >> >> >> > User@lists.neo4j.org > >> >> >> > https://lists.neo4j.org/mailman/listinfo/user > >> >> >> > > >> >> >> > >> >> >> > >> >> >> > >> >> >> -- > >> >> >> Mattias Persson, [matt...@neotechnology.com] > >> >> >> Neo Technology, www.neotechnology.com > >> >> >> _______________________________________________ > >> >> >> Neo mailing list > >> >> >> User@lists.neo4j.org > >> >> >> https://lists.neo4j.org/mailman/listinfo/user > >> >> >> > >> >> > _______________________________________________ > >> >> > Neo mailing list > >> >> > User@lists.neo4j.org > >> >> > https://lists.neo4j.org/mailman/listinfo/user > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> Mattias Persson, [matt...@neotechnology.com] > >> >> Neo Technology, www.neotechnology.com > >> >> _______________________________________________ > >> >> Neo mailing list > >> >> User@lists.neo4j.org > >> >> https://lists.neo4j.org/mailman/listinfo/user > >> >> > >> > > >> > _______________________________________________ > >> > Neo mailing list > >> > User@lists.neo4j.org > >> > https://lists.neo4j.org/mailman/listinfo/user > >> > > >> > > >> > >> > >> > >> -- > >> Mattias Persson, [matt...@neotechnology.com] > >> Neo Technology, www.neotechnology.com > >> _______________________________________________ > >> Neo mailing list > >> User@lists.neo4j.org > >> https://lists.neo4j.org/mailman/listinfo/user > >> > > > > _______________________________________________ > > Neo mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > > > -- > Mattias Persson, [matt...@neotechnology.com] > Neo Technology, www.neotechnology.com > _______________________________________________ > Neo mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > _______________________________________________ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
