Hi again, Núria (it was I, Mattias who asked for the sample code). Well... the fact that you parse 4 csv files doesn't really help me setup a test for this... I mean how can I know that my test will be similar to yours? Would it be ok to attach your code/csv files as well?
/ Mattias 2009/12/9 Núria Trench <nuriatre...@gmail.com>: > Hi Todd, > > The sample code creates nodes and relationships by parsing 4 csv files. > Thank you for trying to trigger this behaviour with this sample. > > Núria > > 2009/12/9 Mattias Persson <matt...@neotechnology.com> > >> Could you provide me with some sample code which can trigger this >> behaviour with the latest index-util-0.9-SNAPSHOT Núria? >> >> 2009/12/9 Núria Trench <nuriatre...@gmail.com>: >> > Todd, >> > >> > I haven't the same problem. In my case, after indexing all the >> > attributes/properties of each node, the application creates all the edges >> by >> > looking up the tail node and the head node. So, it calls the method >> > "org.neo4j.util.index. >> > LuceneIndexBatchInserterImpl.getSingleNode" which returns -1 (no found >> node) >> > in many occasions. >> > >> > Any one has an alternative to get a node with indexex >> attributes/properties? >> > >> > Thank you, >> > >> > Núria. >> > >> > >> > 2009/12/7 Mattias Persson <matt...@neotechnology.com> >> > >> >> Todd, are you sure you have the latest index-util 0.9-SNAPSHOT? This >> >> is a bug that we fixed yesterday... (assuming it's the same bug). >> >> >> >> 2009/12/7 Todd Stavish <toddstav...@gmail.com>: >> >> > Hi Mattias, Núria. >> >> > >> >> > I am also running into scalability problems with the Lucene batch >> >> > inserter at much smaller numbers, 30,000 indexed nodes. I tried >> >> > calling optimize more. Increasing ulimit didn't help. >> >> > >> >> > INFO] Exception in thread "main" java.lang.RuntimeException: >> >> > java.io.FileNotFoundException: >> >> > >> >> >> /Users/todd/Code/neo4Jprototype/target/classes/data/graph/lucene/name/_0.cfx >> >> > (Too many open files) >> >> > [INFO] at >> >> >> org.neo4j.util.index.LuceneIndexBatchInserterImpl.getNodes(LuceneIndexBatchInserterImpl.java:186) >> >> > [INFO] at >> >> >> org.neo4j.util.index.LuceneIndexBatchInserterImpl.getSingleNode(LuceneIndexBatchInserterImpl.java:238) >> >> > [INFO] at >> >> com.collectiveintelligence.QueryNeo.loadDataToGraph(QueryNeo.java:277) >> >> > [INFO] at com.collectiveintelligence.QueryNeo.main(QueryNeo.java:57) >> >> > [INFO] Caused by: java.io.FileNotFoundException: >> >> > >> >> >> /Users/todd/Code/neo4Jprototype/target/classes/data/graph/lucene/name/_0.cfx >> >> > (Too many open files) >> >> > >> >> > I tried breaking up to separate batchinserter instances, and it hangs >> >> > now. Can I create more than one batch inserter per process if they run >> >> > sequentially and non-threaded? >> >> > >> >> > Thanks, >> >> > Todd >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > On Mon, Dec 7, 2009 at 7:28 AM, Núria Trench <nuriatre...@gmail.com> >> >> wrote: >> >> >> Hi again Mattias, >> >> >> >> >> >> I have tried to execute my application with the last version >> available >> >> in >> >> >> the maven repository and I still have the same problem. After >> creating >> >> and >> >> >> indexing all the nodes, the application calls the "optimize" method >> and, >> >> >> then, it creates all the edges by calling the method "getNodes" in >> order >> >> to >> >> >> select the tail and head node of the edge, but it doesn't work >> because >> >> many >> >> >> nodes are not found. >> >> >> >> >> >> I have tried to create only 30 nodes and 15 edges and it works >> properly, >> >> but >> >> >> if I try to create a big graph (180 million edges + 20 million nodes) >> it >> >> >> doesn't. >> >> >> >> >> >> I have also tried to call the "optimize" method every time the >> >> application >> >> >> has been created 1 million nodes but it doesn't work. >> >> >> >> >> >> Have you tried to create as many nodes as I have said with the newer >> >> >> index-util version? >> >> >> >> >> >> Thank you, >> >> >> >> >> >> Núria. >> >> >> >> >> >> 2009/12/4 Núria Trench <nuriatre...@gmail.com> >> >> >> >> >> >>> Hi Mattias, >> >> >>> >> >> >>> Thank you very much for fixing the problem so fast. I will try it as >> >> soon >> >> >>> as the new changes will be available in the maven repository. >> >> >>> >> >> >>> Núria. >> >> >>> >> >> >>> >> >> >>> 2009/12/4 Mattias Persson <matt...@neotechnology.com> >> >> >>> >> >> >>>> I fixed the problem and also added a cache per key for faster >> >> >>>> getNodes/getSingleNode lookup during the insert process. However >> the >> >> >>>> cache assumes that there's nothing in the index when the process >> >> >>>> starts (which almost always will be true) to speed things up even >> >> >>>> further. >> >> >>>> >> >> >>>> You can control the cache size and if it should be used by >> overriding >> >> >>>> the (this is also documented in the Javadoc): >> >> >>>> >> >> >>>> boolean useCache() >> >> >>>> int getMaxCacheSizePerKey() >> >> >>>> >> >> >>>> methods in your LuceneIndexBatchInserterImpl instance. The new >> changes >> >> >>>> should be available in the maven repository within an hour. >> >> >>>> >> >> >>>> 2009/12/4 Mattias Persson <matt...@neotechnology.com>: >> >> >>>> > I think I found the problem... it's indexing as it should, but it >> >> >>>> > isn't reflected in getNodes/getSingleNode properly until you >> >> >>>> > flush/optimize/shutdown the index. I'll try to fix it today! >> >> >>>> > >> >> >>>> > 2009/12/3 Núria Trench <nuriatre...@gmail.com>: >> >> >>>> >> Thank you very much for your response. >> >> >>>> >> If you need more information, you only have to send an e-mail >> and I >> >> >>>> will try >> >> >>>> >> to explain it better. >> >> >>>> >> >> >> >>>> >> Núria. >> >> >>>> >> >> >> >>>> >> 2009/12/3 Mattias Persson <matt...@neotechnology.com> >> >> >>>> >> >> >> >>>> >>> This is something I'd like to reproduce and I'll do some >> testing >> >> on >> >> >>>> >>> this tomorrow >> >> >>>> >>> >> >> >>>> >>> 2009/12/3 Núria Trench <nuriatre...@gmail.com>: >> >> >>>> >>> > Hello, >> >> >>>> >>> > >> >> >>>> >>> > Last week, I decided to download your graph database core in >> >> order >> >> >>>> to use >> >> >>>> >>> > it. First, I created a new project to parse my CSV files and >> >> create >> >> >>>> a new >> >> >>>> >>> > graph database with Neo4j. This CSV files contain 150 milion >> >> edges >> >> >>>> and 20 >> >> >>>> >>> > milion nodes. >> >> >>>> >>> > >> >> >>>> >>> > When I finished to write the code which will create the graph >> >> >>>> database, I >> >> >>>> >>> > executed it and, after six hours of execution, the program >> >> crashes >> >> >>>> >>> because >> >> >>>> >>> > of a Lucene exception. The exception is related to the index >> >> merging >> >> >>>> and >> >> >>>> >>> it >> >> >>>> >>> > has the following message: >> >> >>>> >>> > "mergeFields produced an invalid result: docCount is >> 385282378 >> >> but >> >> >>>> fdx >> >> >>>> >>> file >> >> >>>> >>> > size is 3082259028; now aborting this merge to prevent index >> >> >>>> corruption" >> >> >>>> >>> > >> >> >>>> >>> > I have searched on the net and I found that it is a lucene >> bug. >> >> The >> >> >>>> >>> > libraries used for executing my project were: >> >> >>>> >>> > neo-1.0-b10 >> >> >>>> >>> > index-util-0.7 >> >> >>>> >>> > lucene-core-2.4.0 >> >> >>>> >>> > >> >> >>>> >>> > So, I decided to use a newer Lucene version. I found that you >> >> have a >> >> >>>> >>> newer >> >> >>>> >>> > index-util version so I updated the libraries: >> >> >>>> >>> > neo-1.0-b10 >> >> >>>> >>> > index-util-0.9 >> >> >>>> >>> > lucene-core-2.9.1 >> >> >>>> >>> > >> >> >>>> >>> > When I had updated those libraries, I tried to execute my >> >> project >> >> >>>> again >> >> >>>> >>> and >> >> >>>> >>> > I found that, in many occassions, it was not indexing >> properly. >> >> So, >> >> >>>> I >> >> >>>> >>> tried >> >> >>>> >>> > to optimize the index after every time I indexed something. >> This >> >> was >> >> >>>> a >> >> >>>> >>> > solution because, after that, it was indexing properly but >> the >> >> time >> >> >>>> >>> > execution increased a lot. >> >> >>>> >>> > >> >> >>>> >>> > I am not using transactions, instead of this, I am using the >> >> Batch >> >> >>>> >>> Inserter >> >> >>>> >>> > with the LuceneIndexBatchInserter. >> >> >>>> >>> > >> >> >>>> >>> > So, my question is: What can I do to solve this problem? If >> use >> >> >>>> >>> > index-util-0.7 I cannot finish the execution of creating the >> >> graph >> >> >>>> >>> database >> >> >>>> >>> > and I use index-util-0.9 I have to optimize the index in >> every >> >> >>>> insertion >> >> >>>> >>> and >> >> >>>> >>> > the execution never ever ends. >> >> >>>> >>> > >> >> >>>> >>> > Thank you very much in advance, >> >> >>>> >>> > >> >> >>>> >>> > Núria. >> >> >>>> >>> > _______________________________________________ >> >> >>>> >>> > Neo mailing list >> >> >>>> >>> > User@lists.neo4j.org >> >> >>>> >>> > https://lists.neo4j.org/mailman/listinfo/user >> >> >>>> >>> > >> >> >>>> >>> >> >> >>>> >>> >> >> >>>> >>> >> >> >>>> >>> -- >> >> >>>> >>> Mattias Persson, [matt...@neotechnology.com] >> >> >>>> >>> Neo Technology, www.neotechnology.com >> >> >>>> >>> _______________________________________________ >> >> >>>> >>> Neo mailing list >> >> >>>> >>> User@lists.neo4j.org >> >> >>>> >>> https://lists.neo4j.org/mailman/listinfo/user >> >> >>>> >>> >> >> >>>> >> _______________________________________________ >> >> >>>> >> Neo mailing list >> >> >>>> >> User@lists.neo4j.org >> >> >>>> >> https://lists.neo4j.org/mailman/listinfo/user >> >> >>>> >> >> >> >>>> > >> >> >>>> > >> >> >>>> > >> >> >>>> > -- >> >> >>>> > Mattias Persson, [matt...@neotechnology.com] >> >> >>>> > Neo Technology, www.neotechnology.com >> >> >>>> > >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> -- >> >> >>>> Mattias Persson, [matt...@neotechnology.com] >> >> >>>> Neo Technology, www.neotechnology.com >> >> >>>> _______________________________________________ >> >> >>>> Neo mailing list >> >> >>>> User@lists.neo4j.org >> >> >>>> https://lists.neo4j.org/mailman/listinfo/user >> >> >>>> >> >> >>> >> >> >>> >> >> >> _______________________________________________ >> >> >> Neo mailing list >> >> >> User@lists.neo4j.org >> >> >> https://lists.neo4j.org/mailman/listinfo/user >> >> >> >> >> > _______________________________________________ >> >> > Neo mailing list >> >> > User@lists.neo4j.org >> >> > https://lists.neo4j.org/mailman/listinfo/user >> >> > >> >> >> >> >> >> >> >> -- >> >> Mattias Persson, [matt...@neotechnology.com] >> >> Neo Technology, www.neotechnology.com >> >> _______________________________________________ >> >> Neo mailing list >> >> User@lists.neo4j.org >> >> https://lists.neo4j.org/mailman/listinfo/user >> >> >> > _______________________________________________ >> > Neo mailing list >> > User@lists.neo4j.org >> > https://lists.neo4j.org/mailman/listinfo/user >> > >> >> >> >> -- >> Mattias Persson, [matt...@neotechnology.com] >> Neo Technology, www.neotechnology.com >> _______________________________________________ >> Neo mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> > > _______________________________________________ > Neo mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > > -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com _______________________________________________ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
