Re: [Neo] LuceneIndexBatchInserter doubt
I think I found the problem... it's indexing as it should, but it isn't reflected in getNodes/getSingleNode properly until you flush/optimize/shutdown the index. I'll try to fix it today! 2009/12/3 Núria Trench nuriatre...@gmail.com: Thank you very much for your response. If you need more information, you only have to send an e-mail and I will try to explain it better. Núria. 2009/12/3 Mattias Persson matt...@neotechnology.com This is something I'd like to reproduce and I'll do some testing on this tomorrow 2009/12/3 Núria Trench nuriatre...@gmail.com: Hello, Last week, I decided to download your graph database core in order to use it. First, I created a new project to parse my CSV files and create a new graph database with Neo4j. This CSV files contain 150 milion edges and 20 milion nodes. When I finished to write the code which will create the graph database, I executed it and, after six hours of execution, the program crashes because of a Lucene exception. The exception is related to the index merging and it has the following message: mergeFields produced an invalid result: docCount is 385282378 but fdx file size is 3082259028; now aborting this merge to prevent index corruption I have searched on the net and I found that it is a lucene bug. The libraries used for executing my project were: neo-1.0-b10 index-util-0.7 lucene-core-2.4.0 So, I decided to use a newer Lucene version. I found that you have a newer index-util version so I updated the libraries: neo-1.0-b10 index-util-0.9 lucene-core-2.9.1 When I had updated those libraries, I tried to execute my project again and I found that, in many occassions, it was not indexing properly. So, I tried to optimize the index after every time I indexed something. This was a solution because, after that, it was indexing properly but the time execution increased a lot. I am not using transactions, instead of this, I am using the Batch Inserter with the LuceneIndexBatchInserter. So, my question is: What can I do to solve this problem? If use index-util-0.7 I cannot finish the execution of creating the graph database and I use index-util-0.9 I have to optimize the index in every insertion and the execution never ever ends. Thank you very much in advance, Núria. ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] LuceneIndexBatchInserter doubt
Hi Mattias, Thank you very much for fixing the problem so fast. I will try it as soon as the new changes will be available in the maven repository. Núria. 2009/12/4 Mattias Persson matt...@neotechnology.com I fixed the problem and also added a cache per key for faster getNodes/getSingleNode lookup during the insert process. However the cache assumes that there's nothing in the index when the process starts (which almost always will be true) to speed things up even further. You can control the cache size and if it should be used by overriding the (this is also documented in the Javadoc): boolean useCache() int getMaxCacheSizePerKey() methods in your LuceneIndexBatchInserterImpl instance. The new changes should be available in the maven repository within an hour. 2009/12/4 Mattias Persson matt...@neotechnology.com: I think I found the problem... it's indexing as it should, but it isn't reflected in getNodes/getSingleNode properly until you flush/optimize/shutdown the index. I'll try to fix it today! 2009/12/3 Núria Trench nuriatre...@gmail.com: Thank you very much for your response. If you need more information, you only have to send an e-mail and I will try to explain it better. Núria. 2009/12/3 Mattias Persson matt...@neotechnology.com This is something I'd like to reproduce and I'll do some testing on this tomorrow 2009/12/3 Núria Trench nuriatre...@gmail.com: Hello, Last week, I decided to download your graph database core in order to use it. First, I created a new project to parse my CSV files and create a new graph database with Neo4j. This CSV files contain 150 milion edges and 20 milion nodes. When I finished to write the code which will create the graph database, I executed it and, after six hours of execution, the program crashes because of a Lucene exception. The exception is related to the index merging and it has the following message: mergeFields produced an invalid result: docCount is 385282378 but fdx file size is 3082259028; now aborting this merge to prevent index corruption I have searched on the net and I found that it is a lucene bug. The libraries used for executing my project were: neo-1.0-b10 index-util-0.7 lucene-core-2.4.0 So, I decided to use a newer Lucene version. I found that you have a newer index-util version so I updated the libraries: neo-1.0-b10 index-util-0.9 lucene-core-2.9.1 When I had updated those libraries, I tried to execute my project again and I found that, in many occassions, it was not indexing properly. So, I tried to optimize the index after every time I indexed something. This was a solution because, after that, it was indexing properly but the time execution increased a lot. I am not using transactions, instead of this, I am using the Batch Inserter with the LuceneIndexBatchInserter. So, my question is: What can I do to solve this problem? If use index-util-0.7 I cannot finish the execution of creating the graph database and I use index-util-0.9 I have to optimize the index in every insertion and the execution never ever ends. Thank you very much in advance, Núria. ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo] Maven Dependency For Neo Util for Lucene BatchInserter?
I am using the Lucene Batch Inserter with attributes that have a space (combined first name and last name). Exception in thread main java.lang.NoSuchMethodError: org.neo4j.impl.util.FileUtils.fixSeparatorsInPath(Ljava/lang/String;)Ljava/lang/String; at org.neo4j.util.index.LuceneIndexBatchInserterImpl.fixPath(LuceneIndexBatchInserterImpl.java:61) at org.neo4j.util.index.LuceneIndexBatchInserterImpl.init(LuceneIndexBatchInserterImpl.java:51) at com.cisco.collectiveintelligence.QueryNeo.addData(QueryNeo.java:135) at com.cisco.collectiveintelligence.QueryNeo.main(QueryNeo.java:43) I tried the obvious: dependency groupIdorg.neo4j/groupId artifactIdutil/artifactId version1.0-b9/version /dependency Anyone know where I can find this? Thanks, Todd ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Maven Dependency For Neo Util for Lucene BatchInserter?
2009/12/4 Todd Stavish toddstav...@gmail.com: I am using the Lucene Batch Inserter with attributes that have a space (combined first name and last name). Exception in thread main java.lang.NoSuchMethodError: org.neo4j.impl.util.FileUtils.fixSeparatorsInPath(Ljava/lang/String;)Ljava/lang/String; at org.neo4j.util.index.LuceneIndexBatchInserterImpl.fixPath(LuceneIndexBatchInserterImpl.java:61) at org.neo4j.util.index.LuceneIndexBatchInserterImpl.init(LuceneIndexBatchInserterImpl.java:51) at com.cisco.collectiveintelligence.QueryNeo.addData(QueryNeo.java:135) at com.cisco.collectiveintelligence.QueryNeo.main(QueryNeo.java:43) I tried the obvious: dependency groupIdorg.neo4j/groupId artifactIdutil/artifactId Try 'neo-utils' instead version1.0-b9/version /dependency Anyone know where I can find this? Thanks, Todd ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Maven Dependency For Neo Util for Lucene BatchInserter?
Thanks Mattias. Just an FYI, there doesn't seem to be 1.0-b10 version build for neo-utils yet. http://m2.neo4j.org/org/neo4j/neo-utils/ Regards, Todd On Fri, Dec 4, 2009 at 12:26 PM, Mattias Persson matt...@neotechnology.com wrote: 2009/12/4 Todd Stavish toddstav...@gmail.com: I am using the Lucene Batch Inserter with attributes that have a space (combined first name and last name). Exception in thread main java.lang.NoSuchMethodError: org.neo4j.impl.util.FileUtils.fixSeparatorsInPath(Ljava/lang/String;)Ljava/lang/String; at org.neo4j.util.index.LuceneIndexBatchInserterImpl.fixPath(LuceneIndexBatchInserterImpl.java:61) at org.neo4j.util.index.LuceneIndexBatchInserterImpl.init(LuceneIndexBatchInserterImpl.java:51) at com.cisco.collectiveintelligence.QueryNeo.addData(QueryNeo.java:135) at com.cisco.collectiveintelligence.QueryNeo.main(QueryNeo.java:43) I tried the obvious: dependency groupIdorg.neo4j/groupId artifactIdutil/artifactId Try 'neo-utils' instead version1.0-b9/version /dependency Anyone know where I can find this? Thanks, Todd ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
