Hi Núria, the current ID-scheme of using Integers for IDs for both Nodes, Relationships and Properties limits the possible node space size to 4 Billion nodes, 4 Billion Relationships and 4 Billion properties. Of course one could switch to Long as IDs, but that will increase the reserved amount of bytes and cause possible performance penalties. However, this is the current limit, after that you have to start thinking about sharding along a suitable domain-specific criteria. What size and domain are you imagining?
However, when dealing with bigger nodespaces you probably want to increase RAM of your server machine and think about SSD in order to keep the often-used parts of your graph cached and minimize IO cost. HTH Cheers, /peter neubauer COO and Sales, Neo Technology GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Relationships count. http://gremlin.tinkerpop.com - PageRank in 2 lines of code. http://www.linkedprocess.org - Computing at LinkedData scale. On Sat, Dec 26, 2009 at 4:10 PM, Núria Trench <nuriatre...@gmail.com> wrote: > Hi, > > I have just finished parsing and creating the database with the latest > index-util-0.9-SNAPSHOT available in your repository. It has been finished > succesfully so I must thank you for your interest and useful help. > And, finally, I have one last question. I have been created 180 million of > edges and 20 million of nodes. Is it possible to create a bigger amount of > edges and nodes with Neo4j? Do you have a limit? > > Thank your very much again. > > 2009/12/21 Núria Trench <nuriatre...@gmail.com> > >> Hi again Mattias, >> >> I'm still trying to parse all the data in order to create the graph. I will >> report the results as soon as possible. >> Thank you very much for your interest. >> >> Núria. >> >> 2009/12/21 Mattias Persson <matt...@neotechnology.com> >> >> Hi again, >>> >>> any luck with this yet? >>> >>> 2009/12/11 Núria Trench <nuriatre...@gmail.com>: >>> > Thank you very much Mattias. I will test it as soon as possible and I'll >>> > will tell you something. >>> > >>> > Núria. >>> > >>> > 2009/12/11 Mattias Persson <matt...@neotechnology.com> >>> > >>> >> I've tried this a couple of times now and first of all I see some >>> >> problems in your code: >>> >> >>> >> 1) In the method createRelationsTitleImage you have an inverted "head >>> >> != -1" check where it should be "head == -1" >>> >> >>> >> 2) You index relationships in createRelationsBetweenTitles method, >>> >> this isn't ok since the index can only manage nodes. >>> >> >>> >> And I recently committed a "fix" which removed the caching layer in >>> >> the LuceneIndexBatchInserterImpl (and therefore also >>> >> LuceneFulltextIndexBatchInserter). This probably fixes your problems. >>> >> I'm also working on a performance fix which makes consecutive getNodes >>> >> calls faster. >>> >> >>> >> So I think that with these fixes (1) and (2) and the latest index-util >>> >> 0.9-SNAPSHOT your sample will run fine. Also you could try without >>> >> calling optimize. See more information at >>> >> http://wiki.neo4j.org/content/Indexing_with_BatchInserter >>> >> >>> >> 2009/12/10 Mattias Persson <matt...@neotechnology.com>: >>> >> > To continue this thread in the user list: >>> >> > >>> >> > Thanks Núria, I've gotten your samples code/files and I'm running it >>> >> > now to try to reproduce you problem. >>> >> > >>> >> > 2009/12/9 Núria Trench <nuriatre...@gmail.com>: >>> >> >> I have finished uploading the 4 csv files. You'll see an e-mail with >>> the >>> >> >> other 3 csv files packed in a rar file. >>> >> >> Thanks, >>> >> >> >>> >> >> Núria. >>> >> >> >>> >> >> 2009/12/9 Núria Trench <nuriatre...@gmail.com> >>> >> >>> >>> >> >>> Yes, you are right. But there is one csv file that is too big to be >>> >> packed >>> >> >>> with other files and I am reducing it. >>> >> >>> I am sending the other files now. >>> >> >>> >>> >> >>> 2009/12/9 Mattias Persson <matt...@neotechnology.com> >>> >> >>>> >>> >> >>>> By the way, you might consider packing those files (with zip or >>> tar.gz >>> >> >>>> or something) cause they will shrink quite well >>> >> >>>> >>> >> >>>> 2009/12/9 Mattias Persson <matt...@neotechnology.com>: >>> >> >>>> > Great, but I only got the images.csv file... I'm starting to >>> test >>> >> with >>> >> >>>> > that at least >>> >> >>>> > >>> >> >>>> > 2009/12/9 Núria Trench <nuriatre...@gmail.com>: >>> >> >>>> >> Hi again, >>> >> >>>> >> >>> >> >>>> >> The errors show up after being parsed 2 csv files to create all >>> the >>> >> >>>> >> nodes, >>> >> >>>> >> just in the moment of calling the method "getSingleNode" for >>> >> looking >>> >> >>>> >> up the >>> >> >>>> >> tail and head node for creating all the edges by reading the >>> other >>> >> two >>> >> >>>> >> csv >>> >> >>>> >> files. >>> >> >>>> >> >>> >> >>>> >> I am sending with Sprend the four csv files that will help you >>> to >>> >> >>>> >> trigger >>> >> >>>> >> index behaviour. >>> >> >>>> >> >>> >> >>>> >> Thank you, >>> >> >>>> >> >>> >> >>>> >> Núria. >>> >> >>>> >> >>> >> >>>> >> 2009/12/9 Mattias Persson <matt...@neotechnology.com> >>> >> >>>> >>> >>> >> >>>> >>> Hmm, I've no idea... but does the errors show up early in the >>> >> process >>> >> >>>> >>> or do you have to insert a LOT of data to trigger it? In such >>> case >>> >> >>>> >>> you >>> >> >>>> >>> could send me a part of them... maybe using >>> http://www.sprend.se, >>> >> >>>> >>> WDYT? >>> >> >>>> >>> >>> >> >>>> >>> 2009/12/9 Núria Trench <nuriatre...@gmail.com>: >>> >> >>>> >>> > Hi Mattias, >>> >> >>>> >>> > >>> >> >>>> >>> > The data isn't confident but the files are very big (5,5 >>> GB). >>> >> >>>> >>> > How can I send you this data? >>> >> >>>> >>> > >>> >> >>>> >>> > 2009/12/9 Mattias Persson <matt...@neotechnology.com> >>> >> >>>> >>> >> >>> >> >>>> >>> >> Yep I got the java code, thanks. Yeah if the data is >>> confident >>> >> or >>> >> >>>> >>> >> sensitive you can just send me the formatting, else >>> consider >>> >> >>>> >>> >> sending >>> >> >>>> >>> >> the files as well (or a subset if they are big). >>> >> >>>> >>> >> >>> >> >>>> >>> >> 2009/12/9 Núria Trench <nuriatre...@gmail.com>: >>> >> >>> >> >>> >> >>> >> -- >>> >> Mattias Persson, [matt...@neotechnology.com] >>> >> Neo Technology, www.neotechnology.com >>> >> _______________________________________________ >>> >> Neo mailing list >>> >> User@lists.neo4j.org >>> >> https://lists.neo4j.org/mailman/listinfo/user >>> >> >>> > _______________________________________________ >>> > Neo mailing list >>> > User@lists.neo4j.org >>> > https://lists.neo4j.org/mailman/listinfo/user >>> > >>> >>> >>> >>> -- >>> Mattias Persson, [matt...@neotechnology.com] >>> Neo Technology, www.neotechnology.com >>> _______________________________________________ >>> Neo mailing list >>> User@lists.neo4j.org >>> https://lists.neo4j.org/mailman/listinfo/user >>> >> >> > _______________________________________________ > Neo mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > _______________________________________________ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
