Linan, that is actually exactly what Neo4j is doing. Indexing is done via the Index Framework, that makes external indicies like Lucene, Redas, BabuDB etc conform to transactional semantics in order to keep consistent between the Neo4j graph engine kernel and the indecies. See http://docs.neo4j.org/chunked/snapshot/indexing.html for more details on this, there are others that we have been testing like BabuDB, BerkeleyDB and Redis (soon to come out).
Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Mon, Sep 5, 2011 at 1:05 PM, Linan Wang <tali.w...@gmail.com> wrote: > thank you very much for the detailed replied. > i'm wondering if neo4j had considered the option to use redis or other > key-value store for storing properties and focus upon pure graph, > similar to the role Lucene plays. i assume the major problem is the > performance problem due to address lookup process inside redis value > reading and communication overhead. but the gain is significant in > scalability. just some random thoughts. > > On Fri, Sep 2, 2011 at 9:50 PM, Peter Neubauer > <peter.neuba...@neotechnology.com> wrote: >> Linan, >> see inline ... >> >> On Fri, Sep 2, 2011 at 9:01 PM, Linan Wang <tali.w...@gmail.com> wrote: >> >>> is it >>> https://github.com/peterneubauer/graph-collections/wiki/Indexed-relationships >>> ? >>> seems not included in the current stable ver. >>> >> No, this is work-in-progress, we are quite strict when it come sto including >> things into the official release, since we need to test and document it >> better. So, feel free to test it, a lot of code in there is stable, and >> expect to fork and contribute if you find things to fix! >> >> >>> >> 4, what's the best practice to do bulk insertion when running (not >>> >> seed initial data)? i read post says that too many insertions within a >>> >> transaction may lead to memory problem? what's the proper mount of >>> >> insertion within a transaction? >>> >> >>> > Yes, transaction data is kept in memory before calling commit and >>> flushing >>> > to disk, so overly large TX might result in memory problems. OTOH small >>> TX >>> > incur higher IO load. >>> i'll probably do it with smaller batches (~1k operations per batch) >>> from an external queue. does it sounds reasonable? >>> > >>> >> Yes. From experience, there seems to be a lot of cases where transactions >> hold between 1K and 10K operations and give a good performance vs. RAM vs. >> persistence balance, if you can afford it for your data. >> >> >>> > >>> >> 5, is there a suggested max length for string/array property? would it >>> >> be better to put into sql? >>> >> >>> > Well, the String store block size is adjustable (and we are working on >>> even >>> > better layouts there), but for big strings like documents, a fiel system >>> or >>> > Key/Value store might be better, and just keeping the reference to the >>> > location makes more sense. >>> ok, i'll use redis for strings. >>> >>> Probably a sensible choice. There might even be an Neo4j index coming out >> for Redis, making it transactional with the graph like Lucene. >> >> >>> > >>> > 6, say a facebook user may "likes" thousands of things, and these >>> >> things are sparsly connected. in this case, things should be modeled >>> > >>> > as nodes or array property? >>> >> >>> > Nodes. Sparse connections are one of the places where Neo4j shines - a >>> > fairly balanced graph where supernodes are seldom. >>> > >>> could you give a bottom number qualifies "supernode"? say 1k >>> connections within a graph of 1m nodes? >>> >>> with the current store layout, probably 1K is a good number. We are working >> with store changes that require less reads, but don't explicitly take care >> of supernodes. The is in plan, in which case this number will change upwards >> with good performance :) >> >>> > >>> >> 7, where can i find an example to use domain models with serverplugin? >>> >> i want to put my data in a standalone server and just use the >>> >> serverplugin, unmanaged extension. should i just put the domain models >>> >> into the same serverplugin jar? >>> >> >>> > Yes, I would do that. However, if you are not expecting to return Nodes, >>> > Relationships or Properties, an unmanaged extension will give you the >>> full >>> > API of REST services. One extension that way is for instance the >>> scripting >>> > extension, see https://github.com/neo4j/script-extension >>> thanks. seems i really should look into github instead of neo4j.org ;) >>> >> Well, it's hard to list everything, we are right now trying to put as much >> as possible into the manual which can be generated, tested and curated, and >> serve as a reference. >> >> >>> > >>> > Sorry for the delay, hope this helps. Let us know if you have more >>> > questions! >>> many thanks! i understand documentation is probably not your top >>> priority at this point, but since we are all programmers, we can read >>> codes. i feel samples on wiki and downloads are not updated to use the >>> most recent release. >>> >> I think what you are seeing is the Wiki getting outdated over time. We are >> in the process of moving the Wiki content into docs.neo4j.org, that one is >> MUCH better up to date - all code you see in there is generated from running >> tests. For real. So I think this will change soon to a point where we can >> delete most of the Wiki pages. Sorry for the inconvenience! >> >> /peter >> _______________________________________________ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> > > > > -- > Best regards > > Linan Wang > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
