Actually the Enhanced API doesn't use any index at this moment. All look ups are done within the graph and are all either ID lookups or a traversal of no more than three relationships. Including DBD would only serve to remove this book keeping from the graph. For example, to reify a relationship, a property is now set on that relationship, storing the ID of the associated node, and the associated node has a property storing the ID of the relationship. Those properties are visible to the user, and even if the API were to hide these properties, it would mean that the name space for properties is reduced. I could opt to store some book keeping in the Lucene index, but that has its drawbacks. Lucene doesn't support unique indexes, and I am not convinced it is really the fastest solution available. Adding a Lucene index would also reduce the index name space available to Neo4j users, and would make the content of the index available for manipulation. Having a BDB store hidden from users would allow transparent book keeping without reducing node or namespace and without users being able to manipulate the book keeping store. Niels
> Date: Thu, 28 Jul 2011 16:02:01 +0200 > From: cyuczie...@gmail.com > To: user@lists.neo4j.org > Subject: Re: [Neo4j] HyperRelationship example > > well if I think about it, maybe Niels could use this index(that neo4j uses) > instead of berkeleydb, that is, unless I'm missing something (other than > add() requiring an entity which is something unneeded with bdb when using > key-value). (but likely he's already making use of it and I didn't really > understand why bdb would be better then) > > On Thu, Jul 28, 2011 at 3:45 PM, John cyuczieekc <cyuczie...@gmail.com>wrote: > > > Roger that. > > don't read the following it's irrelevant(don't even know why I sent it): > > > > Btw, seems to me that (since the underlaying index storage is BTree - just > > guessing from the speed) > > I could store the ID of the nodes in two indexes and use only those as a > > base for creating node to node relationships :) that is, kind of emulating a > > key-value database, tho it seems kind of useless at this time :) > > indexForward: > > A-->B > > indexBackward: > > B-->A > > > > this way, I could check if A-->B exists, but could also check all modes > > (ie. A) that point to B, by using Backward index; else just for checking if > > A-->B exists, only one index would be needed; > > though the add() does take an entity, would not need that; > > > > http://components.neo4j.org/neo4j/1.4/apidocs/org/neo4j/graphdb/index/Index.html > > > > > > > > > > On Thu, Jul 28, 2011 at 3:32 PM, Anders Nawroth > > <and...@neotechnology.com>wrote: > > > >> Hi! > >> > >> Seems right to me. And yes, key/value would typically be storing the > >> relationship type. We should bring this up again next week, when Mattias > >> who wrote the indexing stuff is back from vacation! > >> > >> /anders > >> > >> On 07/28/2011 03:05 PM, John cyuczieekc wrote: > >> > with relationshipindex seems to be working as fast, though I am not sure > >> if > >> > I am using it right ie. > >> > doing this first time: > >> > RelationshipIndex ri = graphDB.index().forRelationships( "relsIndex" ); > >> > > >> > and on each relationship created between sNode-->eNode where eNode is > >> any > >> > random node, and sNode is the same on each call (in my case) > >> > Relationship rel = sNode.createRelationshipTo( eNode, linkedRelType ); > >> > ri.add( rel, "key", "value" ); > >> > > >> > and when checking if sNode-->eNode exists: > >> > final Relationship rel = ri.query( "key", "value", sNode, eNode > >> > ).getSingle(); > >> > if ( null == rel ) { > >> > return false; > >> > } else { > >> > return true; > >> > } > >> > > >> > seems to me that using those `key` and `value` are useless, unless I'm > >> > missing something; I'm probably using them wrongly but in my case I only > >> > have one type of relationship. > >> > > >> > In either case, the timings as good ~1ms, and no memory increase, so > >> this > >> > would seem like a good workaround; with findSinglePath the memory would > >> > increase by 1 gig (for my test) > >> > > >> > Thanks for suggesting to revisit RelationshipIndex, last time I dropped > >> it I > >> > think because I didn't know what to put on key/value. > >> > > >> > Also, I get what Niels meant now by that play nice with transactions, > >> that > >> > if both neo4j and bdb recover the same things after crash/recovery or > >> not... > >> > > >> > > >> > On Thu, Jul 28, 2011 at 2:05 PM, Anders Nawroth< > >> and...@neotechnology.com>wrote: > >> > > >> >> Hi! > >> >> > >> >> I think the hard part about transactions is recovering after crashes > >> and > >> >> such. > >> >> > >> >> Regarding finding A-->B, have you tried using a relationship index? > >> See: > >> >> > >> >> > >> http://components.neo4j.org/neo4j/1.4/apidocs/org/neo4j/graphdb/index/ReadableRelationshipIndex.html > >> >> > >> >> /anders > >> >> > >> >> On 07/28/2011 01:35 PM, John cyuczieekc wrote: > >> >>> I don't know what you mean by this: > >> >>> "I don't know how nicely BDB plays with Neo4J transactions." > >> >>> I have some small experience with bdb java edition that is, but I'm > >> not > >> >> sure > >> >>> what would their transaction have to do with neo4j transactions... if > >> you > >> >>> meant if you could make a wrapper such that you could use the same > >> >>> format/interface neo4j uses for their transactions, then you can, I > >> did > >> >> some > >> >>> attempt to that it works for me, also BDB Java Edition doesn't support > >> >>> nested transactions either (the C++ version does), but emulating them > >> to > >> >> use > >> >>> the same root/parent transaction is easy, my attempt is here: > >> >>> > >> >> > >> https://github.com/13th-floor/neo4john/blob/6c0371e82b7fc5b5f45d7c0ea9fb03ee4d241df9/src.obsolete/org/bdb/BETransaction.java > >> >>> probably not much relevant though. But this file here: > >> >>> > >> >> > >> https://github.com/13th-floor/neo4john/blob/master/src/org/benchtests/neo4j/TestLinkage.java > >> >>> I made to use both neo4j and bdb to do the same thing, that is: > >> >>> create nodes(uppercase named ones) with these rels: > >> >>> ROOT_LIST --> START > >> >>> ROOT_LIST --> <half a million unique nodes> > >> >>> ROOT_LIST --> MIDDLE > >> >>> ROOT_LIST --> <another half a million unique nodes> > >> >>> ROOT_LIST --> END > >> >>> > >> >>> then make both bdb and neo4j check if the following rels exist: > >> >>> ROOT_LIST --> START > >> >>> ROOT_LIST --> MIDDLE > >> >>> ROOT_LIST --> END > >> >>> (you probably saw this already in another post) > >> >>> But both bdb and neo4j now use transactions... that is, in my test > >> file. > >> >>> > >> >>> About licensing, I'm not much into that but here's the license for > >> >> Berkeley > >> >>> DB Java Edition: > >> >>> > >> >> > >> http://www.oracle.com/technetwork/database/berkeleydb/downloads/jeoslicense-086837.html > >> >>> Looks like New(or normal?) BSD license or something ... > >> >>> also > >> >>> " > >> >>> Licensing > >> >>> > >> >>> Berkeley DB is available under dual license: > >> >>> > >> >>> - Public license that requires that software that uses the > >> Berkeley > >> >> DB > >> >>> code be free/open source software; and > >> >>> - Closed source license for non-open source software. > >> >>> > >> >>> If your code is not redistributed, no license is required (free for > >> >> in-house > >> >>> use). > >> >>> > >> >>> " > >> >>> > >> >>> from http://www.orafaq.com/wiki/Berkeley_DB#Licensing > >> >>> > >> >>> > >> >>> I would totally use neo4j, if it would be as fast at searches :/ ie. > >> >> BTree > >> >>> storage of nodes/rels? (guessing) > >> >>> > >> >>> But having 10mil rels, and seeing BDB checking if A-->B in 0ms, and > >> neo4j > >> >> in > >> >>> like 0 to 66 to 310 seconds (depending on its position) > >> >>> > >> >>> is a show stopper for me, especially because I want to base everything > >> on > >> >>> just nodes (without properties) and their relationships. ie. make a > >> set > >> >> or > >> >>> list of things, without having A ---[ENTRY]--> e ---[NEXT] ---> e2 > >> but > >> >>> instead A->b->e->c->e2 where b and c are just nodes, and also > >> >>> AllEntries->b and AllNexts->c (silly example with such less info > >> tho) > >> >>> > >> >>> Point is, I would do lots of searches a lot (imagine a real time > >> program > >> >>> running on top of nodes/rels, that is it's defined in and can access > >> only > >> >>> nodes), this would likely cause those ms to add up to seconds... > >> >>> > >> >>> > >> >>> I installed maven (m2e) again, I guess I could use it, but it seems it > >> >>> creates .jar , not sure if that's useful to me while I am coding... > >> seems > >> >>> better to use project/sources no? and maven only when ready to > >> >> publish/get > >> >>> the jar ; anyway I need to learn how to use it otherwise I'm getting > >> >> errors > >> >>> like this , when trying to build: > >> >>> > >> >>> [ERROR] The project org.neo4j:neo4j-graph-collections:1.5-SNAPSHOT > >> >>> (E:\wrkspc\graph-collections\pom.xml) has 1 error > >> >>> [ERROR] Non-resolvable parent POM: The repository system is > >> offline > >> >> but > >> >>> the artifact org.neo4j:parent-central:pom:21 is not available in the > >> >> local > >> >>> repositor > >> >>> y. and 'parent.relativePath' points at wrong local POM @ line 4, > >> column > >> >> 11 > >> >>> -> [Help 2] > >> >>> > >> >>> > >> >>> > >> >>> Anyway, with normal eclipse, I'm still showing 2 different errors: > >> >>> > >> >>> 1) in org.neo4j.collections.graphdb.ComparablePropertyType<T> > >> >>> > >> >>> line 29: super(name, graphDb); > >> >>> > >> >>> The constructor PropertyType<T>(String, GraphDatabaseService) is not > >> >> visible > >> >>> > >> >>> 2) org.neo4j.collections.graphdb.impl.NodeLikeImpl.getRelationships() > >> >>> The return type is incompatible with > >> >>> RelationshipContainer.getRelationships() > >> >>> > >> >>> 3) > >> >>> > >> >> > >> org.neo4j.collections.graphdb.impl.NodeLikeImpl.getRelationships(RelationshipType...) > >> >>> The return type is incompatible with > >> >>> RelationshipContainer.getRelationships(RelationshipType[]) > >> >>> > >> >>> > >> >>> John. > >> >>> > >> >>> On Thu, Jul 28, 2011 at 12:52 PM, Niels Hoogeveen< > >> >> pd_aficion...@hotmail.com > >> >>>> wrote: > >> >>> > >> >>>> > >> >>>> Hi John, > >> >>>> Thanks for showing an interest. > >> >>>> The compile error you got was due to the fact that a removed class > >> was > >> >>>> still hanging around in the Git repo. I renamed > >> BinaryRelationshipRoles > >> >> into > >> >>>> BinaryRelationshipRole, but the original file was still active in the > >> >> Git > >> >>>> repo. I fixed that. > >> >>>> I have been thinking about BDB too for this situation, because the > >> graph > >> >>>> database now stores some information about the associated nodes and > >> >> their > >> >>>> reverse lookup. This of course polutes the name/node space. It would > >> be > >> >> neat > >> >>>> to offload this book keeping information to some persistent hashmap, > >> so > >> >> the > >> >>>> implementation is completely transparent to the user. > >> >>>> I don't know how nicely BDB plays with Neo4J transactions. Does > >> anyone > >> >> have > >> >>>> experience with this? > >> >>>> Another aspect is licencing. I am no legal buff, so maybe someone > >> else > >> >> can > >> >>>> jump in and answer this. > >> >>>> Personally, I don't mind adding BDB as a dependency, but it has to > >> work > >> >>>> well at the transaction level and licence wise, otherwise it's a no > >> go > >> >> for > >> >>>> me. > >> >>>> I would recommend you to start using maven. There is an Eclipse > >> plugin > >> >>>> m2eclipse, which allows you to use/maintain Maven projects from > >> within > >> >>>> Eclipse. > >> >>>> Niels > >> >>>> > >> >>>>> Date: Thu, 28 Jul 2011 05:09:54 +0200 > >> >>>>> From: cyuczie...@gmail.com > >> >>>>> To: user@lists.neo4j.org > >> >>>>> Subject: Re: [Neo4j] HyperRelationship example > >> >>>>> > >> >>>>> Hey Niels, > >> >>>>> > >> >>>>> I like xD > >> >>>>> this seems like a lot of work and professionally done; ie. something > >> I > >> >>>> could > >> >>>>> not have done (I don't have that kind of experience and focus). > >> Gratz > >> >> on > >> >>>>> that, I really appreciate seeing this. > >> >>>>> > >> >>>>> I cloned the repo from git, manually, with eclipse (not using maven > >> - > >> >>>> don't > >> >>>>> know how with eclipse) > >> >>>>> I am getting only about 3 compile errors, like: > >> >>>>> 1) The type BinaryRelationshipRoles<T> must implement the > >> inherited > >> >>>> abstract > >> >>>>> method PropertyContainer.getId() > >> >>>>> 2) The constructor PropertyType<T>(String, GraphDatabaseService) is > >> not > >> >>>>> visible > >> >>>>> 3) The return type is incompatible with > >> >>>>> RelationshipContainer.getRelationships() > >> >>>>> for > >> >>>>> > >> >>>> > >> >> > >> org.neo4j.collections.graphdb.impl.RelationshipIterable.RelationshipIterable(Iterable<Relationship> > >> >>>>> rels) > >> >>>>> > >> >>>>> > >> >>>>> Also, I am thinking to try and implement this on top of > >> berkeleydb > >> >> just > >> >>>>> for fun/benchmarking (so to speak) to compare between that and neo4j > >> - > >> >>>> since > >> >>>>> I am currently unsure which one to use for my hobby project (I like > >> >> that > >> >>>>> berkeleydb's searches are 0-1ms instead of few seconds) > >> >>>>> > >> >>>>> Btw, would it be any interest to you if I were to fork your repo and > >> >> add > >> >>>> ie. > >> >>>>> AllTests.java for junit and the .project and related files for > >> eclipse > >> >>>>> project in a pull or two ? as long as it doesn't seem useless or > >> >>>>> cluttering... (note however I never actually, yet, used fork&pull > >> but > >> >>>> only > >> >>>>> read about it on github xD) > >> >>>>> > >> >>>>> Thanks to all, for wasting some time reading this, > >> >>>>> Greeting and salutations, > >> >>>>> John > >> >>>>> > >> >>>>> On Wed, Jul 27, 2011 at 8:48 PM, Niels Hoogeveen > >> >>>>> <pd_aficion...@hotmail.com>wrote: > >> >>>>> > >> >>>>>> > >> >>>>>> I just posted an example on how to use HyperRelationships: > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>> > >> >> > >> https://github.com/peterneubauer/graph-collections/wiki/HyperRelationship-example > >> >>>>>> > >> >>>>>> There is now a proper test for HyperRelationships, so I hereby push > >> >> the > >> >>>>>> software to Beta status. > >> >>>>>> > >> >>>>>> Please try out the Enhanced API and HyperRelationships and let me > >> know > >> >>>> what > >> >>>>>> needs improvement. > >> >>>>>> > >> >>>>>> Niels > >> >>>>>> _______________________________________________ > >> >>>>>> Neo4j mailing list > >> >>>>>> User@lists.neo4j.org > >> >>>>>> https://lists.neo4j.org/mailman/listinfo/user > >> >>>>>> > >> >>>>> _______________________________________________ > >> >>>>> Neo4j mailing list > >> >>>>> User@lists.neo4j.org > >> >>>>> https://lists.neo4j.org/mailman/listinfo/user > >> >>>> > >> >>>> _______________________________________________ > >> >>>> Neo4j mailing list > >> >>>> User@lists.neo4j.org > >> >>>> https://lists.neo4j.org/mailman/listinfo/user > >> >>>> > >> >>> _______________________________________________ > >> >>> Neo4j mailing list > >> >>> User@lists.neo4j.org > >> >>> https://lists.neo4j.org/mailman/listinfo/user > >> >> _______________________________________________ > >> >> Neo4j mailing list > >> >> User@lists.neo4j.org > >> >> https://lists.neo4j.org/mailman/listinfo/user > >> >> > >> > _______________________________________________ > >> > Neo4j mailing list > >> > User@lists.neo4j.org > >> > https://lists.neo4j.org/mailman/listinfo/user > >> _______________________________________________ > >> Neo4j mailing list > >> User@lists.neo4j.org > >> https://lists.neo4j.org/mailman/listinfo/user > >> > > > > > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
