I think sorting would need to be optional, since it is likely to be a performance and memory hug on large traversals. I think one of the key benefits of the traversal framework in the Embedded API is being able to traverse and 'stream' a very large graph without occupying much memory. If this can be achieved in the REST API (through pagination), that is a very good thing. I assume the main challenge is being able to freeze a traverser and keep it on hold between client requests for the next page. Perhaps you have already solved that bit?
In my opinion, I would code the sorting as a characteristic of the graph itself, in order to avoid having to sort in the server (and incur the memory/performance hit). So that means I would use a domain-specific solution to sorting. Of course, generic sorting is nice also, but make it optional. On Wed, Apr 20, 2011 at 11:19 AM, Jacob Hansson <ja...@voltvoodoo.com>wrote: > On Tue, Apr 19, 2011 at 10:17 PM, Michael DeHaan > <michael.deh...@gmail.com>wrote: > > > On Tue, Apr 19, 2011 at 10:58 AM, Jim Webber <j...@neotechnology.com> > > wrote: > > >>> I'd like to propose that we put this functionality into the plugin ( > > https://github.com/skanjila/gremlin-translation-plugin) that Peter and I > > are currently working on, thoughts? > > > > > > I'm thinking that, if we do it, it should be handled through content > > negotiation. That is if you ask for application/atom then you get paged > > lists of results. I don't necessarily think that's a plugin, it's more > > likely part of the representation logic in server itself. > > > > This is something I've been wondering about as I may have the need to > > feed very large graphs into the system and am wondering how the REST > > API will hold up compared to the native interface. > > > > What happens if the result of an index query (or traversal, whatever) > > legitimately needs to return 100k results? > > > > Wouldn't that be a bit large for one request? If anything, it's a > > lot of JSON to decode at once. > > > > > Yeah, we can't do this right now, and implementing it is harder than it > seems at first glance, since we first need to implement sorting of results, > otherwise the paged result will be useless. Like Jim said though, this is > another one of those *must be done* features. > > > > Feeds make sense for things that are feed-like, but do atom feeds > > really make sense for results of very dynamic queries that don't get > > subscribed to? > > Or, related question, is there a point where the result sets of > > operations get so large that things start to break down? What do > > people find this to generally be? > > > > I'm sure there are some awesome content types out there that we can look at > that will fit our uses, I don't feel confident to say if Atom is a good > choice, I've never worked with it.. > > The point where this breaks down I'm gonna guess is in server-side > serialization, because we currently don't stream the serialized data, but > build it up in memory and ship it off when it's done. I'd say you'll run > out > of memory after 10000 nodes or so on a small server, which I think > underlines how important this is to fix. > > > > > > Maybe it's not an issue, but pointers to any problems REST API usage > > has with large data sets (and solutions?) would be welcome. > > > > Not aware of anyone bumping into these limits yet, but I'm sure we'll start > hearing about it.. The only current solution I can think of is a server > plugin that emulates this, but it would have to sort the result, and I'm > afraid that it will be hard (probably not impossible, but hard) to > implement > that in a memory-efficient way that far away from the kernel. You may just > end up moving the OutOfMemeoryExceptions' to the plugin instead of the > serialization system. > > > > > > --Michael > > _______________________________________________ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > -- > Jacob Hansson > Phone: +46 (0) 763503395 > Twitter: @jakewins > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
