On Tue, Apr 19, 2011 at 10:17 PM, Michael DeHaan <michael.deh...@gmail.com>wrote:
> On Tue, Apr 19, 2011 at 10:58 AM, Jim Webber <j...@neotechnology.com> > wrote: > >>> I'd like to propose that we put this functionality into the plugin ( > https://github.com/skanjila/gremlin-translation-plugin) that Peter and I > are currently working on, thoughts? > > > > I'm thinking that, if we do it, it should be handled through content > negotiation. That is if you ask for application/atom then you get paged > lists of results. I don't necessarily think that's a plugin, it's more > likely part of the representation logic in server itself. > > This is something I've been wondering about as I may have the need to > feed very large graphs into the system and am wondering how the REST > API will hold up compared to the native interface. > > What happens if the result of an index query (or traversal, whatever) > legitimately needs to return 100k results? > > Wouldn't that be a bit large for one request? If anything, it's a > lot of JSON to decode at once. > > Yeah, we can't do this right now, and implementing it is harder than it seems at first glance, since we first need to implement sorting of results, otherwise the paged result will be useless. Like Jim said though, this is another one of those *must be done* features. > Feeds make sense for things that are feed-like, but do atom feeds > really make sense for results of very dynamic queries that don't get > subscribed to? > Or, related question, is there a point where the result sets of > operations get so large that things start to break down? What do > people find this to generally be? > I'm sure there are some awesome content types out there that we can look at that will fit our uses, I don't feel confident to say if Atom is a good choice, I've never worked with it.. The point where this breaks down I'm gonna guess is in server-side serialization, because we currently don't stream the serialized data, but build it up in memory and ship it off when it's done. I'd say you'll run out of memory after 10000 nodes or so on a small server, which I think underlines how important this is to fix. > > Maybe it's not an issue, but pointers to any problems REST API usage > has with large data sets (and solutions?) would be welcome. > Not aware of anyone bumping into these limits yet, but I'm sure we'll start hearing about it.. The only current solution I can think of is a server plugin that emulates this, but it would have to sort the result, and I'm afraid that it will be hard (probably not impossible, but hard) to implement that in a memory-efficient way that far away from the kernel. You may just end up moving the OutOfMemeoryExceptions' to the plugin instead of the serialization system. > > --Michael > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Jacob Hansson Phone: +46 (0) 763503395 Twitter: @jakewins _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
