Re: [Neo4j] Paged traversal REST API suggestion for improvement
> All of this works, without JS. That means the client (your browser) isn't > the party doing the paging or keeping track of all results. It's something > on the server side. That "something on the server side" is called "Google". That is, there is an organisation which has set up databases, processing, caching, and a myriad of other things and exposed them through a Web server. Saying Neo4j should do this because Google does isn't a fair comparison - Google has more than a database, it has an entire application. I'd say your app should do this, and if you can find a performant way of doing it that doesn't hold server-side state then I'll be happy to take a patch for the REST API. Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Paged traversal REST API suggestion for improvement
Hi Aseem, > Jim, thanks for the explanation. I understand your constraints, but thinking > about it more, I'm actually even more baffled -- how can we actually make > use of this paged traversal API in a web scenario? Neo4j's traversers (which is what the REST paging API really is) knows nothing about the back button. It is simply a way of getting bite sized chunks of results back to the client while minimising client, server, and network overhead. It's more discrete streaming than a hypermedia set of pages. If your client wants to provide navigation, then it can. But it is your client's job to cache previous pages that it's pulled from the server. The server does not support prev navigation because the aggregate costs to the server are prohibitive. [snip] > It seems to me that the root cause of this dilemma is the Iterable > constraint you mentioned -- it only goes forward. Yes it does, and for good performance reasons. > Perhaps a better API/model then would be to use an offset parameter. This is > different than "page" because it doesn't require the server to keep state; > it just tells the server, return me the results starting *after* this result > (which would generally be the last result I saw on my previous page of > results). The offset would require us to go forward through the iterator to the offset. Then return from there. I'd considered it briefly, but then dropped the idea since it's extremely wasteful. > - "Get me the results of this traverse" would return everything That's what the normal (non-paged) traversal API does. > - "Get me the results of this traverse, paged with size 20" would return 20 > results That's what the paged traversal API does. > - "Get me the results of this traverse, paged with size 20, starting with > the node/rel after this one [given by an ID]" would return the next 20 > results That's what (sort of) repeated GETs on the traversal API does, with the caveat that you already specified the page size when you set up the traverser. But we can't give you the ability to look up an arbitrary page on the server. That doesn't even make sense since between invocations the graph may mutate and you'll get different results back anyway. Jim PS - if you want to be prev friendly, then think about making the entire prev page cacheable rather than placing another call on the server for the previous page and doing the hard work of rendering it all over again. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Paged traversal REST API suggestion for improvement
Haha. I think we're speaking past each other. Here's my point: Disable JS. Open up Google. Do a search. Click to the next page. Click to back to the previous page. Click page 3. Refresh. Etc. All of this works, without JS. That means the client (your browser) isn't the party doing the paging or keeping track of all results. It's something on the server side. My original question was: I don't see how one can achieve this using Neo4j's paged traversal API. Is that not a reasonable scenario / use case for it? If not, what is? My follow-up suggestion was: could we not support this scenario / use case by changing (or expanding) the API to support an "offset" parameter? In other words, random access -- like keying into a graph by index and then following edges -- instead of forward-only and once-only Iterable. Aseem On Thu, Jul 28, 2011 at 4:27 PM, Rick Bullotta wrote: > Um, I'm guessing that you aren't aware of how Google's UI/API works... > > Open up Firebug, Chrome Tools, or Fiddler, and you'll see that the biggest > chunk of traffic when you switch from one page of Google search results to > another page (via the main Google search page) is a JSON packet...which uses > their search API...which returns JSON. Do they page? Yes. Can you call > the API and get hundreds of results at once? Yes... > > > > -Original Message- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > On Behalf Of Aseem Kishore > Sent: Thursday, July 28, 2011 6:42 PM > To: Neo4j user discussions > Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement > > Sorry, I don't understand. Are you going to tell Google and Bing to send > JSON for their search results also, and page on the client side? There is a > very valid use case for not wanting to require JavaScript to page. > > Aseem > > On Thu, Jul 28, 2011 at 3:25 PM, Rick Bullotta > wrote: > > > Not really, unless the JSON content is HUGE - which is rare. The small > > price in bandwidth versus a responsive UI and the need for not managing > > state is a good tradeoff. The only exception might be mobile apps, but > in > > that case, I'd just fetch a smaller block (maybe 4 or 5 pages worth). > Very > > few users have the patience to page through more than a few pages worth > of > > information. > > > > -Original Message----- > > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > > On Behalf Of Daniel Gasienica > > Sent: Thursday, July 28, 2011 6:20 PM > > To: Neo4j user discussions > > Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement > > > > Sure but isn't it a huge waste of bandwidth if you load hundreds of > results > > and the user only looks at the first dozen? > > > > On Thu, Jul 28, 2011 at 15:16, Rick Bullotta < > rick.bullo...@thingworx.com > > >wrote: > > > > > BTW, "paging" is a relic of the dial-up modem days, IMNSHO. > > > > > > If a machine is the client of the REST API call, it should get all the > > data > > > in a single, atomic call. If it is a browser or mobile app, there's a > > > natural limit to the # of items that a human will bother paging > through, > > > which is fairly small (20 -> 500 typically), so again, it is painless > to > > get > > > all the data in a single call. > > > > > > -Original Message- > > > From: user-boun...@lists.neo4j.org [mailto: > user-boun...@lists.neo4j.org] > > > On Behalf Of Rick Bullotta > > > Sent: Thursday, July 28, 2011 6:11 PM > > > To: Neo4j user discussions > > > Subject: Re: [Neo4j] Paged traversal REST API suggestion for > improvement > > > > > > If you don't keep "state" paging will not work properly if the data > > changes > > > often. What may have been record #21 when you are viewing the first > page > > of > > > 20 result might not be record #21 when you go to fetch the 2nd "page". > > > > > > If you're only concerned with pages of 20 or so, you should absolutely, > > > positively do the paging on the browser. Anything up to a couple > > thousand > > > rows can *easily* be handled by any modern browser. Keep the parsed > JSON > > > data on the client side and page it there. This approach will be very > > > performant on both the client and server, and won't require any > "state". > > > > > > -Original Message- > > > From: user-boun...@lists.neo4j.o
Re: [Neo4j] Paged traversal REST API suggestion for improvement
Um, I'm guessing that you aren't aware of how Google's UI/API works... Open up Firebug, Chrome Tools, or Fiddler, and you'll see that the biggest chunk of traffic when you switch from one page of Google search results to another page (via the main Google search page) is a JSON packet...which uses their search API...which returns JSON. Do they page? Yes. Can you call the API and get hundreds of results at once? Yes... -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Aseem Kishore Sent: Thursday, July 28, 2011 6:42 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Sorry, I don't understand. Are you going to tell Google and Bing to send JSON for their search results also, and page on the client side? There is a very valid use case for not wanting to require JavaScript to page. Aseem On Thu, Jul 28, 2011 at 3:25 PM, Rick Bullotta wrote: > Not really, unless the JSON content is HUGE - which is rare. The small > price in bandwidth versus a responsive UI and the need for not managing > state is a good tradeoff. The only exception might be mobile apps, but in > that case, I'd just fetch a smaller block (maybe 4 or 5 pages worth). Very > few users have the patience to page through more than a few pages worth of > information. > > -Original Message- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > On Behalf Of Daniel Gasienica > Sent: Thursday, July 28, 2011 6:20 PM > To: Neo4j user discussions > Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement > > Sure but isn't it a huge waste of bandwidth if you load hundreds of results > and the user only looks at the first dozen? > > On Thu, Jul 28, 2011 at 15:16, Rick Bullotta >wrote: > > > BTW, "paging" is a relic of the dial-up modem days, IMNSHO. > > > > If a machine is the client of the REST API call, it should get all the > data > > in a single, atomic call. If it is a browser or mobile app, there's a > > natural limit to the # of items that a human will bother paging through, > > which is fairly small (20 -> 500 typically), so again, it is painless to > get > > all the data in a single call. > > > > -Original Message- > > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > > On Behalf Of Rick Bullotta > > Sent: Thursday, July 28, 2011 6:11 PM > > To: Neo4j user discussions > > Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement > > > > If you don't keep "state" paging will not work properly if the data > changes > > often. What may have been record #21 when you are viewing the first page > of > > 20 result might not be record #21 when you go to fetch the 2nd "page". > > > > If you're only concerned with pages of 20 or so, you should absolutely, > > positively do the paging on the browser. Anything up to a couple > thousand > > rows can *easily* be handled by any modern browser. Keep the parsed JSON > > data on the client side and page it there. This approach will be very > > performant on both the client and server, and won't require any "state". > > > > -Original Message- > > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > > On Behalf Of Aseem Kishore > > Sent: Thursday, July 28, 2011 3:01 PM > > To: Neo4j user discussions > > Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement > > > > Jim, thanks for the explanation. I understand your constraints, but > > thinking > > about it more, I'm actually even more baffled -- how can we actually make > > use of this paged traversal API in a web scenario? > > > > [neo4j db] < [web server] < [web client, e.g. browser] > > > > If I want to show the results of a search, let's say, in a way that > > supports > > paging, it just seems like I can't take advantage of the Neo4j API at all > > here. > > > > If the user wants to be able to refresh the page, or click "Prev", the > > browser isn't keeping state across page changes, so it has to rely on > the > > web server to give it the right page. > > > > If the server now has to keep state of all results, it defeats the > purpose > > of using the Neo4j paging API altogether: the server necessarily has to > > fetch everything and handle paging itself. > > > > Am I missing something? Or is this (typical in our case) scenario not > > something y
Re: [Neo4j] Paged traversal REST API suggestion for improvement
Sorry, I don't understand. Are you going to tell Google and Bing to send JSON for their search results also, and page on the client side? There is a very valid use case for not wanting to require JavaScript to page. Aseem On Thu, Jul 28, 2011 at 3:25 PM, Rick Bullotta wrote: > Not really, unless the JSON content is HUGE - which is rare. The small > price in bandwidth versus a responsive UI and the need for not managing > state is a good tradeoff. The only exception might be mobile apps, but in > that case, I'd just fetch a smaller block (maybe 4 or 5 pages worth). Very > few users have the patience to page through more than a few pages worth of > information. > > -Original Message- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > On Behalf Of Daniel Gasienica > Sent: Thursday, July 28, 2011 6:20 PM > To: Neo4j user discussions > Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement > > Sure but isn’t it a huge waste of bandwidth if you load hundreds of results > and the user only looks at the first dozen? > > On Thu, Jul 28, 2011 at 15:16, Rick Bullotta >wrote: > > > BTW, "paging" is a relic of the dial-up modem days, IMNSHO. > > > > If a machine is the client of the REST API call, it should get all the > data > > in a single, atomic call. If it is a browser or mobile app, there's a > > natural limit to the # of items that a human will bother paging through, > > which is fairly small (20 -> 500 typically), so again, it is painless to > get > > all the data in a single call. > > > > -Original Message- > > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > > On Behalf Of Rick Bullotta > > Sent: Thursday, July 28, 2011 6:11 PM > > To: Neo4j user discussions > > Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement > > > > If you don't keep "state" paging will not work properly if the data > changes > > often. What may have been record #21 when you are viewing the first page > of > > 20 result might not be record #21 when you go to fetch the 2nd "page". > > > > If you're only concerned with pages of 20 or so, you should absolutely, > > positively do the paging on the browser. Anything up to a couple > thousand > > rows can *easily* be handled by any modern browser. Keep the parsed JSON > > data on the client side and page it there. This approach will be very > > performant on both the client and server, and won't require any "state". > > > > -Original Message- > > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > > On Behalf Of Aseem Kishore > > Sent: Thursday, July 28, 2011 3:01 PM > > To: Neo4j user discussions > > Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement > > > > Jim, thanks for the explanation. I understand your constraints, but > > thinking > > about it more, I'm actually even more baffled -- how can we actually make > > use of this paged traversal API in a web scenario? > > > > [neo4j db] < [web server] < [web client, e.g. browser] > > > > If I want to show the results of a search, let's say, in a way that > > supports > > paging, it just seems like I can't take advantage of the Neo4j API at all > > here. > > > > If the user wants to be able to refresh the page, or click "Prev", the > > browser isn't keeping state across page changes, so it has to rely on > the > > web server to give it the right page. > > > > If the server now has to keep state of all results, it defeats the > purpose > > of using the Neo4j paging API altogether: the server necessarily has to > > fetch everything and handle paging itself. > > > > Am I missing something? Or is this (typical in our case) scenario not > > something you guys are designing for? (And if not, what *is* the use case > > you guys envisioned for this?) > > > > It seems to me that the root cause of this dilemma is the Iterable > > constraint you mentioned -- it only goes forward. > > > > Perhaps a better API/model then would be to use an offset parameter. This > > is > > different than "page" because it doesn't require the server to keep > state; > > it just tells the server, return me the results starting *after* this > > result > > (which would generally be the last result I saw on my previous page of > > results). > > > > E.g. the following requests: > > >
Re: [Neo4j] Paged traversal REST API suggestion for improvement
Not really, unless the JSON content is HUGE - which is rare. The small price in bandwidth versus a responsive UI and the need for not managing state is a good tradeoff. The only exception might be mobile apps, but in that case, I'd just fetch a smaller block (maybe 4 or 5 pages worth). Very few users have the patience to page through more than a few pages worth of information. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Daniel Gasienica Sent: Thursday, July 28, 2011 6:20 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Sure but isn’t it a huge waste of bandwidth if you load hundreds of results and the user only looks at the first dozen? On Thu, Jul 28, 2011 at 15:16, Rick Bullotta wrote: > BTW, "paging" is a relic of the dial-up modem days, IMNSHO. > > If a machine is the client of the REST API call, it should get all the data > in a single, atomic call. If it is a browser or mobile app, there's a > natural limit to the # of items that a human will bother paging through, > which is fairly small (20 -> 500 typically), so again, it is painless to get > all the data in a single call. > > -Original Message- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > On Behalf Of Rick Bullotta > Sent: Thursday, July 28, 2011 6:11 PM > To: Neo4j user discussions > Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement > > If you don't keep "state" paging will not work properly if the data changes > often. What may have been record #21 when you are viewing the first page of > 20 result might not be record #21 when you go to fetch the 2nd "page". > > If you're only concerned with pages of 20 or so, you should absolutely, > positively do the paging on the browser. Anything up to a couple thousand > rows can *easily* be handled by any modern browser. Keep the parsed JSON > data on the client side and page it there. This approach will be very > performant on both the client and server, and won't require any "state". > > -Original Message- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > On Behalf Of Aseem Kishore > Sent: Thursday, July 28, 2011 3:01 PM > To: Neo4j user discussions > Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement > > Jim, thanks for the explanation. I understand your constraints, but > thinking > about it more, I'm actually even more baffled -- how can we actually make > use of this paged traversal API in a web scenario? > > [neo4j db] < [web server] < [web client, e.g. browser] > > If I want to show the results of a search, let's say, in a way that > supports > paging, it just seems like I can't take advantage of the Neo4j API at all > here. > > If the user wants to be able to refresh the page, or click "Prev", the > browser isn't keeping state across page changes, so it has to rely on the > web server to give it the right page. > > If the server now has to keep state of all results, it defeats the purpose > of using the Neo4j paging API altogether: the server necessarily has to > fetch everything and handle paging itself. > > Am I missing something? Or is this (typical in our case) scenario not > something you guys are designing for? (And if not, what *is* the use case > you guys envisioned for this?) > > It seems to me that the root cause of this dilemma is the Iterable > constraint you mentioned -- it only goes forward. > > Perhaps a better API/model then would be to use an offset parameter. This > is > different than "page" because it doesn't require the server to keep state; > it just tells the server, return me the results starting *after* this > result > (which would generally be the last result I saw on my previous page of > results). > > E.g. the following requests: > > - "Get me the results of this traverse" would return everything > - "Get me the results of this traverse, paged with size 20" would return 20 > results > - "Get me the results of this traverse, paged with size 20, starting with > the node/rel after this one [given by an ID]" would return the next 20 > results > - etc. > > Aseem > > > On Thu, Jul 28, 2011 at 5:09 AM, Jim Webber wrote: > > > Hi Aseem, > > > > When you GET a resource, you're asking for its state at a particular > > instant in time. GET's don't always return the same content, what's > > important about them is that clients can't be held responsible for any > > (
Re: [Neo4j] Paged traversal REST API suggestion for improvement
Sure but isn’t it a huge waste of bandwidth if you load hundreds of results and the user only looks at the first dozen? On Thu, Jul 28, 2011 at 15:16, Rick Bullotta wrote: > BTW, "paging" is a relic of the dial-up modem days, IMNSHO. > > If a machine is the client of the REST API call, it should get all the data > in a single, atomic call. If it is a browser or mobile app, there's a > natural limit to the # of items that a human will bother paging through, > which is fairly small (20 -> 500 typically), so again, it is painless to get > all the data in a single call. > > -Original Message- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > On Behalf Of Rick Bullotta > Sent: Thursday, July 28, 2011 6:11 PM > To: Neo4j user discussions > Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement > > If you don't keep "state" paging will not work properly if the data changes > often. What may have been record #21 when you are viewing the first page of > 20 result might not be record #21 when you go to fetch the 2nd "page". > > If you're only concerned with pages of 20 or so, you should absolutely, > positively do the paging on the browser. Anything up to a couple thousand > rows can *easily* be handled by any modern browser. Keep the parsed JSON > data on the client side and page it there. This approach will be very > performant on both the client and server, and won't require any "state". > > -Original Message- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > On Behalf Of Aseem Kishore > Sent: Thursday, July 28, 2011 3:01 PM > To: Neo4j user discussions > Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement > > Jim, thanks for the explanation. I understand your constraints, but > thinking > about it more, I'm actually even more baffled -- how can we actually make > use of this paged traversal API in a web scenario? > > [neo4j db] < [web server] < [web client, e.g. browser] > > If I want to show the results of a search, let's say, in a way that > supports > paging, it just seems like I can't take advantage of the Neo4j API at all > here. > > If the user wants to be able to refresh the page, or click "Prev", the > browser isn't keeping state across page changes, so it has to rely on the > web server to give it the right page. > > If the server now has to keep state of all results, it defeats the purpose > of using the Neo4j paging API altogether: the server necessarily has to > fetch everything and handle paging itself. > > Am I missing something? Or is this (typical in our case) scenario not > something you guys are designing for? (And if not, what *is* the use case > you guys envisioned for this?) > > It seems to me that the root cause of this dilemma is the Iterable > constraint you mentioned -- it only goes forward. > > Perhaps a better API/model then would be to use an offset parameter. This > is > different than "page" because it doesn't require the server to keep state; > it just tells the server, return me the results starting *after* this > result > (which would generally be the last result I saw on my previous page of > results). > > E.g. the following requests: > > - "Get me the results of this traverse" would return everything > - "Get me the results of this traverse, paged with size 20" would return 20 > results > - "Get me the results of this traverse, paged with size 20, starting with > the node/rel after this one [given by an ID]" would return the next 20 > results > - etc. > > Aseem > > > On Thu, Jul 28, 2011 at 5:09 AM, Jim Webber wrote: > > > Hi Aseem, > > > > When you GET a resource, you're asking for its state at a particular > > instant in time. GET's don't always return the same content, what's > > important about them is that clients can't be held responsible for any > > (unintended) side-effects of a GET operation. For example, if you GET the > > BBC new page now, and then GET it again in 20 minutes, the bytes you > receive > > will almost certainly be different. > > > > I think what's perhaps a bit odd in the paged traverser case is that > > clients do in fact trigger a state transition on GET which materially > > changes the response. Adding /next to the paged traverser URI does not > > change this, it just lengthens the URI. > > > > Under the covers, the REST API is just using the traversal API and so > gets > > back an Iterable which is forward-onl
Re: [Neo4j] Paged traversal REST API suggestion for improvement
BTW, "paging" is a relic of the dial-up modem days, IMNSHO. If a machine is the client of the REST API call, it should get all the data in a single, atomic call. If it is a browser or mobile app, there's a natural limit to the # of items that a human will bother paging through, which is fairly small (20 -> 500 typically), so again, it is painless to get all the data in a single call. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Rick Bullotta Sent: Thursday, July 28, 2011 6:11 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement If you don't keep "state" paging will not work properly if the data changes often. What may have been record #21 when you are viewing the first page of 20 result might not be record #21 when you go to fetch the 2nd "page". If you're only concerned with pages of 20 or so, you should absolutely, positively do the paging on the browser. Anything up to a couple thousand rows can *easily* be handled by any modern browser. Keep the parsed JSON data on the client side and page it there. This approach will be very performant on both the client and server, and won't require any "state". -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Aseem Kishore Sent: Thursday, July 28, 2011 3:01 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Jim, thanks for the explanation. I understand your constraints, but thinking about it more, I'm actually even more baffled -- how can we actually make use of this paged traversal API in a web scenario? [neo4j db] < [web server] < [web client, e.g. browser] If I want to show the results of a search, let's say, in a way that supports paging, it just seems like I can't take advantage of the Neo4j API at all here. If the user wants to be able to refresh the page, or click "Prev", the browser isn't keeping state across page changes, so it has to rely on the web server to give it the right page. If the server now has to keep state of all results, it defeats the purpose of using the Neo4j paging API altogether: the server necessarily has to fetch everything and handle paging itself. Am I missing something? Or is this (typical in our case) scenario not something you guys are designing for? (And if not, what *is* the use case you guys envisioned for this?) It seems to me that the root cause of this dilemma is the Iterable constraint you mentioned -- it only goes forward. Perhaps a better API/model then would be to use an offset parameter. This is different than "page" because it doesn't require the server to keep state; it just tells the server, return me the results starting *after* this result (which would generally be the last result I saw on my previous page of results). E.g. the following requests: - "Get me the results of this traverse" would return everything - "Get me the results of this traverse, paged with size 20" would return 20 results - "Get me the results of this traverse, paged with size 20, starting with the node/rel after this one [given by an ID]" would return the next 20 results - etc. Aseem On Thu, Jul 28, 2011 at 5:09 AM, Jim Webber wrote: > Hi Aseem, > > When you GET a resource, you're asking for its state at a particular > instant in time. GET's don't always return the same content, what's > important about them is that clients can't be held responsible for any > (unintended) side-effects of a GET operation. For example, if you GET the > BBC new page now, and then GET it again in 20 minutes, the bytes you receive > will almost certainly be different. > > I think what's perhaps a bit odd in the paged traverser case is that > clients do in fact trigger a state transition on GET which materially > changes the response. Adding /next to the paged traverser URI does not > change this, it just lengthens the URI. > > Under the covers, the REST API is just using the traversal API and so gets > back an Iterable which is forward-only. We can't (or rather won't) do > things like ?page=2 (or similar) because that would involve holding all the > traversal state on the server which is one of the things paged traversers > were meant to avoid (being kind to both the server and clients). > > If you want to remember all traversal state on the client, that's OK. You > can page through your own local data structures. In fact given the number of > clients is likely to be larger than the number of servers, this makes sense > since the memory burden is shared amongst the many clients, rather than > being placed on the single server.
Re: [Neo4j] Paged traversal REST API suggestion for improvement
If you don't keep "state" paging will not work properly if the data changes often. What may have been record #21 when you are viewing the first page of 20 result might not be record #21 when you go to fetch the 2nd "page". If you're only concerned with pages of 20 or so, you should absolutely, positively do the paging on the browser. Anything up to a couple thousand rows can *easily* be handled by any modern browser. Keep the parsed JSON data on the client side and page it there. This approach will be very performant on both the client and server, and won't require any "state". -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Aseem Kishore Sent: Thursday, July 28, 2011 3:01 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Jim, thanks for the explanation. I understand your constraints, but thinking about it more, I'm actually even more baffled -- how can we actually make use of this paged traversal API in a web scenario? [neo4j db] < [web server] < [web client, e.g. browser] If I want to show the results of a search, let's say, in a way that supports paging, it just seems like I can't take advantage of the Neo4j API at all here. If the user wants to be able to refresh the page, or click "Prev", the browser isn't keeping state across page changes, so it has to rely on the web server to give it the right page. If the server now has to keep state of all results, it defeats the purpose of using the Neo4j paging API altogether: the server necessarily has to fetch everything and handle paging itself. Am I missing something? Or is this (typical in our case) scenario not something you guys are designing for? (And if not, what *is* the use case you guys envisioned for this?) It seems to me that the root cause of this dilemma is the Iterable constraint you mentioned -- it only goes forward. Perhaps a better API/model then would be to use an offset parameter. This is different than "page" because it doesn't require the server to keep state; it just tells the server, return me the results starting *after* this result (which would generally be the last result I saw on my previous page of results). E.g. the following requests: - "Get me the results of this traverse" would return everything - "Get me the results of this traverse, paged with size 20" would return 20 results - "Get me the results of this traverse, paged with size 20, starting with the node/rel after this one [given by an ID]" would return the next 20 results - etc. Aseem On Thu, Jul 28, 2011 at 5:09 AM, Jim Webber wrote: > Hi Aseem, > > When you GET a resource, you're asking for its state at a particular > instant in time. GET's don't always return the same content, what's > important about them is that clients can't be held responsible for any > (unintended) side-effects of a GET operation. For example, if you GET the > BBC new page now, and then GET it again in 20 minutes, the bytes you receive > will almost certainly be different. > > I think what's perhaps a bit odd in the paged traverser case is that > clients do in fact trigger a state transition on GET which materially > changes the response. Adding /next to the paged traverser URI does not > change this, it just lengthens the URI. > > Under the covers, the REST API is just using the traversal API and so gets > back an Iterable which is forward-only. We can't (or rather won't) do > things like ?page=2 (or similar) because that would involve holding all the > traversal state on the server which is one of the things paged traversers > were meant to avoid (being kind to both the server and clients). > > If you want to remember all traversal state on the client, that's OK. You > can page through your own local data structures. In fact given the number of > clients is likely to be larger than the number of servers, this makes sense > since the memory burden is shared amongst the many clients, rather than > being placed on the single server. > > So, I'll buy that we've conflated a little too much into a single GET, but > it's motivated by a sincere trade off to not overburden the server. > > Jim > > > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Paged traversal REST API suggestion for improvement
Jim, thanks for the explanation. I understand your constraints, but thinking about it more, I'm actually even more baffled -- how can we actually make use of this paged traversal API in a web scenario? [neo4j db] < [web server] < [web client, e.g. browser] If I want to show the results of a search, let's say, in a way that supports paging, it just seems like I can't take advantage of the Neo4j API at all here. If the user wants to be able to refresh the page, or click "Prev", the browser isn't keeping state across page changes, so it has to rely on the web server to give it the right page. If the server now has to keep state of all results, it defeats the purpose of using the Neo4j paging API altogether: the server necessarily has to fetch everything and handle paging itself. Am I missing something? Or is this (typical in our case) scenario not something you guys are designing for? (And if not, what *is* the use case you guys envisioned for this?) It seems to me that the root cause of this dilemma is the Iterable constraint you mentioned -- it only goes forward. Perhaps a better API/model then would be to use an offset parameter. This is different than "page" because it doesn't require the server to keep state; it just tells the server, return me the results starting *after* this result (which would generally be the last result I saw on my previous page of results). E.g. the following requests: - "Get me the results of this traverse" would return everything - "Get me the results of this traverse, paged with size 20" would return 20 results - "Get me the results of this traverse, paged with size 20, starting with the node/rel after this one [given by an ID]" would return the next 20 results - etc. Aseem On Thu, Jul 28, 2011 at 5:09 AM, Jim Webber wrote: > Hi Aseem, > > When you GET a resource, you're asking for its state at a particular > instant in time. GET's don't always return the same content, what's > important about them is that clients can't be held responsible for any > (unintended) side-effects of a GET operation. For example, if you GET the > BBC new page now, and then GET it again in 20 minutes, the bytes you receive > will almost certainly be different. > > I think what's perhaps a bit odd in the paged traverser case is that > clients do in fact trigger a state transition on GET which materially > changes the response. Adding /next to the paged traverser URI does not > change this, it just lengthens the URI. > > Under the covers, the REST API is just using the traversal API and so gets > back an Iterable which is forward-only. We can't (or rather won't) do > things like ?page=2 (or similar) because that would involve holding all the > traversal state on the server which is one of the things paged traversers > were meant to avoid (being kind to both the server and clients). > > If you want to remember all traversal state on the client, that's OK. You > can page through your own local data structures. In fact given the number of > clients is likely to be larger than the number of servers, this makes sense > since the memory burden is shared amongst the many clients, rather than > being placed on the single server. > > So, I'll buy that we've conflated a little too much into a single GET, but > it's motivated by a sincere trade off to not overburden the server. > > Jim > > > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Paged traversal REST API suggestion for improvement
Hi Aseem, When you GET a resource, you're asking for its state at a particular instant in time. GET's don't always return the same content, what's important about them is that clients can't be held responsible for any (unintended) side-effects of a GET operation. For example, if you GET the BBC new page now, and then GET it again in 20 minutes, the bytes you receive will almost certainly be different. I think what's perhaps a bit odd in the paged traverser case is that clients do in fact trigger a state transition on GET which materially changes the response. Adding /next to the paged traverser URI does not change this, it just lengthens the URI. Under the covers, the REST API is just using the traversal API and so gets back an Iterable which is forward-only. We can't (or rather won't) do things like ?page=2 (or similar) because that would involve holding all the traversal state on the server which is one of the things paged traversers were meant to avoid (being kind to both the server and clients). If you want to remember all traversal state on the client, that's OK. You can page through your own local data structures. In fact given the number of clients is likely to be larger than the number of servers, this makes sense since the memory burden is shared amongst the many clients, rather than being placed on the single server. So, I'll buy that we've conflated a little too much into a single GET, but it's motivated by a sincere trade off to not overburden the server. Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Paged traversal REST API suggestion for improvement
Hey there, Neo4j's REST API is overall quite awesome and impressive -- it's just such a great *actual* REST API. I was just reading into paged traversals ( http://docs.neo4j.org/chunked/stable/rest-api-traverse.html ) and couldn't help but feel that this isn't the best way to do paging over REST. I dig that you POST to create a paged traverser, which returns a 201 with a Location set to the traverser -- the traverser is indeed a real resource on the server. But, it caught me off guard that you page by repeatedly GETting the same resource over and over. In other words, the resource is changing each time you GET it. Doesn't that violate GET, that state on the server shouldn't be visibly changed, or that GETting the same resource multiple times should return that same resource each time? I've seen other paging APIs return a "next" token in each response, and you include that token in your next request, either as a header or in the query string. I've also seen the simple "?page=2" or "/page/2" approach. This is obviously not a dealbreaker, but it is a limitation -- I have to remember the results on my end and can't refetch them if e.g. the user refreshes the page in their browser. That kinda sucks. What are your guys' thoughts? Aseem ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
