Re: [Neo] Retrieving nodes ordered by property
Getting ordered results from any system always requires sorting, unless the ordering property is stored. And sorting always requires (at least) O(n) memory and O( log(n!) ) time for comparison sort, possibly O( n ) time for sorting integer keys. So if you want results to be sorted on an arbitrary property you will have to sort the entire result set and keep it around during your pagination process (possibly redoing the query + entire sorting + skipping a few pages, to preserve memory). If you know which properties you will want to order your results by when you are designing your database you can store the ordering information in the database. I would suggest a linked list of relationships in between the nodes, in the natural order for the sorting property. You need to be aware of two things with this approach though. The first one is that if the sorting property is unrelated to the filtering property, if you want something like give me all nodes where x=15 ordered by y, you will either have to store separate linked lists for each value of x, filter the result set while traversing through the results as they are ordered by y, or revert to the sorting approach. The second thing to be aware of is that insertion (and changing the value of the ordering property in some node) will require some overhead, to preserve the order. If your queries are as simple as give me all nodes where xLOWER_LIMIT and xUPPER_LIMIT ordered by x, and you can reduce the x property to a long integer value, then the timeline index will do this for you. Otherwise there is no ready made component for this today. Happy hacking, Tobias On Mon, Jan 4, 2010 at 1:30 AM, Raul Raja Martinez raulr...@gmail.comwrote: Hi, Anybody has any experience returning indexed nodes ordered by a given property?. For example return all nodes ordered by creationDate. I understand that if the node property is not indexed I'd have to iterate over all nodes first then order then limit the results which seems overkill to me. I'd like to be able to do... get me all nodes from start to limit ordered by property. This is necessary when the data is iterated over using pagination and the order determines what the next start node is next. ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Tobias Ivarsson tobias.ivars...@neotechnology.com Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Retrieving nodes ordered by property
Hi Tobias, Thanks for the info! I understand the implications of returning ordered nodes. Do you guys plan to build such support natively to neo4j even if it is for a basic set of node properties? While retrieving results ordered by property values or more extensive queries capabilities is provably not easy when dealing with a graph, it is currently one of the main things a programmer encounters when comparing neo4j to relational databases for storage and info retrieval. Would it make sense to have something like node.setOrderedProperty(property, value); so that the linked list based on the natural order of the value is kept internally by neo4j and then have another param in the traversers to specify that the traversal should follow the relationships established by the ordered linked nodes? If you guys think ordering based on property is out of the scope that's fine, I'm just asking in case we can avoid having to roll our own hack or component to get it working since ordered entities is a requirement in all of our current projects where we plan to use neo4j Thanks Raul 2010/1/4 Tobias Ivarsson tobias.ivars...@neotechnology.com: Getting ordered results from any system always requires sorting, unless the ordering property is stored. And sorting always requires (at least) O(n) memory and O( log(n!) ) time for comparison sort, possibly O( n ) time for sorting integer keys. So if you want results to be sorted on an arbitrary property you will have to sort the entire result set and keep it around during your pagination process (possibly redoing the query + entire sorting + skipping a few pages, to preserve memory). If you know which properties you will want to order your results by when you are designing your database you can store the ordering information in the database. I would suggest a linked list of relationships in between the nodes, in the natural order for the sorting property. You need to be aware of two things with this approach though. The first one is that if the sorting property is unrelated to the filtering property, if you want something like give me all nodes where x=15 ordered by y, you will either have to store separate linked lists for each value of x, filter the result set while traversing through the results as they are ordered by y, or revert to the sorting approach. The second thing to be aware of is that insertion (and changing the value of the ordering property in some node) will require some overhead, to preserve the order. If your queries are as simple as give me all nodes where xLOWER_LIMIT and xUPPER_LIMIT ordered by x, and you can reduce the x property to a long integer value, then the timeline index will do this for you. Otherwise there is no ready made component for this today. Happy hacking, Tobias On Mon, Jan 4, 2010 at 1:30 AM, Raul Raja Martinez raulr...@gmail.comwrote: Hi, Anybody has any experience returning indexed nodes ordered by a given property?. For example return all nodes ordered by creationDate. I understand that if the node property is not indexed I'd have to iterate over all nodes first then order then limit the results which seems overkill to me. I'd like to be able to do... get me all nodes from start to limit ordered by property. This is necessary when the data is iterated over using pagination and the order determines what the next start node is next. ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Tobias Ivarsson tobias.ivars...@neotechnology.com Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Raul Raja ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Retrieving nodes ordered by property
There's an addition in LuceneIndexService which can sort the returned results. Take a look at http://wiki.neo4j.org/content/Indexing_with_IndexService#Sorting for more information about that. For range queries, take a look at LuceneFulltextQueryIndexService (http://wiki.neo4j.org/content/Indexing_with_IndexService#Fulltext_indexing_with_lucene_query_syntax). I'll verify that sorting can be done there too. 2010/1/4 Raul Raja Martinez raulr...@gmail.com: Hi, Anybody has any experience returning indexed nodes ordered by a given property?. For example return all nodes ordered by creationDate. I understand that if the node property is not indexed I'd have to iterate over all nodes first then order then limit the results which seems overkill to me. I'd like to be able to do... get me all nodes from start to limit ordered by property. This is necessary when the data is iterated over using pagination and the order determines what the next start node is next. ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Retrieving nodes ordered by property
There are no plans to include any support for setting ordered properties or any of the likes in core Neo4j. Features like this would instead probably find its way into auxiliary components (such as the index utilities), either an existing component where it would fit, or in some new component. As Mattias just said there is support for getting ordered results in the lucene index service already. It doesn't look to me as if the sorting there is very customizable, but I think the most commonly desired orderings are there. /Tobias On Mon, Jan 4, 2010 at 10:17 AM, Raul Raja Martinez raulr...@gmail.comwrote: Hi Tobias, Thanks for the info! I understand the implications of returning ordered nodes. Do you guys plan to build such support natively to neo4j even if it is for a basic set of node properties? While retrieving results ordered by property values or more extensive queries capabilities is provably not easy when dealing with a graph, it is currently one of the main things a programmer encounters when comparing neo4j to relational databases for storage and info retrieval. Would it make sense to have something like node.setOrderedProperty(property, value); so that the linked list based on the natural order of the value is kept internally by neo4j and then have another param in the traversers to specify that the traversal should follow the relationships established by the ordered linked nodes? If you guys think ordering based on property is out of the scope that's fine, I'm just asking in case we can avoid having to roll our own hack or component to get it working since ordered entities is a requirement in all of our current projects where we plan to use neo4j Thanks Raul 2010/1/4 Tobias Ivarsson tobias.ivars...@neotechnology.com: Getting ordered results from any system always requires sorting, unless the ordering property is stored. And sorting always requires (at least) O(n) memory and O( log(n!) ) time for comparison sort, possibly O( n ) time for sorting integer keys. So if you want results to be sorted on an arbitrary property you will have to sort the entire result set and keep it around during your pagination process (possibly redoing the query + entire sorting + skipping a few pages, to preserve memory). If you know which properties you will want to order your results by when you are designing your database you can store the ordering information in the database. I would suggest a linked list of relationships in between the nodes, in the natural order for the sorting property. You need to be aware of two things with this approach though. The first one is that if the sorting property is unrelated to the filtering property, if you want something like give me all nodes where x=15 ordered by y, you will either have to store separate linked lists for each value of x, filter the result set while traversing through the results as they are ordered by y, or revert to the sorting approach. The second thing to be aware of is that insertion (and changing the value of the ordering property in some node) will require some overhead, to preserve the order. If your queries are as simple as give me all nodes where xLOWER_LIMIT and xUPPER_LIMIT ordered by x, and you can reduce the x property to a long integer value, then the timeline index will do this for you. Otherwise there is no ready made component for this today. Happy hacking, Tobias On Mon, Jan 4, 2010 at 1:30 AM, Raul Raja Martinez raulr...@gmail.com wrote: Hi, Anybody has any experience returning indexed nodes ordered by a given property?. For example return all nodes ordered by creationDate. I understand that if the node property is not indexed I'd have to iterate over all nodes first then order then limit the results which seems overkill to me. I'd like to be able to do... get me all nodes from start to limit ordered by property. This is necessary when the data is iterated over using pagination and the order determines what the next start node is next. ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Tobias Ivarsson tobias.ivars...@neotechnology.com Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Raul Raja ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Tobias Ivarsson tobias.ivars...@neotechnology.com Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Retrieving nodes ordered by property
I think a generic solution would mean a generic numerical index. I wrote an index recently that is very similar in principle to the timeline index Tobias mentioned below, except I allow multiple dimensions and indexing over any numerical primitive type. Then we get to the next point, combining indexes. Like in Tobias example: give me all nodes where x=15 ordered by y. This requires an index that works on x and y at the same time. Otherwise you have to index one, and do a brute force search on the other in the result set. I have an idea about dynamically generating a combined index based on queries received by the database. So if the user does a query like the one above, involving both x and y, and both of these are indexed, then a combined index can be added dynamically. The first query will not be fast, but subsequent ones would be. And as far as I can see, a combined index should be as simple as linking nodes between the un-combined indexes. Now who's up for a n-way combination of k-dimensional indexes? I think I'm getting a headache just thinking about it :-) On Mon, Jan 4, 2010 at 10:17 AM, Raul Raja Martinez raulr...@gmail.comwrote: Hi Tobias, Thanks for the info! I understand the implications of returning ordered nodes. Do you guys plan to build such support natively to neo4j even if it is for a basic set of node properties? While retrieving results ordered by property values or more extensive queries capabilities is provably not easy when dealing with a graph, it is currently one of the main things a programmer encounters when comparing neo4j to relational databases for storage and info retrieval. Would it make sense to have something like node.setOrderedProperty(property, value); so that the linked list based on the natural order of the value is kept internally by neo4j and then have another param in the traversers to specify that the traversal should follow the relationships established by the ordered linked nodes? If you guys think ordering based on property is out of the scope that's fine, I'm just asking in case we can avoid having to roll our own hack or component to get it working since ordered entities is a requirement in all of our current projects where we plan to use neo4j Thanks Raul 2010/1/4 Tobias Ivarsson tobias.ivars...@neotechnology.com: Getting ordered results from any system always requires sorting, unless the ordering property is stored. And sorting always requires (at least) O(n) memory and O( log(n!) ) time for comparison sort, possibly O( n ) time for sorting integer keys. So if you want results to be sorted on an arbitrary property you will have to sort the entire result set and keep it around during your pagination process (possibly redoing the query + entire sorting + skipping a few pages, to preserve memory). If you know which properties you will want to order your results by when you are designing your database you can store the ordering information in the database. I would suggest a linked list of relationships in between the nodes, in the natural order for the sorting property. You need to be aware of two things with this approach though. The first one is that if the sorting property is unrelated to the filtering property, if you want something like give me all nodes where x=15 ordered by y, you will either have to store separate linked lists for each value of x, filter the result set while traversing through the results as they are ordered by y, or revert to the sorting approach. The second thing to be aware of is that insertion (and changing the value of the ordering property in some node) will require some overhead, to preserve the order. If your queries are as simple as give me all nodes where xLOWER_LIMIT and xUPPER_LIMIT ordered by x, and you can reduce the x property to a long integer value, then the timeline index will do this for you. Otherwise there is no ready made component for this today. Happy hacking, Tobias On Mon, Jan 4, 2010 at 1:30 AM, Raul Raja Martinez raulr...@gmail.com wrote: Hi, Anybody has any experience returning indexed nodes ordered by a given property?. For example return all nodes ordered by creationDate. I understand that if the node property is not indexed I'd have to iterate over all nodes first then order then limit the results which seems overkill to me. I'd like to be able to do... get me all nodes from start to limit ordered by property. This is necessary when the data is iterated over using pagination and the order determines what the next start node is next. ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Tobias Ivarsson tobias.ivars...@neotechnology.com Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___
Re: [Neo] Retrieving nodes ordered by property
Thanks for all your answers, Everything said makes sense. I will get back to you guys with what we end up doing in case it is of any value for anyone else. 2010/1/4 Craig Taverner cr...@amanzi.com: I think a generic solution would mean a generic numerical index. I wrote an index recently that is very similar in principle to the timeline index Tobias mentioned below, except I allow multiple dimensions and indexing over any numerical primitive type. Then we get to the next point, combining indexes. Like in Tobias example: give me all nodes where x=15 ordered by y. This requires an index that works on x and y at the same time. Otherwise you have to index one, and do a brute force search on the other in the result set. I have an idea about dynamically generating a combined index based on queries received by the database. So if the user does a query like the one above, involving both x and y, and both of these are indexed, then a combined index can be added dynamically. The first query will not be fast, but subsequent ones would be. And as far as I can see, a combined index should be as simple as linking nodes between the un-combined indexes. Now who's up for a n-way combination of k-dimensional indexes? I think I'm getting a headache just thinking about it :-) On Mon, Jan 4, 2010 at 10:17 AM, Raul Raja Martinez raulr...@gmail.comwrote: Hi Tobias, Thanks for the info! I understand the implications of returning ordered nodes. Do you guys plan to build such support natively to neo4j even if it is for a basic set of node properties? While retrieving results ordered by property values or more extensive queries capabilities is provably not easy when dealing with a graph, it is currently one of the main things a programmer encounters when comparing neo4j to relational databases for storage and info retrieval. Would it make sense to have something like node.setOrderedProperty(property, value); so that the linked list based on the natural order of the value is kept internally by neo4j and then have another param in the traversers to specify that the traversal should follow the relationships established by the ordered linked nodes? If you guys think ordering based on property is out of the scope that's fine, I'm just asking in case we can avoid having to roll our own hack or component to get it working since ordered entities is a requirement in all of our current projects where we plan to use neo4j Thanks Raul 2010/1/4 Tobias Ivarsson tobias.ivars...@neotechnology.com: Getting ordered results from any system always requires sorting, unless the ordering property is stored. And sorting always requires (at least) O(n) memory and O( log(n!) ) time for comparison sort, possibly O( n ) time for sorting integer keys. So if you want results to be sorted on an arbitrary property you will have to sort the entire result set and keep it around during your pagination process (possibly redoing the query + entire sorting + skipping a few pages, to preserve memory). If you know which properties you will want to order your results by when you are designing your database you can store the ordering information in the database. I would suggest a linked list of relationships in between the nodes, in the natural order for the sorting property. You need to be aware of two things with this approach though. The first one is that if the sorting property is unrelated to the filtering property, if you want something like give me all nodes where x=15 ordered by y, you will either have to store separate linked lists for each value of x, filter the result set while traversing through the results as they are ordered by y, or revert to the sorting approach. The second thing to be aware of is that insertion (and changing the value of the ordering property in some node) will require some overhead, to preserve the order. If your queries are as simple as give me all nodes where xLOWER_LIMIT and xUPPER_LIMIT ordered by x, and you can reduce the x property to a long integer value, then the timeline index will do this for you. Otherwise there is no ready made component for this today. Happy hacking, Tobias On Mon, Jan 4, 2010 at 1:30 AM, Raul Raja Martinez raulr...@gmail.com wrote: Hi, Anybody has any experience returning indexed nodes ordered by a given property?. For example return all nodes ordered by creationDate. I understand that if the node property is not indexed I'd have to iterate over all nodes first then order then limit the results which seems overkill to me. I'd like to be able to do... get me all nodes from start to limit ordered by property. This is necessary when the data is iterated over using pagination and the order determines what the next start node is next. ___ Neo mailing list User@lists.neo4j.org
[Neo] Retrieving nodes ordered by property
Hi, Anybody has any experience returning indexed nodes ordered by a given property?. For example return all nodes ordered by creationDate. I understand that if the node property is not indexed I'd have to iterate over all nodes first then order then limit the results which seems overkill to me. I'd like to be able to do... get me all nodes from start to limit ordered by property. This is necessary when the data is iterated over using pagination and the order determines what the next start node is next. ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
