Re: difference between geospatial search from database angle and from solr angle
Wait! How can you do distance calculations across different shards efficiently? On Thu, Apr 7, 2011 at 7:19 AM, Smiley, David W. dsmi...@mitre.org wrote: I haven't used PostGIS so I can't offer a real comparison. I think if you were to try out both, you'd be impressed with Solr's performance/scalability thanks in large part to its sharding. But for functionality richness in so far as geospatial is concerned, that's where Solr currently comes short. It just has the basic stuff 80% of people want. ~ David Smiley Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/ On Apr 7, 2011, at 2:24 AM, Sean Bigdatafun wrote: Thanks, David. I am thinking of a scenario that billions of objects, whose indices are too big for a single machine to serve the indexing, to serve the querying. Is there any sharding mechanism? Can you give a comparison between solr-based geospatial search and PostGIS based geospatial search? * scalability * functionality richness * incremental indexing (re-indexing) cost * query cost * sharding scheme support -- Lance Norskog goks...@gmail.com
Re: difference between geospatial search from database angle and from solr angle
On Sun, Apr 10, 2011 at 5:24 PM, Lance Norskog goks...@gmail.com wrote: Wait! How can you do distance calculations across different shards efficiently? Basic spatial search (bounding box filter, radius filter, sort by distance) has no cross-document component, so it just works with distributed search. -Yonik http://www.lucenerevolution.org -- Lucene/Solr User Conference, May 25-26, San Francisco
Re: difference between geospatial search from database angle and from solr angle
Thanks, David. I am thinking of a scenario that billions of objects, whose indices are too big for a single machine to serve the indexing, to serve the querying. Is there any sharding mechanism? Can you give a comparison between solr-based geospatial search and PostGIS based geospatial search? * scalability * functionality richness * incremental indexing (re-indexing) cost * query cost * sharding scheme support On Wed, Apr 6, 2011 at 9:42 PM, David Smiley (@MITRE.org) dsmi...@mitre.org wrote: Sean, Geospatial search in Lucene/Solr is of course implemented based on Lucene's underlying index technology. That technology was originally just for text but it's been adapted very successfully for numerics and querying ranges too. The only mature geospatial field type in Solr 3.1 is LatLonType which under the hood is simply a pair of latitude longitude numeric fields. There really isn't anything sophisticated (geospatially speaking) in Solr 3.1. I'm not sure what sort of geospatial DB research you have in mind but I would expect other systems would be free to use an indexing strategy designed for spatial such as R-Trees. Nevertheless, I think Lucene offers the underlying primitives to compete with systems using other technologies. Case in point is my patch SOLR-2155 which indexes a single point in the form of a geohash at multiple resolutions (geohash lengths AKA spatial prefixes / grids) and uses a recursive algorithm to efficiently query an arbitrary shape. It's quite fast and bests LatLonType already; and there's a lot more I can do to make it faster. This is definitely a field of interest and a growing one in the Lucene/Solr community. There are even some external spatial providers (JTeam, MetaCarta) and I'm partnering with other individuals to create a new one. Expect to see more in the coming months. If you're looking for some specific geospatial capabilities then let us know. ~ David Smiley Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/ - Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book -- View this message in context: http://lucene.472066.n3.nabble.com/difference-between-geospatial-search-from-database-angle-and-from-solr-angle-tp2788442p2788972.html Sent from the Solr - User mailing list archive at Nabble.com. -- --Sean
Re: difference between geospatial search from database angle and from solr angle
Have you looked at solr sharding? Best Erick On Thu, Apr 7, 2011 at 2:24 AM, Sean Bigdatafun sean.bigdata...@gmail.comwrote: Thanks, David. I am thinking of a scenario that billions of objects, whose indices are too big for a single machine to serve the indexing, to serve the querying. Is there any sharding mechanism? Can you give a comparison between solr-based geospatial search and PostGIS based geospatial search? * scalability * functionality richness * incremental indexing (re-indexing) cost * query cost * sharding scheme support On Wed, Apr 6, 2011 at 9:42 PM, David Smiley (@MITRE.org) dsmi...@mitre.org wrote: Sean, Geospatial search in Lucene/Solr is of course implemented based on Lucene's underlying index technology. That technology was originally just for text but it's been adapted very successfully for numerics and querying ranges too. The only mature geospatial field type in Solr 3.1 is LatLonType which under the hood is simply a pair of latitude longitude numeric fields. There really isn't anything sophisticated (geospatially speaking) in Solr 3.1. I'm not sure what sort of geospatial DB research you have in mind but I would expect other systems would be free to use an indexing strategy designed for spatial such as R-Trees. Nevertheless, I think Lucene offers the underlying primitives to compete with systems using other technologies. Case in point is my patch SOLR-2155 which indexes a single point in the form of a geohash at multiple resolutions (geohash lengths AKA spatial prefixes / grids) and uses a recursive algorithm to efficiently query an arbitrary shape. It's quite fast and bests LatLonType already; and there's a lot more I can do to make it faster. This is definitely a field of interest and a growing one in the Lucene/Solr community. There are even some external spatial providers (JTeam, MetaCarta) and I'm partnering with other individuals to create a new one. Expect to see more in the coming months. If you're looking for some specific geospatial capabilities then let us know. ~ David Smiley Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/ - Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book -- View this message in context: http://lucene.472066.n3.nabble.com/difference-between-geospatial-search-from-database-angle-and-from-solr-angle-tp2788442p2788972.html Sent from the Solr - User mailing list archive at Nabble.com. -- --Sean
Re: difference between geospatial search from database angle and from solr angle
I haven't used PostGIS so I can't offer a real comparison. I think if you were to try out both, you'd be impressed with Solr's performance/scalability thanks in large part to its sharding. But for functionality richness in so far as geospatial is concerned, that's where Solr currently comes short. It just has the basic stuff 80% of people want. ~ David Smiley Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/ On Apr 7, 2011, at 2:24 AM, Sean Bigdatafun wrote: Thanks, David. I am thinking of a scenario that billions of objects, whose indices are too big for a single machine to serve the indexing, to serve the querying. Is there any sharding mechanism? Can you give a comparison between solr-based geospatial search and PostGIS based geospatial search? * scalability * functionality richness * incremental indexing (re-indexing) cost * query cost * sharding scheme support
Re: difference between geospatial search from database angle and from solr angle
Sean, Geospatial search in Lucene/Solr is of course implemented based on Lucene's underlying index technology. That technology was originally just for text but it's been adapted very successfully for numerics and querying ranges too. The only mature geospatial field type in Solr 3.1 is LatLonType which under the hood is simply a pair of latitude longitude numeric fields. There really isn't anything sophisticated (geospatially speaking) in Solr 3.1. I'm not sure what sort of geospatial DB research you have in mind but I would expect other systems would be free to use an indexing strategy designed for spatial such as R-Trees. Nevertheless, I think Lucene offers the underlying primitives to compete with systems using other technologies. Case in point is my patch SOLR-2155 which indexes a single point in the form of a geohash at multiple resolutions (geohash lengths AKA spatial prefixes / grids) and uses a recursive algorithm to efficiently query an arbitrary shape. It's quite fast and bests LatLonType already; and there's a lot more I can do to make it faster. This is definitely a field of interest and a growing one in the Lucene/Solr community. There are even some external spatial providers (JTeam, MetaCarta) and I'm partnering with other individuals to create a new one. Expect to see more in the coming months. If you're looking for some specific geospatial capabilities then let us know. ~ David Smiley Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/ - Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book -- View this message in context: http://lucene.472066.n3.nabble.com/difference-between-geospatial-search-from-database-angle-and-from-solr-angle-tp2788442p2788972.html Sent from the Solr - User mailing list archive at Nabble.com.
