difference between geospatial search from database angle and from solr angle

[Sean Bigdatafun]

I understand Solr can do pretty powerful geospatial search
http://www.ibm.com/developerworks/java/library/j-spatial/

But I also
understand lots of DB researchers have done lots of geospatial related work,
can someone give an overview of the difference from the different angel?

Thanks,
-- 
--Sean

Re: difference between geospatial search from database angle and from solr angle

[David Smiley (@MITRE.org)]

Sean,
Geospatial search in Lucene/Solr is of course implemented based on
Lucene's underlying index technology. That technology was originally just
for text but it's been adapted very successfully for numerics and querying
ranges too. The only mature geospatial field type in Solr 3.1 is LatLonType
which under the hood is simply a pair of latitude & longitude numeric
fields.  There really isn't anything sophisticated (geospatially speaking)
in Solr 3.1. I'm not sure what sort of geospatial DB research you have in
mind but I would expect other systems would be free to use an indexing
strategy designed for spatial such as "R-Trees". Nevertheless, I think
Lucene offers the underlying primitives to compete with systems using other
technologies.  Case in point is my patch SOLR-2155 which indexes a single
point in the form of a "geohash" at multiple resolutions (geohash lengths
AKA spatial prefixes / grids) and uses a recursive algorithm to efficiently
query an arbitrary shape.  It's quite fast and bests LatLonType already; and
there's a lot more I can do to make it faster.
This is definitely a field of interest and a growing one in the
Lucene/Solr community.  There are even some external spatial providers
(JTeam, MetaCarta) and I'm partnering with other individuals to create a new
one.  Expect to see more in the coming months.  If you're looking for some
specific geospatial capabilities then let us know.

~ David Smiley 
Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/

-
 Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
--
View this message in context: 
http://lucene.472066.n3.nabble.com/difference-between-geospatial-search-from-database-angle-and-from-solr-angle-tp2788442p2788972.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: difference between geospatial search from database angle and from solr angle

[Sean Bigdatafun]

Thanks, David.

I am thinking of a scenario that billions of objects, whose indices are too
big for a single machine to serve the indexing, to serve the querying. Is
there any sharding mechanism?


Can you give a comparison between solr-based geospatial search and PostGIS
based geospatial search?
  * scalability
  * functionality richness
  * incremental indexing (re-indexing) cost
  * query cost
  * sharding scheme support



On Wed, Apr 6, 2011 at 9:42 PM, David Smiley (@MITRE.org)  wrote:

> Sean,
>Geospatial search in Lucene/Solr is of course implemented based on
> Lucene's underlying index technology. That technology was originally just
> for text but it's been adapted very successfully for numerics and querying
> ranges too. The only mature geospatial field type in Solr 3.1 is LatLonType
> which under the hood is simply a pair of latitude & longitude numeric
> fields.  There really isn't anything sophisticated (geospatially speaking)
> in Solr 3.1. I'm not sure what sort of geospatial DB research you have in
> mind but I would expect other systems would be free to use an indexing
> strategy designed for spatial such as "R-Trees". Nevertheless, I think
> Lucene offers the underlying primitives to compete with systems using other
> technologies.  Case in point is my patch SOLR-2155 which indexes a single
> point in the form of a "geohash" at multiple resolutions (geohash lengths
> AKA spatial prefixes / grids) and uses a recursive algorithm to efficiently
> query an arbitrary shape.  It's quite fast and bests LatLonType already;
> and
> there's a lot more I can do to make it faster.
>This is definitely a field of interest and a growing one in the
> Lucene/Solr community.  There are even some external spatial providers
> (JTeam, MetaCarta) and I'm partnering with other individuals to create a
> new
> one.  Expect to see more in the coming months.  If you're looking for some
> specific geospatial capabilities then let us know.
>
> ~ David Smiley
> Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/
>
> -
>  Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/difference-between-geospatial-search-from-database-angle-and-from-solr-angle-tp2788442p2788972.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



-- 
--Sean

Re: difference between geospatial search from database angle and from solr angle

[Erick Erickson]

Have you looked at solr sharding?

Best
Erick

On Thu, Apr 7, 2011 at 2:24 AM, Sean Bigdatafun
wrote:

> Thanks, David.
>
> I am thinking of a scenario that billions of objects, whose indices are too
> big for a single machine to serve the indexing, to serve the querying. Is
> there any sharding mechanism?
>
>
> Can you give a comparison between solr-based geospatial search and PostGIS
> based geospatial search?
>  * scalability
>  * functionality richness
>  * incremental indexing (re-indexing) cost
>  * query cost
>  * sharding scheme support
>
>
>
> On Wed, Apr 6, 2011 at 9:42 PM, David Smiley (@MITRE.org) <
> dsmi...@mitre.org
> > wrote:
>
> > Sean,
> >Geospatial search in Lucene/Solr is of course implemented based on
> > Lucene's underlying index technology. That technology was originally just
> > for text but it's been adapted very successfully for numerics and
> querying
> > ranges too. The only mature geospatial field type in Solr 3.1 is
> LatLonType
> > which under the hood is simply a pair of latitude & longitude numeric
> > fields.  There really isn't anything sophisticated (geospatially
> speaking)
> > in Solr 3.1. I'm not sure what sort of geospatial DB research you have in
> > mind but I would expect other systems would be free to use an indexing
> > strategy designed for spatial such as "R-Trees". Nevertheless, I think
> > Lucene offers the underlying primitives to compete with systems using
> other
> > technologies.  Case in point is my patch SOLR-2155 which indexes a single
> > point in the form of a "geohash" at multiple resolutions (geohash lengths
> > AKA spatial prefixes / grids) and uses a recursive algorithm to
> efficiently
> > query an arbitrary shape.  It's quite fast and bests LatLonType already;
> > and
> > there's a lot more I can do to make it faster.
> >This is definitely a field of interest and a growing one in the
> > Lucene/Solr community.  There are even some external spatial providers
> > (JTeam, MetaCarta) and I'm partnering with other individuals to create a
> > new
> > one.  Expect to see more in the coming months.  If you're looking for
> some
> > specific geospatial capabilities then let us know.
> >
> > ~ David Smiley
> > Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/
> >
> > -
> >  Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
> > --
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/difference-between-geospatial-search-from-database-angle-and-from-solr-angle-tp2788442p2788972.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> --Sean
>

Re: difference between geospatial search from database angle and from solr angle

[Smiley, David W.]

I haven't used PostGIS so I can't offer a real comparison. I think if you were 
to try out both, you'd be impressed with Solr's performance/scalability thanks 
in large part to its sharding.  But for "functionality richness" in so far as 
geospatial is concerned, that's where Solr currently comes short. It just has 
the basic stuff 80% of people want.

~ David Smiley
Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/

On Apr 7, 2011, at 2:24 AM, Sean Bigdatafun wrote:

> Thanks, David.
> 
> I am thinking of a scenario that billions of objects, whose indices are too
> big for a single machine to serve the indexing, to serve the querying. Is
> there any sharding mechanism?
> 
> 
> Can you give a comparison between solr-based geospatial search and PostGIS
> based geospatial search?
>  * scalability
>  * functionality richness
>  * incremental indexing (re-indexing) cost
>  * query cost
>  * sharding scheme support

Re: difference between geospatial search from database angle and from solr angle

[Lance Norskog]

Wait! How can you do distance calculations across different shards efficiently?

On Thu, Apr 7, 2011 at 7:19 AM, Smiley, David W.  wrote:
> I haven't used PostGIS so I can't offer a real comparison. I think if you 
> were to try out both, you'd be impressed with Solr's performance/scalability 
> thanks in large part to its sharding.  But for "functionality richness" in so 
> far as geospatial is concerned, that's where Solr currently comes short. It 
> just has the basic stuff 80% of people want.
>
> ~ David Smiley
> Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/
>
> On Apr 7, 2011, at 2:24 AM, Sean Bigdatafun wrote:
>
>> Thanks, David.
>>
>> I am thinking of a scenario that billions of objects, whose indices are too
>> big for a single machine to serve the indexing, to serve the querying. Is
>> there any sharding mechanism?
>>
>>
>> Can you give a comparison between solr-based geospatial search and PostGIS
>> based geospatial search?
>>          * scalability
>>          * functionality richness
>>          * incremental indexing (re-indexing) cost
>>          * query cost
>>          * sharding scheme support
>
>
>
>
>
>
>



-- 
Lance Norskog
goks...@gmail.com

Re: difference between geospatial search from database angle and from solr angle

[Yonik Seeley]

On Sun, Apr 10, 2011 at 5:24 PM, Lance Norskog  wrote:
> Wait! How can you do distance calculations across different shards 
> efficiently?

Basic spatial search (bounding box filter, radius filter, sort by
distance) has no cross-document component, so "it just works" with
distributed search.

-Yonik
http://www.lucenerevolution.org -- Lucene/Solr User Conference, May
25-26, San Francisco