David,
Firstly, thanks for putting together such a thorough email it helps a lot to
understand some of the things we were just guessing at because (as you
mentioned a few times) the documentation around all of this is rather sparse.
I’ll explain the context around the use case we’re trying to solve and then
attempt to respond as best I can to each of your points. What we have is a
list of documents that in our case the location is sometimes a point and
sometimes a circle. These basically represent (in our case) inventory at a
physical location (point) or inventory that can be delivered to you within X km
(configurable per document) which represents the circle use case. We want to
be able to allow a user to say I want all documents within X distance of my
location, but also all documents that are able to be delivered to your point
where the delivery distance is defined on the inventory (creating the circle).
This is why we were actually trying to combine both point based data and
poly/circle data into a single geospatial field, since I don’t believe you
could do something like fq=geofilt(latlng, x, y, d) OR geofilt(latlngCircle, x,
y, 1) but perhaps we’re just not getting quite the right syntax, etc.
* Personally, I find it highly confusing to have a field named "latlng" and
have it be anything other than a simple point -- it's all you have if given a
single latitude longitude pair. If you intend for the data to be a circle
(either exactly or approximated) then perhaps call it latLngCircle
- This is happening because we’re trying to combine two different use
cases into a single field, since I don’t think we have that option from the
query side. The name is really just us re-using our current field for this
exploration, but would probably end up being named something different.
* geodist() and for that matter any other attempt to get the distance to a
non-point shape is not going to work -- either error or confusing results; I
forget. This is hard to do and the logic isn't there for it, and probably
wouldn't perform to user's expectations if it did. This ought to be documented
but seems not to be.
-Good to know, so no matter what we’ll have to have a point value
stored somewhere for each document and calculate geodist on that.
* Generally RptWithGeometrySpatialField should be used over
SpatialRecursivePrefixTreeFieldType unless you want heatmaps or are willing to
make trade-offs in higher index size and lossy precision in order to get faster
search. It's up to you; if you benchmark both I'd love to hear how it went.
-We may explore both but typically we’re more interested in speed than
accuracy, benchmarking it may be a very interesting exercise however. For
sorting for instance we’re actually using sqedist instead of geodist because
we’re not overly concerned about sorting accuracy.
* In WKT format, the ordinate order is "X Y" (thus longitude then latitude).
Looking at your triangle, it is extremely close to Antarctica, and I'm
skeptical you intended that. This is not directly documented AFAICT but it's
such a common mistake that it ought to be called out in the docs.
-Definitely did not intend it to be close to Antarctica, I think we
tried both but probably went back to lat,long and was definitely more common in
our (failed) testing.
* I see you are using Geo3D, which is not the default. Geo3D is strict about
the coordinate order -- counter-clickwise. Your triangle is clockwise and thus
it has an inverted interpretation -- thus it's a shape that covers nearly the
whole globe. I recently documented this
https://issues.apache.org/jira/browse/SOLR-13467 but it's not published yet
since it's so new.
- Thanks for this clarification as well. I had read this in the WKT
docs too, again something we tried but really weren’t sure about what the right
answer was and had been going back and forth on. The documentation seems to
specify that you need to specify either JTS or Geo3d, but doesn’t provide much
info/guidance about which to use when and since JTS required adding another jar
manually and therefore complicates our build process significantly (at least vs
using Geo3D) we tried Geo3D. I’d love to hear more about the tradeoffs and
other considerations between the two, but sounds like we should switch to JTS
(the default, correct?)
* You can absolutely index a circle in Solr -- this is something cool and
somewhat unique. And you don't need format=legacy. The documentation needs to
call this out better, though it at least refers to circles as a "buffered
point" which is the currently supported way of representing it, and it does
have one example. Search for "BUFFER" and you'll see a WKT-like syntax to do
it. BUFFER is not standard WKT; it was added on to do this. The first arg is
a X Y center, and 2nd arg is a distance in decimal degrees (not km). BTW Geo3D
is a good choice here but not essential either.
- This sounds very promising and we’ll definitely spend some time here
because it may ultimately be what we really want to use, sounds like Geo3D may
actually be the right choice now?
One other question I have is what the behavior will be if both my point and my
search radius are inside of the circle/polygon entirely? Like geofilt(x,y,10)
and a buffered point (in km instead decimal degrees for simplicity) of BUFFER(x
y 20km). Will this document return even though my filter is entirely inside
the polygon, or is it looking for edge intersections?
Thanks so much for the response/help!!!
-Marshall Sanders
On 7/26/19, 12:01 AM, "David Smiley" <david.w.smi...@gmail.com> wrote:
Hello Marshall,
I worked on a lot of this functionality. I have lots to say:
* Personally, I find it highly confusing to have a field named "latlng" and
have it be anything other than a simple point -- it's all you have if given
a single latitude longitude pair. If you intend for the data to be a
circle (either exactly or approximated) then perhaps call it latLngCircle
* geodist() and for that matter any other attempt to get the distance to a
non-point shape is not going to work -- either error or confusing results;
I forget. This is hard to do and the logic isn't there for it, and
probably wouldn't perform to user's expectations if it did. This ought to
be documented but seems not to be.
* Generally RptWithGeometrySpatialField should be used
over SpatialRecursivePrefixTreeFieldType unless you want heatmaps or are
willing to make trade-offs in higher index size and lossy precision in
order to get faster search. It's up to you; if you benchmark both I'd love
to hear how it went.
* In WKT format, the ordinate order is "X Y" (thus longitude then
latitude). Looking at your triangle, it is extremely close to Antarctica,
and I'm skeptical you intended that. This is not directly documented AFAICT
but it's such a common mistake that it ought to be called out in the docs.
* I see you are using Geo3D, which is not the default. Geo3D is strict
about the coordinate order -- counter-clickwise. Your triangle is
clockwise and thus it has an inverted interpretation -- thus it's a shape
that covers nearly the whole globe. I recently documented this
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_SOLR-2D13467&d=DwIFaQ&c=hrETxhO8sRCXAcJITi-bu62jJ43QQVS6-BatTNT-3bs&r=3lL1Fjs6t-l8MLo9jYFBo_kY10nHjYnyV94cayAiWXc&m=qODo0eqhR9JwDbcVfadRxP0k6Pc2jQizOaf9CBky6Ow&s=gRY9FeOVxx3Oe1nLMsc4d2ATvX81qtF0UmuyCRI2fRc&e=
but it's not published yet
since it's so new.
* You can absolutely index a circle in Solr -- this is something cool and
somewhat unique. And you don't need format=legacy. The documentation needs
to call this out better, though it at least refers to circles as a
"buffered point" which is the currently supported way of representing it,
and it does have one example. Search for "BUFFER" and you'll see a
WKT-like syntax to do it. BUFFER is not standard WKT; it was added on to
do this. The first arg is a X Y center, and 2nd arg is a distance in
decimal degrees (not km). BTW Geo3D is a good choice here but not
essential either.
Back to your core requirement -- you want to index circles and sort results
by distance. Can you please elaborate better on this... distance to the
outer ring of the circle or the center point? Center point is easy to do
simply by putting the center point additionally in a field using
LatLonPointSpatialField and use geodist referring to that. Also,
FYI geodist() is a function that can take arguments directly which makes
more sense when multiple spatial fields are in play. Sadly this aspect is
not documented. Suffice it to say, if you do geodist(latLng) (maybe
quoted?) then it'll use that field, and parse "pt" param from the request.
~ David Smiley
Apache Lucene/Solr Search Developer
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_in_davidwsmiley&d=DwIFaQ&c=hrETxhO8sRCXAcJITi-bu62jJ43QQVS6-BatTNT-3bs&r=3lL1Fjs6t-l8MLo9jYFBo_kY10nHjYnyV94cayAiWXc&m=qODo0eqhR9JwDbcVfadRxP0k6Pc2jQizOaf9CBky6Ow&s=naIBT_8ZUFFX6dx4Oxe3SqU-K5xw51R2C2dsoalFcDY&e=
On Tue, Jul 23, 2019 at 2:32 PM Sanders, Marshall (CAI - Atlanta) <
marshall.sande...@coxautoinc.com> wrote:
> We’re trying to index a polygon into solr and then filter/calculate
> geodist on the polygon (ideally we actually want a circle, but it looks
> like that’s not really supported officially by wkt/geojson and instead you
> have to switch format=”legacy” which seems like something that might be
> removed in the future so don’t want to rely on it).
>
> Here’s the info from schema:
> <field name="latlng" type="location_rpt" indexed="true" stored="true"
> multiValued="true"/>
>
> <fieldType name="location_rpt"
> class="solr.SpatialRecursivePrefixTreeFieldType"
> geo="true" distErrPct="0.025" maxDistErr="0.000009"
> distanceUnits="kilometers"
> spatialContextFactory="Geo3D"/>
>
>
> We’ve tried indexing some different data, but to keep it as simple as
> possible we started with a triangle (will eventually add more points to
> approximate a circle). Here’s an example document that we’ve added just
> for testing:
>
> {
> "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
> 33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> "ID": "284598223"
> }
>
>
> However, it seems like filtering/distance calculations aren’t working (at
> least not the way we are used to doing it for points). Here’s an example
> query where the pt is several hundred kilometers away from the polygon,
yet
> the document still returns. Also, it seems that regardless of origin
point
> or polygon location the calculated geodist is always 20015.115
>
> Example query:
>
>
select?d=1&fl=ID,latlng,geodist()&fq=%7B!geofilt%7D&indent=on&pt=33.9798087,-94.3286133&q=*:*&sfield=latlng&wt=json
>
> Example documents coming back anyway:
> "docs": [
> {
> "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
> 33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> "ID": "284598223",
> "geodist()": 20015.115
> },
> {
> "latlng": ["POLYGON((33.7942704 -84.4412613, 33.7100611 -84.4028091,
> 33.7802888 -84.3279648, 33.7942704 -84.4412613))"],
> "ID": "284600596",
> "geodist()": 20015.115
> }
> ]
>
>
> Anyone who has experience in this area can you point us in the right
> direction about what we’re doing incorrectly with either how we are
> indexing the data and/or how we are querying against the polygons.
>
> Thank you,
>
>
> --
> Marshall Sanders
> Principal Software Engineer
> Autotrader.com
> marshall.sande...@coxautoinc.com<mailto:marshall.sande...@coxautoinc.com>
>
>
>
