Re: Geoquery Against Pre-Indexed Shape is Failing [Second Try]
Hey Waldo,
I had a look over some of the code we have for Pelias
https://github.com/pelias/pelias and I managed to get it to work, the
trick being that you need to index the business location as a 'geo-shape'
'point' type:
https://gist.github.com/missinglink/d5d37082303bfdb5529c#file-gis-pre-indexed-sh-L50
https://gist.github.com/missinglink/d5d37082303bfdb5529c#file-gis-pre-indexed-sh-L62
Working example: https://gist.github.com/missinglink/d5d37082303bfdb5529c
Visual
representation: https://gist.github.com/anonymous/5384833492ecbaf814e5
When we index openstreetmap data in elasticsearch, we first find the
intersection of the lat/lon within multiple indeces containing polygon
information sourced from the quattroshapes project.
The results are then stored as 'admin' properties against each record
http://pelias.mapzen.com/reverse?lat=38.0509lon=-78.4522size=20 before
indexing the POI, which means doing a lookup for all places inside polygon
becomes a simple string match rather than a complex query-time computation.
Would love to have a chat some time about collaborating on US polygon data
and sources of open address data.
All the best,
-P
On Tuesday, 6 January 2015 02:52:05 UTC, Waldo Jaquith wrote:
This is fascinating—it hadn't crossed my mind that I might be doing
something that simply wasn't supported by Elasticsearch. As many times as I
read the documentation, I think I was reading for what I *wanted* it to do,
rather than as a blank slate. Perhaps I'm trying to do something that is
supported, but I'm doing it wrong?
This is for all businesses registered in a U.S. state (Virginia), to
identify which ones are registered in a given county or city (a sub-unit of
a state). The UK equivalent of this would be a list of all businesses
registered in the East Midlands, and wanting to identify all of the
businesses in Nottingham. I want somebody running a search for a business
name to be able to limit that search to a given county or city. Perhaps I'm
doing that all wrong, in terms of how Elasticsearch works. (At present, I'm
doing this by including the shapes in my HTML, rather than pre-indexing
them.) Is there a more Elasticsearch-y way to accomplish this, or is it
really best for me to open an issue proposing such a new feature?
My apologies for the errors in my gist! Thank you for forking it and
fixing them. This is my first time putting together a complete test case of
a problem for a mailing list, so when the test case failed, it only served
to (wrongly) reinforce that I'd correctly reproduced the problem!
Best,
Waldo
---
Waldo Jaquith
Director
U.S. Open Data
http://usopendata.org/
202-719-5315
On Monday, January 5, 2015 9:15:51 AM UTC-5, Peter Johnson wrote:
Jörg is correct in saying that the 'geo_shape' filter only supports
finding shapes within shapes and not points within shapes.
It would be great if there was a filter which *did* support finding all
points within a pre-indexed shape!
You may want to open an issue regarding extending the 'geo-polygon'
filter to support 'indexed_shape' instead of just 'points'.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-polygon-filter.html
As it currently stands I don't think this is a bug, but the above feature
request sounds like it could be useful to others.
I also found a bunch of mistakes in your gist from spelling mistakes
(businesses!=business) to syntax errors and things like not actually
indexing a 'location' field for your business!?
https://gist.github.com/waldoj/004f77755179d862fe3a#file-elasticsearch-sh-L226
Feel free to use my minimal testcase as a template to make your debugging
easier:
https://gist.github.com/missinglink/4bf3e262fbd0a89d3f5e
-P
On Sunday, 4 January 2015 23:28:01 UTC, Jörg Prante wrote:
I think you already found the answer so I hesitate to comment. If you
set location to type geo_point and not to geo_shape, you can not execute
a geo shape filter on it. There are geo bounding box / polygon filters that
can be executed on geo points.
Best,
Jörg
On Sun, Jan 4, 2015 at 8:53 PM, Waldo Jaquith wa...@usodi.org wrote:
Folks,
[I sent the following on December 7, and got no reply. I’m sending it
again and, if nobody tells me otherwise, I’ll assume that this is a bug,
and file a report accordingly.]
I’m trying to run a geo_shape filter query using a pre-indexed shape,
and it’s failing for reasons that I cannot identify. This is with
Elasticsearch v1.4.1-1. I’ve documented how to reproduce the problem at
https://gist.github.com/waldoj/004f77755179d862fe3a.
The pre-indexed shapes are GeoJSON converted from Census Bureau
municipal geodata. The indexed documents are state corporate registration
records, each of which have an address that I’ve geocoded into a latitude
and longitude (a field named “location”). The indexed shapes are in a
geometry”:{ “coordinates: { [1, 2
Re: Geoquery Against Pre-Indexed Shape is Failing [Second Try]
Jörg is correct in saying that the 'geo_shape' filter only supports finding
shapes within shapes and not points within shapes.
It would be great if there was a filter which *did* support finding all
points within a pre-indexed shape!
You may want to open an issue regarding extending the 'geo-polygon' filter
to support 'indexed_shape' instead of just 'points'.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-polygon-filter.html
As it currently stands I don't think this is a bug, but the above feature
request sounds like it could be useful to others.
I also found a bunch of mistakes in your gist from spelling mistakes
(businesses!=business) to syntax errors and things like not actually
indexing a 'location' field for your business!?
https://gist.github.com/waldoj/004f77755179d862fe3a#file-elasticsearch-sh-L226
Feel free to use my minimal testcase as a template to make your debugging
easier:
https://gist.github.com/missinglink/4bf3e262fbd0a89d3f5e
-P
On Sunday, 4 January 2015 23:28:01 UTC, Jörg Prante wrote:
I think you already found the answer so I hesitate to comment. If you set
location to type geo_point and not to geo_shape, you can not execute a
geo shape filter on it. There are geo bounding box / polygon filters that
can be executed on geo points.
Best,
Jörg
On Sun, Jan 4, 2015 at 8:53 PM, Waldo Jaquith wa...@usodi.org
javascript: wrote:
Folks,
[I sent the following on December 7, and got no reply. I’m sending it
again and, if nobody tells me otherwise, I’ll assume that this is a bug,
and file a report accordingly.]
I’m trying to run a geo_shape filter query using a pre-indexed shape, and
it’s failing for reasons that I cannot identify. This is with Elasticsearch
v1.4.1-1. I’ve documented how to reproduce the problem at
https://gist.github.com/waldoj/004f77755179d862fe3a.
The pre-indexed shapes are GeoJSON converted from Census Bureau municipal
geodata. The indexed documents are state corporate registration records,
each of which have an address that I’ve geocoded into a latitude and
longitude (a field named “location”). The indexed shapes are in a
geometry”:{ “coordinates: { [1, 2] [3, 5] } } format. In the query I
specify path: “geometry”, which generates this error:
Failed to find geo_shape field [location]].”
Based on the error, it looks like Elasticsearch is looking for *shapes*
that fall within the pre-indexed shape, rather than *points*, and so while
the indexed documents have a geo_point field named “location,” there’s no
geo_shape field of that name. FWIW, I’m following the instructions in the
manual at
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html#_pre_indexed_shape
.
I’d be grateful to anybody who could offer some guidance. Thank you.
Best,
Waldo
---
Waldo Jaquith
Director
U.S. Open Data Institute
http://usodi.org/
202-719-5315
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com javascript:.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/450CBB58-AA14-4106-A6C2-5762DB4845F5%40usodi.org
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3ea4e571-e192-4136-b860-65c19801193c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: How is data stored
have a look at https://github.com/polyfractal/elasticsearch-inquisitor#analyzer-testing On Thursday, 1 January 2015 17:05:25 UTC, Bruno Kamiche wrote: Thanks for your replies, that gave me the clue for what I was looking for, and now it is solved! On Wednesday, December 31, 2014 10:36:27 PM UTC-5, Bruno Kamiche wrote: Hello, I'm new in using elasticsearch, so maybe this is a basic question... Is there any way to see how is the text stored or at least how it would look once the filters defined for a field (in the analyzer) are applied? I know that the actual field is stored as is, and the filters are considered for indexing purposes, but i want to see what is the result of that operation. Why do I ask this, the reason is that I'm applying filters to a field (for example removing urls), and performing aggregations on that field, but the aggregations return http as a used word, and guess that is not correct, as it shall be removed... -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/59accfbd-46a4-437c-81b1-6978b752ac4c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: GIS Query not working on ES as expected
Nice. Out of curiosity are the polygons ope ndata? If so could you share the source please? -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/112d8bf5-ecb1-40dd-bac3-b9ec5276615b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: GIS Query not working on ES as expected
I think I figured out what's going on, see the github issue.
-P
On Sunday, 28 December 2014 05:21:07 UTC, AndyGIS wrote:
Filed https://github.com/elasticsearch/elasticsearch/issues/9079
FYI!
On Saturday, December 27, 2014 9:07:50 PM UTC-8, AndyGIS wrote:
What i have is a simple test case for the feature (this is not a corner
case)Is this feature expected to work in some other coordinate system?
Any workaround to overcome this bug? Thanks!
Andy
On Saturday, December 27, 2014 6:40:44 AM UTC-8, Peter Johnson wrote:
Hey Andy,
I tried to fix this but you're right, it seems to be a bug. I've
attached a full bug report which you might want to add to a new github
issue.
https://gist.github.com/anonymous/c250d602d1e7fe6d3655
https://gist.github.com/missinglink/6e96f06e9e6032aa6416
I also tried using 'geohash' instead of 'quadtree' with the same result.
-P
On Friday, 26 December 2014 22:51:56 UTC, AndyGIS wrote:
Hi,
I am new to GIS w/ ES.
Below is my request/ response. I am puzzled why do i get a polygon
returned when it does not intersect w/ envelope defined in request. What
am i missing? Appreciate quick response. Thanks,
Andy
*Request*:
*_search*
{size: 51000,query: { geo_shape: { POLYS: {shape: {type:
envelope, coordinates: [*[-118.58, 35.32],[-118.68, 35.30]*]},
relation: intersects
*Response*:
took : 1172,
timed_out : false,
_shards : {
total : 5,
successful : 5,
failed : 0
},
hits : {
total : 94420,
max_score : 1.0,
hits : [ {
_index : XXX,
_type : XXX_data,
_id : 1234,
_score : 1.0,
_source:{POLYS:{type:Polygon,coordinates:
*[[[-117.7656797176,35.2420721325],[-117.766565557,35.2429646794],[-117.7675000712,35.2429532681],[-117.7661768866,35.2415486409],[-117.7661640858,35.2415341862],[-117.765523769,35.2419167046],[-117.7656797176,35.2420721325]*]]},
STATE: XX, ID: 1234 ,COUNTY_NAME: YY}]}
}, {
_index : XXX,
_type : XXX_data,
_id : 1235,
_score : 1.0,
_mapping?pretty'
{
XXX : {
mappings : {
XXX_data : {
properties : {
COUNTY_NAME : {
type : string
},
ID : {
type : long
},
POLYS : {
type : geo_shape,
tree : quadtree,
tree_levels : 26
},
STATE : {
type : string
}
}
}
}
}
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0b3a1d4f-7ad5-4163-947d-992a33b71ec7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: GIS Query not working on ES as expected
Hey Andy,
I tried to fix this but you're right, it seems to be a bug. I've attached a
full bug report which you might want to add to a new github issue.
https://gist.github.com/anonymous/c250d602d1e7fe6d3655
https://gist.github.com/missinglink/6e96f06e9e6032aa6416
I also tried using 'geohash' instead of 'quadtree' with the same result.
-P
On Friday, 26 December 2014 22:51:56 UTC, AndyGIS wrote:
Hi,
I am new to GIS w/ ES.
Below is my request/ response. I am puzzled why do i get a polygon
returned when it does not intersect w/ envelope defined in request. What
am i missing? Appreciate quick response. Thanks,
Andy
*Request*:
*_search*
{size: 51000,query: { geo_shape: { POLYS: {shape: {type:
envelope, coordinates: [*[-118.58, 35.32],[-118.68, 35.30]*]},
relation: intersects
*Response*:
took : 1172,
timed_out : false,
_shards : {
total : 5,
successful : 5,
failed : 0
},
hits : {
total : 94420,
max_score : 1.0,
hits : [ {
_index : XXX,
_type : XXX_data,
_id : 1234,
_score : 1.0,
_source:{POLYS:{type:Polygon,coordinates:
*[[[-117.7656797176,35.2420721325],[-117.766565557,35.2429646794],[-117.7675000712,35.2429532681],[-117.7661768866,35.2415486409],[-117.7661640858,35.2415341862],[-117.765523769,35.2419167046],[-117.7656797176,35.2420721325]*]]},
STATE: XX, ID: 1234 ,COUNTY_NAME: YY}]}
}, {
_index : XXX,
_type : XXX_data,
_id : 1235,
_score : 1.0,
_mapping?pretty'
{
XXX : {
mappings : {
XXX_data : {
properties : {
COUNTY_NAME : {
type : string
},
ID : {
type : long
},
POLYS : {
type : geo_shape,
tree : quadtree,
tree_levels : 26
},
STATE : {
type : string
}
}
}
}
}
}
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f1c49b57-721d-4abd-bb1d-ac54631e543a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: search point in polygon
Did you get this working in the end Maarten? I have the same problem with the way 'intersects' works and Jilles's solution doesn't work for me; possibly due to the 'tree_levels' accuracy for quad tree. As a kind of workaround, I was thinking that you could draw 2 'envelope' geo_shape intersection; one vertical and one horizontal. Using a boolean filter you could reduce your results to only the polygons which intersect BOTH intersections. It would look something like this: https://lh3.googleusercontent.com/-NkNXyNjCLwQ/U6wBLrRIbNI/JkU/383LpZNwVzE/s1600/bbox.png Here's an example of the query: https://gist.github.com/missinglink/ce25f1f9f7554ff1010f Unfortunately I couldn't get this working either, do you think this may be a simpler workaround for the 'intersects' issue? -P On Tuesday, 5 March 2013 14:17:01 UTC+1, Jilles van Gurp wrote: Actually, I was wrong again. I did make it work in the end. If others want to replicate my experiment, I pasted a few sample queries and responses as well as the data and mapping I indexed here: http://pastebin.com/QpXvYK5B The first query shows that a query with a circle of 1m radius returns Berlin and Mitte for a circle near Rosenthalerplatz. So, the 1m circle polygon intersects with both polygons that fully contain the circle polygon. If I increase the radius to 300 meter it also finds the nearby pois. There is a problem with accuracy though. I noticed that 100 meter doesn't return any of the pois. even though I know that it should have returned a few. If I increase to 150 it returns the Prenzlauerberg polygon but still no POIs. At 157 still the same. At 158, I suddenly get 8 results. I might be able to improve the accuracy by playing with the levels in the mapping. So what might be happening is that you are running into this accuracy issue here as well. If you haven't already, upgrading to a post 0.90-beta1 snapshot build might help you. That would get you the changes made a few days ago to the geo_shape implementation. Jilles On Monday, March 4, 2013 6:44:10 PM UTC+1, Maarten Janssen wrote: Or even better a new ShapeRelation should be produced called CONTAINS or IS_PART_OF where you get do the other way around. In our case (I think we need the same approach) we could go top down and match first all the polygons, and then use indeed a contains method, but in my opnion it should be part of the spatial search capabilities. On Mon, Mar 4, 2013 at 6:33 PM, Jilles van Gurp jilles...@gmail.com wrote: Hmm, I just tried this and it seems you are right. Very annoying. I guess was assuming too much about intersects. I guess you could turn things around and use a really large circle that for sure includes all the polygons you care about and then do a filter on polygon containment after you get the results. My library offers a contains function. But that is far from ideal. This sounds like it might be a bug though since technically a large feature should intersect with any small feature contained by it. Jilles -- You received this message because you are subscribed to a topic in the Google Groups elasticsearch group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/bDUQKTK0oe0/unsubscribe?hl=en-US . To unsubscribe from this group and all its topics, send an email to elasticsearc...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- Maarten Janssen CTO Infohubble Email: mjan...@infohubble.com URL:www.infohubble.com Mobile: +31620006790 Card: eee.am/mjanssen *Address:* Infohubble BV Gebouw Sevilla Entrada 304 1096 ED Amsterdam The Netherlands -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/efa8d361-fc4f-46fb-9e99-d40910caa395%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
