Re: Geoquery Against Pre-Indexed Shape is Failing [Second Try]

[Waldo Jaquith]

This is just wonderful, Peter—thank you. (Providing a complete, tested, working 
example is probably more than I’ve ever done for anybody seeking debugging 
help!) It looks like I went and made the indexed location data more complicated 
than it needed to be, ginning up an object / geo_point structure, when all I 
really needed was "type: point” and “coordinates: -78, 38”. This has been a 
blocker on a project for so long, I scarcely remember how to proceed from here. 
:) My thanks to you and the Elasticsearch community for your help!

And, yes, I’d be happy to talk about US polygon data and sources of open 
address data. My organization is keenly interested in improving the state of 
open geodata in the US, and I’m eager to talk to anybody about how U.S. Open 
Data could help to improve things.

Best,
Waldo

---
Waldo Jaquith
Director
U.S. Open Data
http://usopendata.org/
202-719-5315

On Jan 6, 2015, at 10:15 AM, Peter Johnson  wrote:
> Hey Waldo,
> 
> I had a look over some of the code we have for Pelias and I managed to get it 
> to work, the trick being that you need to index the business location as a 
> 'geo-shape' 'point' type:
> https://gist.github.com/missinglink/d5d37082303bfdb5529c#file-gis-pre-indexed-sh-L50
> https://gist.github.com/missinglink/d5d37082303bfdb5529c#file-gis-pre-indexed-sh-L62
>  
> 
> Working example: https://gist.github.com/missinglink/d5d37082303bfdb5529c
> Visual representation: https://gist.github.com/anonymous/5384833492ecbaf814e5
> 
> When we index openstreetmap data in elasticsearch, we first find the 
> intersection of the lat/lon within multiple indeces containing polygon 
> information sourced from the quattroshapes project.
> The results are then stored as 'admin' properties against each record before 
> indexing the POI, which means doing a lookup for all places inside polygon 
> becomes a simple string match rather than a complex query-time computation.
> 
> Would love to have a chat some time about collaborating on US polygon data 
> and sources of open address data.
> 
> All the best,
> -P
> 
> On Tuesday, 6 January 2015 02:52:05 UTC, Waldo Jaquith wrote:
> This is fascinating—it hadn't crossed my mind that I might be doing something 
> that simply wasn't supported by Elasticsearch. As many times as I read the 
> documentation, I think I was reading for what I *wanted* it to do, rather 
> than as a blank slate. Perhaps I'm trying to do something that is supported, 
> but I'm doing it wrong?
> 
> This is for all businesses registered in a U.S. state (Virginia), to identify 
> which ones are registered in a given county or city (a sub-unit of a state). 
> The UK equivalent of this would be a list of all businesses registered in the 
> East Midlands, and wanting to identify all of the businesses in Nottingham. I 
> want somebody running a search for a business name to be able to limit that 
> search to a given county or city. Perhaps I'm doing that all wrong, in terms 
> of how Elasticsearch works. (At present, I'm doing this by including the 
> shapes in my HTML, rather than pre-indexing them.) Is there a more 
> Elasticsearch-y way to accomplish this, or is it really best for me to open 
> an issue proposing such a new feature?
> 
> My apologies for the errors in my gist! Thank you for forking it and fixing 
> them. This is my first time putting together a complete test case of a 
> problem for a mailing list, so when the test case failed, it only served to 
> (wrongly) reinforce that I'd correctly reproduced the problem!
> 
> Best,
> Waldo
> 
> ---
> Waldo Jaquith
> Director
> U.S. Open Data
> http://usopendata.org/
> 202-719-5315

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/E4F6AC07-3A57-404C-B73D-2DBA3266207C%40usodi.org.
For more options, visit https://groups.google.com/d/optout.

Re: Geoquery Against Pre-Indexed Shape is Failing [Second Try]

[Peter Johnson]

Hey Waldo,

I had a look over some of the code we have for Pelias 
 and I managed to get it to work, the 
trick being that you need to index the business location as a 'geo-shape' 
'point' type:
https://gist.github.com/missinglink/d5d37082303bfdb5529c#file-gis-pre-indexed-sh-L50
https://gist.github.com/missinglink/d5d37082303bfdb5529c#file-gis-pre-indexed-sh-L62
 

Working example: https://gist.github.com/missinglink/d5d37082303bfdb5529c
Visual 
representation: https://gist.github.com/anonymous/5384833492ecbaf814e5

When we index openstreetmap data in elasticsearch, we first find the 
intersection of the lat/lon within multiple indeces containing polygon 
information sourced from the quattroshapes project.
The results are then stored as 'admin' properties against each record 
 before 
indexing the POI, which means doing a lookup for all places inside polygon 
becomes a simple string match rather than a complex query-time computation.

Would love to have a chat some time about collaborating on US polygon data 
and sources of open address data.

All the best,
-P

On Tuesday, 6 January 2015 02:52:05 UTC, Waldo Jaquith wrote:
>
> This is fascinating—it hadn't crossed my mind that I might be doing 
> something that simply wasn't supported by Elasticsearch. As many times as I 
> read the documentation, I think I was reading for what I *wanted* it to do, 
> rather than as a blank slate. Perhaps I'm trying to do something that is 
> supported, but I'm doing it wrong?
>
> This is for all businesses registered in a U.S. state (Virginia), to 
> identify which ones are registered in a given county or city (a sub-unit of 
> a state). The UK equivalent of this would be a list of all businesses 
> registered in the East Midlands, and wanting to identify all of the 
> businesses in Nottingham. I want somebody running a search for a business 
> name to be able to limit that search to a given county or city. Perhaps I'm 
> doing that all wrong, in terms of how Elasticsearch works. (At present, I'm 
> doing this by including the shapes in my HTML, rather than pre-indexing 
> them.) Is there a more Elasticsearch-y way to accomplish this, or is it 
> really best for me to open an issue proposing such a new feature?
>
> My apologies for the errors in my gist! Thank you for forking it and 
> fixing them. This is my first time putting together a complete test case of 
> a problem for a mailing list, so when the test case failed, it only served 
> to (wrongly) reinforce that I'd correctly reproduced the problem!
>
> Best,
> Waldo
>
> ---
> Waldo Jaquith
> Director
> U.S. Open Data
> http://usopendata.org/
> 202-719-5315
>
>
> On Monday, January 5, 2015 9:15:51 AM UTC-5, Peter Johnson wrote:
>>
>> Jörg is correct in saying that the 'geo_shape' filter only supports 
>> finding shapes within shapes and not points within shapes.
>>
>> It would be great if there was a filter which *did* support finding all 
>> points within a pre-indexed shape!
>>
>> You may want to open an issue regarding extending the 'geo-polygon' 
>> filter to support 'indexed_shape' instead of just 'points'.
>>
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-polygon-filter.html
>>
>> As it currently stands I don't think this is a bug, but the above feature 
>> request sounds like it could be useful to others.
>>
>> I also found a bunch of mistakes in your gist from spelling mistakes 
>> (businesses!=business) to syntax errors and things like not actually 
>> indexing a 'location' field for your business!?
>>
>> https://gist.github.com/waldoj/004f77755179d862fe3a#file-elasticsearch-sh-L226
>>
>> Feel free to use my minimal testcase as a template to make your debugging 
>> easier:
>> https://gist.github.com/missinglink/4bf3e262fbd0a89d3f5e
>>
>> -P
>>
>>
>> On Sunday, 4 January 2015 23:28:01 UTC, Jörg Prante wrote:
>>>
>>> I think you already found the answer so I hesitate to comment. If you 
>>> set "location" to type geo_point and not to geo_shape, you can not execute 
>>> a geo shape filter on it. There are geo bounding box / polygon filters that 
>>> can be executed on geo points.
>>>
>>> Best,
>>>
>>> Jörg
>>>
>>> On Sun, Jan 4, 2015 at 8:53 PM, Waldo Jaquith  wrote:
>>>
 Folks,

 [I sent the following on December 7, and got no reply. I’m sending it 
 again and, if nobody tells me otherwise, I’ll assume that this is a bug, 
 and file a report accordingly.]

 I’m trying to run a geo_shape filter query using a pre-indexed shape, 
 and it’s failing for reasons that I cannot identify. This is with 
 Elasticsearch v1.4.1-1. I’ve documented how to reproduce the problem at <
 https://gist.github.com/waldoj/004f77755179d862fe3a>.

 The pre-indexed shapes are GeoJSON converted from Census Bureau 
 municipal geodata. The indexed documents are state corporate registration 
 recor

Re: Geoquery Against Pre-Indexed Shape is Failing [Second Try]

[Waldo Jaquith]

This is fascinating—it hadn't crossed my mind that I might be doing 
something that simply wasn't supported by Elasticsearch. As many times as I 
read the documentation, I think I was reading for what I *wanted* it to do, 
rather than as a blank slate. Perhaps I'm trying to do something that is 
supported, but I'm doing it wrong?

This is for all businesses registered in a U.S. state (Virginia), to 
identify which ones are registered in a given county or city (a sub-unit of 
a state). The UK equivalent of this would be a list of all businesses 
registered in the East Midlands, and wanting to identify all of the 
businesses in Nottingham. I want somebody running a search for a business 
name to be able to limit that search to a given county or city. Perhaps I'm 
doing that all wrong, in terms of how Elasticsearch works. (At present, I'm 
doing this by including the shapes in my HTML, rather than pre-indexing 
them.) Is there a more Elasticsearch-y way to accomplish this, or is it 
really best for me to open an issue proposing such a new feature?

My apologies for the errors in my gist! Thank you for forking it and fixing 
them. This is my first time putting together a complete test case of a 
problem for a mailing list, so when the test case failed, it only served to 
(wrongly) reinforce that I'd correctly reproduced the problem!

Best,
Waldo

---
Waldo Jaquith
Director
U.S. Open Data
http://usopendata.org/
202-719-5315


On Monday, January 5, 2015 9:15:51 AM UTC-5, Peter Johnson wrote:
>
> Jörg is correct in saying that the 'geo_shape' filter only supports 
> finding shapes within shapes and not points within shapes.
>
> It would be great if there was a filter which *did* support finding all 
> points within a pre-indexed shape!
>
> You may want to open an issue regarding extending the 'geo-polygon' filter 
> to support 'indexed_shape' instead of just 'points'.
>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-polygon-filter.html
>
> As it currently stands I don't think this is a bug, but the above feature 
> request sounds like it could be useful to others.
>
> I also found a bunch of mistakes in your gist from spelling mistakes 
> (businesses!=business) to syntax errors and things like not actually 
> indexing a 'location' field for your business!?
>
> https://gist.github.com/waldoj/004f77755179d862fe3a#file-elasticsearch-sh-L226
>
> Feel free to use my minimal testcase as a template to make your debugging 
> easier:
> https://gist.github.com/missinglink/4bf3e262fbd0a89d3f5e
>
> -P
>
>
> On Sunday, 4 January 2015 23:28:01 UTC, Jörg Prante wrote:
>>
>> I think you already found the answer so I hesitate to comment. If you set 
>> "location" to type geo_point and not to geo_shape, you can not execute a 
>> geo shape filter on it. There are geo bounding box / polygon filters that 
>> can be executed on geo points.
>>
>> Best,
>>
>> Jörg
>>
>> On Sun, Jan 4, 2015 at 8:53 PM, Waldo Jaquith  wrote:
>>
>>> Folks,
>>>
>>> [I sent the following on December 7, and got no reply. I’m sending it 
>>> again and, if nobody tells me otherwise, I’ll assume that this is a bug, 
>>> and file a report accordingly.]
>>>
>>> I’m trying to run a geo_shape filter query using a pre-indexed shape, 
>>> and it’s failing for reasons that I cannot identify. This is with 
>>> Elasticsearch v1.4.1-1. I’ve documented how to reproduce the problem at <
>>> https://gist.github.com/waldoj/004f77755179d862fe3a>.
>>>
>>> The pre-indexed shapes are GeoJSON converted from Census Bureau 
>>> municipal geodata. The indexed documents are state corporate registration 
>>> records, each of which have an address that I’ve geocoded into a latitude 
>>> and longitude (a field named “location”). The indexed shapes are in a 
>>> "geometry”:{ “coordinates": { [1, 2] [3, 5] } } format. In the query I 
>>> specify "path": “geometry”, which generates this error:
>>>
>>> "Failed to find geo_shape field [location]].”
>>>
>>> Based on the error, it looks like Elasticsearch is looking for *shapes* 
>>> that fall within the pre-indexed shape, rather than *points*, and so while 
>>> the indexed documents have a geo_point field named “location,” there’s no 
>>> geo_shape field of that name. FWIW, I’m following the instructions in the 
>>> manual at <
>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html#_pre_indexed_shape
>>> >.
>>>
>>> I’d be grateful to anybody who could offer some guidance. Thank you.
>>>
>>> Best,
>>> Waldo
>>>
>>> ---
>>> Waldo Jaquith
>>> Director
>>> U.S. Open Data Institute
>>> http://usodi.org/
>>> 202-719-5315
>>>
>>> --
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/450CB

Re: Geoquery Against Pre-Indexed Shape is Failing [Second Try]

[Peter Johnson]

Jörg is correct in saying that the 'geo_shape' filter only supports finding 
shapes within shapes and not points within shapes.

It would be great if there was a filter which *did* support finding all 
points within a pre-indexed shape!

You may want to open an issue regarding extending the 'geo-polygon' filter 
to support 'indexed_shape' instead of just 'points'.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-polygon-filter.html

As it currently stands I don't think this is a bug, but the above feature 
request sounds like it could be useful to others.

I also found a bunch of mistakes in your gist from spelling mistakes 
(businesses!=business) to syntax errors and things like not actually 
indexing a 'location' field for your business!?
https://gist.github.com/waldoj/004f77755179d862fe3a#file-elasticsearch-sh-L226

Feel free to use my minimal testcase as a template to make your debugging 
easier:
https://gist.github.com/missinglink/4bf3e262fbd0a89d3f5e

-P


On Sunday, 4 January 2015 23:28:01 UTC, Jörg Prante wrote:
>
> I think you already found the answer so I hesitate to comment. If you set 
> "location" to type geo_point and not to geo_shape, you can not execute a 
> geo shape filter on it. There are geo bounding box / polygon filters that 
> can be executed on geo points.
>
> Best,
>
> Jörg
>
> On Sun, Jan 4, 2015 at 8:53 PM, Waldo Jaquith  > wrote:
>
>> Folks,
>>
>> [I sent the following on December 7, and got no reply. I’m sending it 
>> again and, if nobody tells me otherwise, I’ll assume that this is a bug, 
>> and file a report accordingly.]
>>
>> I’m trying to run a geo_shape filter query using a pre-indexed shape, and 
>> it’s failing for reasons that I cannot identify. This is with Elasticsearch 
>> v1.4.1-1. I’ve documented how to reproduce the problem at <
>> https://gist.github.com/waldoj/004f77755179d862fe3a>.
>>
>> The pre-indexed shapes are GeoJSON converted from Census Bureau municipal 
>> geodata. The indexed documents are state corporate registration records, 
>> each of which have an address that I’ve geocoded into a latitude and 
>> longitude (a field named “location”). The indexed shapes are in a 
>> "geometry”:{ “coordinates": { [1, 2] [3, 5] } } format. In the query I 
>> specify "path": “geometry”, which generates this error:
>>
>> "Failed to find geo_shape field [location]].”
>>
>> Based on the error, it looks like Elasticsearch is looking for *shapes* 
>> that fall within the pre-indexed shape, rather than *points*, and so while 
>> the indexed documents have a geo_point field named “location,” there’s no 
>> geo_shape field of that name. FWIW, I’m following the instructions in the 
>> manual at <
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html#_pre_indexed_shape
>> >.
>>
>> I’d be grateful to anybody who could offer some guidance. Thank you.
>>
>> Best,
>> Waldo
>>
>> ---
>> Waldo Jaquith
>> Director
>> U.S. Open Data Institute
>> http://usodi.org/
>> 202-719-5315
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/450CBB58-AA14-4106-A6C2-5762DB4845F5%40usodi.org
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3ea4e571-e192-4136-b860-65c19801193c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Geoquery Against Pre-Indexed Shape is Failing [Second Try]

[joergpra...@gmail.com]

I think you already found the answer so I hesitate to comment. If you set
"location" to type geo_point and not to geo_shape, you can not execute a
geo shape filter on it. There are geo bounding box / polygon filters that
can be executed on geo points.

Best,

Jörg

On Sun, Jan 4, 2015 at 8:53 PM, Waldo Jaquith  wrote:

> Folks,
>
> [I sent the following on December 7, and got no reply. I’m sending it
> again and, if nobody tells me otherwise, I’ll assume that this is a bug,
> and file a report accordingly.]
>
> I’m trying to run a geo_shape filter query using a pre-indexed shape, and
> it’s failing for reasons that I cannot identify. This is with Elasticsearch
> v1.4.1-1. I’ve documented how to reproduce the problem at <
> https://gist.github.com/waldoj/004f77755179d862fe3a>.
>
> The pre-indexed shapes are GeoJSON converted from Census Bureau municipal
> geodata. The indexed documents are state corporate registration records,
> each of which have an address that I’ve geocoded into a latitude and
> longitude (a field named “location”). The indexed shapes are in a
> "geometry”:{ “coordinates": { [1, 2] [3, 5] } } format. In the query I
> specify "path": “geometry”, which generates this error:
>
> "Failed to find geo_shape field [location]].”
>
> Based on the error, it looks like Elasticsearch is looking for *shapes*
> that fall within the pre-indexed shape, rather than *points*, and so while
> the indexed documents have a geo_point field named “location,” there’s no
> geo_shape field of that name. FWIW, I’m following the instructions in the
> manual at <
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html#_pre_indexed_shape
> >.
>
> I’d be grateful to anybody who could offer some guidance. Thank you.
>
> Best,
> Waldo
>
> ---
> Waldo Jaquith
> Director
> U.S. Open Data Institute
> http://usodi.org/
> 202-719-5315
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/450CBB58-AA14-4106-A6C2-5762DB4845F5%40usodi.org
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHuSsEkNEwwLnM-T4Kp-zV6Dku%3DZpwf9J9s3%3DR%3DvqF1ww%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Geoquery Against Pre-Indexed Shape is Failing [Second Try]

[Waldo Jaquith]

Folks,

[I sent the following on December 7, and got no reply. I’m sending it again 
and, if nobody tells me otherwise, I’ll assume that this is a bug, and file a 
report accordingly.]

I’m trying to run a geo_shape filter query using a pre-indexed shape, and it’s 
failing for reasons that I cannot identify. This is with Elasticsearch 
v1.4.1-1. I’ve documented how to reproduce the problem at 
.

The pre-indexed shapes are GeoJSON converted from Census Bureau municipal 
geodata. The indexed documents are state corporate registration records, each 
of which have an address that I’ve geocoded into a latitude and longitude (a 
field named “location”). The indexed shapes are in a "geometry”:{ 
“coordinates": { [1, 2] [3, 5] } } format. In the query I specify "path": 
“geometry”, which generates this error:

"Failed to find geo_shape field [location]].”

Based on the error, it looks like Elasticsearch is looking for *shapes* that 
fall within the pre-indexed shape, rather than *points*, and so while the 
indexed documents have a geo_point field named “location,” there’s no geo_shape 
field of that name. FWIW, I’m following the instructions in the manual at 
.

I’d be grateful to anybody who could offer some guidance. Thank you.

Best,
Waldo

---
Waldo Jaquith
Director
U.S. Open Data Institute
http://usodi.org/
202-719-5315

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/450CBB58-AA14-4106-A6C2-5762DB4845F5%40usodi.org.
For more options, visit https://groups.google.com/d/optout.