Re: Geoquery Against Pre-Indexed Shape is Failing [Second Try]
This is just wonderful, Peter—thank you. (Providing a complete, tested, working example is probably more than I’ve ever done for anybody seeking debugging help!) It looks like I went and made the indexed location data more complicated than it needed to be, ginning up an object / geo_point structure, when all I really needed was type: point” and “coordinates: -78, 38”. This has been a blocker on a project for so long, I scarcely remember how to proceed from here. :) My thanks to you and the Elasticsearch community for your help! And, yes, I’d be happy to talk about US polygon data and sources of open address data. My organization is keenly interested in improving the state of open geodata in the US, and I’m eager to talk to anybody about how U.S. Open Data could help to improve things. Best, Waldo --- Waldo Jaquith Director U.S. Open Data http://usopendata.org/ 202-719-5315 On Jan 6, 2015, at 10:15 AM, Peter Johnson pe...@missinglink.co.nz wrote: Hey Waldo, I had a look over some of the code we have for Pelias and I managed to get it to work, the trick being that you need to index the business location as a 'geo-shape' 'point' type: https://gist.github.com/missinglink/d5d37082303bfdb5529c#file-gis-pre-indexed-sh-L50 https://gist.github.com/missinglink/d5d37082303bfdb5529c#file-gis-pre-indexed-sh-L62 Working example: https://gist.github.com/missinglink/d5d37082303bfdb5529c Visual representation: https://gist.github.com/anonymous/5384833492ecbaf814e5 When we index openstreetmap data in elasticsearch, we first find the intersection of the lat/lon within multiple indeces containing polygon information sourced from the quattroshapes project. The results are then stored as 'admin' properties against each record before indexing the POI, which means doing a lookup for all places inside polygon becomes a simple string match rather than a complex query-time computation. Would love to have a chat some time about collaborating on US polygon data and sources of open address data. All the best, -P On Tuesday, 6 January 2015 02:52:05 UTC, Waldo Jaquith wrote: This is fascinating—it hadn't crossed my mind that I might be doing something that simply wasn't supported by Elasticsearch. As many times as I read the documentation, I think I was reading for what I *wanted* it to do, rather than as a blank slate. Perhaps I'm trying to do something that is supported, but I'm doing it wrong? This is for all businesses registered in a U.S. state (Virginia), to identify which ones are registered in a given county or city (a sub-unit of a state). The UK equivalent of this would be a list of all businesses registered in the East Midlands, and wanting to identify all of the businesses in Nottingham. I want somebody running a search for a business name to be able to limit that search to a given county or city. Perhaps I'm doing that all wrong, in terms of how Elasticsearch works. (At present, I'm doing this by including the shapes in my HTML, rather than pre-indexing them.) Is there a more Elasticsearch-y way to accomplish this, or is it really best for me to open an issue proposing such a new feature? My apologies for the errors in my gist! Thank you for forking it and fixing them. This is my first time putting together a complete test case of a problem for a mailing list, so when the test case failed, it only served to (wrongly) reinforce that I'd correctly reproduced the problem! Best, Waldo --- Waldo Jaquith Director U.S. Open Data http://usopendata.org/ 202-719-5315 -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/E4F6AC07-3A57-404C-B73D-2DBA3266207C%40usodi.org. For more options, visit https://groups.google.com/d/optout.
Re: Geoquery Against Pre-Indexed Shape is Failing [Second Try]
Hey Waldo,
I had a look over some of the code we have for Pelias
https://github.com/pelias/pelias and I managed to get it to work, the
trick being that you need to index the business location as a 'geo-shape'
'point' type:
https://gist.github.com/missinglink/d5d37082303bfdb5529c#file-gis-pre-indexed-sh-L50
https://gist.github.com/missinglink/d5d37082303bfdb5529c#file-gis-pre-indexed-sh-L62
Working example: https://gist.github.com/missinglink/d5d37082303bfdb5529c
Visual
representation: https://gist.github.com/anonymous/5384833492ecbaf814e5
When we index openstreetmap data in elasticsearch, we first find the
intersection of the lat/lon within multiple indeces containing polygon
information sourced from the quattroshapes project.
The results are then stored as 'admin' properties against each record
http://pelias.mapzen.com/reverse?lat=38.0509lon=-78.4522size=20 before
indexing the POI, which means doing a lookup for all places inside polygon
becomes a simple string match rather than a complex query-time computation.
Would love to have a chat some time about collaborating on US polygon data
and sources of open address data.
All the best,
-P
On Tuesday, 6 January 2015 02:52:05 UTC, Waldo Jaquith wrote:
This is fascinating—it hadn't crossed my mind that I might be doing
something that simply wasn't supported by Elasticsearch. As many times as I
read the documentation, I think I was reading for what I *wanted* it to do,
rather than as a blank slate. Perhaps I'm trying to do something that is
supported, but I'm doing it wrong?
This is for all businesses registered in a U.S. state (Virginia), to
identify which ones are registered in a given county or city (a sub-unit of
a state). The UK equivalent of this would be a list of all businesses
registered in the East Midlands, and wanting to identify all of the
businesses in Nottingham. I want somebody running a search for a business
name to be able to limit that search to a given county or city. Perhaps I'm
doing that all wrong, in terms of how Elasticsearch works. (At present, I'm
doing this by including the shapes in my HTML, rather than pre-indexing
them.) Is there a more Elasticsearch-y way to accomplish this, or is it
really best for me to open an issue proposing such a new feature?
My apologies for the errors in my gist! Thank you for forking it and
fixing them. This is my first time putting together a complete test case of
a problem for a mailing list, so when the test case failed, it only served
to (wrongly) reinforce that I'd correctly reproduced the problem!
Best,
Waldo
---
Waldo Jaquith
Director
U.S. Open Data
http://usopendata.org/
202-719-5315
On Monday, January 5, 2015 9:15:51 AM UTC-5, Peter Johnson wrote:
Jörg is correct in saying that the 'geo_shape' filter only supports
finding shapes within shapes and not points within shapes.
It would be great if there was a filter which *did* support finding all
points within a pre-indexed shape!
You may want to open an issue regarding extending the 'geo-polygon'
filter to support 'indexed_shape' instead of just 'points'.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-polygon-filter.html
As it currently stands I don't think this is a bug, but the above feature
request sounds like it could be useful to others.
I also found a bunch of mistakes in your gist from spelling mistakes
(businesses!=business) to syntax errors and things like not actually
indexing a 'location' field for your business!?
https://gist.github.com/waldoj/004f77755179d862fe3a#file-elasticsearch-sh-L226
Feel free to use my minimal testcase as a template to make your debugging
easier:
https://gist.github.com/missinglink/4bf3e262fbd0a89d3f5e
-P
On Sunday, 4 January 2015 23:28:01 UTC, Jörg Prante wrote:
I think you already found the answer so I hesitate to comment. If you
set location to type geo_point and not to geo_shape, you can not execute
a geo shape filter on it. There are geo bounding box / polygon filters that
can be executed on geo points.
Best,
Jörg
On Sun, Jan 4, 2015 at 8:53 PM, Waldo Jaquith wa...@usodi.org wrote:
Folks,
[I sent the following on December 7, and got no reply. I’m sending it
again and, if nobody tells me otherwise, I’ll assume that this is a bug,
and file a report accordingly.]
I’m trying to run a geo_shape filter query using a pre-indexed shape,
and it’s failing for reasons that I cannot identify. This is with
Elasticsearch v1.4.1-1. I’ve documented how to reproduce the problem at
https://gist.github.com/waldoj/004f77755179d862fe3a.
The pre-indexed shapes are GeoJSON converted from Census Bureau
municipal geodata. The indexed documents are state corporate registration
records, each of which have an address that I’ve geocoded into a latitude
and longitude (a field named “location”). The indexed shapes are in a
geometry”:{ “coordinates: { [1, 2]
Re: Geoquery Against Pre-Indexed Shape is Failing [Second Try]
Jörg is correct in saying that the 'geo_shape' filter only supports finding
shapes within shapes and not points within shapes.
It would be great if there was a filter which *did* support finding all
points within a pre-indexed shape!
You may want to open an issue regarding extending the 'geo-polygon' filter
to support 'indexed_shape' instead of just 'points'.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-polygon-filter.html
As it currently stands I don't think this is a bug, but the above feature
request sounds like it could be useful to others.
I also found a bunch of mistakes in your gist from spelling mistakes
(businesses!=business) to syntax errors and things like not actually
indexing a 'location' field for your business!?
https://gist.github.com/waldoj/004f77755179d862fe3a#file-elasticsearch-sh-L226
Feel free to use my minimal testcase as a template to make your debugging
easier:
https://gist.github.com/missinglink/4bf3e262fbd0a89d3f5e
-P
On Sunday, 4 January 2015 23:28:01 UTC, Jörg Prante wrote:
I think you already found the answer so I hesitate to comment. If you set
location to type geo_point and not to geo_shape, you can not execute a
geo shape filter on it. There are geo bounding box / polygon filters that
can be executed on geo points.
Best,
Jörg
On Sun, Jan 4, 2015 at 8:53 PM, Waldo Jaquith wa...@usodi.org
javascript: wrote:
Folks,
[I sent the following on December 7, and got no reply. I’m sending it
again and, if nobody tells me otherwise, I’ll assume that this is a bug,
and file a report accordingly.]
I’m trying to run a geo_shape filter query using a pre-indexed shape, and
it’s failing for reasons that I cannot identify. This is with Elasticsearch
v1.4.1-1. I’ve documented how to reproduce the problem at
https://gist.github.com/waldoj/004f77755179d862fe3a.
The pre-indexed shapes are GeoJSON converted from Census Bureau municipal
geodata. The indexed documents are state corporate registration records,
each of which have an address that I’ve geocoded into a latitude and
longitude (a field named “location”). The indexed shapes are in a
geometry”:{ “coordinates: { [1, 2] [3, 5] } } format. In the query I
specify path: “geometry”, which generates this error:
Failed to find geo_shape field [location]].”
Based on the error, it looks like Elasticsearch is looking for *shapes*
that fall within the pre-indexed shape, rather than *points*, and so while
the indexed documents have a geo_point field named “location,” there’s no
geo_shape field of that name. FWIW, I’m following the instructions in the
manual at
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html#_pre_indexed_shape
.
I’d be grateful to anybody who could offer some guidance. Thank you.
Best,
Waldo
---
Waldo Jaquith
Director
U.S. Open Data Institute
http://usodi.org/
202-719-5315
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com javascript:.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/450CBB58-AA14-4106-A6C2-5762DB4845F5%40usodi.org
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3ea4e571-e192-4136-b860-65c19801193c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Geoquery Against Pre-Indexed Shape is Failing [Second Try]
This is fascinating—it hadn't crossed my mind that I might be doing
something that simply wasn't supported by Elasticsearch. As many times as I
read the documentation, I think I was reading for what I *wanted* it to do,
rather than as a blank slate. Perhaps I'm trying to do something that is
supported, but I'm doing it wrong?
This is for all businesses registered in a U.S. state (Virginia), to
identify which ones are registered in a given county or city (a sub-unit of
a state). The UK equivalent of this would be a list of all businesses
registered in the East Midlands, and wanting to identify all of the
businesses in Nottingham. I want somebody running a search for a business
name to be able to limit that search to a given county or city. Perhaps I'm
doing that all wrong, in terms of how Elasticsearch works. (At present, I'm
doing this by including the shapes in my HTML, rather than pre-indexing
them.) Is there a more Elasticsearch-y way to accomplish this, or is it
really best for me to open an issue proposing such a new feature?
My apologies for the errors in my gist! Thank you for forking it and fixing
them. This is my first time putting together a complete test case of a
problem for a mailing list, so when the test case failed, it only served to
(wrongly) reinforce that I'd correctly reproduced the problem!
Best,
Waldo
---
Waldo Jaquith
Director
U.S. Open Data
http://usopendata.org/
202-719-5315
On Monday, January 5, 2015 9:15:51 AM UTC-5, Peter Johnson wrote:
Jörg is correct in saying that the 'geo_shape' filter only supports
finding shapes within shapes and not points within shapes.
It would be great if there was a filter which *did* support finding all
points within a pre-indexed shape!
You may want to open an issue regarding extending the 'geo-polygon' filter
to support 'indexed_shape' instead of just 'points'.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-polygon-filter.html
As it currently stands I don't think this is a bug, but the above feature
request sounds like it could be useful to others.
I also found a bunch of mistakes in your gist from spelling mistakes
(businesses!=business) to syntax errors and things like not actually
indexing a 'location' field for your business!?
https://gist.github.com/waldoj/004f77755179d862fe3a#file-elasticsearch-sh-L226
Feel free to use my minimal testcase as a template to make your debugging
easier:
https://gist.github.com/missinglink/4bf3e262fbd0a89d3f5e
-P
On Sunday, 4 January 2015 23:28:01 UTC, Jörg Prante wrote:
I think you already found the answer so I hesitate to comment. If you set
location to type geo_point and not to geo_shape, you can not execute a
geo shape filter on it. There are geo bounding box / polygon filters that
can be executed on geo points.
Best,
Jörg
On Sun, Jan 4, 2015 at 8:53 PM, Waldo Jaquith wa...@usodi.org wrote:
Folks,
[I sent the following on December 7, and got no reply. I’m sending it
again and, if nobody tells me otherwise, I’ll assume that this is a bug,
and file a report accordingly.]
I’m trying to run a geo_shape filter query using a pre-indexed shape,
and it’s failing for reasons that I cannot identify. This is with
Elasticsearch v1.4.1-1. I’ve documented how to reproduce the problem at
https://gist.github.com/waldoj/004f77755179d862fe3a.
The pre-indexed shapes are GeoJSON converted from Census Bureau
municipal geodata. The indexed documents are state corporate registration
records, each of which have an address that I’ve geocoded into a latitude
and longitude (a field named “location”). The indexed shapes are in a
geometry”:{ “coordinates: { [1, 2] [3, 5] } } format. In the query I
specify path: “geometry”, which generates this error:
Failed to find geo_shape field [location]].”
Based on the error, it looks like Elasticsearch is looking for *shapes*
that fall within the pre-indexed shape, rather than *points*, and so while
the indexed documents have a geo_point field named “location,” there’s no
geo_shape field of that name. FWIW, I’m following the instructions in the
manual at
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html#_pre_indexed_shape
.
I’d be grateful to anybody who could offer some guidance. Thank you.
Best,
Waldo
---
Waldo Jaquith
Director
U.S. Open Data Institute
http://usodi.org/
202-719-5315
--
You received this message because you are subscribed to the Google
Groups elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/450CBB58-AA14-4106-A6C2-5762DB4845F5%40usodi.org
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
Geoquery Against Pre-Indexed Shape is Failing [Second Try]
Folks,
[I sent the following on December 7, and got no reply. I’m sending it again
and, if nobody tells me otherwise, I’ll assume that this is a bug, and file a
report accordingly.]
I’m trying to run a geo_shape filter query using a pre-indexed shape, and it’s
failing for reasons that I cannot identify. This is with Elasticsearch
v1.4.1-1. I’ve documented how to reproduce the problem at
https://gist.github.com/waldoj/004f77755179d862fe3a.
The pre-indexed shapes are GeoJSON converted from Census Bureau municipal
geodata. The indexed documents are state corporate registration records, each
of which have an address that I’ve geocoded into a latitude and longitude (a
field named “location”). The indexed shapes are in a geometry”:{
“coordinates: { [1, 2] [3, 5] } } format. In the query I specify path:
“geometry”, which generates this error:
Failed to find geo_shape field [location]].”
Based on the error, it looks like Elasticsearch is looking for *shapes* that
fall within the pre-indexed shape, rather than *points*, and so while the
indexed documents have a geo_point field named “location,” there’s no geo_shape
field of that name. FWIW, I’m following the instructions in the manual at
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html#_pre_indexed_shape.
I’d be grateful to anybody who could offer some guidance. Thank you.
Best,
Waldo
---
Waldo Jaquith
Director
U.S. Open Data Institute
http://usodi.org/
202-719-5315
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/450CBB58-AA14-4106-A6C2-5762DB4845F5%40usodi.org.
For more options, visit https://groups.google.com/d/optout.
Re: Geoquery Against Pre-Indexed Shape is Failing [Second Try]
I think you already found the answer so I hesitate to comment. If you set
location to type geo_point and not to geo_shape, you can not execute a
geo shape filter on it. There are geo bounding box / polygon filters that
can be executed on geo points.
Best,
Jörg
On Sun, Jan 4, 2015 at 8:53 PM, Waldo Jaquith wa...@usodi.org wrote:
Folks,
[I sent the following on December 7, and got no reply. I’m sending it
again and, if nobody tells me otherwise, I’ll assume that this is a bug,
and file a report accordingly.]
I’m trying to run a geo_shape filter query using a pre-indexed shape, and
it’s failing for reasons that I cannot identify. This is with Elasticsearch
v1.4.1-1. I’ve documented how to reproduce the problem at
https://gist.github.com/waldoj/004f77755179d862fe3a.
The pre-indexed shapes are GeoJSON converted from Census Bureau municipal
geodata. The indexed documents are state corporate registration records,
each of which have an address that I’ve geocoded into a latitude and
longitude (a field named “location”). The indexed shapes are in a
geometry”:{ “coordinates: { [1, 2] [3, 5] } } format. In the query I
specify path: “geometry”, which generates this error:
Failed to find geo_shape field [location]].”
Based on the error, it looks like Elasticsearch is looking for *shapes*
that fall within the pre-indexed shape, rather than *points*, and so while
the indexed documents have a geo_point field named “location,” there’s no
geo_shape field of that name. FWIW, I’m following the instructions in the
manual at
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-filter.html#_pre_indexed_shape
.
I’d be grateful to anybody who could offer some guidance. Thank you.
Best,
Waldo
---
Waldo Jaquith
Director
U.S. Open Data Institute
http://usodi.org/
202-719-5315
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/450CBB58-AA14-4106-A6C2-5762DB4845F5%40usodi.org
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
elasticsearch group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHuSsEkNEwwLnM-T4Kp-zV6Dku%3DZpwf9J9s3%3DR%3DvqF1ww%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
