Summary: If geolocation data is mapped to geo_points in ElasticSearch (to enable bounding box/polygon/pin and distance searches), Graylog maps no longer function with the new data.
System: Graylog 2.1 OVA, ES and everything else as shipped in the OVA. Background: Enabled the geolocation processor on 27th September 2016 (over 4 months ago). The maps worked since that time without issue (I have another problem with the maps - revolving around the apparent inability to graph/map anything other than the message count - but that is a separate discussion and I'll start a new thread for that). Decided to try and run some geo bounding box searches on ElasticSearch using the geolocation info acquired (from 4 months ago to current). Only to discover that ES requires a mapping of the geolocation points to be actual geo_points rather than strings (as the geolocation plugin creates them now). Result: that 4 months worth of data is unusable. So the geolocation processor/plugin is loading geolocation data into a geolocation-aware database, but *not* in a format that the geolocation-aware database can use to do a native geo-based search. This in itself appears counterintuitive. So I added the mapping to ES (after a lot of digging - eventually found the 1st reference below). And the bounding box search works beautifully (using a python script to query ES directly). However, all the maps are now broken and return status 500 errors when clicking the "World Map" option in a geolocation enabled field (message shown in the red background at lower part of screen): Could not load map information Loading map information failed: cannot POST http://localhost/api/plugins/org.graylog.plugins.map/mapdata (500) Maps of the data before the change to the ES mapping still work as expected - so can still map the old data. So it seems that I can have mapping ability in Graylog, _or_ I can do a (python) bounding box/polygon/pin and distance search - but not simultaneously have both? It would seem to me to be a bug - when I enabled the geolocation processor I expected: 1) to be able to do a bounding box/polygon/pin-distance style search at a later time (ie:that the location data is put in the db in a format the db can _natively_ use in a geo query - a standard function of said db) 2) to have maps in the graylog interface 3) for 1 and 2 not to be mutually exclusive "1)" is currently not the case unless the geo_point mapping is added. "2)" only works while "1)" is not enabled. So the question is: How to fix maps in graylog without removing the geo_point mapping from elasticsearch? Am confident this was a minor oversight when the map processor/plugin was built. Note: The format of the data loaded into ES is unchanged - it is still a string as such in query results eg: "srcip_geolocation" : "-37.7204,145.0698" - it is only the mapping in ES that has been changed. References: Graylog plugin - elastic field types: Last post describes the way to enable correct ES mapping of the geo_points. https://groups.google.com/forum/?hl=en#!searchin/graylog2/geopoint%7Csort:relevance/graylog2/G1hOxGMFls4/FDAsN3h3BQAJ Not sure if relevant, but this page indicated additional requirement for the mapping, namely "lat_lon" : true - unsure if that is required/necessary but it didn't cause any errors - I obviously changed the URL to suit my graylog instance: ie: curl -XPUT 'localhost:9200/listing/listing/_mapping?pretty' -d '{"listing": {"properties": {"rent": {"type": "intger"}, "location": {"type": "geo_point", *"lat_lon": true*}}}}' https://gist.github.com/mastermunj/a58298bc56615b54d643 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/d631139d-77bd-451b-a819-5c5f8e96f0fc%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
