Re: How to get Elasticsearch boolean match working for multiple fields
Hi - thanks again - I was misunderstanding the following : path : { type : string, norms : { enabled : false }, fields : { raw : { type : string, index : not_analyzed, ignore_above : 256 } } } This is saying that the path is analyzed (default analyzer, and no 'index: not_analyzed'), but that the field 'raw' is not analyzed. One solution for me will be to simply use the path.raw field instead of the path field. I'll also try the regexp. Thanks again for the help! Dom On Fri, May 8, 2015 at 10:35 AM, Allan Mitchell casfanal...@gmail.com wrote: Dominic Normal nomenclature is that Field is analyzed and Field.raw is not analyzed. Not sure why you would have both as not analyzed given they would do the same thing, all else being equal When performing your original query above on fields I know are not_analyzed I get no results because there are no strings in the fields that match those terms exactly. I could of course look to do a regex query GET /testingindex/mytesttype/_search { query: { bool: { must: [ { regexp : { message : .*Failed password for.* } }, { regexp : { path : .*/var/log/secure.* } } ] } } } On 8 May 2015 at 15:03, Dominic Nicholas dominic.s.nicho...@gmail.com wrote: Hi Alan, I really appreciate the thoughtful response. One comment before I try what you are suggesting... Our path and message fields mappings indicate not_analyzed, and we don't want to change them at this point. Someone suggested using the .raw versions of the fields (path.raw and message.raw, which does work. However, it leaves me with the question : If the original field mappings indicate the fields are not_analyzed, why is it necessary to use the .raw version ? Cheers Dom On Fri, May 8, 2015 at 6:37 AM, Allan Mitchell casfanal...@gmail.com wrote: Hi Have a look at the below and see if it is what you want. DELETE /testingindex PUT /testingindex { settings : { number_of_shards : 1 }, mappings : { mytesttype : { _source : { enabled : false }, properties : { message : { type : string, index : analyzed }, path : {type: string, index: analyzed } } } } } POST /testingindex/mytesttype/1 { message: Failed password for some user or another, path:/wrong/path/ } POST /testingindex/mytesttype/2 { message: Not the right message but the right path, path:/var/log/secure } POST /testingindex/mytesttype/3 { message: Failed password for some user or another, path:/var/log/secure } POST /testingindex/mytesttype/4 { message: Nothing is right here, path:/wrong/path/too } GET /testingindex/mytesttype/_search GET /testingindex/mytesttype/_search { query: { bool: { must: [ { match_phrase : { message : Failed password for some } }, { match_phrase : { path : /var/log/secure } } ] } } } On 8 May 2015 at 02:07, Dominic Nicholas dominic.s.nicho...@gmail.com wrote: Hi, I need some expert guidance on trying to get a bool match working. I'd like the query to only return a successful search result if *both* 'message' matches 'Failed password for', *and* 'path' matches '/var/log/secure'. This is my query : curl -s -XGET 'http://localhost:9200/logstash-2015.05.07/syslog/_search?pretty=true' -d '{ filter : { range : { @timestamp : { gte : now-1h } } }, query : { bool : { must : [ { match_phrase : { message : Failed password for } }, { match_phrase : { path: /var/log/secure } } ] } } } ' Here is the start of the output from the search : { took : 3, timed_out : false, _shards : { total : 5, successful : 5, failed : 0 }, hits : { total : 46, max_score : 13.308596, hits : [ { _index : logstash-2015.05.07, _type : syslog, _id : AU0wzLEqqCKq_IPSp_8k, _score : 13.308596, _source:{message:May 7 16:53:50 s_local@logstash-02 sshd[17970]: Failed password for fred from 172.28.111.200 port 43487 ssh2,@version:1,@timestamp:2015-05-07T16:53:50.554-07:00,type:syslog,host:logstash-02,path:/var/log/secure} }, ... The problem is if I change '/var/log/secure' to just 'var' say, and run the query, I still get a result, just with a lower score. I understood the bool...must construct meant both match terms here would need to be successful. What I'm after is *no* result if 'path' doesn't exactly match '/var/log/secure'... { took : 3, timed_out : false, _shards : { total : 5, successful : 5, failed : 0 }, hits : { total : 46
How to get Elasticsearch boolean match working for multiple fields
Hi, I need some expert guidance on trying to get a bool match working. I'd like the query to only return a successful search result if *both* 'message' matches 'Failed password for', *and* 'path' matches '/var/log/secure'. This is my query : curl -s -XGET 'http://localhost:9200/logstash-2015.05.07/syslog/_search?pretty=true' -d '{ filter : { range : { @timestamp : { gte : now-1h } } }, query : { bool : { must : [ { match_phrase : { message : Failed password for } }, { match_phrase : { path: /var/log/secure } } ] } } } ' Here is the start of the output from the search : { took : 3, timed_out : false, _shards : { total : 5, successful : 5, failed : 0 }, hits : { total : 46, max_score : 13.308596, hits : [ { _index : logstash-2015.05.07, _type : syslog, _id : AU0wzLEqqCKq_IPSp_8k, _score : 13.308596, _source:{message:May 7 16:53:50 s_local@logstash-02 sshd[17970]: Failed password for fred from 172.28.111.200 port 43487 ssh2,@version:1,@timestamp:2015-05-07T16:53:50.554-07:00,type:syslog,host:logstash-02,path:/var/log/secure} }, ... The problem is if I change '/var/log/secure' to just 'var' say, and run the query, I still get a result, just with a lower score. I understood the bool...must construct meant both match terms here would need to be successful. What I'm after is *no* result if 'path' doesn't exactly match '/var/log/secure'... { took : 3, timed_out : false, _shards : { total : 5, successful : 5, failed : 0 }, hits : { total : 46, max_score : 10.354593, hits : [ { _index : logstash-2015.05.07, _type : syslog, _id : AU0wzLEqqCKq_IPSp_8k, _score : 10.354593, _source:{message:May 7 16:53:50 s_local@logstash-02 sshd[17970]: Failed password for fred from 172.28.111.200 port 43487 ssh2,@version:1,@timestamp:2015-05-07T16:53:50.554-07:00,type:syslog,host:logstash-02,path:/var/log/secure} },... I checked the mappings for these fields to check that they are not analyzed : curl -X GET 'http://localhost:9200/logstash-2015.05.07/_mapping?pretty=true' I think these fields are non analyzed and so I believe the search will not be analyzed too (based on some training documentation I read recently from elasticsearch). Here is a snippet of the output _mapping for this index below. message : { type : string, norms : { enabled : false }, fields : { raw : { type : string, index : not_analyzed, ignore_above : 256 } } }, path : { type : string, norms : { enabled : false }, fields : { raw : { type : string, index : not_analyzed, ignore_above : 256 } } }, Where am I going wrong (in a bunch of places I'm sure), what am I misunderstanding here (probably a lot!) ? Any help would be much appreciated! Thanks -- Please update your bookmarks! We moved to https://discuss.elastic.co/ --- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0470f9df-8d9a-48ef-9dbd-a90c8f2db194%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
elasticsearch init script for centos or rhel ?
Hi - can someone please point me to an /etc/init.d script for elasticsearch 1.0.1 for CentOS or RHEL ? Thanks -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/25064596-595d-4227-be37-d20f267edc5b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
help with setting up mappings (Perl)
Hi, I'm using the Elasticsearch Perl module and need guidance on setting up mappings. I'm using the bulk() method to index data. Here is an example of the structure of the data : $response = $e-bulk( index : idx-2014.03.10, type : my_type, body : [ { index : { _index : idx-2014.03.10, _id : 4410, _type : my_type } }, { something : interesting, somethingelse : also interesting }, { index : { _index : idx-2014.03.10, _id : 4411, _type : my_type } }, { something : very interesting, somethingelse : not interesting } ] ); How do I set up mappings on various fields in the above example for 'something' and 'somethingelse' fields ? Also, how do I turn off the analyzer for an index (index: not_analyzed) too ? I know there are several ways of setting up mappings such as : - when creating an index - by using the dedicated update mapping api - using index templates Ideally I'd like to use the dedicated update mapping api but am unclear how to use that through the Perl library interface (eg use transport-perform_request() ?). Thanks for any guidance and help. Dom -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f310961e-c389-49ea-82c9-c47d71d32209%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: help with setting up mappings (Perl)
Hi Clinton, I really appreciate the fast reply. We're now using Search::Elasticsearch. I'm still having a problem getting mappings set and hope you can help. What I'm trying to do is turn of the analyzer for certain fields. Here is what I have : In Perl : %mappings = ( 'index' = 'idx-2014.03.10', 'type' = 'my_type', 'body' = { 'my_type' = { properties = { 'somefield' = 'not_analyzed' } } } ); eval { $es_result = $elastic_search_object-indices-put_mapping( \%mappings ) ; }; etc etc The put_mapping call does not return an error. In JSON format, the mappings hash is : { index : idx-2014.03.10, type : my_type body : { my_type : { properties : { somefield : not_analyzed } } } } When I try to get the mappings for the this type/index with : curl -XGET 'localhost:9200/idx-2014.03.10/my_type/_mapping' I get {} which I think means there are no mappings yet set for this index and type. Is the body section above of the correct structure ? Is 'properties' always required ? Any guidance would be much appreciated again. I've been using these as guides so far : - https://metacpan.org/pod/Search::Elasticsearch::Client::Direct::Indices#put_mapping - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-put-mapping.html - some training docs Thanks again Dom On Monday, March 10, 2014 10:35:21 AM UTC-4, Dominic Nicholas wrote: Hi, I'm using the Elasticsearch Perl module and need guidance on setting up mappings. I'm using the bulk() method to index data. Here is an example of the structure of the data : $response = $e-bulk( index : idx-2014.03.10, type : my_type, body : [ { index : { _index : idx-2014.03.10, _id : 4410, _type : my_type } }, { something : interesting, somethingelse : also interesting }, { index : { _index : idx-2014.03.10, _id : 4411, _type : my_type } }, { something : very interesting, somethingelse : not interesting } ] ); How do I set up mappings on various fields in the above example for 'something' and 'somethingelse' fields ? Also, how do I turn off the analyzer for an index (index: not_analyzed) too ? I know there are several ways of setting up mappings such as : - when creating an index - by using the dedicated update mapping api - using index templates Ideally I'd like to use the dedicated update mapping api but am unclear how to use that through the Perl library interface (eg use transport-perform_request() ?). Thanks for any guidance and help. Dom -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f23f9a91-3176-4bea-8258-c9d38d1dd34f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.