Re: Advice for implementing a secure graph index with ElasticSearch

[mohit kaushik]

And i also want to ask you, are you from altamira? i found you on 
lumify.


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e8f576c8-5d3f-4402-915d-8d0e9eaf8c10%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Advice for implementing a secure graph index with ElasticSearch

[mohit kaushik]

Hi! jeff
You said, you are using lumify. And lumify user secure-graph which 
implicitly implements cell level security that you all need. You can easily 
have access controls on your user and query returns the vertices in the way 
you want. I recently started working with secure-graph and want to 
implement the class 
"/securegraph-core/src/main/java/org/securegraph/query/GraphQuery.java" 
wchich is provided in the package and as it has been much days to your post 
so hope you might have figured out it. If you have please notify me.


Thanks 
Mohit kaushik

On Wednesday, March 5, 2014 10:40:11 PM UTC+5:30, Jeff Kunkle wrote:
>
> I've been trying to figure out how I can index a graph data structure 
> using ElasticSearch and could really use some advice from someone more 
> knowledgeable than me. First, let me explain the challenge. The graph model 
> has individual access controls at the vertex (node), edge (relationship), 
> and property level. I'd like my users to be able to search the graph for 
> vertices or edges containing matching properties, with two caveats:
>
>1. They should not get vertex or edge results they don't have 
>permission to see.
>2. Properties a user does not have access to see should not be 
>evaluated in the query.
>
> My first thought was to index properties as either nested or child 
> documents of a vertex/edge and use a custom filter to remove properties a 
> user didn't have access to. The first problem I run into is when I try a 
> boolean query across properties. For example, assume I want to query a 
> person vertex by first name and date of birth. Since these properties are 
> indexed as separate documents there is never a match. 
>
> What I essentially need is the ability to query across nested or child 
> documents and return the parent only when there are matches across the 
> child documents. For example, assume a parent vertex with one property 
> document called "full_name" set to Barak Obama and another property 
> document named "political_party" set to Democrat. Is there any way for me 
> to query for the parent document of these two properties by asking for one 
> property with full_name="Barak Obama" and another property with 
> political_party="Democrat"?
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0ff88ff7-425f-40ff-91be-826962c904aa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Java issue when trying to send requests to ElasticSearch

[Charles Martinot]

The tutorial was done with older versions of logstash, kibana and
elasticsearch so I guess that those issues didn't show up.

I did a lot of things... And I spent so much time making it work, and the
tutorial is clearly not working out of the box. I also modified their
logstash cookbook so it could do what I want it to do.

Oh and by the way I'm not using rabbitMQ but AWS SQS instead.


On Fri, May 16, 2014 at 1:59 AM, Patrick Marx wrote:

> Hey,
> Did you ever find a solution to this? I'm running in to the same issue.
> Now that I think of it, I don't understand how the tutorial even worked
> unless they removed the  "elasticsearch-http-basic" plugin altogether.
>
>
> On Friday, March 21, 2014 4:12:10 AM UTC-7, cha...@pocketplaylab.comwrote:
>>
>> Oh, ok thanks...
>>
>> I had to update ES because the version of Kibana I am using wasn't
>> supporting the previous one. I guess I'll have to downgrade everything or
>> wait.
>>
>> Thanks a lot!
>>
>> On Friday, March 21, 2014 6:01:51 PM UTC+7, Kevin Wang wrote:
>>>
>>> It looks like you are using "elasticsearch-http-basic" plugin and that
>>> plugin doesn't support ES 1.0
>>> https://github.com/Asquera/elasticsearch-http-basic/issues/9
>>>
>>>
>>> On Friday, March 21, 2014 9:50:02 PM UTC+11, cha...@pocketplaylab.comwrote:

 Hi all,

 I am currently trying to set up a complete ElasticSearch + LogStash +
 Kibana stack on Amazon Web Services OpsWorks using the following tutorial :
 http://devblog.springest.com/complete-logstash-stack-
 on-aws-opsworks-in-15-minutes/

 Most of the things run fine except for ElasticSearch. When the process
 is started, if I try to do a simple *c**url -X GET
 http://localhost:9200/ *, I get the following
 answer : *curl: (52) Empty reply from server*

 In my cluster's log, I see the hereunder java error. Did anybody
 experience that ? Any suggestions ?

 Thanks for your help,

 Charles.

 Java error :

 *[2014-03-21 10:46:48,657][WARN ][http.netty   ] [Cecilia
 Reyes] Caught exception while handling client http traffic, closing
 connection [id: 0xf290eec5, /127.0.0.1:60355  =>
 /127.0.0.1:9200 ]*

 *java.lang.IncompatibleClassChangeError: Found class
 org.elasticsearch.http.HttpRequest, but interface was expected*

 * at
 com.asquera.elasticsearch.plugins.http.HttpBasicServer.shouldLetPass(HttpBasicServer.java:43)*

 * at
 com.asquera.elasticsearch.plugins.http.HttpBasicServer.internalDispatchRequest(HttpBasicServer.java:35)*

 * at
 org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)*

 * at
 org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:291)*

 * at
 org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:43)*

 * at
 org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)*

 * at
 org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)*

 * at
 org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)*

 * at
 org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)*

 * at
 org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)*

 * at
 org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)*

 * at
 org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)*

 * at
 org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)*

 * at
 org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)*

 * at
 org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)*

 * at
 org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)*

 * at
 org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)*

 * at
 org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)*

 * at
 org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)*

 * at
 org.elasticsearch.common.netty.OpenChannel

Re: Help with ES 1.x percolator query plz

[JGL]

Hi Martijin,

Thanks for the reply. The analyzer breaking up the UUID explains a lot why 
the UUIDs are not matched as a whole.

I am still wondering if we can register other types of queries other than 
match query into percolator.  We would like to put a list of values into a 
query for the "id" field, which is meant as a device ID, so that when we 
percolate a document with a device ID, all percolator queries whose ID list 
contains the device ID can be considered as a match.

But according to our experimentation, queries like the following are not 
working with percolator, which seems only happy with match queries:

{
  "_index" : "my_idx",
  "_type" : ".percolator",
  "_id" : "inQuery",
  "_score" : 1.0, "_source" : 
{"query":{"terms":{"id":["1aa808dc-48f0-4de3-8978-a0293d54b852","6b256fd1-cd04-4e3c-8f38-aaa87ac2220d"]}}}
},


{
  "_index" : "my_idx",
  "_type" : ".percolator",
  "_id" : "inFilterQ",
  "_score" : 1.0, "_source" : 
{"query":{"filtered":{"query":{"match_all":{}},"filter":{"terms":{"id":["1aa808dc-48f0-4de3-8978-a0293d50b852","6b256fd1-cd04-4e3c-8f38-aaa87ac2220d"]}
}, 


I could not find any resources clearly state that percolator can only work 
with match queries. Is it actually the case?

Thanks,
Jason

On Friday, May 9, 2014 10:04:51 PM UTC+12, Martijn v Groningen wrote:
>
> I think the issue here is that the 'id' field is analyzed and your UUIDS 
> are broken up into separate tokens. The standard analyzer is responsible 
> for breaking up by '-'. If you use the analyze api you can see what happens 
> with your uuids:
> curl -XGET 
> 'localhost:9200/_analyze?text=1aa808dc-48f0-4de3-8978-a0293d54b852 
> 6b256fd1-cd04-4e3c-8f38-aaa87ac2220d 
> 1234fd1a-cd04-4e3c-8f38-aaa87142380d&tokenizer=standard'
>
> The 'id' field in ES is not used as the id field. In ES the _id field is 
> used to store the unique identifier and that field is not analyzed. 
> Assuming that the 'id' field has the same value as the id of a document 
> then you can use the `ids` query instead in your percolator queries: 
>
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-ids-query.html#query-dsl-ids-query
>
> Martijn
>
>
> On 9 May 2014 09:20, JGL > wrote:
>
>> Can anybody help plz?
>>
>>
>> On Wednesday, May 7, 2014 6:29:35 PM UTC+12, JGL wrote:
>>>
>>> Can anybody help plz?
>>>
>>> On Tuesday, May 6, 2014 11:53:32 AM UTC+12, JGL wrote:


 Can anybody help plz?

 On Monday, May 5, 2014 10:24:09 AM UTC+12, JGL wrote:
>
>
> Hi Martjin,
>
> The percolator query in the 1st post above is what we registered to 
> the percolator and kinda working, which consolidate all IDs in one query 
> string for a match query, which seems not quite a elegant solution to us. 
>
> {
>   "_index" : "my_idx",
>   "_type" : ".percolator",
>   "_id" : "my_query_id",
>   "_score" : 1.0, 
>   "_source" : {
> "query":{
>"match":{
>   "id":{
>   "query":"id1 id2 id3",
>
>
>   "type":"boolean"
>}
>}
> }
>   }
> }
>
>
> Another issue is that the above solution is not quite accurate when 
> the IDs are UUIDs. For example, if the query we register is as the 
> following
>
> {
>   "_index" : "my_idx",
>   "_type" : ".percolator",
>   "_id" : "my_query_id",
>   "_score" : 1.0, 
>   "_source" : {
> "query":{
>"match":{
>   "id":{
>   
> "query":"1aa808dc-48f0-4de3-8978-*a0293d54b852* 
> 6b256fd1-cd04-4e3c-8f38-aaa87ac2220d 
> 1234fd1a-cd04-4e3c-8f38-aaa87142380d",
>
>
>   "type":"boolean"
>}
>}
> }
>   }
> }
>
>
> , the percolator return the above query as a match if the document we 
> try to percolate is "{"doc" : {"id":"1aa808dc-48f0-4de3-8978-
> *00293d54b852*"}}", though we are expecting a no match response here 
> as the id in the document does not have a matched ID in the query String. 
>
> Such false positive response, according to the experimentations we 
> had, happens when the doc UUID is almost the same to one of the IDs in 
> the 
> query except the the last part of ID. Wondering if there is an 
> explanation 
> for such behavior of elasticsearch?
>
> Our another question is if there is any way we could put the UUID list 
> as a list into a query that is working with 

Re: [hadoop] Extra Documents in Elastic Search

[Napoleon T.]

I did not entirely solve this issue. But it looks like ES is dropping some 
requests when it's overloaded.  As my hadoop cluster can handle 42 mappers, 
I had 42 tasks trying to send write requests to only 1 ES node (I believe 
all the requests only go to one node is ES).  Most of the time, many tasks 
will fail and my hadoop job will fail. But sometimes, hadoop returns a 
success and not all the data has been successfully written.
Reducing the number of mappers should have helped, but for some reasons 
running pig with the property -Dmapred.tasktracker.map.tasks.maximum=1 did 
not do the trick.
Limiting the number of mappers directly in the cluster conf files seems to 
have solved the problem. 

On Wednesday, April 23, 2014 4:15:19 PM UTC-5, Napoleon T. wrote:
>
> Hi,
>
> I'm trying to store a lot of documents into ES using pig. The pig job ends 
> successfully but I end up with more documents in Elasticsearch than the 
> number of rows in my input.
> My pig script is 3 lines: 
> REGISTER 'local/path/to/m2.jar'
> data = load 'path/to/hdfs/file.tsv' as (field1: chararray, field2: long, 
> field3: long, field4: long)
> store data into 'index/type' using 
> org.elasticsearch.hadoop.pig.EsStorage('es.nodes=node2.domain.com', 
> 'es.rersource=index/type');
>
> I have speculative execution disabled for map and reduce when running this 
> pig script. 
>
>
> Hadoop states that 54,723,557 records were written (console output and 
> job tracker UI).
> ES head plugin claims that I have docs: 57,344,987 (57,344,987).
>
> My environment:
> hadoop: 1.2.1 with 6 nodes cluster
> elasticsearch: 1.0.0. 6 node cluster. Different than hadoop nodes.
> elasticsearch-hadoop version M2. 
> Pig version: 0.12.0
>
> Any ideas of what is going on here?
>
> Thanks.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b819e576-f0ef-41d4-854a-63bab811951a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Extremely long time to create/delete alias

[E. A. Graham. Jr.]

Currently using ES 1.0.1 with 3 masters, 4 data nodes, and 2 "query" (REST 
endpoints) nodes.

   1. We are using aliases to apply routing to our queries
   2. We have multiple "ingesting applications", each supporting a 
   different index (currently 10)
  - Each uses NodeClient to connect to the ES cluster
  3. Each index has the same number of aliases (prefix matches index, 
   suffix is identifier) - around 2000
   4. Our environment ends up triggering large numbers (>20) of alias 
   creation/deletions
  - Each "ingesting application" creates/deletes the aliases linearly 
  5. Observed by logging and profiling, it takes up to *15 seconds* for 
   an alias to be created

Is this expected? Are we expecting too much from this particular operation? 
Is there some thread/queue somewhere that can be tweaked?

Thanks...

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/44a30d46-d86e-4615-91ef-767e8d6036a8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Java issue when trying to send requests to ElasticSearch

[Patrick Marx]

Hey,
Did you ever find a solution to this? I'm running in to the same issue. Now 
that I think of it, I don't understand how the tutorial even worked unless 
they removed the  "elasticsearch-http-basic" plugin altogether.


On Friday, March 21, 2014 4:12:10 AM UTC-7, cha...@pocketplaylab.com wrote:
>
> Oh, ok thanks...
>
> I had to update ES because the version of Kibana I am using wasn't 
> supporting the previous one. I guess I'll have to downgrade everything or 
> wait.
>
> Thanks a lot!
>
> On Friday, March 21, 2014 6:01:51 PM UTC+7, Kevin Wang wrote:
>>
>> It looks like you are using "elasticsearch-http-basic" plugin and that 
>> plugin doesn't support ES 1.0
>> https://github.com/Asquera/elasticsearch-http-basic/issues/9
>>
>>
>> On Friday, March 21, 2014 9:50:02 PM UTC+11, cha...@pocketplaylab.comwrote:
>>>
>>> Hi all,
>>>
>>> I am currently trying to set up a complete ElasticSearch + LogStash + 
>>> Kibana stack on Amazon Web Services OpsWorks using the following tutorial : 
>>> http://devblog.springest.com/complete-logstash-stack-on-aws-opsworks-in-15-minutes/
>>>
>>> Most of the things run fine except for ElasticSearch. When the process 
>>> is started, if I try to do a simple *c**url -X GET 
>>> http://localhost:9200/ *, I get the following 
>>> answer : *curl: (52) Empty reply from server*
>>>
>>> In my cluster's log, I see the hereunder java error. Did anybody 
>>> experience that ? Any suggestions ?
>>>
>>> Thanks for your help,
>>>
>>> Charles.
>>>
>>> Java error :
>>>
>>> *[2014-03-21 10:46:48,657][WARN ][http.netty   ] [Cecilia 
>>> Reyes] Caught exception while handling client http traffic, closing 
>>> connection [id: 0xf290eec5, /127.0.0.1:60355  => 
>>> /127.0.0.1:9200 ]*
>>>
>>> *java.lang.IncompatibleClassChangeError: Found class 
>>> org.elasticsearch.http.HttpRequest, but interface was expected*
>>>
>>> * at 
>>> com.asquera.elasticsearch.plugins.http.HttpBasicServer.shouldLetPass(HttpBasicServer.java:43)*
>>>
>>> * at 
>>> com.asquera.elasticsearch.plugins.http.HttpBasicServer.internalDispatchRequest(HttpBasicServer.java:35)*
>>>
>>> * at 
>>> org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)*
>>>
>>> * at 
>>> org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:291)*
>>>
>>> * at 
>>> org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:43)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)*
>>>
>>> * at 
>>> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)*
>>>
>>> * a

Re: modify explain string in elasticsearch plugin

[Srinivasan Ramaswamy]

Does the current explain gives details about the function/script in
function score query ? If so how is it achieved ? I would like to add
explanation for my custom scoring so that its easy to investigate. How do
people achieve this typically using elasticsearch ?

Thanks
Srini


On Wed, May 14, 2014 at 4:28 PM, Ivan Brusic  wrote:

> The explain string is basically a JSON formatted string of Lucene's
> explain object. I doubt this formatter is pluggable or that you can change
> the Explain object.
>
> --
> Ivan
>
>
> On Tue, May 13, 2014 at 11:36 PM, Srinivasan Ramaswamy  > wrote:
>
>> Hi
>>
>> I am overriding the scoring function using a plugin as described in
>>
>> https://github.com/imotov/elasticsearch-native-script-example
>>
>> I would like to know how to override the explain string, so that i can
>> add explanation for all my custom scoring.
>>
>> Thanks
>> Srini
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/b4485fa8-39f0-40f3-b614-9842c200a0e2%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/7e2Xe_mpJYU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAmU3cKM%3Ddw2NyV_7BZfZn6MnBJ8zy7SX0Ri45epZUPcA%40mail.gmail.com
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAL1MvVzRZ%2B5eSn85uNFi8BX963Q_QojcYp72m6unTrmm5SdLUw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Elastic Search - Query multiple indices

[CC]

Hi! I have data stored in 3 indices that I want to query, but these indices 
have different fields. If I try it like this:

curl -X GET 'http://host:9200/myindex1,myindex2,myindex3/_search?pretty' -d 
'{"query":{"bool":{"must":[{"query_string":{"query":"myquery","default_operator":"AND"}},
 
{"range":{"due_date":{"from":"2011-11-30T19:09:06.275Z","to":"2013-07-07T02:52:37.012Z"}}}]}},
 
"facets":{"facet1":{"terms":{"field":"text_content", "size":100' 

the results returned by ElasticSearch refer only to the first index, namely 
myindex1. Another issue is that I really need to filter these results by 
due_date, but only myIndex1 contains this field. The text_content field can 
also vary per index, i.e that in myindex2 and myindex3 it has got a 
different name.

How can I achieve my purpose using ElasticSearch? Many thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e911d3e5-1213-46bb-8ca2-04b411b04abf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Accessing Parameter values inside a MVEL script

[ravimbhatt]

Here is my full ES query in python 

res1 = es.search(index="myIndex", body={"query":{ "bool": { "must" : {
"terms":{ "type.itemid1":wiids  }  }, "must_not" : { "terms" : {
"type.itemid2":wiids } }  } },"facets":{ "itemid2Stats": {"terms_stats" : { 
"key_field": "type.itemid2", "value_script":"( iid1=  
doc['itemid1'].value;r = doc['internalScore'].value * 
srcVals.containsKey(iid1) ? srcVals.get(iid1): 11; return r )", "params": { 
"srcVals": sources  }, "size": 1000, "shard_size": 5000 ,"order": "mean"  
}  } }}, sort=["type.itemid1:desc","type.similarity:desc"], size=1)

and the error i get is:

sort[!,!]:
 
Query Failed [Failed to execute main query]]; nested: NullPointerException; 
}

On Thursday, 15 May 2014 17:26:49 UTC+1, ravim...@gmail.com wrote:
>
> Hi All, 
>
> I am trying to run a faceted query with a custom script on the facet 
> value. I need to pass a "map" of values into the script with params. Below 
> id my *script*, 
>
> "*value_script*":"( iid1= new String(doc['itemid1'].value) ;r = 
> doc['similarity'].value * srcVals.containsKey(iid1) ? srcVals.get(iid1): -1 
> ; return r )"
>
> For some reason, my *containsKey* returns true but get on the map results 
> in a *nullpointer* exception. 
>
> I pass params as :
>
> "*params*": { "srcVals": sources  }
>
> Where sources is a python dict. example dict is:
>
> {'3058211': '3', '3473092': '4', '3097044': '3', '2491239': '4', 
> '2371363': '4', '3049884': '5', '2212351': '4', '3064057': '3'}
>
> As you can see, i need to get a number associated with some other number. 
> I pass the association as a param. but accessing the param is returning a 
> nullpointer. 
>
> Thanks!
> Ravi
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d7753b14-c793-4c45-8aec-ac0fd1b386f9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Accessing Parameter values inside a MVEL script

[ravimbhatt]

Hi All, 

I am trying to run a faceted query with a custom script on the facet value. 
I need to pass a "map" of values into the script with params. Below id my 
*script*, 

"*value_script*":"( iid1= new String(doc['itemid1'].value) ;r = 
doc['similarity'].value * srcVals.containsKey(iid1) ? srcVals.get(iid1): -1 
; return r )"

For some reason, my *containsKey* returns true but get on the map results 
in a *nullpointer* exception. 

I pass params as :

"*params*": { "srcVals": sources  }

Where sources is a python dict. example dict is:

{'3058211': '3', '3473092': '4', '3097044': '3', '2491239': '4', '2371363': 
'4', '3049884': '5', '2212351': '4', '3064057': '3'}

As you can see, i need to get a number associated with some other number. I 
pass the association as a param. but accessing the param is returning a 
nullpointer. 

Thanks!
Ravi

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/31746a14-4c9c-4fe3-b8f2-c59947f31706%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

What Are Valid Geometries

[Brian Behling]

Hello ES Users.

Does anyone have some documentation as to what ES considers valid 
geometries? Polygons that I am trying to load into ES are created in an 
ESRI/SQL stack, and we use the validation methods in this stack to validate 
our geometrres . But when I try to load some of the geometries into ES, I 
get a self-intersecting polygon error. 

This is fine, I can modify the geometries to support ES, but I'm not sure 
as to what I'm looking for to make a valid ES geometry.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/93d3f489-0a54-4bad-9639-4f36e377a113%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Conditional query for geo location lookup

[Utkarsh Sengar]

Bumping this one up. Any advice on the query?


On Tue, May 13, 2014 at 6:17 PM, Utkarsh Sengar wrote:

> I have a usecase where I have 2 types of locations (i.e. with geo_point
> type):
>
> 1. Location 1: Has a lat/lon with say radius=90 miles (it will vary) and
> type=outgoing
> 2. Location 2: Has a lat/lon with no radius and type=incoming
>
> Now, when a query comes in with: lat/lon and radius=20, I expect this to
> happen:
>
> 1. Simple geo lookup: If the input lat/lon is within 20miles of location
> 2, return location 2.
> 2. If input lat/lon is within 90 miles of Location 1, return location 1
> too in the result. If you notice, I want input radius to be overwritten by
> the saved radius for a specific type of location.
>
> This is what I have come up with using script:
> {
>   "query": {
> "match_all": {}
>   },
>   "filter": {
> "script": {
>   "script": "!doc['geopoint'].empty && doc['coverage_type'].value ==
> 'outgoing' ? doc['geopoint'].distanceInMiles(37,-121) <=
> doc['radius'].value : doc['geopoint'].distanceInMiles(37,-121) <= 20"
> }
>   }
> }
> Where "37,-121" is input lat/lon and 20 is the input radius.
>
>
> What do you think?
>
> --
> Thanks,
> -Utkarsh
>



-- 
Thanks,
-Utkarsh

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CADjjot9Qxaq2Zd%2BdsCcX3YjJ4%3DKhAQ_fzv0BsUva5R-KWcSKOg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

ElasticSearch not seen in JPS

[Greg Bui]

Hello,

I encounter a weird problem, jps is not returning elasticsearch whereas it 
is running and (seems) working fine. I remember that was the case before. 
jstat is not working as well...

# jps
243157 jruby-complete-1.7.11.jar
50184 Jps


# ps aux | grep elasti 
494   48931 54.3 13.0 383893516 4303840 ?   Sl   17:29   3:07 
/usr/java/default/bin/java -Xms16g -Xmx16g -Xss256k 
-Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC 
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
-XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=256m -Delasticsearch 
-Des.pidfile=/var/run/elasticsearch/elasticsearch.pid 
-Des.path.home=/usr/share/elasticsearch -cp 
:/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/*
 
-Des.default.path.home=/usr/share/elasticsearch 
-Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/data 
-Des.default.path.work=/tmp/elasticsearch 
-Des.default.path.conf=/etc/elasticsearch 
org.elasticsearch.bootstrap.Elasticsearch


# jstat -gc 48931
48931 not found


jmap -heap PID is working though.

Any ideas how to fix this ? I'm using JPS for one of my monitoring probe.


I'm using CentOS 6.5 with ES 1.1.1 from RPM and java 1.7
# java -version
java version "1.7.0_55"
Java(TM) SE Runtime Environment (build 1.7.0_55-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)


$ curl http://localhost:9200/
{
  "status" : 200,
  "name" : "logstash_master",
  "version" : {
"number" : "1.1.1",
"build_hash" : "f1585f096d3f3985e73456debdc1a0745f512bbc",
"build_timestamp" : "2014-04-16T14:27:12Z",
"build_snapshot" : false,
"lucene_version" : "4.7"
  },
  "tagline" : "You Know, for Search"
}


Thanks a lot in advance,
Greg

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3792f6e6-a1fa-4658-a36e-959a2c65b52c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch-php and function_score

[chris]

i've found my problem...

an intermediate array was missing (under [functions] see below) in the 
query construction with elasticsearch-php :

[function_score] =>

Array
(

[query] => Array
(
[terms] => Array
(
[category] => Array
(
[0] => toto
[1] => tutu
)

)

)

[functions] => Array
(
[0] => Array
(
[filter] => Array
(
[terms] => Array
(
[tags] => Array
(
[0] => truc
)

)

)

[boost_factor] => 2
)

)

)


Le mardi 6 mai 2014 17:51:13 UTC+2, chris a écrit :
>
> hi !
>
> the query below is working but when i use the php official client but i 
> still have the exception : *malformed query, expected a START_OBJECT*
>
>
> "query": {
>   "function_score": {
>  "query": {
> "terms": {
>"categoryName": [
>   "toto",
>   "tutu"
>],
>"minimum_match": 1
> }
>  },
>  "functions": [
> {
>"filter": {
>   "terms": {
>  "tags": [
> "truc",
> "bidule"
>  ]
>   }
>},
>"boost_factor": 2
> }
>  ]
>   }
>}
>
> in my script the query is divided 
> both $category and $boost are arrays of values
>
> // function_score --> init main query  
> $functionScoreQuery = array( 'query' => array(
> 'terms' => array(
> 'categoryName' => $category
>  )
> ));  
> 
> // function_score functions --> init boost 
> $functionScoreFunctions = array( 'functions' => array(
> 'filter' => array( 
> 'terms' => array(
> 'tags' => $boost
> )
> ),
> 'boost_factor' => 2
> ));   
>  
> $defaultSubQuery['function_score'] =  $functionScoreQuery; 
> if (($boost)) {
> $defaultSubQuery['function_score'] += 
>  $functionScoreFunctions;  
> }   
>
> a print_r() of the request show :
>
> Array
> (
> [index] => myIndex
> [body] => Array
> (
> [query] => Array
> (
> [function_score] => Array
> (
> [query] => Array
> (
> [terms] => Array
> (
> [categoryName] => Array
> (
> [0] => toto
> [1] => tutu
> )
>
> )
>
> )
>
> [functions] => Array
> (
> [filter] => Array
> (
> [terms] => Array
> (
> [tags] => Array
> (
> [0] => truc
> [1] => bidule
> )
>
> )
>
> )
>
> [boost_factor] => 2
> )
>
> )
>
>   

running out of memory indexing on linux

[Vince Wadhwani]

I recently switched from a mac to using debian linux. Same data set, 
indexing method, and elastic version. On OS X I can index my records (about 
1 million) but on linux I can see memory usage increasing over time until I 
finally run out and the process is killed. I've got 8GB of RAM and 16GB of 
Swap but I still can't get through the records I could with half that RAM 
on OSX.

Is there a config option somewhere I can toggle? (I've already tried 
enabling bootstrap.mlockall: true) Or is there something else going on?

-Vince

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e0268477-5cc2-49a9-ad95-f90960884578%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Increase in Indexing Time and big Merges

[elias]

Hi Mihir,
i had the same problem, index time increasing from about 3 sec for a bulk of
100k, to over 500sec. After increasing the number of shards from 1 to 4 per
node, and setting indices.memory.index_buffer_size to 20%, the indextime is
quite constant around 3-5 sec. 



--
View this message in context: 
http://elasticsearch-users.115913.n3.nabble.com/Increase-in-Indexing-Time-and-big-Merges-tp4055918p4055924.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1400138792860-4055924.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

Timestamp on ES using Mongo River

[karnamonkster]

Hi, 

Here are my version details: 

ES 1.0.0, MongoDB 2.4.9, mongo-river 2.0.0(with the driver that is required 
for the river plugin) 
The cluster health is green, I have ES 2 nodes on the same machine to make
sure the shards are indexed properly. 
I have a river with mongo which fetches events from the remote mongodb. 

Now i have everything up an running but the only issue is with the "time 
format" which is still in UNIX epoch for:

1.event_time 
2.timestamp 
3.app_event_time 

here is the mapping. 

curl -XGET http://ESserverIP:9200/secana/_mapping?pretty=true
{ 
  "secana" : { 
"mappings" : { 
  "messages" : { 
"properties" : { 
  "app_event_time" : { 
"type" : "long" 
  }, 
  "event_time" : { 
"type" : "long" 
  }, 
  "frecency" : { 
"type" : "long" 
  }, 
  "hostname_id" : { 
"type" : "string" 
  }, 
  "message" : { 
"type" : "string" 
  }, 
  "netloc" : { 
"type" : "string" 
  }, 
  "signum_id" : { 
"type" : "string" 
  }, 
  "source" : { 
"type" : "string" 
  }, 
  "source2" : { 
"type" : "string" 
  }, 
  "title" : { 
"type" : "string" 
  }, 
  "url" : { 
"type" : "string" 
  }, 
  "visit_count" : { 
"type" : "long" 
  } 
} 
  } 
} 
  } 
} 


- 

Now i am still not able to get the correct date format as mentioned in my
mapping. 

By default it is picking up " long". 

I have tried to update the mapping properties using 

PUT /_river/secana/messages/_mapping 

{"secana":{"mappings":{"messages":{"properties":{"_timestamp":{"enabled":"true","store":"yes"},"app_event_time":{"type":"date","format":"-MM-dd
HH:mm:ss"},"event_time":{"type":"date","format":"-MM-dd
HH:mm:ss"},"firstName":{"type":"string"},"frecency":{"type":"long"},"hostname_id":{"type":"string"},"lastName":{"type":"string"},"message":{"type":"string"},"netloc":{"type":"string"},"signum_id":{"type":"string"},"source":{"type":"string"},"source2":{"type":"string"},"timestamp":{"type":"date","format":"-MM-dd
HH:mm:ss"},"title":{"type":"string"},"url":{"type":"string"},"visit_count":{"type":"long"}}
 

But this is not updated. 
Using the head plugin, I could see that the index metadata is still giving
me the incorrect details for date fields. 
1.event_time 
2.timestamp 
3.app_event_time 
So i could sense this might be the issue. 
I have tried several to DELETE the index and _river many times with the
correct mapping and index fields. 


{ 

state: open 
settings: { 
index: { 
uuid: 6UBnCIgFRt-77OexGcIdnQ 
number_of_replicas: 1 
number_of_shards: 5 
version: { 
created: 199 
} 
} 
} 
mappings: { 
messages: { 
properties: { 
visit_count: { 
type: long 
} 
message: { 
type: string 
} 
title: { 
type: string 
} 
signum_id: { 
type: string 
} 
source: { 
type: string 
} 
app_event_time: { 
type: long 
} 
hostname_id: { 
type: string 
} 
event_time: { 
type: long 
} 
netloc: { 
type: string 
} 
source2: { 
type: string 
} 
frecency: { 
type: long 
} 
url: { 
type: string 
} 
} 
} 
} 
aliases: [ ] 

}



--
View this message in context: 
http://elasticsearch-users.115913.n3.nabble.com/Timestamp-on-ES-using-Mongo-River-tp4055931.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1400152265559-4055931.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

Lucene syntax for phrase prefix?

[Vladimir Khazin]

Hey everybody,

I am looking for Lucene not ElasticSeach syntax to format phrase prefix 
query to find people.

Sample request:
post /person/_search
{
  "query" : {
"query_string": {
  "query": "Title:Jake AND Title:J*"
}
  },
  "_source":[
  "Title"
  ]
}

Sample Response:
{
   "took": 0,
   "timed_out": false,
   "_shards": {
  "total": 1,
  "successful": 1,
  "failed": 0
   },
   "hits": {
  "total": 2,
  "max_score": 5.546068,
  "hits": [
 {
"_index": "18c252a5-82d3-45f0-9fa3-4bf799194037",
"_type": "person",
"_id": "d04e1479-0a85-4c24-be80-5cff8f8d8a7c",
"_score": 5.546068,
"_source": {
   "Title": "Jake Johnson"
}
 },
 {
"_index": "18c252a5-82d3-45f0-9fa3-4bf799194037",
"_type": "person",
"_id": "51deb8bb-b1d7-471e-9cbb-ab9e853976a7",
"_score": 5.546068,
"_source": {
   "Title": "Jake Lacy"
}
 }
  ]
   }
}

Problem: 'Jake Lacy' from human perspective should not have been in the 
results.

I appreciate there is a ElasticSearch specific solution to the problem with 
following problem:
post person/_search
{
  "query" : {
"match_phrase_prefix": {
"Title": "Jake J"
  }
  },
  "_source":[
  "Title"
  ]
}


With desired results:
{
   "took": 0,
   "timed_out": false,
   "_shards": {
  "total": 1,
  "successful": 1,
  "failed": 0
   },
   "hits": {
  "total": 1,
  "max_score": 279.47867,
  "hits": [
 {
"_index": "18c252a5-82d3-45f0-9fa3-4bf799194037",
"_type": "person",
"_id": "d04e1479-0a85-4c24-be80-5cff8f8d8a7c",
"_score": 279.47867,
"_source": {
   "Title": "Jake Johnson"
}
 }
  ]
   }
}

I am however stubbornly insistent n finding out whether there is a Lucene 
syntax solution to that challenge. 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f5997311-ae6c-4404-bc46-05995d972afa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is it possible to get the name of the field above an element?

[maximilian . brodhun]

Additional:

At the beginning I had all entries in an array. But By searching for the 
strings I got also back the wohle array.
The document looks in this way:

{
"project" : 

[ 
  {   
 "title" : "A",
 "id" : "1234",
 "fulltext" : "Some Text"

  },
  
{
   "title": "Believe",
"id": "5678",
"fulltext": "inside the fied"

  },
  
  {
"title": "imagination",
"id": "9",
"fulltext": "makes trouble" 
  }
]
}


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0e2a1f79-712b-4012-ab95-0eac36f70399%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Is it possible to get the name of the field above an element?

[maximilian . brodhun]

Hello dear all,

the last days I struggle with queries and structures to index and query 
stuff like this.


{"project" : 

  {
  "set1" : {
  "title" : "A",
  "id" : "1234",
  "fulltext" : "Some Text"
  },
  
"set2" : {
"title": "Believe",
"id": "5678",
"fulltext": "inside the fied"
  },
  
  "set3" : {
"title": "imagination",
"id": "9",
"fulltext": "makes trouble" 
  }
  }
 
 }






So, when I'm searching for "inside the field" I just want to get back all 
field from "set1".

The problem is, that the exact name of the field "set*" ist not known. I 
know that the stuff I want to do is a little bit cruel. But it would be 
nice to get it working.

I tried queries like this:

{
  "query" : {
"query_string" : {
  "query" : "praedicatum"
}
  },
  "_source" : {
"includes" : [ "project" ],
"excludes" : [ null ]
  }
}


But in this way, of course I get back the all entries of the field 
"project".



thanks to you!


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/594a7efd-a12b-40ec-a13f-63d869350d5b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Is it possible to update/set fielddata cache expire time per index

[keety]

Is fielddata cache expire 
time
 a 
per-index-level setting ?

Looking at the documentation for index settings it doesn't seem to be 
available although the filter cache seems to have an index level expiry 
setting
.

If not available is there a drawback of allowing this setting per index 
level ?


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6d6e797f-7cf6-477e-82fe-bddabce3badd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How to know if my curator instance is running fine ?

[Guillaume boufflers]

Hello buds !

I've installed Elasticsearch, Logstash and Kibana lately ..
I've been playing with those tools for a couple of days now and it's been 
so cool, great tool ! 
Now that i've got data in my elasticsearch base, i'd like to remove the 
indexes that are older than an hour.
I made some searches on the web and i found the curator tool. seems great.
I did follow the instructions on the github webpage.
This is the command i ran :

curator --host 192.168.1.12 --port 9200 -T hours -d 1


And Here is the output :

2014-05-15T15:34:44.412 INFOmain:333  Job starting
...

2014-05-15T15:34:44.413 INFO   _new_conn:180  Starting newHTTP 
connection 
(1): 192.168.1.12

2014-05-15T15:34:44.415 INFO log_request_success:49   GET 
http://192.168.1.12:9200/ 
[status:200 request:0.002s]

2014-05-15T15:34:44.416 INFOmain:359  Deletingindices 
older than 
1 hours...

2014-05-15T15:34:44.418 INFO log_request_success:49   GET 
http://192.168.1.12:9200/logstash-*/_settings?expand_wildcards=closed 
[status:200 request:0.001s]

2014-05-15T15:34:44.418 INFO  index_loop:309  DELETE index 
operations completed.

2014-05-15T15:34:44.419 INFOmain:379  Done in 0:00:
00.010193.

Seems pretty good in fact but the problem (or not) is that i don't see any 
difference in my Kibana monitor, i'm still seeing graph from this morning,

i was just wondering if this was a normal behavior or if i did something 
wrong with the command i ran.

Is there a way to concretely see if my curator instance is doing the job 
just fine ?

Thank you for reading and helping.
Regards, Guillaume.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d01beb3e-c35d-439b-bb18-44b3a5c57557%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: problem stop indexing data when I restart Elasticsearch ?

[Tanguy Bernard]

Thank you very much Jörg. It works perfectly.

Tanguy

Le jeudi 15 mai 2014 15:17:04 UTC+2, Jörg Prante a écrit :
>
> After you have indexed your data and all the work is done, you should 
> remove the river.
>
> curl -XDELETE '0:9200/_river/user/'
>
> Otherwise, the river will be automatically started again when the node 
> starts again.
>
> Jörg
>
>
> On Thu, May 15, 2014 at 2:59 PM, Tanguy Bernard 
> 
> > wrote:
>
>> Hello,
>> I indexing my data : everything fine, but when I restart Elasticsearch, 
>> he reindex me my data, My problem is that I have twice the same data.
>> Can you help me to solve this problem ?
>>
>> Thanks in advance.
>>
>> My code :
>>
>> PUT /my_index/_mapping/user
>> {
>>   "mappings" : {
>> "user" : {
>>  "properties" : {
>> 
>> "name_user":{
>> "type":"string"
>> }
>>
>>  }
>> }
>>   },
>>
>>   "settings" : {
>> "analysis" : {
>>   "analyzer" : {
>>  "str_search_analyzer" : {
>>
>>   "tokenizer" : "standard",
>>   "filter" : ["lowercase", "asciifolding"]
>> },
>>
>> "str_index_analyzer" : {
>>   "tokenizer" : "standard",
>>   "filter" : ["lowercase","asciifolding"]
>> }
>>   },
>>
>> 
>> }
>>   }
>> }
>>
>>
>> PUT /_river/user/_meta
>> {
>> "type" : "jdbc",
>> "jdbc" : {
>>
>> "url" : "my_url",
>> "user" : "user",
>> "password" : "password",
>> "sql" : "select name_user from user",
>> "index" : "my_index",
>> "type" : "user",
>> "max_bulk_requests" : 5  
>>
>>
>> }
>> }
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/c74975a9-71f1-4c40-a3de-b7907be2c401%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/96290862-0d71-4d68-b4bc-f2fe73bb8ff6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: problem stop indexing data when I restart Elasticsearch ?

[Tanguy Bernard]

Thank you very much Jörg. It's work perfectly.

Tanguy 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6991c45a-e716-4106-9e00-01ebe849941a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: problem stop indexing data when I restart Elasticsearch ?

[joergpra...@gmail.com]

After you have indexed your data and all the work is done, you should
remove the river.

curl -XDELETE '0:9200/_river/user/'

Otherwise, the river will be automatically started again when the node
starts again.

Jörg


On Thu, May 15, 2014 at 2:59 PM, Tanguy Bernard  wrote:

> Hello,
> I indexing my data : everything fine, but when I restart Elasticsearch, he
> reindex me my data, My problem is that I have twice the same data.
> Can you help me to solve this problem ?
>
> Thanks in advance.
>
> My code :
>
> PUT /my_index/_mapping/user
> {
>   "mappings" : {
> "user" : {
>  "properties" : {
>
> "name_user":{
> "type":"string"
> }
>
>  }
> }
>   },
>
>   "settings" : {
> "analysis" : {
>   "analyzer" : {
>  "str_search_analyzer" : {
>
>   "tokenizer" : "standard",
>   "filter" : ["lowercase", "asciifolding"]
> },
>
> "str_index_analyzer" : {
>   "tokenizer" : "standard",
>   "filter" : ["lowercase","asciifolding"]
> }
>   },
>
>
> }
>   }
> }
>
>
> PUT /_river/user/_meta
> {
> "type" : "jdbc",
> "jdbc" : {
>
> "url" : "my_url",
> "user" : "user",
> "password" : "password",
> "sql" : "select name_user from user",
> "index" : "my_index",
> "type" : "user",
> "max_bulk_requests" : 5
>
>
> }
> }
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/c74975a9-71f1-4c40-a3de-b7907be2c401%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHTejtO%3DOn-Bp_QEA-Jw4Lf0MNbUp80eQs%3DKniq9FWQqA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

problem stop indexing data when I restart Elasticsearch ?

[Tanguy Bernard]

Hello,
I indexing my data : everything fine, but when I restart Elasticsearch, he 
reindex me my data, My problem is that I have twice the same data.
Can you help me to solve this problem ?

Thanks in advance.

My code :

PUT /my_index/_mapping/user
{
  "mappings" : {
"user" : {
 "properties" : {

"name_user":{
"type":"string"
}
   
 }
}
  },

  "settings" : {
"analysis" : {
  "analyzer" : {
 "str_search_analyzer" : {

  "tokenizer" : "standard",
  "filter" : ["lowercase", "asciifolding"]
},

"str_index_analyzer" : {
  "tokenizer" : "standard",
  "filter" : ["lowercase","asciifolding"]
}
  },


}
  }
}


PUT /_river/user/_meta
{
"type" : "jdbc",
"jdbc" : {

"url" : "my_url",
"user" : "user",
"password" : "password",
"sql" : "select name_user from user",
"index" : "my_index",
"type" : "user",
"max_bulk_requests" : 5  


}
}

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c74975a9-71f1-4c40-a3de-b7907be2c401%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: WebServer on Same Machine As ES Instances

[joergpra...@gmail.com]

Yes, I use nginx in front of Wildfly app servers that has TransportClients,
because of middleware, not because of network latency.

Internal network (LAN) latency can be neglected when you expose service to
the outside from your DC, the outside network (WAN) latency is much bigger.

Jörg


On Thu, May 15, 2014 at 6:01 AM, webish  wrote:

> Has anyone experimented with putting web servers (nginx) on the same
> machine as each ES node?
>
> For APIs that require many queries and network round trips this could
> reduce network latency
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/63ea4b36-394f-4976-b5ac-fbf5a8ad3b3d%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFZxWe3oE4s%3DtL4bobEkJ24Q7PPHMnNFr_Cr-z3bbtUqA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: error message in logfile : Parse Failure [No mapping found for [@timestamp] in order to sort on]

[Firass Gzayil]

Goto your dashboard->settings->index and replace [_all] with [logstash*]

Firass

On Monday, May 5, 2014 1:18:18 PM UTC+3, Alexander Reelsen wrote:
>
> Hey,
>
> the problem is, that you are trying to search over all indices. One of 
> those indices contains your kibana dashboards - which is just fine. However 
> that index (named kibana-int) does not contain a timestamp field.
>
> The most simple solution might be, to change your search to not search in 
> that index or only include the indices you want to search in.
>
>
> --Alex
>
>
> On Tue, Apr 29, 2014 at 8:47 AM, HansPeterSloot 
> 
> > wrote:
>
>> Well I am using elasticsearch as a logstash repository.
>>
>> Can you give me a curl statement to check whether there are indexes 
>> without @timestamp?
>>
>> Op maandag 28 april 2014 15:45:05 UTC+2 schreef Binh Ly:
>>>
>>> Is it possible that one of in the indexes you're querying against does 
>>> not have the field @timestamp?
>>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/d5f858f8-5ab7-4b86-9c3d-bcb41647b548%40googlegroups.com
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3e65ee59-3d96-42cb-8507-9feba27db3f9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Stats combination

[oz . shelach]

I'd like to calculate the relation between the stats of some fields. For 
example, the average revenue per rental day would be the total of NET 
divided by the total of DURATION. Is this an answer ES can give me 
directly, or must send two queries and do my own manipulation of the 
responses ?

I've read and re-read the documentation for various facets, which seems to 
be the place to look. Perhaps I'm missing the correct place to look, or the 
appropriate example...

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b0a8438d-8a4f-4fca-ab04-309c83af046b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch supports multiple not RAID 0 Data Paths?

[Mark Walkom]

Assuming these are all in the same server; You can't do this unless you run
multiple instances and then tell each instance which directory (mount) to
store the data.

You'd then need to use something like this
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-allocation.html

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 15 May 2014 20:57, horst knete  wrote:

> Hello,
>
> we are currently running our Elasticsearch on 1 single node and get about
> 20 Million Logs per Day(40 GB/daily indices). Since this is a much Stuff to
> handle with, the indices takes a lot of disk space on our server.
>
> What we like to implement:
>
> - 1 Data directory which is stored on our SSDs and contains the indices of
> the last 7 days for quick access.
> - 1 Data directory which is stored on normal HDDs and contains indices of
> last 3 months for normals speed acces.
> - 1 Data directory which is stored on slow 5400 rpm HDDs and contains the
> indices of the last 2 years for access if needed.
>
> Well it´s not problem to tell ES multiple data paths but if you do this,
> ES will stripe (RAID 0) the indices on all 3 data directories.
>
> But thats not what we want. We want do copy the indices with a script to
> the matching directories ( a index which is older than 8 days gets
> automatically moved to normal HDDs and so on).
>
> Is there any way to make this work?
>
> Thanks for your feedback.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/e8e34044-5895-4ccd-bac4-5ef11ea81204%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624Yr3MpLCi2wiRmogkdozPFG5_7WZnD3un-dEJhU_mToiA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch supports multiple not RAID 0 Data Paths?

[horst knete]

Hello,

we are currently running our Elasticsearch on 1 single node and get about 
20 Million Logs per Day(40 GB/daily indices). Since this is a much Stuff to 
handle with, the indices takes a lot of disk space on our server.

What we like to implement:

- 1 Data directory which is stored on our SSDs and contains the indices of 
the last 7 days for quick access.
- 1 Data directory which is stored on normal HDDs and contains indices of 
last 3 months for normals speed acces.
- 1 Data directory which is stored on slow 5400 rpm HDDs and contains the 
indices of the last 2 years for access if needed.

Well it´s not problem to tell ES multiple data paths but if you do this, ES 
will stripe (RAID 0) the indices on all 3 data directories.

But thats not what we want. We want do copy the indices with a script to 
the matching directories ( a index which is older than 8 days gets 
automatically moved to normal HDDs and so on).

Is there any way to make this work?

Thanks for your feedback.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e8e34044-5895-4ccd-bac4-5ef11ea81204%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: time problem

[Quentin Guiot]

ok so i put "_all" in default index and the errors are gone 

But now i need to input the three files that are in my "logstash.conf" file 
and i really don't know how 
i tried a lot of things, checked on google so many times but there are few 
tutorials on the net and they didn't help me at all 

Maybe someone here could 

thanks 

Le mercredi 14 mai 2014 16:49:53 UTC+2, Quentin Guiot a écrit :
>
> hi 
> thanks for the reply
> but when i do what you say, i got another error 
>
> *Oops!* IndexMissingException[[INDEX_NONE] missing]
>
> Le mercredi 14 mai 2014 15:56:30 UTC+2, Binh Ly a écrit :
>>
>> If you don't have logstash style time based indexes, you'll need to open 
>> your dashboard config (gear icon top right), go to to Index tab, change 
>> Timestamping: none, and Default Index: . Click Save, go 
>> back to your dashboard and remove any time filters up on the top (if any).
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5eee26d1-debf-4b9b-9bc6-36b698e4f280%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch scenarios HA

[davor . sharic]

Tnx, got it :)

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/45941719-c1c9-41d4-b33d-264507c6834f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Index size

[Lukáš Homza]

Hey,

I am using logstash and elasticsearch for logs. I have noticed a huge 
difference in index size between test and production environment. In 
testing environment - index size with 20k documents is 15MB, however in 
production with 60k documents - index size is 2GB.

Everything logged in production is the same as in test. The only difference 
is speed at which logs are logged.

Why is there such huge difference in size? Could it be that the index cant 
keep up with optimizing itself (after I hit optimize, size dropped by 200MB 
and in test environment index size was going between 15-300MB)? What can be 
done do decrease index size?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/71056765-4fc4-4caf-9afc-653a73d38416%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Multi DC cluster or separate cluster per DC?

[Sebastian Łaskawiec]

We are still thinking about production configuration and here is a short 
list of single/separate cluster's advantages and disadvantages...

Single cluster:

   - (+) If you have single cluster - you perform single query to the 
   database. In case of having cluster per DC - each cluster needs to query DB 
   separately
   - (+) Data consistency - in the matter of fact this is achieved by 
   single query to the DB
   - (+) You can introduce new DC easily
   - (+) True active-active configuration
   - (-) Split brain and pretty complicated configuration (to avoid split 
   brain in case when DC link is down)
   - (-) node.master setting can not be changed in runtime (take a look at 
   my first post and split brain solution)
   - (-) In case of a disaster we need to operate on single DC. If you use 
   single cluster per 2 DCs you can't really tell if a single DC is strong 
   enough to handle query and indexing load
   - (-) In pessimistic scenario data travels through WAN 2 times (first 
   time - database replication, second time - ES replication)
   - (-) You can't really tell which node will respond to the query. Let's 
   assume that you have full index in each DC (force awareness option). ES 
   might decide to gather results from the remote DC and not from the local 
   one. This way you need to add WAN latency into your query time.
   - (-) You need to turn off whole cluster or perform cycle restarts 
   during upgrade

Separate cluster per DC:

   - (+) No Split brain
   - (+) You can tell precisely when you are out of resources to handle 
   load in ES cluster in each DC
   - (+) You can experiment with different settings on production. If 
   something goes wrong - just switch clients to standby DC.
   - (+) Full failover - in case of any problems - just switch to the other 
   DC
   - (+) Upgrades are easy and you have no down time (upgrade first DC, 
   stabilize it, test it, and then to the same to the other DC)
   - (+) Since these are 2 separate clusters you can avoid data traveling 
   through WAN during queries. Each DC queries nodes locally.
   - (-) It is not a full active-active configuration. It's more like an 
   active-standby configuration
   - (-) Data inconsistency might occur (different results when queried 
   local and remote DC)
   - (-) Each DC will query DB separately. This will generate additional 
   load to the DB

Right now we think we should go for 2 separate clusters. DB load is a thing 
which worries me the most (we have really complicated query with a lot of 
left joins). However we think that in our case having to separate DC have 
more advantages then disadvantages.

If you have some more arguments or comments - please let us know :)

Regards
Sebastian

W dniu poniedziałek, 12 maja 2014 20:02:35 UTC+2 użytkownik Deepak Jha 
napisał:
>
> Having a separate cluster is definitely a better way to go. OR, you can 
> control the shard, replica placement so that they are always placed in the 
> same DC. In this way, you can avoid interDC issues still having a single 
> cluster. I have the similar issue and I am looking at it as one of the 
> alternative. 
>
> On Saturday, May 10, 2014 1:05:08 AM UTC-7, Sebastian Łaskawiec wrote:
>>
>> Thanks for the answer! We've been talking with several other teams in our 
>> company and it looks like this is the most recommended and stable setup.
>>
>> Regards
>> Sebastian
>>
>> W dniu środa, 7 maja 2014 03:23:43 UTC+2 użytkownik Mark Walkom napisał:
>>>
>>> Go the latter method and have two clusters, ES can be very sensitive to 
>>> network latency and you'll likely end up with more problems than it is 
>>> worth. 
>>> Given you already have the data source of truth being replicated, it's 
>>> the sanest option to just read that locally.
>>>
>>> Regards,
>>> Mark Walkom
>>>
>>> Infrastructure Engineer
>>> Campaign Monitor
>>> email: ma...@campaignmonitor.com
>>> web: www.campaignmonitor.com
>>>
>>>
>>> On 6 May 2014 23:51, Sebastian Łaskawiec  wrote:
>>>
 Hi!

 I'd like to ask for advice about deployment in multi DC scenario.

 Currently we operate on 2 Data Centers in active/standby mode.  like to 
 opeIn case of ES we'd like to have different approach - we'drate in 
 active-active mode (we want to optimize our resources especially for 
 querying). 
 Here are some details about target configuration:

- 4 ES instances per DC. Full cluster will have 8 instances.
- Up to 1 TB of data 
- Data pulled from database using JDBC River
- Database is replicated asynchronously between DCs. Each DC will 
have its own database instance to pull data. 
- Average latency between DCs is about several miliseconds
- We need to operate when passive DC is down

 We know that multi DC configuration might end with Split Brain issue. 
 Here is how we want to prevent it:

- Set node.master: true only in 4 nodes in active DC
- Set n