Re: Write a plugin to query and aggregate results from multiple shards

[Sandeep Ramesh Khanzode]

Appreciate the response as always.

Please bear with my technical understanding of ES :)

In the TransportSearchAction, the doExecute() delegates to one of the six
different search types. It is inside the execute methods of those
individual six actions, that they will look at the shards. Correct me if I
am wrong. Even if I modify the SearchRequest in this case, I will not be
able to identify the shardId that is operating on this request object
before doing the modification.

Which class/module should I use to write my custom action? If I write a
seventh SearchAction type extending TransportSearchTypeAction, then in the
executeQuery()/executeFetch() first/second phases, I will probably get a
hold of the shardsearchrequest object. Are you saying that I should modify
it there? Please let me know. My concern is that if I write a new
searchaction custom plugin like this, I will not be able to replicate ES
functionality of the existing six search types and I will also be referring
to some Internal classes of ES. Will that be okay?

Maybe I am missing something. But I need to modify the request object with
the insertion of the shardID identifier when it is known. The reason I need
to insert the shardID is so that the Filter Parser Plugin can then take a
look at the SearchRequest (or ShardSearchRequest at that time) and do
something specific for that shard.


Thanks,
Sandeep

On Mon, Sep 15, 2014 at 1:08 PM, joergpra...@gmail.com <
joergpra...@gmail.com> wrote:

> See the TransportSearchAction, in the doExecute() method, the
> SearchRequest is dispatched to several transport actions of the search
> types.
>
> Assuming you write your own custom action: the shard level request
> is ShardSearchRequest. It is easier to add information to SearchRequest,
> pass it down, and extract relevant parts from the SearchRequest later. See
> ShardSearchRequest constructor for repacking the parameters and delegate it
> to a shard.
>
> Maybe it is possible to add info to extrasource.
>
> Jörg
>
> On Mon, Sep 15, 2014 at 8:57 AM, 'Sandeep Ramesh Khanzode' via
> elasticsearch  wrote:
>
>> Hi,
>>
>>
>> I am looking through the sources, and I am not sure whether this is
>> possible. What I am looking to is the possibility to manipulate the
>> SearchRequest object when it reaches the SearchShards level.
>> Since I need to update the object with some value that is shard specific.
>>
>> For this, I was checking the TransportBroadcastOperationAction which
>> actually allows to hit multiple shards and we can inject a SearchService.
>> However, in the response aggregation, we may have to write our own logic to
>> call SearchPhaseController::merge() or something. Not sure if this will be
>> a problem when the same code in ElasticSearch changes over releases.
>>
>> There are also other classes like SearchServiceTransportAction and we can
>> also probably extend TransportSearchTypeAction like the other QAF, DFS_QAF,
>> QTF, DFS_QTF, etc. However, what I want to know is whether this is standard
>> practice and should be done this way? Or is there any other plugin that
>> allows me to do this?
>>
>>
>> Thanks,
>> Sandeep
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/e1f52da2-bb05-4005-bf88-8031f5440225%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/BS64wfqrHNM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGMr6K81CSWid1%2BoPP_fZ6i1_4J%2B0uNXHuqjCkVB5MXWw%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKnM90YKM68sc-_mB_yGhP-93N%2BE0fipfzWyX8jDZwX-phPZsA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Highly variable query performance with ES 1.3.2 (filter + aggregations)

[Craig Wittenberg]

Hi Peter,

No, old or long gcs are not showing up here.  I use bigdesk among other 
tools and also inspect the logs.  Bigdesk shows a very regular sawtooth 
pattern and nearly always young gcs of less than 1s.  The logs show the 
same, of course.  It is very rare that we ever get a gc of the old 
generation, much less a long one.  

As a side note, that's one of the reasons why I switched to g1.  When we 
were on ES 1.1.2 using g1 we would get an old gc over a minute when we had 
lots of incremental snapshots and occasionally after a lot of merges.  With 
ES 1.3.2 I don't have enough experience yet on the cases, but they are rare 
to date.  

Craig.

On Monday, September 15, 2014 8:37:47 PM UTC-7, Peter van der Weerd wrote:

> Is it possible that the JVM is doing large GC's ?
>
> Peter
>
>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1d01653f-c7e1-4b1c-8e53-048d90ab0726%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Highly variable query performance with ES 1.3.2 (filter + aggregations)

[Peter van der Weerd]

Is it possible that the JVM is doing large GC's ?

Peter


Op dinsdag 16 september 2014 02:24:42 UTC+2 schreef Craig Wittenberg:
>
> Hardware problems in Azure?  Hmm.  I suppose so :)
>
> In this case I restarted just ES (i.e., shutdown ES using 
> http://localhost:9200/_cluster/nodes/_local/_shutdown and then ran 
> elasticsearch.bat again) and thus didn't reboot the VM or anything so 
> dramatic.  All of the ES node files remained in place and the node 
> reconnected with the cluster with no issue.  My rdp session remained.  I 
> suspect if this were a hardware issue a) we would have seen this long ago 
> and b) it would have persisted after I restarted ES.
>
> Related, we actually have two clusters of the same size and same data that 
> have exhibited the same behavior.  The cluster in which I restarted three 
> ES nodes remains responsive.  The other cluster in which I have not (yet) 
> restarted nodes remains with the highly variable response time (200-4ms 
> with 34415ms being a recent one).
>
> Thanks for replying.
>
> Craig.
>
> On Monday, September 15, 2014 2:58:22 PM UTC-7, Jörg Prante wrote:
>
>> You said you restarted a node, and since then, the effect has gone, is 
>> that true? Such high spikes like 40sec response might indicate problems in 
>> the hardware (disk failure, network latency or timeout, DNS), is this 
>> possible?
>>
>> Jörg
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/87e682c1-717e-4d29-b9bf-82078616744f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Highly variable query performance with ES 1.3.2 (filter + aggregations)

[Craig Wittenberg]

Hardware problems in Azure?  Hmm.  I suppose so :)

In this case I restarted just ES (i.e., shutdown ES using 
http://localhost:9200/_cluster/nodes/_local/_shutdown and then ran 
elasticsearch.bat again) and thus didn't reboot the VM or anything so 
dramatic.  All of the ES node files remained in place and the node 
reconnected with the cluster with no issue.  My rdp session remained.  I 
suspect if this were a hardware issue a) we would have seen this long ago 
and b) it would have persisted after I restarted ES.

Related, we actually have two clusters of the same size and same data that 
have exhibited the same behavior.  The cluster in which I restarted three 
ES nodes remains responsive.  The other cluster in which I have not (yet) 
restarted nodes remains with the highly variable response time (200-4ms 
with 34415ms being a recent one).

Thanks for replying.

Craig.

On Monday, September 15, 2014 2:58:22 PM UTC-7, Jörg Prante wrote:

> You said you restarted a node, and since then, the effect has gone, is 
> that true? Such high spikes like 40sec response might indicate problems in 
> the hardware (disk failure, network latency or timeout, DNS), is this 
> possible?
>
> Jörg
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9d74b9a8-cdcc-474f-9a34-9e2fa8240c74%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Is it possible to annotate each result in a result set?

[James Addison]

 

I have two types stored in an index: locations and activities. An activity 
has a 'relation' to a location - ie. an activity takes place at a location.

Is it possible to get a location search result set that includes the count 
of activities at each location? Sort of like annotating each item in the 
location results?

(This is my second attempt to post - apologies if this is a duplicate post!)

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c39df045-7570-4fb4-83d6-702b4891a503%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Version 1.3.2] Root type mapping not empty after parsing! Remaining fields:

[Jack Park]

I got this on 1.2.2 and found on the web that it was a bug. So, I upgraded
to 1.3.2 and got the same bug.

There was a note here that suggested adding "mappings": to the top of the
mappings doc. Didn't help on 1.2.2
http://stackoverflow.com/questions/24254081/elasticsearch-1-2-1-exception-root-type-mapping-not-empty-after-parsing

And didn't change anything on 1.3.2

Here is the trace without "mappings"

 Root type mapping not empty after parsing! Remaining fields:   [xvertices
: {core={properties={label={index=analyzed, store=true, type=string},
InVertexId={index=not_analyzed, store=true, type=string},
lox={index=not_analyzed, store=true, type=string},
OutVertexId={index=not_analyzed, store=true, type=string},
lists={properties={outEdgeIdList={index=not_analyzed, store=true,
type=string}, inEdgeIdList={index=not_analyzed, store=true,
type=string}}]

Here was where I had the same problem in my nodejs client:
http://elasticsearch-users.115913.n3.nabble.com/A-fresh-variant-of-MapperParsingException-td4061732.html

and it was fixed by precisely the document I am using on the java client.

My code which reads a given file and index name from a config file and
performs the mappings is here:
https://gist.github.com/KnowledgeGarden/b67b9f38a2ea9a29c21a

It's somewhat hairy code. If the index exists, it takes the time to dump
the mappings to see what's in there. Otherwise, it brings in the json file,
turns it into a JSONObject, and passes that in as mappings.

the mappings file which fails is below

Many thanks for ideas
Jack


{
"core": {
"properties": {
"lox": {
"index": "not_analyzed",
"type": "string",
"store": true
},
"InVertexId": {
"index": "not_analyzed",
"type": "string",
"store": true
},
"OutVertexId": {
"index": "not_analyzed",
"type": "string",
"store": true
},
"label": {
"index": "analyzed",
"type": "string",
"store": true
},
"lists": {
"properties": {
"inEdgeIdList": {
"index": "not_analyzed",
"type": "string",
"store": true
},
"outEdgeIdList": {
"index": "not_analyzed",
"type": "string",
"store": true
}
}
}
}
}

}

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAH6s0fyS2x2PYKhaqFcX%3D3dWvC-eGcBR6v-yYr5apKrNOv%2Bq7w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is there any way to prevent ES from disclosing exception details in REST response?

[Alex Roytman]

I guess I could but it would mean passing a response wrapper to capture
output stream and then copy it to real request or discard it in case of an
error. That would be a second copy of response - the first one being done
in the NodeServlet - will hurt performance for large responses :-(

On Mon, Sep 15, 2014 at 6:40 PM, joergpra...@gmail.com <
joergpra...@gmail.com> wrote:

> Then why don't you simply add a servlet filter that filters unwanted
> responses away?
>
> Jörg
>
> On Tue, Sep 16, 2014 at 12:21 AM, Alex Roytman  wrote:
>
>> Thanks Jorg,
>>
>> Unfortunately it is not an option - we are not at liberty to touch
>> anything beyond our app servers. We are using transport-wares servlet for
>> ES and I could easily tweak AbstractServletRestChannel to handle Rest
>> Channel response with codes 400,500 but I would like to avoid modifying the
>> code directly and there is no way to do it nicely. I put a request on
>> github for enhancements of the NodeServlet but was hoping ES may have an
>> option to turn error details on/off. I think it would be nice to control
>> error level  in REST responses with three levels -
>> suppress/message/stack-trace
>>
>> On Mon, Sep 15, 2014 at 6:01 PM, joergpra...@gmail.com <
>> joergpra...@gmail.com> wrote:
>>
>>> You can put a revers proxy like nginx between ES cluster and the rest of
>>> the world and filter away all HTTP status 500 responses.
>>>
>>> Jörg
>>>
>>> On Mon, Sep 15, 2014 at 11:57 PM, AlexR  wrote:
>>>
 We expose ES _search endpoint directly to consumers. When our REST API
 get scanned for security vulnerabilities it complains on ES returning
 exception details. For example a malformed query will be included in the
 response along with exception. While it is more or a less harmless the tool
 complains of various injections and internals disclosures. I would like to
 be able to turn error message in the response off (or substitute it with a
 generic message) in production while keeping normal response logic in
 development.

 Is there any way I can do it?

 --
 You received this message because you are subscribed to the Google
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/83e6b126-4db1-44cc-9a0a-0dd6c6d44a64%40googlegroups.com
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "elasticsearch" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/elasticsearch/Qx1TPSMS9ro/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHdM-ih%2BW_XCgHiphemWSSGtmK2Z5C%3D%2BiWD-ciBokbWHg%40mail.gmail.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAAY7rMRi4zsnv7QapKEr3iwyWxY0EDOnDczDeYa_0nzsFYT9sQ%40mail.gmail.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/Qx1TPSMS9ro/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH8fWhWpAKU4MT6oVxgRNG%2BBS1Y3QdOyn2eWiqwCErsJQ%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe fro

Re: Is there any way to prevent ES from disclosing exception details in REST response?

[joergpra...@gmail.com]

Then why don't you simply add a servlet filter that filters unwanted
responses away?

Jörg

On Tue, Sep 16, 2014 at 12:21 AM, Alex Roytman  wrote:

> Thanks Jorg,
>
> Unfortunately it is not an option - we are not at liberty to touch
> anything beyond our app servers. We are using transport-wares servlet for
> ES and I could easily tweak AbstractServletRestChannel to handle Rest
> Channel response with codes 400,500 but I would like to avoid modifying the
> code directly and there is no way to do it nicely. I put a request on
> github for enhancements of the NodeServlet but was hoping ES may have an
> option to turn error details on/off. I think it would be nice to control
> error level  in REST responses with three levels -
> suppress/message/stack-trace
>
> On Mon, Sep 15, 2014 at 6:01 PM, joergpra...@gmail.com <
> joergpra...@gmail.com> wrote:
>
>> You can put a revers proxy like nginx between ES cluster and the rest of
>> the world and filter away all HTTP status 500 responses.
>>
>> Jörg
>>
>> On Mon, Sep 15, 2014 at 11:57 PM, AlexR  wrote:
>>
>>> We expose ES _search endpoint directly to consumers. When our REST API
>>> get scanned for security vulnerabilities it complains on ES returning
>>> exception details. For example a malformed query will be included in the
>>> response along with exception. While it is more or a less harmless the tool
>>> complains of various injections and internals disclosures. I would like to
>>> be able to turn error message in the response off (or substitute it with a
>>> generic message) in production while keeping normal response logic in
>>> development.
>>>
>>> Is there any way I can do it?
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/83e6b126-4db1-44cc-9a0a-0dd6c6d44a64%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/Qx1TPSMS9ro/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHdM-ih%2BW_XCgHiphemWSSGtmK2Z5C%3D%2BiWD-ciBokbWHg%40mail.gmail.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAAY7rMRi4zsnv7QapKEr3iwyWxY0EDOnDczDeYa_0nzsFYT9sQ%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH8fWhWpAKU4MT6oVxgRNG%2BBS1Y3QdOyn2eWiqwCErsJQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is there any way to prevent ES from disclosing exception details in REST response?

[Alex Roytman]

Thanks Jorg,

Unfortunately it is not an option - we are not at liberty to touch anything
beyond our app servers. We are using transport-wares servlet for ES and I
could easily tweak AbstractServletRestChannel to handle Rest Channel
response with codes 400,500 but I would like to avoid modifying the code
directly and there is no way to do it nicely. I put a request on github for
enhancements of the NodeServlet but was hoping ES may have an option to
turn error details on/off. I think it would be nice to control error level
 in REST responses with three levels - suppress/message/stack-trace

On Mon, Sep 15, 2014 at 6:01 PM, joergpra...@gmail.com <
joergpra...@gmail.com> wrote:

> You can put a revers proxy like nginx between ES cluster and the rest of
> the world and filter away all HTTP status 500 responses.
>
> Jörg
>
> On Mon, Sep 15, 2014 at 11:57 PM, AlexR  wrote:
>
>> We expose ES _search endpoint directly to consumers. When our REST API
>> get scanned for security vulnerabilities it complains on ES returning
>> exception details. For example a malformed query will be included in the
>> response along with exception. While it is more or a less harmless the tool
>> complains of various injections and internals disclosures. I would like to
>> be able to turn error message in the response off (or substitute it with a
>> generic message) in production while keeping normal response logic in
>> development.
>>
>> Is there any way I can do it?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/83e6b126-4db1-44cc-9a0a-0dd6c6d44a64%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/Qx1TPSMS9ro/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHdM-ih%2BW_XCgHiphemWSSGtmK2Z5C%3D%2BiWD-ciBokbWHg%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAAY7rMRi4zsnv7QapKEr3iwyWxY0EDOnDczDeYa_0nzsFYT9sQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is there any way to prevent ES from disclosing exception details in REST response?

[joergpra...@gmail.com]

You can put a revers proxy like nginx between ES cluster and the rest of
the world and filter away all HTTP status 500 responses.

Jörg

On Mon, Sep 15, 2014 at 11:57 PM, AlexR  wrote:

> We expose ES _search endpoint directly to consumers. When our REST API get
> scanned for security vulnerabilities it complains on ES returning exception
> details. For example a malformed query will be included in the response
> along with exception. While it is more or a less harmless the tool
> complains of various injections and internals disclosures. I would like to
> be able to turn error message in the response off (or substitute it with a
> generic message) in production while keeping normal response logic in
> development.
>
> Is there any way I can do it?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/83e6b126-4db1-44cc-9a0a-0dd6c6d44a64%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHdM-ih%2BW_XCgHiphemWSSGtmK2Z5C%3D%2BiWD-ciBokbWHg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Highly variable query performance with ES 1.3.2 (filter + aggregations)

[joergpra...@gmail.com]

You said you restarted a node, and since then, the effect has gone, is that
true? Such high spikes like 40sec response might indicate problems in the
hardware (disk failure, network latency or timeout, DNS), is this possible?

Jörg

On Mon, Sep 15, 2014 at 11:51 PM, Craig Wittenberg 
wrote:

> Hello,
>
> Our team recently upgraded from ES 1.1.2 to 1.3.2 and are happy with the
> improvements ... except for one perplexing situation.
>
> We are running on Azure worker roles with Oracle Java 1.8u11 and using the
> G1 gc.  It is possible this is due to G1, but please consider all of the
> data below before you pull out a pat response on g1.
>
> Our cluster has 18 nodes, 3 of which are dedicated masters.  We have
> three indexes, 5 shards and one replica each.  The primary index is about
> 30gb total (5.9gb per shard and the shards are the same size).  We have
> five types in the main index and are about 10 fields each, a mix of
> strings, dates, bools, longs.  None of the strings are analyzed.
>
> All of the 18 nodes are client nodes and Azure is set up to round robin
> requests.  We have considered creating dedicated client nodes, but
> haven't done so yet.
>
> The query I have been using is a combination of a non-trivial filter, a
> terms aggregation and two sum aggregations nested beneath the terms
> aggregation:
>
> { "query": { "filtered": { "filter": { "bool": { … } } } },
>
> "aggs": { "name1": { "terms": { "field": "stringfield1" },
>
> "aggs": { "sum1": { "sum": { "field": "longfield1" } },
>
> "sum2 ": { "sum": { "field": "longfield2" } } } } } }
>
> I have run the tests on the cluster when it was lightly loaded (some
> indexing plus lightweight metrics queries) and run the tests when there was
> no load.  I’ll be the first to admit I can be even more systematic, but
> the results I have are consistent enough and hard to explain enough that
> wanted to write this community.
>
> The primary test uses a filter which always results in an empty set.  The
> filter contains two must terms, one must range and three mustnot terms.  Since
> I only care about the aggregation results, this is a search_type=count
> query.
>
> If I run the query/filter without the aggregations, the time taken
> (results.took from ES) is ~0 (sometimes as high as 15ms).   That makes
> sense.
>
> The case that doesn’t make sense is that I run the same filter on the same
> cluster under the same condition this time WITH the aggregations and I get
> anywhere from 200ms to 4ms.  Yes, a factor of 200x.  I could believe
> 200ms to account for some overhead of the aggregations machinery, but
> 4ms?  And there is no pattern that I can tell as to when 200ms is
> returned .vs. 4ms.
>
> Given that Azure round robins the queries, I can imagine that depending on
> which nodes are involved, the query might take more or less of the time.  In
> fact, I would expect some variations.
>
> The other piece of data is that in trying to debug this I restarted ES on
> some of the nodes.  By the time I had restarted the third node the
> query/filter + all agregations case now returned 200ms consistently.
>
> My question is how it is possible for an empty filter + aggregations to
> result in 4ms time.  I tried the same filter and only the terms
> aggregation (not the sums); the result was in the 3500-4000ms range – in
> case that matters.
>
> Hopefully this makes sense to someone.  I’m pulling my hair out and my
> colleagues on our internal ES alias are stumped as well.
>
> Thanks for any help,
>
> Craig.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/7f16f9e6-52e7-4d5c-854a-a7bd409e2040%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG-TuLrUOMqpt24vu2FE-%3D320%3DOSfTbsaYyRCiswaN9cg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Is there any way to prevent ES from disclosing exception details in REST response?

[AlexR]

We expose ES _search endpoint directly to consumers. When our REST API get 
scanned for security vulnerabilities it complains on ES returning exception 
details. For example a malformed query will be included in the response 
along with exception. While it is more or a less harmless the tool 
complains of various injections and internals disclosures. I would like to 
be able to turn error message in the response off (or substitute it with a 
generic message) in production while keeping normal response logic in 
development. 

Is there any way I can do it?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/83e6b126-4db1-44cc-9a0a-0dd6c6d44a64%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ann] Log4j2 Elasticsearch HTTP appender 1.0.0

[joergpra...@gmail.com]

Hi,

I just released a small, dependency-free Java logging tool, an appender for
Log4j2, using the Java JDK HTTP client to connect to the _bulk endpoint.

https://github.com/jprante/log4j2-elasticsearch-http

All comments and feedback is welcome!

Best,

Jörg

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHb7SR2fOwwtBKgUhAZCH4wPqUvGd5KTjQMktixXjF7zg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Highly variable query performance with ES 1.3.2 (filter + aggregations)

[Craig Wittenberg]

 

Hello,

Our team recently upgraded from ES 1.1.2 to 1.3.2 and are happy with the 
improvements ... except for one perplexing situation.

We are running on Azure worker roles with Oracle Java 1.8u11 and using the 
G1 gc.  It is possible this is due to G1, but please consider all of the 
data below before you pull out a pat response on g1.  

Our cluster has 18 nodes, 3 of which are dedicated masters.  We have three 
indexes, 5 shards and one replica each.  The primary index is about 30gb 
total (5.9gb per shard and the shards are the same size).  We have five 
types in the main index and are about 10 fields each, a mix of strings, 
dates, bools, longs.  None of the strings are analyzed.

All of the 18 nodes are client nodes and Azure is set up to round robin 
requests.  We have considered creating dedicated client nodes, but haven't 
done so yet.

The query I have been using is a combination of a non-trivial filter, a 
terms aggregation and two sum aggregations nested beneath the terms 
aggregation:

{ "query": { "filtered": { "filter": { "bool": { … } } } },

"aggs": { "name1": { "terms": { "field": "stringfield1" },

"aggs": { "sum1": { "sum": { "field": "longfield1" } },

"sum2 ": { "sum": { "field": "longfield2" } } } } } }

I have run the tests on the cluster when it was lightly loaded (some 
indexing plus lightweight metrics queries) and run the tests when there was 
no load.  I’ll be the first to admit I can be even more systematic, but the 
results I have are consistent enough and hard to explain enough that wanted 
to write this community.

The primary test uses a filter which always results in an empty set.  The 
filter contains two must terms, one must range and three mustnot terms.  Since 
I only care about the aggregation results, this is a search_type=count 
query.

If I run the query/filter without the aggregations, the time taken 
(results.took from ES) is ~0 (sometimes as high as 15ms).   That makes 
sense.

The case that doesn’t make sense is that I run the same filter on the same 
cluster under the same condition this time WITH the aggregations and I get 
anywhere from 200ms to 4ms.  Yes, a factor of 200x.  I could believe 
200ms to account for some overhead of the aggregations machinery, but 
4ms?  And there is no pattern that I can tell as to when 200ms is 
returned .vs. 4ms. 

Given that Azure round robins the queries, I can imagine that depending on 
which nodes are involved, the query might take more or less of the time.  In 
fact, I would expect some variations.

The other piece of data is that in trying to debug this I restarted ES on 
some of the nodes.  By the time I had restarted the third node the 
query/filter + all agregations case now returned 200ms consistently.

My question is how it is possible for an empty filter + aggregations to 
result in 4ms time.  I tried the same filter and only the terms 
aggregation (not the sums); the result was in the 3500-4000ms range – in 
case that matters.

Hopefully this makes sense to someone.  I’m pulling my hair out and my 
colleagues on our internal ES alias are stumped as well.

Thanks for any help,

Craig.

 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7f16f9e6-52e7-4d5c-854a-a7bd409e2040%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Aggregations: Year over Year, Week over Week

[nascosto]

Assuming I have a date field on a document, I know using the date_histogram 
aggregation I can get a document count by day, month, year, etc. 

What I want to do is get the average document count for January, February, 
March, etc. over several given years. The same goes for Monday, Tuesday, 
Wednesday, etc. over several given weeks. Is there a way to do this having 
just that same date field or what is the best way to accomplish this with 
Elasticsearch?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/027a9e40-d307-4caa-8482-4830108ad353%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [JAVA API] Retrieve all indexes

[Luan Garrido]

Vineeth, can you tell me how can I get the types of this index?
I tried:

GetMappingsResponse response = 
this.client.admin().indices().prepareGetMappings(index).execute().actionGet();
for(ObjectCursor st : response.mappings().keys()){
System.out.println(st.value);
}
But it returned me the name of the index.

Thank you


Em segunda-feira, 15 de setembro de 2014 08h22min40s UTC-3, Luan Garrido 
escreveu:
>
> Hi all,
> First of all I took a look and did not find the answer for my question.
> Can anyone teach me how can I find all indexes from a cluster using the 
> JAVA API? 
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/960d95d7-1c21-43f3-b95f-98d6f2cf5e26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Received response for a request that has timed out Error

[shriyansh jain]

Hi,

I am getting the following error in the elasticsearch log file. I have a 
cluster of 2 elasticsearch nodes, and have a setup of ELK stack with redis 
as a buffer. Everything was running fine but suddenly, logstash is not able 
to parse logs (not showing any error messages) and elasticsearch is giving 
the following error message.
Can anybody guide me on what is this error message about. 

[2014-09-15 11:55:30,607][WARN ][transport] [Node1] 
Received response for a request that has timed out, sent [41307ms] ago, 
timed out [11307ms] ago, action [discovery/zen/fd/ping], node 
[[Node2][01qr5MvqQaGhGYgfAZE_IQ][sjain-centos6-3][inet[/10.12.10.235:9300]]], 
id [332]


Any input is greatly appreciated.

Thank you,
Shriyansh

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e9a04c4c-be87-4e5a-a2dd-3a76062e4ce1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Revisiting query parse (no field after start_object) errors

[Jack Park]

Updating my query.

I ran the same test just using this query:
{
  "term" : {
"sbOf" : "TypeType"
  }
}

with absolutely no failure.

But, if I put from and size in front of it, with or without "query", there
is a failure.
So, this query does run, though from and size are meaningless
{
"term": {
"sbOf": "TypeType"
},
"from": 0,
"size": 30
}

The final solution appears to be to use the SearchBuilder to set from and
size, code like this:

SearchRequestBuilder builder = prepareSearch(index)
.setSearchType(SearchType.QUERY_THEN_FETCH)
.setTypes(types)
.setQuery(queryString)
.setFrom(start);
if (count > -1)
 builder.setSize(count);

Interesting thought:
With the NodeJS client, folding from and size into the query string works
just fine; in the Java client, it fails rather spectacularly.

No clue why, but it's bound to be something in my code...


On Mon, Sep 15, 2014 at 8:45 AM, Jack Park  wrote:

> This query:
>
> {
> "from": 0,
> "size": 30,
> "query": {
> "term": {
> "sbOf": "TypeType"
> }
> }
> }
>
> is giving me fits. I recently migrated from 0.90 to 1.2.2 and went through
> this with my node.js client, got many ideas in this forum and solved the
> problem. The query above works fine from node.js, but fails  (listing
> below) from the java client running against the same index as the node
> client.
>
> I realize this is a potentially high-dimensional issue with many actors
> other than the query string in play. Just looking for hints.
>
> I have tested with and without "query". No difference.
>
> Many thanks in advance
> Jack
>
> listing:
> ERROR 2014-09-15 08:28:50,882 [main] - Failed to execute phase [query],
> all shards failed; shardFailures {[vIdL7fWlT1en9lKkGS8NyA][topics][2]:
> SearchParseException[[topics][2]: from[-1],size[-1]: Parse Failure [Failed
> to parse source
> [{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
> nested: QueryParsingException[[topics] [_na] query malformed, no field
> after start_object]; }{[vIdL7fWlT1en9lKkGS8NyA][topics][3]:
> SearchParseException[[topics][3]: from[-1],size[-1]: Parse Failure [Failed
> to parse source
> [{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
> nested: QueryParsingException[[topics] [_na] query malformed, no field
> after start_object]; }{[vIdL7fWlT1en9lKkGS8NyA][topics][0]:
> SearchParseException[[topics][0]: from[-1],size[-1]: Parse Failure [Failed
> to parse source
> [{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
> nested: QueryParsingException[[topics] [_na] query malformed, no field
> after start_object]; }{[vIdL7fWlT1en9lKkGS8NyA][topics][1]:
> SearchParseException[[topics][1]: from[-1],size[-1]: Parse Failure [Failed
> to parse source
> [{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
> nested: QueryParsingException[[topics] [_na] query malformed, no field
> after start_object]; }{[vIdL7fWlT1en9lKkGS8NyA][topics][4]:
> SearchParseException[[topics][4]: from[-1],size[-1]: Parse Failure [Failed
> to parse source
> [{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
> nested: QueryParsingException[[topics] [_na] query malformed, no field
> after start_object]; }
> org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to
> execute phase [query], all shards failed; shardFailures
> {[vIdL7fWlT1en9lKkGS8NyA][topics][2]: SearchParseException[[topics][2]:
> from[-1],size[-1]: Parse Failure [Failed to parse source
> [{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
> nested: QueryParsingException[[topics] [_na] query malformed, no field
> after start_object]; }{[vIdL7fWlT1en9lKkGS8NyA][topics][3]:
> SearchParseException[[topics][3]: from[-1],size[-1]: Parse Failure [Failed
> to parse source
> [{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
> nested: QueryParsingException[[topics] [_na] query malformed, no field
> after start_object]; }{[vIdL7fWlT1en9lKkGS8NyA][topics][0]:
> SearchParseException[[topics][0]: from[-1],size[-1]: Parse Failure [Failed
> to parse source
> [{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
> nested: QueryParsingException[[topics] [_na] query malformed, no field
> after start_object]; }{[vIdL7fWlT1en9lKkGS8NyA][topics][1]:
> SearchParseException[[topics][1]: from[-1],size[-1]: Parse Failure [Failed
> to parse source
> [{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
> nested: QueryParsingException[[topics] [_na] query malformed, no field
> after start_object]; }{[vIdL7fWlT1en9lKkGS8NyA][

Filtered not working with has_parent in elasticsearch 1.3.0

[Roopendra Vishwakarma]

In elasticsearch 1.3.0 *filtered* not working with *has_parent*. In 
*elasticsearch 
0.90.5* its working fine.

I am using below query. In this query I need add filtered inside 
*has_parent->query->filtered->bool->must*. But my basic query is not 
working. 

{
   "query": {
  "has_parent": {
 "parent_type": "url",
 "query": {
   "match": {
  "affiliate_id": "1718489511"
   }
 }
  },
  "filtered": {
 "filter": {
"bool": {
   "must": {
  "terms": {
 "country": [
"india"
 ]
  }
   }
}
 }
  }
   }
}



Getting error message* ElasticsearchParseException[Expected field name but 
got START_OBJECT "filtered"*

*Complete Error Message:-*

error: SearchPhaseExecutionException[Failed to execute phase [query], all 
shards failed;
 shardFailures {[hZgYLUmsQL2T8z1MY2eyNA][esindex][5]: 
SearchParseException[[esindex][5]: 
 query[parent_filter[url](filtered(affiliate_id:1718489511)->cache(_type
:url))],from[-1],size[-1]: 
 Parse Failure [Failed to parse source [{"query":{"has_parent":{
"parent_type":"url","query":{"match":{"affiliate_id":"1718489511"}}},
 "filtered":{"filter":{"bool":{"must":{"terms":{"country":["india"
]}}}]]];
 nested: ElasticsearchParseException[Expected field name but got 
START_OBJECT "filtered"]; }
 {[OPG4zuQlQr2dFxUvRe2GWg][esindex][4]: RemoteTransportException[[
server1][inet[/127.0.0.1:9300]]
 [search/phase/query]]; nested: SearchParseException[[esindex][4]: query
[parent_filter[url]
 (filtered(affiliate_id:1718489511)->cache(_type:url))],from[-1],size[-1
]: Parse Failure 
 [Failed to parse source [{"query":{"has_parent":{"parent_type":"url",
"query":{"match":{"affiliate_id":"1718489511"}}},
 "filtered":{"filter":{"bool":{"must":{"terms":{"country":["india"
]}}}]]]; 
 nested: ElasticsearchParseException[Expected field name but got 
START_OBJECT "filtered"]; }

Any suggestion? 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/683129a1-c757-4250-a97e-a15a4efbbaa5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Elastic Search Design question.

[taimoorkhan1]

Hi, I am trying to figure out how to best model my data in elasticsearch 
for the following scenario.

Scenario:
Imagine a chat service which can have a number of participants (in tens of 
thousands).  Messages are not pushed out to members but rather members can 
query for new messages based on which messages they have received so far.  
So we need to keep track of the last message sent to the channel and also 
the last message which each member has received.  This comparison will 
allow us to determine if there are any new messages for this member.

The frequency of messages being sent is high (which may require frequent 
re-indexing and replica updates).  I would also like to avoid double read 
i.e. read last message time from chat and then make another query 

Possible models:
1.  Nested objects.  (Concern: is only root document is returned and not 
nested objects?  Performance of re-indexing based on high frequency of 
udpates?  Large replica updates between elastic search nodes).

2.  Parent Child.  (Would like to know more about this).

3.  Sort of parent child without linking the documents with the '_parent' 
property.  So each member will keep track of both the last message sent to 
the chat as well as the last message they have seen so far. (Concern: a 
single message sent to a chat will required tens of thousands of updates to 
all its members).

4.  A single large document with individual members nested inside.  
(Concern:  The size of the document and frequency of updates will thrash 
the system?)


I would like to understand how elasticsearch can best model the above 
scenario or even if it is the right tool for this problem.

Thanks.






-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e7850f0a-25d3-4fd0-8595-a33ec38485d8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] New IMAP and POP3 email river for Elasticsearch - elasticsearch-river-imap

[Marc Laporte]

Hi!

Thanks for this.

I want to use Zarafa with Elasticsearch on ClearOS, in the context of Tiki 
Suite.

Tiki Suite is a selection of Free / Libre / Open Source Software (FLOSS) 
server, web, mobile and desktop apps with a concerted focus on greater 
interoperability, security and adaptability, which is aimed at small & 
medium-sized organizations. The Tiki Suite is especially suited to 
decentralized and knowledge-centric organizations and offers most (80%+) of 
the features all organizations need, such as: Email, Website & Blog, 
Shopping Cart, Intranet & Project Management, E-learning, Social 
Networking, Knowledge base, File sharing, Issue Tracker, 
Video-conferencing, LDAP, VPN, Gateway, Network, etc.

The main server apps are: ClearOS (server), Tiki Wiki CMS Groupware, Zarafa 
(Mail), Prosody (XMPP) and Syncthing (file sync).

Tiki is the 
http://tiki.org/FLOSS+Web+Application+with+the+most+built-in+features and 
the content is searchable via Elasticsearch. It's all filtered by 
permission, and permits goodies like More Like This, Facets, Percolator 
(Stored searches), and more is on the way: http://doc.tiki.org/Elasticsearch

This is all nice and well. However I would like to add emails, XMPP logs, 
and the files from the shared/sync folders. So looking at any item 
(e-mails, wiki pages, files, tasks, bugs, forum posts, blog posts, XMPP 
chat logs, etc.) can show me related entries.

"Introduced with Tiki14, the federated search allows to integrate results 
from other sources into the main search results."
http://doc.tiki.org/Federated+Search 

So I'd like to include emails (and contacts and events) in these search 
results, with proper permissions. So 
https://github.com/salyh/elasticsearch-river-imap is of great interest to 
me.

I also know of:
http://manifoldcf.apache.org/release/trunk/en_US/end-user-documentation.html#emailrepository


Thoughts? Advice?

Thanks!


https://suite.tiki.org/Tiki+Suite


On Tuesday, April 22, 2014 11:36:11 AM UTC-4, Hendrik wrote:
>
> Hi,
> i'd like to announce a IMAP and POP river for Elasticsearch.
>
> It can be found here:
> https://github.com/salyh/elasticsearch-river-imap
>
> This river connects to IMAP4 or POP3 servers polls your mail and index it. 
> The emails on the server will be never modified or removed from the server. 
> The river tracks (after the first initial full load) which mails are new 
> or deleted and then only update the index for this mails.
>
> Features:
>
>- Incremental indexing of e-mails from a IMAP or POP3 server
>- Support indexing of attachments (in conjunction with 
>https://github.com/elasticsearch/elasticsearch-mapper-attachments/)
>- Support for UTF-7 encoded e-mails (through jutf7)
>- SSL, STARTTLS and SASL are supported (through JavaMail API)
>- IMAP only: Folders which should be indexed can be specified with 
>a regex pattern
>- IMAP only: Subfolders can also be indexed (whole traversal of 
>all folders)
>- No special server capabilities needed
>- Bulk indexing
>
> The river acts currently as a disconnected client. This means that the 
> river is polling and for every indexing run a new server connections are 
> opened and,
> after work is done, closed. At a later time is planned to use additionally 
> the IMAP IDLE feature (if server supports it).
>
> Suggestions, corrections, improvements are very welcome!
> Thanks and best regards
> Hendrik
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/82b9855d-548d-43e4-a871-91a4377ca172%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Suggestion needed on Indexing Performance

[Anantha Govindarajan]

Hi,

Here is our ES ( elasticsearch-1.2.1 ) cluster details

   - 20 Data Nodes (physical machines 2TB normal disk,32GB RAM,8 cpu 
   cores). Both Xms & Xmx 14GB.  Xmn 7GB
   - 2 Master Nodes (virtual machines 16GB RAM)
   - Cluster Size ~17 TB (out of 20 * 2 = 40TB)
   - Number of records ~70 billions
   - Number of indices ~3500
   - Number of shards ~1

Indices config

   - index.refresh_interval: 40s
   - index.compound_format: true
   - index.compound_on_flush: true
   - index.merge.policy.max_merged_segment: 500mb
   - index.codec.postings_format.no_bloom.type: default
   - indices.cache.filter.size: 1000mb
   - indices.store.throttle.type: merge
   - indices.store.throttle.max_bytes_per_sec: 30mb
   - index.routing.allocation.total_shards_per_node: 2

We create 100 daily indices.  We keep indices maximum of 30 days.

   - Small size indices (1GB to 5GB / per day - one shard & one replica) .
   - Large size indices (95GB to 125GB / per day - 12 shards & one replica)
   - We issue bulk indexing request . 1000 docs / per request.
   - docs may be simple tomcat logs / exceptions

10 - 15 search queries per day. 

   - Queries may be simple term query or phrase query. 
   - 80% of the queries response time fall under 10s. (Which is acceptable).
   - Queries on large indices (12 shards) , response time fall under 20s to 
   30s

Problem  (Suggestions needed)

   - indexing time for bulk request(1000 per req) for first 5 to 8 requests 
   is less than 500 ms .
   - Then next 5 to 8 indexing bulk requests(1000 per req), response time 
   is 2000ms to 5000ms (Some times 2 ms or above)
   - Then next 5 to 8 indexing bulk requests(1000 per req), response time 
   is 1000ms to 2000ms
   - Then next 5 to 8 indexing bulk requests(1000 per req), response time 
   is less than 500ms

This variation in response time continues as cycle only in large indices 
(12 shards). 

As a result it takes nearly 10 to 15 mins to index a file which contains 
~40 (4 lakh) docs on large indices (12 shards)

I don't change any settings related with Lucene Merge .

Over the time indexing process lagging far behind, then i update 
 index.number_of_replicas as 0 to large indices. (But we need 
HighAvailability.)


   - What makes indexing slow ?  ( Lucene Merge ? ES flush ?  ES refresh ? 
   ) How to identify ?
   - 12 shards - too big ? 
   - I dont want to change index.number_of_replicas as 0 each time. 
   Suggestions ?
   - Some times even i reduced the copy, indexing not getting catch up. 
   - 10% of memory for indexing 1400mb for ~20 shards on each node for 
   active indices (Obviously today indices.)
   - Therefore 70mb for each index. Whether i need to increase ?

FYI : I went through Mike's recent blog on performance for ES indexing.

Regards,
Anantha Govindarajan

 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d49106f0-3169-4bf6-83f4-c2782d51ab4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

likes and favourite item search and storage

[Michael]

Hi,

I'm looking for the best practise for recording 'likes' of an item.

If I have a document

{
  'name' : 'Dave',
  'age' : 40,
  'album' : 'foo'
}


If people decide to 'like' 'Daves' album 'foo'. Where is the best place to 
store these 'likes'? in the document?

{
  'name' : 'Dave',
  'age' : 40,
  'album' : 'foo',
  'likes' : {
   'user1',
   'user2',
  }
}

Or store the likes in a different type? Because When I return results for 
albums, Dave's album should rank higher because that album has more likes. 
But also I want to show how many likes the album has. But with people 
liking and un-liking albums, I want to avoid conflicts with updates.

So what is the best approach? Should I put everything into a queue? How 
will I query and arrange by number of likes if the documents are separated?

Thanks. 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a5238715-5df0-40a9-b06f-f33f8d22cc92%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Filtered Query not working with Elasticsearch 1.3.0

[David Pilato]

Did you read this?

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/breaking-changes-1.0.html#breaking-changes-1.0

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

> Le 15 sept. 2014 à 18:08, Roopendra Vishwakarma  a 
> écrit :
> 
> Filtered query not working with Elasticsearch 1.3.0. I upgrade elasticsearch 
> 0.90.5 to 1.3.0. 
> I am using elasticsearch below Query DSL. In Elasticsearch 0.90.5 below query 
> DSL is working fine. But in Elasticsearch 1.3.0 getting error 
> ElasticsearchParseException[Expected field name but got START_OBJECT
> 
> {
> "query": {
> "match_all": {},
> "filtered": {
>"filter": {
>"bool": {
>"must": [
>   {
>   "terms": {
>  "status": [
> "publish"
>  ]
>   }
>   }
>]
>}
>}
> }
> }
> }
> 
> 
> 
> Complete Error Message:
> 
> {
> 
> error: SearchPhaseExecutionException[Failed to execute phase [query], 
> all shards failed; shardFailures {[8FQelETaR22xvV1SvynPEQ][esindex][0]: 
> RemoteTransportException[[server1][inet[/127.0.0.1:9300]][search/phase/query]];
>  nested: SearchParseException[[esindex][0]: 
> query[ConstantScore(*:*)],from[-1],size[1]: Parse Failure [Failed to parse 
> source 
> [{"size":1,"query":{"match_all":{},"filtered":{"filter":{"bool":{"must":[{"terms":{"status":["publish"]}}]}]]];
>  nested: ElasticsearchParseException[Expected field name but got START_OBJECT 
> "filtered"]; }{[8FQelETaR22xvV1SvynPEQ][esindex][1]: 
> RemoteTransportException[[server1][inet[/127.0.0.1:9300]][search/phase/query]];
>  nested: SearchParseException[[esindex][1]: 
> query[ConstantScore(*:*)],from[-1],size[1]: Parse Failure [Failed to parse 
> source 
> [{"size":1,"query":{"match_all":{},"filtered":{"filter":{"bool":{"must":[{"terms":{"status":["publish"]}}]}]]];
>  nested: ElasticsearchParseException[Expected field name but got START_OBJECT 
> "filtered"]; }{[hZgYLUmsQL2T8z1MY2eyNA][esindex][2]: 
> SearchParseException[[esindex][2]: 
> query[ConstantScore(*:*)],from[-1],size[1]: Parse Failure [Failed to parse 
> source 
> [{"size":1,"query":{"match_all":{},"filtered":{"filter":{"bool":{"must":[{"terms":{"status":["publish"]}}]}]]];
>  nested: ElasticsearchParseException[Expected field name but got START_OBJECT 
> "filtered"]; }]
> status: 400
> 
> }  
> 
>   
> 
> Any suggestion?
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/032e7ac5-8663-4793-8eec-3661fadceef3%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/DF1CC3CD-71E0-4DD4-8B62-30BA8286F0C7%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Re: How to get current date in script.

[vineeth mohan]

Hello Abhishek ,

Something similar to this should work -

{
  "script_fields": {
"test": {
  "script": "doc['date'].value - new Date().getTime() * 1000"
}
  }
}

Thanks
  Vineeth

On Mon, Sep 15, 2014 at 9:41 PM, Abhishek Gupta 
wrote:

> Hi,
>
> I want to write a script to calculate the number of days from current date
> on a date field. I can use a script like:
> "script_fields": {
> "test": {
>   "script": "doc['date'].date.year * doc['date'].date.dayOfYear - 
> * date>*"
> }
>  }
>
> How to get the current date? No date related function is mentioned in the
> help(
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html
> ).
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/81d6726b-88ca-4a62-829d-0a7b970b8669%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nzE6oDppCzf67ED3d%2Bhgq995wPDTi7f7rOPSBsyE578Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to do case insensitive search on terms?

[John Smith]

Thanks match works also.

On Monday, 15 September 2014 12:09:43 UTC-4, Nikolas Everett wrote:
>
> Use a match instead of a query_string - query_string has funky syntax that 
> can activate fuzzy matching and search on other fields and stuff - probably 
> not what you want.  Otherwise it shouldn't have any real impact on 
> performance.
>
> On Mon, Sep 15, 2014 at 12:04 PM, John Smith  > wrote:
>
>> Thanks!
>>
>> Yes I want case insensitive search and yes I'm using default analyzer.
>>
>> Just as you guys answered I tried this...
>>
>> {
>>   "query": {
>> "query_string": {
>>   "fields": ["logType"],
>>   "query": "ABC"
>> }
>>   }
>> }
>>
>> and it worked. Since I'm searching for "exact" matches (not wildards or 
>> anything like that) does it make a difference in performance? So far it 
>> seems like no at least when testing through Sense on the same amount of 
>> data.
>>
>> On Monday, 15 September 2014 11:49:33 UTC-4, Nikolas Everett wrote:
>>>
>>> Or if you want case insensitive search use a match query 
>>> 
>>> .
>>>
>>> On Mon, Sep 15, 2014 at 11:47 AM, joerg...@gmail.com >> > wrote:
>>>
 I assume you use the standard analyzer which uses by default a token 
 filter "lowercase".

 Just use a custom analyzer, without "lowercase" token filter, and you 
 will get case-sensitive search.

 Jörg

 On Mon, Sep 15, 2014 at 5:44 PM, John Smith  
 wrote:

> Using ES 1.3.2
>
> The current application I'm building only uses term queries for exact 
> matches.
>
> Example query
>
>   "query": {
> "term": {
>   "logType": "abc"
> }
>
>
> The field logType is pulled from external DB as all caps so for 
> instance ABC
>
> If i send the query
>
>   "query": {
> "term": {
>   "logType": "ABC"
> }
>
>
> I get no results.
>
> If I send the query
>
>   "query": {
> "term": {
>   "logType": "abc"
> }
>
> I get results.
>
> So does this mean I need to toLower the input before building the 
> query or there's an ES way of doing this?
>
> Thanks
>
>
>
>
>  -- 
> You received this message because you are subscribed to the Google 
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to elasticsearc...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/elasticsearch/8fec1f56-1a59-4d17-bb01-f2ecd62bfbb4%
> 40googlegroups.com 
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

  -- 
 You received this message because you are subscribed to the Google 
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/CAKdsXoHi-483Tgf5DmwXEfKrGRye1vjvGU4jFL0
 o8guNsGBaCw%40mail.gmail.com 
 
 .

 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/b72784e7-48c1-4c10-ab69-77df2c1b6d42%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/07bb0325-ee98-4fe9-a2f1-a88a50a555f8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How to get current date in script.

[Abhishek Gupta]

Hi,

I want to write a script to calculate the number of days from current date 
on a date field. I can use a script like:
"script_fields": {
"test": {
  "script": "doc['date'].date.year * doc['date'].date.dayOfYear - **"
}
 }

How to get the current date? No date related function is mentioned in the 
help(
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html
).

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/81d6726b-88ca-4a62-829d-0a7b970b8669%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to do case insensitive search on terms?

[Nikolas Everett]

Use a match instead of a query_string - query_string has funky syntax that
can activate fuzzy matching and search on other fields and stuff - probably
not what you want.  Otherwise it shouldn't have any real impact on
performance.

On Mon, Sep 15, 2014 at 12:04 PM, John Smith  wrote:

> Thanks!
>
> Yes I want case insensitive search and yes I'm using default analyzer.
>
> Just as you guys answered I tried this...
>
> {
>   "query": {
> "query_string": {
>   "fields": ["logType"],
>   "query": "ABC"
> }
>   }
> }
>
> and it worked. Since I'm searching for "exact" matches (not wildards or
> anything like that) does it make a difference in performance? So far it
> seems like no at least when testing through Sense on the same amount of
> data.
>
> On Monday, 15 September 2014 11:49:33 UTC-4, Nikolas Everett wrote:
>>
>> Or if you want case insensitive search use a match query
>> 
>> .
>>
>> On Mon, Sep 15, 2014 at 11:47 AM, joerg...@gmail.com 
>> wrote:
>>
>>> I assume you use the standard analyzer which uses by default a token
>>> filter "lowercase".
>>>
>>> Just use a custom analyzer, without "lowercase" token filter, and you
>>> will get case-sensitive search.
>>>
>>> Jörg
>>>
>>> On Mon, Sep 15, 2014 at 5:44 PM, John Smith  wrote:
>>>
 Using ES 1.3.2

 The current application I'm building only uses term queries for exact
 matches.

 Example query

   "query": {
 "term": {
   "logType": "abc"
 }


 The field logType is pulled from external DB as all caps so for
 instance ABC

 If i send the query

   "query": {
 "term": {
   "logType": "ABC"
 }


 I get no results.

 If I send the query

   "query": {
 "term": {
   "logType": "abc"
 }

 I get results.

 So does this mean I need to toLower the input before building the query
 or there's an ES way of doing this?

 Thanks




  --
 You received this message because you are subscribed to the Google
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/8fec1f56-1a59-4d17-bb01-f2ecd62bfbb4%
 40googlegroups.com
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/CAKdsXoHi-483Tgf5DmwXEfKrGRye1vjvGU4jFL0
>>> o8guNsGBaCw%40mail.gmail.com
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/b72784e7-48c1-4c10-ab69-77df2c1b6d42%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0Bqco40M1102FXXY8DpyV9Yen0OwaXdE1wHND%3DsaKPNw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Filtered Query not working with Elasticsearch 1.3.0

[Roopendra Vishwakarma]

Filtered query not working with *Elasticsearch 1.3.0*. I upgrade 
elasticsearch *0.90.5* to *1.3.0*. 
I am using elasticsearch below Query DSL. In *Elasticsearch 0.90.5* below 
query DSL is working fine. But in *Elasticsearch 1.3.0* getting error 
*ElasticsearchParseException[Expected 
field name but got START_OBJECT*

{
"query": {
"match_all": {},
"filtered": {
   "filter": {
   "bool": {
   "must": [
  {
  "terms": {
 "status": [
"publish"
 ]
  }
  }
   ]
   }
   }
}
}
}



*Complete Error Message:*

{

error: SearchPhaseExecutionException[Failed to execute phase [query
], all shards failed; shardFailures {[8FQelETaR22xvV1SvynPEQ][esindex][0]: 
RemoteTransportException[[server1][inet[/127.0.0.1:9300]][search/phase/query
]]; nested: SearchParseException[[esindex][0]: query[ConstantScore(*:*)],
from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,"query":{
"match_all":{},"filtered":{"filter":{"bool":{"must":[{"terms":{"status":[
"publish"]}}]}]]]; nested: ElasticsearchParseException[Expected field 
name but got START_OBJECT "filtered"]; }{[8FQelETaR22xvV1SvynPEQ][esindex][1
]: RemoteTransportException[[server1][inet[/127.0.0.1:9300]][search/phase/
query]]; nested: SearchParseException[[esindex][1]: query[ConstantScore
(*:*)],from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,
"query":{"match_all":{},"filtered":{"filter":{"bool":{"must":[{"terms":{
"status":["publish"]}}]}]]]; nested: ElasticsearchParseException[
Expected field name but got START_OBJECT "filtered"]; }{[
hZgYLUmsQL2T8z1MY2eyNA][esindex][2]: SearchParseException[[esindex][2]: 
query[ConstantScore(*:*)],from[-1],size[1]: Parse Failure [Failed to parse 
source [{"size":1,"query":{"match_all":{},"filtered":{"filter":{"bool":{
"must":[{"terms":{"status":["publish"]}}]}]]]; nested: 
ElasticsearchParseException[Expected field name but got START_OBJECT 
"filtered"]; }]
status: 400

}  

  

Any suggestion?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/032e7ac5-8663-4793-8eec-3661fadceef3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to do case insensitive search on terms?

[John Smith]

Thanks!

Yes I want case insensitive search and yes I'm using default analyzer.

Just as you guys answered I tried this...

{
  "query": {
"query_string": {
  "fields": ["logType"],
  "query": "ABC"
}
  }
}

and it worked. Since I'm searching for "exact" matches (not wildards or 
anything like that) does it make a difference in performance? So far it 
seems like no at least when testing through Sense on the same amount of 
data.

On Monday, 15 September 2014 11:49:33 UTC-4, Nikolas Everett wrote:
>
> Or if you want case insensitive search use a match query 
> 
> .
>
> On Mon, Sep 15, 2014 at 11:47 AM, joerg...@gmail.com  <
> joerg...@gmail.com > wrote:
>
>> I assume you use the standard analyzer which uses by default a token 
>> filter "lowercase".
>>
>> Just use a custom analyzer, without "lowercase" token filter, and you 
>> will get case-sensitive search.
>>
>> Jörg
>>
>> On Mon, Sep 15, 2014 at 5:44 PM, John Smith > > wrote:
>>
>>> Using ES 1.3.2
>>>
>>> The current application I'm building only uses term queries for exact 
>>> matches.
>>>
>>> Example query
>>>
>>>   "query": {
>>> "term": {
>>>   "logType": "abc"
>>> }
>>>
>>>
>>> The field logType is pulled from external DB as all caps so for instance 
>>> ABC
>>>
>>> If i send the query
>>>
>>>   "query": {
>>> "term": {
>>>   "logType": "ABC"
>>> }
>>>
>>>
>>> I get no results.
>>>
>>> If I send the query
>>>
>>>   "query": {
>>> "term": {
>>>   "logType": "abc"
>>> }
>>>
>>> I get results.
>>>
>>> So does this mean I need to toLower the input before building the query 
>>> or there's an ES way of doing this?
>>>
>>> Thanks
>>>
>>>
>>>
>>>
>>>  -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com .
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/8fec1f56-1a59-4d17-bb01-f2ecd62bfbb4%40googlegroups.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHi-483Tgf5DmwXEfKrGRye1vjvGU4jFL0o8guNsGBaCw%40mail.gmail.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b72784e7-48c1-4c10-ab69-77df2c1b6d42%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Filtered Query not working with Elasticsearch 1.3.0

[Roopendra Vishwakarma]

Filtered query not working with Elasticsearch 1.3.0. I upgrade 
elasticsearch 0.90.5 to 1.3.0. 
I am using elasticsearch Query DSL;

{
"query": {
"match_all": {},
"filtered": {
   "filter": {
   "bool": {
   "must": [
  {
  "terms": {
 "status": [
"publish"
 ]
  }
  }
   ]
   }
   }
}
}
}




In Elasticsearch 0.90.5 below query DSL is working fine. But in 
Elasticsearch 1.3.0 getting error. 

  
   ElasticsearchParseException[Expected field name but got START_OBJECT



**Complete Error Message:**

{

error: SearchPhaseExecutionException[Failed to execute phase [query
], all shards failed; shardFailures {[8FQelETaR22xvV1SvynPEQ][esindex][0]: 
RemoteTransportException[[server1][inet[/127.0.0.1:9300]][search/phase/query
]]; nested: SearchParseException[[esindex][0]: query[ConstantScore(*:*)],
from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,"query":{
"match_all":{},"filtered":{"filter":{"bool":{"must":[{"terms":{"status":[
"publish"]}}]}]]]; nested: ElasticsearchParseException[Expected field 
name but got START_OBJECT "filtered"]; }{[8FQelETaR22xvV1SvynPEQ][esindex][1
]: RemoteTransportException[[server1][inet[/127.0.0.1:9300]][search/phase/
query]]; nested: SearchParseException[[esindex][1]: query[ConstantScore
(*:*)],from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,
"query":{"match_all":{},"filtered":{"filter":{"bool":{"must":[{"terms":{
"status":["publish"]}}]}]]]; nested: ElasticsearchParseException[
Expected field name but got START_OBJECT "filtered"]; }{[
hZgYLUmsQL2T8z1MY2eyNA][esindex][2]: SearchParseException[[esindex][2]: 
query[ConstantScore(*:*)],from[-1],size[1]: Parse Failure [Failed to parse 
source [{"size":1,"query":{"match_all":{},"filtered":{"filter":{"bool":{
"must":[{"terms":{"status":["publish"]}}]}]]]; nested: 
ElasticsearchParseException[Expected field name but got START_OBJECT 
"filtered"]; }]
status: 400

}


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e0e632cb-1f58-417e-b642-eabfd8514942%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to do case insensitive search on terms?

[joergpra...@gmail.com]

I assume you use the standard analyzer which uses by default a token filter
"lowercase".

Just use a custom analyzer, without "lowercase" token filter, and you will
get case-sensitive search.

Jörg

On Mon, Sep 15, 2014 at 5:44 PM, John Smith  wrote:

> Using ES 1.3.2
>
> The current application I'm building only uses term queries for exact
> matches.
>
> Example query
>
>   "query": {
> "term": {
>   "logType": "abc"
> }
>
>
> The field logType is pulled from external DB as all caps so for instance
> ABC
>
> If i send the query
>
>   "query": {
> "term": {
>   "logType": "ABC"
> }
>
>
> I get no results.
>
> If I send the query
>
>   "query": {
> "term": {
>   "logType": "abc"
> }
>
> I get results.
>
> So does this mean I need to toLower the input before building the query or
> there's an ES way of doing this?
>
> Thanks
>
>
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/8fec1f56-1a59-4d17-bb01-f2ecd62bfbb4%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHi-483Tgf5DmwXEfKrGRye1vjvGU4jFL0o8guNsGBaCw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to do case insensitive search on terms?

[Nikolas Everett]

Or if you want case insensitive search use a match query

.

On Mon, Sep 15, 2014 at 11:47 AM, joergpra...@gmail.com <
joergpra...@gmail.com> wrote:

> I assume you use the standard analyzer which uses by default a token
> filter "lowercase".
>
> Just use a custom analyzer, without "lowercase" token filter, and you will
> get case-sensitive search.
>
> Jörg
>
> On Mon, Sep 15, 2014 at 5:44 PM, John Smith 
> wrote:
>
>> Using ES 1.3.2
>>
>> The current application I'm building only uses term queries for exact
>> matches.
>>
>> Example query
>>
>>   "query": {
>> "term": {
>>   "logType": "abc"
>> }
>>
>>
>> The field logType is pulled from external DB as all caps so for instance
>> ABC
>>
>> If i send the query
>>
>>   "query": {
>> "term": {
>>   "logType": "ABC"
>> }
>>
>>
>> I get no results.
>>
>> If I send the query
>>
>>   "query": {
>> "term": {
>>   "logType": "abc"
>> }
>>
>> I get results.
>>
>> So does this mean I need to toLower the input before building the query
>> or there's an ES way of doing this?
>>
>> Thanks
>>
>>
>>
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/8fec1f56-1a59-4d17-bb01-f2ecd62bfbb4%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHi-483Tgf5DmwXEfKrGRye1vjvGU4jFL0o8guNsGBaCw%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3q6SNUP6gBDADV802JGSZyO0M6%2BDKP%2BBvBNRwaEr8p0A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to do case insensitive search on terms?

[joergpra...@gmail.com]

Ugh. Read my post exactly the other way round.

=> standard analyzer is *not* using token filter "lowercase"

=> you can build a custom analyzer *with* token filter 'lowercase" for
*case-insensitive* search.

Jörg

On Mon, Sep 15, 2014 at 5:47 PM, joergpra...@gmail.com <
joergpra...@gmail.com> wrote:

> I assume you use the standard analyzer which uses by default a token
> filter "lowercase".
>
> Just use a custom analyzer, without "lowercase" token filter, and you will
> get case-sensitive search.
>
> Jörg
>
> On Mon, Sep 15, 2014 at 5:44 PM, John Smith 
> wrote:
>
>> Using ES 1.3.2
>>
>> The current application I'm building only uses term queries for exact
>> matches.
>>
>> Example query
>>
>>   "query": {
>> "term": {
>>   "logType": "abc"
>> }
>>
>>
>> The field logType is pulled from external DB as all caps so for instance
>> ABC
>>
>> If i send the query
>>
>>   "query": {
>> "term": {
>>   "logType": "ABC"
>> }
>>
>>
>> I get no results.
>>
>> If I send the query
>>
>>   "query": {
>> "term": {
>>   "logType": "abc"
>> }
>>
>> I get results.
>>
>> So does this mean I need to toLower the input before building the query
>> or there's an ES way of doing this?
>>
>> Thanks
>>
>>
>>
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/8fec1f56-1a59-4d17-bb01-f2ecd62bfbb4%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEHpcSbsbC3S%3DMd9KeCZP0ikhPtCBZu-fVJ8NZod33THg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Naive question about multiple clients

[Jack Park]

Hi David,

I am not certain how to interpret your question. Here is the precise
scenario:
The node client is driving this website: http://www.topicquests.org:3000/
 which is a very early-stage experiment in developing online collective
intelligence tools, the theory behind which is sketched in these slides
http://www.slideshare.net/jackpark/issip20140910

But, this is all "topic map" based, which means that there are background
agents running which are mostly Java/Scala/Clojure based. Those agents
receive TCP-sent messages from the node-based server when a new topic is
created; they then set about to study that new document and perform various
processes on the index in the background.

Right now, my java ES client is giving me fits mostly associated with a
migration from 0.90 to 1.2.2 , which Ill mention in a different thread.

On Mon, Sep 15, 2014 at 7:56 AM, David Pilato  wrote:

> Why not using the JavaScript client?
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
> Le 15 sept. 2014 à 16:39, Jack Park  a écrit :
>
> Thanks!
> I have the system running under nodejs, but want to start a java client
> against the same cluster (right now, nodejs is talking to it over http: the
> java client wants to talk to it by way of embedded, though still over the
> localhost socket.
>
> On Mon, Sep 15, 2014 at 7:22 AM, David Pilato  wrote:
>
>> Yes. You can have many client talking to es cluster at the same time.
>> Wondering if you are talking about Node Client here. Which is is
>> basically a node running inside the cluster and which can provide access to
>> Client features.
>>
>> If you use Java Transport Client, I don't think any file is generated.
>> If you use other language clients (PHP, Python, Perl…), I can't see that
>> kind of file either.
>>
>> About Java, note that it's a good idea to share the same Client across
>> all threads. Java client is thread safe.
>>
>>
>> Hope this helps
>>
>> --
>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com
>> *
>> @dadoonet  | @elasticsearchfr
>> 
>>
>>
>> Le 15 septembre 2014 à 15:58:29, Jack Park (jackp...@topicquests.org) a
>> écrit:
>>
>> A really simple question is this: can more than one client be talking to
>> an ES cluster at the same time?  I ask because I notice that there is a
>> lock file in a shard directory.
>>
>> Thanks!
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAH6s0fy%3DgQtfSNRdYxH7SV_Jw%2BJ7VsTMWKt-PfTzKyBuyALT_g%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/etPan.5416f60e.77465f01.3c6%40MacBook-Air-de-David.local
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAH6s0fyFRN2yDA%2B%3DDdUkj5tJqhRSrOP65_eSTeV5QJK5AV1ZcA%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/D12FFA2C-3687-4DB7-9C6B-5C36F14D192B%40pilato.fr
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"e

Revisiting query parse (no field after start_object) errors

[Jack Park]

This query:

{
"from": 0,
"size": 30,
"query": {
"term": {
"sbOf": "TypeType"
}
}
}

is giving me fits. I recently migrated from 0.90 to 1.2.2 and went through
this with my node.js client, got many ideas in this forum and solved the
problem. The query above works fine from node.js, but fails  (listing
below) from the java client running against the same index as the node
client.

I realize this is a potentially high-dimensional issue with many actors
other than the query string in play. Just looking for hints.

I have tested with and without "query". No difference.

Many thanks in advance
Jack

listing:
ERROR 2014-09-15 08:28:50,882 [main] - Failed to execute phase [query], all
shards failed; shardFailures {[vIdL7fWlT1en9lKkGS8NyA][topics][2]:
SearchParseException[[topics][2]: from[-1],size[-1]: Parse Failure [Failed
to parse source
[{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
nested: QueryParsingException[[topics] [_na] query malformed, no field
after start_object]; }{[vIdL7fWlT1en9lKkGS8NyA][topics][3]:
SearchParseException[[topics][3]: from[-1],size[-1]: Parse Failure [Failed
to parse source
[{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
nested: QueryParsingException[[topics] [_na] query malformed, no field
after start_object]; }{[vIdL7fWlT1en9lKkGS8NyA][topics][0]:
SearchParseException[[topics][0]: from[-1],size[-1]: Parse Failure [Failed
to parse source
[{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
nested: QueryParsingException[[topics] [_na] query malformed, no field
after start_object]; }{[vIdL7fWlT1en9lKkGS8NyA][topics][1]:
SearchParseException[[topics][1]: from[-1],size[-1]: Parse Failure [Failed
to parse source
[{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
nested: QueryParsingException[[topics] [_na] query malformed, no field
after start_object]; }{[vIdL7fWlT1en9lKkGS8NyA][topics][4]:
SearchParseException[[topics][4]: from[-1],size[-1]: Parse Failure [Failed
to parse source
[{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
nested: QueryParsingException[[topics] [_na] query malformed, no field
after start_object]; }
org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to
execute phase [query], all shards failed; shardFailures
{[vIdL7fWlT1en9lKkGS8NyA][topics][2]: SearchParseException[[topics][2]:
from[-1],size[-1]: Parse Failure [Failed to parse source
[{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
nested: QueryParsingException[[topics] [_na] query malformed, no field
after start_object]; }{[vIdL7fWlT1en9lKkGS8NyA][topics][3]:
SearchParseException[[topics][3]: from[-1],size[-1]: Parse Failure [Failed
to parse source
[{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
nested: QueryParsingException[[topics] [_na] query malformed, no field
after start_object]; }{[vIdL7fWlT1en9lKkGS8NyA][topics][0]:
SearchParseException[[topics][0]: from[-1],size[-1]: Parse Failure [Failed
to parse source
[{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
nested: QueryParsingException[[topics] [_na] query malformed, no field
after start_object]; }{[vIdL7fWlT1en9lKkGS8NyA][topics][1]:
SearchParseException[[topics][1]: from[-1],size[-1]: Parse Failure [Failed
to parse source
[{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
nested: QueryParsingException[[topics] [_na] query malformed, no field
after start_object]; }{[vIdL7fWlT1en9lKkGS8NyA][topics][4]:
SearchParseException[[topics][4]: from[-1],size[-1]: Parse Failure [Failed
to parse source
[{"query_binary":"eyJmcm9tIjowLCJzaXplIjozMCwicXVlcnkiOnsidGVybSI6eyJzYk9mIjoiVHlwZVR5cGUifX19"}]]];
nested: QueryParsingException[[topics] [_na] query malformed, no field
after start_object]; }
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:233)

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAH6s0fzRqQpTCc83N_vdwT15eAS-%2BH6OzUQqCWR2FWZsgnsNKQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

How to do case insensitive search on terms?

[John Smith]

Using ES 1.3.2

The current application I'm building only uses term queries for exact 
matches.

Example query

  "query": {
"term": {
  "logType": "abc"
}


The field logType is pulled from external DB as all caps so for instance ABC

If i send the query

  "query": {
"term": {
  "logType": "ABC"
}


I get no results.

If I send the query

  "query": {
"term": {
  "logType": "abc"
}

I get results.

So does this mean I need to toLower the input before building the query or 
there's an ES way of doing this?

Thanks




-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8fec1f56-1a59-4d17-bb01-f2ecd62bfbb4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Returning exact matches with ElasticSearch 'Lite'

[joergpra...@gmail.com]

Yes, you can select a tokenizer for controlling how to split words.

If you index "bar-baz", you index two words, bar and baz, because standard
analyzer delimits tokens according to Unicode text segmentation rules in
http://unicode.org/reports/tr29/

So the easiest method to use standard tokenizer is probably by phrase
search, and it works like this:

PUT /test/doc/1
{
"foo" : "bar"
}

PUT /test/doc/2
{
"foo" : "bar-baz"
}

GET /test/_refresh

GET /test/_search?q=%22bar%2Dbaz%22

Jörg

On Mon, Sep 15, 2014 at 5:04 PM, Neil Middleton  wrote:

> Percent encoding the string appears to make no difference.  Passing in
> "+foo:bar\-baz” or "%2Bfoo%3Abar-baz” yields exactly the same resultset.
>
> Is this potentially to do with the analysis of the strings in question?
>  For instance - I assume by not analysing the `foo` field the propensity to
> return ‘similar’ results will be removed as ES won’t have done the analysis
> on the parts of the word?
>
> For instance: `/_analyze?text=foo-bar` yields something different to say
> `/_analyze?text=foo-bar&analyzer=whitespace`.
>
> —
> N
>
>
> On Mon, Sep 15, 2014 at 3:12 PM, joergpra...@gmail.com <
> joergpra...@gmail.com> wrote:
>
>> Use URI escaping (percent encoding). It is perfectly described on the
>> page
>> http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/search-lite.html
>>
>> Jörg
>>
>>
>>
>> On Mon, Sep 15, 2014 at 2:03 PM, Neil Middleton  wrote:
>>
>>>  I'm trying to use ES's search lite
>>> 
>>>  to
>>> query a dataset which contains a key foo. An example of this is:
>>>
>>> .../_search?q=+foo:bar\-baz
>>>
>>> Now, my understanding is is that the addition of + to the foo attribute
>>> should make this search return only exactly matches where foo is bar-baz.
>>> However, I'm finding that I'm also getting other results coming back, for
>>> instance foo-baz or baz-baz etc.
>>>
>>> What am I doing wrong here which is causing these other results to be
>>> returned?   How can I resolve this?
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/f77af3af-8615-44c2-9d86-f9b6c1692f63%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/X4wS16I10Ko/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF7%2BF-sQpm2ypm3YWfVgu7H%2BbO6vEAPbw9e4m824Hd8Aw%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/1410793495416.33166361%40Nodemailer
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGnLQ1d5AZQym9n1wWsY_aC0hfWtZRfouRqHK2jsA%3DdSw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Naive question about multiple clients

[David Pilato]

Why not using the JavaScript client?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

> Le 15 sept. 2014 à 16:39, Jack Park  a écrit :
> 
> Thanks!
> I have the system running under nodejs, but want to start a java client 
> against the same cluster (right now, nodejs is talking to it over http: the 
> java client wants to talk to it by way of embedded, though still over the 
> localhost socket.
> 
>> On Mon, Sep 15, 2014 at 7:22 AM, David Pilato  wrote:
>> Yes. You can have many client talking to es cluster at the same time.
>> Wondering if you are talking about Node Client here. Which is is basically a 
>> node running inside the cluster and which can provide access to Client 
>> features.
>> 
>> If you use Java Transport Client, I don't think any file is generated.
>> If you use other language clients (PHP, Python, Perl…), I can't see that 
>> kind of file either.
>> 
>> About Java, note that it's a good idea to share the same Client across all 
>> threads. Java client is thread safe.
>> 
>> 
>> Hope this helps
>> 
>> -- 
>> David Pilato | Technical Advocate | Elasticsearch.com
>> @dadoonet | @elasticsearchfr
>> 
>> 
>>> Le 15 septembre 2014 à 15:58:29, Jack Park (jackp...@topicquests.org) a 
>>> écrit:
>>> 
>>> A really simple question is this: can more than one client be talking to an 
>>> ES cluster at the same time?  I ask because I notice that there is a lock 
>>> file in a shard directory.
>>> 
>>> Thanks!
>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/CAH6s0fy%3DgQtfSNRdYxH7SV_Jw%2BJ7VsTMWKt-PfTzKyBuyALT_g%40mail.gmail.com.
>>> For more options, visit https://groups.google.com/d/optout.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/etPan.5416f60e.77465f01.3c6%40MacBook-Air-de-David.local.
>> For more options, visit https://groups.google.com/d/optout.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/CAH6s0fyFRN2yDA%2B%3DDdUkj5tJqhRSrOP65_eSTeV5QJK5AV1ZcA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/D12FFA2C-3687-4DB7-9C6B-5C36F14D192B%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Re: Returning exact matches with ElasticSearch 'Lite'

[Neil Middleton]

Percent encoding the string appears to make no difference.  Passing in 
"+foo:bar\-baz” or "%2Bfoo%3Abar-baz” yields exactly the same resultset.




Is this potentially to do with the analysis of the strings in question?  For 
instance - I assume by not analysing the `foo` field the propensity to return 
‘similar’ results will be removed as ES won’t have done the analysis on the 
parts of the word?




For instance: `/_analyze?text=foo-bar` yields something different to say 
`/_analyze?text=foo-bar&analyzer=whitespace`.



—
N

On Mon, Sep 15, 2014 at 3:12 PM, joergpra...@gmail.com
 wrote:

> Use URI escaping (percent encoding). It is perfectly described on the page
> http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/search-lite.html
> Jörg
> On Mon, Sep 15, 2014 at 2:03 PM, Neil Middleton  wrote:
>> I'm trying to use ES's search lite
>> 
>>  to
>> query a dataset which contains a key foo. An example of this is:
>>
>> .../_search?q=+foo:bar\-baz
>>
>> Now, my understanding is is that the addition of + to the foo attribute
>> should make this search return only exactly matches where foo is bar-baz.
>> However, I'm finding that I'm also getting other results coming back, for
>> instance foo-baz or baz-baz etc.
>>
>> What am I doing wrong here which is causing these other results to be
>> returned?   How can I resolve this?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/f77af3af-8615-44c2-9d86-f9b6c1692f63%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> -- 
> You received this message because you are subscribed to a topic in the Google 
> Groups "elasticsearch" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/elasticsearch/X4wS16I10Ko/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF7%2BF-sQpm2ypm3YWfVgu7H%2BbO6vEAPbw9e4m824Hd8Aw%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1410793495416.33166361%40Nodemailer.
For more options, visit https://groups.google.com/d/optout.

Re: Naive question about multiple clients

[Jack Park]

Thanks!
I have the system running under nodejs, but want to start a java client
against the same cluster (right now, nodejs is talking to it over http: the
java client wants to talk to it by way of embedded, though still over the
localhost socket.

On Mon, Sep 15, 2014 at 7:22 AM, David Pilato  wrote:

> Yes. You can have many client talking to es cluster at the same time.
> Wondering if you are talking about Node Client here. Which is is basically
> a node running inside the cluster and which can provide access to Client
> features.
>
> If you use Java Transport Client, I don't think any file is generated.
> If you use other language clients (PHP, Python, Perl…), I can't see that
> kind of file either.
>
> About Java, note that it's a good idea to share the same Client across all
> threads. Java client is thread safe.
>
>
> Hope this helps
>
> --
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
> @dadoonet  | @elasticsearchfr
> 
>
>
> Le 15 septembre 2014 à 15:58:29, Jack Park (jackp...@topicquests.org) a
> écrit:
>
> A really simple question is this: can more than one client be talking to
> an ES cluster at the same time?  I ask because I notice that there is a
> lock file in a shard directory.
>
> Thanks!
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAH6s0fy%3DgQtfSNRdYxH7SV_Jw%2BJ7VsTMWKt-PfTzKyBuyALT_g%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/etPan.5416f60e.77465f01.3c6%40MacBook-Air-de-David.local
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAH6s0fyFRN2yDA%2B%3DDdUkj5tJqhRSrOP65_eSTeV5QJK5AV1ZcA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [JAVA API] Retrieve all indexes

[Luan Garrido]

Thank you for your healp Vineeth 

Em segunda-feira, 15 de setembro de 2014 08h22min40s UTC-3, Luan Garrido 
escreveu:
>
> Hi all,
> First of all I took a look and did not find the answer for my question.
> Can anyone teach me how can I find all indexes from a cluster using the 
> JAVA API? 
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/29528fd4-7f84-4a4a-b689-1b7665a4714b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Naive question about multiple clients

[David Pilato]

Yes. You can have many client talking to es cluster at the same time.
Wondering if you are talking about Node Client here. Which is is basically a 
node running inside the cluster and which can provide access to Client features.

If you use Java Transport Client, I don't think any file is generated.
If you use other language clients (PHP, Python, Perl…), I can't see that kind 
of file either.

About Java, note that it's a good idea to share the same Client across all 
threads. Java client is thread safe.


Hope this helps

-- 
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr


Le 15 septembre 2014 à 15:58:29, Jack Park (jackp...@topicquests.org) a écrit:

A really simple question is this: can more than one client be talking to an ES 
cluster at the same time?  I ask because I notice that there is a lock file in 
a shard directory.

Thanks!
--
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAH6s0fy%3DgQtfSNRdYxH7SV_Jw%2BJ7VsTMWKt-PfTzKyBuyALT_g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/etPan.5416f60e.77465f01.3c6%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

Re: Returning exact matches with ElasticSearch 'Lite'

[joergpra...@gmail.com]

Use URI escaping (percent encoding). It is perfectly described on the page
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/search-lite.html

Jörg



On Mon, Sep 15, 2014 at 2:03 PM, Neil Middleton  wrote:

> I'm trying to use ES's search lite
> 
>  to
> query a dataset which contains a key foo. An example of this is:
>
> .../_search?q=+foo:bar\-baz
>
> Now, my understanding is is that the addition of + to the foo attribute
> should make this search return only exactly matches where foo is bar-baz.
> However, I'm finding that I'm also getting other results coming back, for
> instance foo-baz or baz-baz etc.
>
> What am I doing wrong here which is causing these other results to be
> returned?   How can I resolve this?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/f77af3af-8615-44c2-9d86-f9b6c1692f63%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF7%2BF-sQpm2ypm3YWfVgu7H%2BbO6vEAPbw9e4m824Hd8Aw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Naive question about multiple clients

[Jack Park]

A really simple question is this: can more than one client be talking to an
ES cluster at the same time?  I ask because I notice that there is a lock
file in a shard directory.

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAH6s0fy%3DgQtfSNRdYxH7SV_Jw%2BJ7VsTMWKt-PfTzKyBuyALT_g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: 'NoNodeAvailable' exception JMeter 10K requests

[joergpra...@gmail.com]

Nodes did not respond, and if the last node disconnected and does not
respond, the TransportClient reports in his log "no node available".

You have to look in the logs of the cluster nodes, there is more
information.

Jörg


On Mon, Sep 15, 2014 at 3:51 PM, Vidosh Sahu  wrote:

> Hi All,
>
> I am trying to do some es perf analysis. I have an AWS machine [quad core
> + 16 gigs]. I have started the ES with 8 gigs. version 1.21
>
> The ES server is a single cluster multi node setup. There are 9 nodes as
> of now.
> 5M nodes - There are 3 nodes having 5M records each with 1/2/4 shard
> config.
> 10M nodes - There are 3 nodes having 10M records each with 1/2/4 shard
> config.
> 20M nodes - There are 3 nodes having 20M records each with 1/2/4 shard
> config.
>
> Now i have written a java class extending 'AbstractJavaSamplerClient' [a
> Jmeter Custom java sampler]which creates some search requests.
> Thing is that it works fine for almost 2500 requests but anything more
> than that, it starts throwing the 'NoNodeAvailable' exception.
>
> This is a jmeter test case that i am running in the non-gui mode. I have
> not used any heavy graphical listeners. I am using 'Aggregate Report'
> listener. But for debugging i have added 'View Results Tree' listener as
> well so that i know what happened to each request. This happens in both the
> cases [the failure after 2500 requests] if i use or don't use the VRT
> listener.
>
> This is the ex stack-trace
> Response message: Exception: No node available
> at
> org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:219)
> at
> org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
> at
> org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:212)
> at
> org.elasticsearch.client.transport.TransportClient.search(TransportClient.java:420)
> at
> org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:1071)
> at
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:85)
> at
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59)
> at
> com.readypulse.utils.ElasticSearchSamplerClient.runTest(ElasticSearchSamplerClient.java:59)
> at
> org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:191)
> at
> org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:429)
> at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:257)
> at java.lang.Thread.run(Thread.java:745)
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/574a3474-d5c3-4d21-8a9e-5abb674f5264%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEcH_pnggU%3DXd_aMgskQeT0DnBxfHPV92g9%2B21jcoT4jw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Seeking opinions on cluster platforms

[Jack Park]

Mark and Jörg

Many thanks for these comments. In a large sense, they confirm my
intuitions.

Cheers,
Jack

On Fri, Sep 12, 2014 at 4:00 PM, joergpra...@gmail.com <
joergpra...@gmail.com> wrote:

> Not sure what is "extreme". The design of ES may be a surprise for those
> who are not familiar with distributed system architecture.
>
> ES can handle faults in software. I pile up cheap 1U rack servers with 32
> cores, 64G RAM, ~1TB RAID 0. All nodes are equally provisioned.
>
> If a server fails, mostly spindle drives or fans, it is decommissioned and
> repaired.
>
> No need to monitor for master failure or making backups. Master is
> switched over automatically by ES, and replica level 1 (or higher) is a
> must.
>
> Jörg
>
>
> On Sat, Sep 13, 2014 at 12:37 AM, Mark Walkom 
> wrote:
>
>> Personally, I'd go with the latter and then let the software handle all
>> the redundancy. You can get super cheap 1RU pizza boxes from Quanta or the
>> like and save yourself a bundle in that area and then leverage automation
>> and configuration using The Foreman and Puppet.
>>
>> Tie a bit more smarts into it and you would have an awesome elastic
>> compute platform. Or just use something like OpenStack, though it might be
>> a bit heavy.
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>>
>> On 13 September 2014 03:16, Jack Park  wrote:
>>
>>> Let me pose a question by suggesting two extremes for hardware to create
>>> and maintain a growing ElasticSearch cluster datacenter (not in the cloud).
>>>
>>> One extreme places redundancy at the server hardware level, by which I
>>> mean:
>>> dual power supplies, RAID hard drives
>>>
>>> Another extreme places redundancy in a multitude of backup servers:
>>> commodity servers, single power supply, no RAID on the disks, low cost,
>>> with a cluster monitor that can advise of a failed master or backup, and
>>> can rebuild the replacement
>>>
>>> I would love to learn how others see or implement within the boundaries
>>> of those extremes, with the understanding that the two poles are just
>>> suggestions, there may be other ways to slice this space.
>>>
>>> Many thanks in advance
>>> Jack
>>> ps: documents I read based on a broad query:
>>> https://github.com/aphyr/partitions-post
>>> http://www.elasticsearch.org/case-study/maptimize/
>>>
>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html
>>>
>>> http://www.slideshare.net/clintongormley/scaling-realtime-search-and-analytics-with-elasticsearch
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/CAH6s0fyHDhB9N8rOCwuf%2B3GR1E8xQ4aqSoQD8cYKZwo72bHw7A%40mail.gmail.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAEM624bho%2BcJStsAAMvx5ZMApNEqCSz3a4oEofrU7VfEeuVX%2Bg%40mail.gmail.com
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFuXTUVQrhKpueJWKPT2ws4CtXm6mB-4rGLB-0DFkC37w%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAH6s0fxkMS9cm5eY7c2j2SkFFwUdvEruWCNX2U_8Zhhr%3DQ3T3A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

'NoNodeAvailable' exception JMeter 10K requests

[Vidosh Sahu]

Hi All,

I am trying to do some es perf analysis. I have an AWS machine [quad core + 
16 gigs]. I have started the ES with 8 gigs. version 1.21

The ES server is a single cluster multi node setup. There are 9 nodes as of 
now.
5M nodes - There are 3 nodes having 5M records each with 1/2/4 shard config.
10M nodes - There are 3 nodes having 10M records each with 1/2/4 shard 
config.
20M nodes - There are 3 nodes having 20M records each with 1/2/4 shard 
config.

Now i have written a java class extending 'AbstractJavaSamplerClient' [a 
Jmeter Custom java sampler]which creates some search requests.
Thing is that it works fine for almost 2500 requests but anything more than 
that, it starts throwing the 'NoNodeAvailable' exception.

This is a jmeter test case that i am running in the non-gui mode. I have 
not used any heavy graphical listeners. I am using 'Aggregate Report' 
listener. But for debugging i have added 'View Results Tree' listener as 
well so that i know what happened to each request. This happens in both the 
cases [the failure after 2500 requests] if i use or don't use the VRT 
listener.

This is the ex stack-trace
Response message: Exception: No node available
at 
org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:219)
at 
org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
at 
org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:212)
at 
org.elasticsearch.client.transport.TransportClient.search(TransportClient.java:420)
at 
org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:1071)
at 
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:85)
at 
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59)
at 
com.readypulse.utils.ElasticSearchSamplerClient.runTest(ElasticSearchSamplerClient.java:59)
at 
org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:191)
at 
org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:429)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:257)
at java.lang.Thread.run(Thread.java:745)

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/574a3474-d5c3-4d21-8a9e-5abb674f5264%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Sorting search results

[matej . zerovnik]

Hello!

I'm trying to create a query, that would return the last(sorted by 
timestamp) 10 hits. I'm using logstash to parse and index my log files...

I tried 2 different queries:
{
"query" : {
"filtered" : {
"query":   {"match" : {"user" : "abc"}},
"query":   {"match" : {"login_succesful" : "true"}}
}
},
"sort": [
{ "timestamp":   { "order": "desc" }}
]
}

and another one
{
  "query" : {
"bool" : {
  "must" : [
{"match" : {"user" : "abc"}},
{"match" : {"login_succesful" : "true"}}
  ]
},
  "sort" : [
{ "timestamp": { "order": "desc" }
  ]
  }
}

The first one lists all hits and doesn't sort and the 2nd one gives me an 
error saying "Parse Failure [No parser for element [timestamp]]]; }]".

What am I doing wrong?
Should I define 'timestamp' field as date? Currently, all fields are mapped 
as "string, index, analyzed"...

Also, which one of the queries is preferred? Filtered or query?
How to apply "must" condition in the 1st search?

Matej

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e956a1d4-a200-48c5-9841-2072e12cc149%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Aggregation framework, Java API

[Emanuel Buzek]

Ivan, thanks a lot for the reply, I switched to FilteredQuery (using 
matchAll when no query is submitted) and that simplified my code a lot. It 
also makes more sense than using post filters and filtered aggregations...
best, emanuel

Dne úterý, 9. září 2014 18:46:25 UTC+2 Ivan Brusic napsal(a):
>
> A filtered query with no explicit query will ultimately be translated into 
> a match-all/constant-score query at the Lucene level. I prefer to 
> explicitly define all my match all queries and use the specific post filter 
> name, and not the old filter name, which was deprecated due to its 
> ambiguity.
>
> Besides, even if you did not have aggregations, you want to do as much 
> pre-filtered as you can. Post filters work on documents that have been 
> scored. No need to score documents that will eventually be filtered. Post 
> filters have some benefits, but it seems like they do not apply in this 
> case.
>
> -- 
> Ivan
>
>
> On Tue, Sep 9, 2014 at 2:26 AM, Emanuel Buzek  > wrote:
>
>> Thanks Ivan.
>>
>> Yes, it was the post filter which was ignored. We use filtered query only 
>> when the user sends a query string, otherwise (when only exact filters for 
>> specific columns are specified) we use the post filter. It seems strange to 
>> me to use the FilteredQuery when the query string is empty, but perhaps 
>> that would be the most straight forward way of doing this.
>>
>> thank you,
>> emanuel
>>
>> Dne pondělí, 8. září 2014 17:21:21 UTC+2 Ivan Brusic napsal(a):
>>>
>>> Which filter was ignored? I am assuming you meant the post filter (which 
>>> might be still called filter at the Java API), which in this case the 
>>> filter is bypassed by design. Post filters allow you to filter the 
>>> documents returned, but leave the aggregations as is. Sounds like you are 
>>> looking for filtered queries. The method name is ambiguous, which is why it 
>>> has been renamed (and should actually be deprecated in the API). 
>>>
>>> Best way to learn the Java API is via the unit tests, but I do agree, 
>>> there is no clean way to write elegant code due to explicit casting.
>>>
>>> https://github.com/elasticsearch/elasticsearch/
>>> tree/master/src/test/java/org/elasticsearch/search/aggregations
>>>
>>> Cheers,
>>>
>>> Ivan
>>>
>>>
>>> On Mon, Sep 8, 2014 at 5:41 AM, mooky  wrote:
>>>
 The aggregation takes into account a query - but not a post-filter. I'm 
 not sure of the rationale behind the difference.

 The java api for traversing results is quite painful - but I think a 
 good part of that is due to Java & the fact that there is very little 
 polymorphic behaviour between aggregation results (some have single 
 results, others have buckets, some have sub-aggregations, some dont).
 The only alternative that I can think of is a completely type-less 
 navigation of the data - which does little more than navigate the JSON 
 document.

 Hope that helps a bit.



 On Monday, 8 September 2014 10:26:44 UTC+1, Emanuel Buzek wrote:
>
> Hi there,
> I just used the elasticsearch aggregations through the Java API for 
> the first time.
>
> All I wanted was a simple min/max/sum/avg, so I used the Stats 
> aggregation. However, I was very surprised that the filter in the 
> SearchRequestBuilder is ignored, so I had to wrap the Stats Aggregation 
> into FilterAggregation.
>
> Getting the aggregation result seems a bit tedious:
>
> InternalStats stats = (InternalStats)((InternalFilter)a).
> getAggregations().asList().get(0);
>
> Maybe I am using the Java API wrong (I hope I am, otherwise it's imho 
> poorly designed.) Can anyone point me to an example how to access the 
> aggregation results from Java better?
>
>
> Also, I think that the aggregation should be filtered by default. If I 
> specify the filter with a query or post filter:
>
> queryBuilder = QueryBuilders.filteredQuery(queryBuilder, 
> filterBuilder);
>
>   searchRequestBuilder.setQuery(queryBuilder);
>
> and then add an aggregation GET to the same searchRequestBuilder, I 
> think it's very unintuitive if the aggregation is computed globally. 
> Anyone 
> has this feeling as well?
>
> thanks, emanuel
>
> -- 
> Emanuel Buzek
> Software Engineer, ROKE.cz 
> tel: +420 776 54 26 26
>  
  -- 
 You received this message because you are subscribed to the Google 
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/a2e3a1e1-912a-4257-b938-6036d0c9d3ff%
 40googlegroups.com 
 

When does a deleted index get removed from disk?

[Neil Andrassy]

Hi,

We recently created a lot of (date suffixed) indexes (about 3,000) on our 
development cluster (1.3.2). Subsequently we deleted many of these indexes. 
The cluster ran, predominantly green, for several days, certainly more than 
a week after that with no sign of the indexes. After that we performed a 
cluster restart and the "recover dangling indexes" functionality 
unexpectedly re-animated all of the deleted indexes. We (think) we 
understand the dangling index recovery functionality (and could potentially 
disable it - but not sure if we really want to go down that road though). 

So, the question is, at what point does an index deleted through the API 
get cleaned from individual node disks? At what stage will a deleted index 
no longer be recovered as dangling? Is this zombie functionality by design? 
Has a physical part of the delete failed in our cluster? Do we need silver 
bullets to kill our indexes once and for all?

The extra problem we faced in this scenario is that the recovered indexes 
placed our elected master under quite a bit of stress, rendering our 
cluster unusable :(

Thanks in advance for any insight,

Neil

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/fb2d385d-1143-48a0-97e5-0b1b90776723%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Returning exact matches with ElasticSearch 'Lite'

[Neil Middleton]

I'm trying to use ES's search lite 

 to 
query a dataset which contains a key foo. An example of this is:

.../_search?q=+foo:bar\-baz

Now, my understanding is is that the addition of + to the foo attribute 
should make this search return only exactly matches where foo is bar-baz. 
However, I'm finding that I'm also getting other results coming back, for 
instance foo-baz or baz-baz etc.

What am I doing wrong here which is causing these other results to be 
returned?   How can I resolve this?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f77af3af-8615-44c2-9d86-f9b6c1692f63%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [JAVA API] Retrieve all indexes

[vineeth mohan]

Hello Luan ,

There are many ways to do it -

Client client = new TransportClient()
.addTransportAddress(new InetSocketTransportAddress("host1", 9300));
GetSettingsRequest req = new GetSettingsRequest();;
GetSettingsResponse response =
client.admin().indices().getSettings(req).actionGet();
ImmutableOpenMap settings = response.getIndexToSettings();
System.out.println("Indices are " + settings.keys());

Thanks
  Vineeth

On Mon, Sep 15, 2014 at 4:52 PM, Luan Garrido  wrote:

> Hi all,
> First of all I took a look and did not find the answer for my question.
> Can anyone teach me how can I find all indexes from a cluster using the
> JAVA API?
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/adf8f2c2-311e-4732-8c5e-97835798e573%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5karShQcL%2Bp0VFfCKgM7tUzmWhdCGfjca3ah7YSedOfTQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to connect elasticsearch instance of docker VMs from two different machine?

[Mark Walkom]

Your problem is a docker networking issue, not an ES one.
I'm not familiar with docker so can't help much more sorry.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 15 September 2014 21:34, Anand kumar  wrote:

>
> Hi Mark,
>
> Thanks for your reply.
>
> Yeah, both the hosts belonged to same network.
> I cant able to use telnet, instead I tried ssh.
>
>
> On Monday, 15 September 2014 16:36:53 UTC+5:30, Mark Walkom wrote:
>>
>> Are they on the same network?
>> Can you telnet to each of the hosts?
>>
>> It looks like the two nodes don't have access to each other, hence the no
>> route to host error.
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>> On 15 September 2014 21:02, Anand kumar  wrote:
>>
>>>
>>> Hi all,
>>>
>>>  I'm trying to connect two ES instances using docker VM and both as
>>> master nodes, but the instances belong to two different hosts from the same
>>> network.
>>>
>>> My docker commands are as below:
>>>
>>> host1 : $ docker run -t -i --name="master-node-01" -p 9200:9200 -p
>>> 9300:9300
>>> host2 : $ docker run -t -i --name="master-node-02" -p 9200:9200 -p
>>> 9300:9300 (as the host were different, am using the same port for both)
>>>
>>> And I used the configurations as follows:
>>>
>>> host1 : discovery.zen.ping.unicast.hosts: ["host2"]
>>> host2 : discovery.zen.ping.unicast.hosts: ["host1"]
>>>
>>> When i tried to start the host2 instance after starting the host1, I
>>> cant able connect the host2 with the host1
>>>
>>> and tried some other configurations like
>>>
>>> host1 : discovery.zen.ping.unicast.hosts: ["host2:9200"]
>>> host2 : discovery.zen.ping.unicast.hosts: ["host1:9200"]
>>>
>>> host1 : discovery.zen.ping.unicast.hosts: ["xxx.xxx.xx.xx:9200"]
>>> host2 : discovery.zen.ping.unicast.hosts: ["xxx.xxx.xx.xx:9200"]
>>>
>>> But I'm getting the following issues, Is it possible to connect the VMs
>>> of different host through ES
>>>
>>> [2014-09-15 09:46:46,920][WARN ][transport.netty  ]
>>> [ES_MASTER_NODE_1] exception caught on transport layer [[id: 0x2f06682e]],
>>> closing connection
>>> java.net.NoRouteToHostException: No route to host
>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>> at sun.nio.ch.SocketChannelImpl.finishConnect(
>>> SocketChannelImpl.java:708)
>>> at org.elasticsearch.common.netty.channel.socket.nio.
>>> NioClientBoss.connect(NioClientBoss.java:150)
>>> at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.
>>> processSelectedKeys(NioClientBoss.java:105)
>>> at org.elasticsearch.common.netty.channel.socket.nio.
>>> NioClientBoss.process(NioClientBoss.java:79)
>>> at org.elasticsearch.common.netty.channel.socket.nio.
>>> AbstractNioSelector.run(AbstractNioSelector.java:318)
>>> at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(
>>> NioClientBoss.java:42)
>>> at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
>>> ThreadRenamingRunnable.java:108)
>>> at org.elasticsearch.common.netty.util.internal.
>>> DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>> ThreadPoolExecutor.java:1145)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:724)
>>>
>>>
>>> Any suggestion would be helpful. Thanks in advance.
>>>
>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/d7450d33-b569-493c-8391-e3c68be0cd02%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/4070c285-0b91-43db-b9ee-61e674202a83%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion o

Re: How to connect elasticsearch instance of docker VMs from two different machine?

[Anand kumar]

Thank you Mark for your precious time.

On Monday, 15 September 2014 17:08:48 UTC+5:30, Mark Walkom wrote:
>
> Your problem is a docker networking issue, not an ES one.
> I'm not familiar with docker so can't help much more sorry.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>
> On 15 September 2014 21:34, Anand kumar > 
> wrote:
>
>>
>> Hi Mark,
>>
>> Thanks for your reply.
>>
>> Yeah, both the hosts belonged to same network. 
>> I cant able to use telnet, instead I tried ssh.
>>
>>
>> On Monday, 15 September 2014 16:36:53 UTC+5:30, Mark Walkom wrote:
>>>
>>> Are they on the same network?
>>> Can you telnet to each of the hosts?
>>>
>>> It looks like the two nodes don't have access to each other, hence the 
>>> no route to host error.
>>>
>>> Regards,
>>> Mark Walkom
>>>
>>> Infrastructure Engineer
>>> Campaign Monitor
>>> email: ma...@campaignmonitor.com
>>> web: www.campaignmonitor.com
>>>
>>> On 15 September 2014 21:02, Anand kumar  wrote:
>>>

 Hi all, 

  I'm trying to connect two ES instances using docker VM and both as 
 master nodes, but the instances belong to two different hosts from the 
 same 
 network.

 My docker commands are as below:

 host1 : $ docker run -t -i --name="master-node-01" -p 9200:9200 -p 
 9300:9300 
 host2 : $ docker run -t -i --name="master-node-02" -p 9200:9200 -p 
 9300:9300 (as the host were different, am using the same port for both)

 And I used the configurations as follows:

 host1 : discovery.zen.ping.unicast.hosts: ["host2"]
 host2 : discovery.zen.ping.unicast.hosts: ["host1"]

 When i tried to start the host2 instance after starting the host1, I 
 cant able connect the host2 with the host1

 and tried some other configurations like

 host1 : discovery.zen.ping.unicast.hosts: ["host2:9200"]
 host2 : discovery.zen.ping.unicast.hosts: ["host1:9200"]

 host1 : discovery.zen.ping.unicast.hosts: ["xxx.xxx.xx.xx:9200"]
 host2 : discovery.zen.ping.unicast.hosts: ["xxx.xxx.xx.xx:9200"]

 But I'm getting the following issues, Is it possible to connect the VMs 
 of different host through ES

 [2014-09-15 09:46:46,920][WARN ][transport.netty  ] 
 [ES_MASTER_NODE_1] exception caught on transport layer [[id: 0x2f06682e]], 
 closing connection
 java.net.NoRouteToHostException: No route to host
 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
 at sun.nio.ch.SocketChannelImpl.finishConnect(
 SocketChannelImpl.java:708)
 at org.elasticsearch.common.netty.channel.socket.nio.
 NioClientBoss.connect(NioClientBoss.java:150)
 at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.
 processSelectedKeys(NioClientBoss.java:105)
 at org.elasticsearch.common.netty.channel.socket.nio.
 NioClientBoss.process(NioClientBoss.java:79)
 at org.elasticsearch.common.netty.channel.socket.nio.
 AbstractNioSelector.run(AbstractNioSelector.java:318)
 at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(
 NioClientBoss.java:42)
 at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
 ThreadRenamingRunnable.java:108)
 at org.elasticsearch.common.netty.util.internal.
 DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(
 ThreadPoolExecutor.java:1145)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(
 ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:724)


 Any suggestion would be helpful. Thanks in advance.


  -- 
 You received this message because you are subscribed to the Google 
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/elasticsearch/d7450d33-b569-493c-8391-e3c68be0cd02%
 40googlegroups.com 
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/4070c285-0b91-43db-b9ee-61e674202a83%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.

Re: How to connect elasticsearch instance of docker VMs from two different machine?

[Anand kumar]

Hi Mark,

Thanks for your reply.

Yeah, both the hosts belonged to same network. 
I cant able to use telnet, instead I tried ssh.


On Monday, 15 September 2014 16:36:53 UTC+5:30, Mark Walkom wrote:
>
> Are they on the same network?
> Can you telnet to each of the hosts?
>
> It looks like the two nodes don't have access to each other, hence the no 
> route to host error.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>
> On 15 September 2014 21:02, Anand kumar > 
> wrote:
>
>>
>> Hi all, 
>>
>>  I'm trying to connect two ES instances using docker VM and both as 
>> master nodes, but the instances belong to two different hosts from the same 
>> network.
>>
>> My docker commands are as below:
>>
>> host1 : $ docker run -t -i --name="master-node-01" -p 9200:9200 -p 
>> 9300:9300 
>> host2 : $ docker run -t -i --name="master-node-02" -p 9200:9200 -p 
>> 9300:9300 (as the host were different, am using the same port for both)
>>
>> And I used the configurations as follows:
>>
>> host1 : discovery.zen.ping.unicast.hosts: ["host2"]
>> host2 : discovery.zen.ping.unicast.hosts: ["host1"]
>>
>> When i tried to start the host2 instance after starting the host1, I cant 
>> able connect the host2 with the host1
>>
>> and tried some other configurations like
>>
>> host1 : discovery.zen.ping.unicast.hosts: ["host2:9200"]
>> host2 : discovery.zen.ping.unicast.hosts: ["host1:9200"]
>>
>> host1 : discovery.zen.ping.unicast.hosts: ["xxx.xxx.xx.xx:9200"]
>> host2 : discovery.zen.ping.unicast.hosts: ["xxx.xxx.xx.xx:9200"]
>>
>> But I'm getting the following issues, Is it possible to connect the VMs 
>> of different host through ES
>>
>> [2014-09-15 09:46:46,920][WARN ][transport.netty  ] 
>> [ES_MASTER_NODE_1] exception caught on transport layer [[id: 0x2f06682e]], 
>> closing connection
>> java.net.NoRouteToHostException: No route to host
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
>> at 
>> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
>> at 
>> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
>> at 
>> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
>> at 
>> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
>> at 
>> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
>> at 
>> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>> at 
>> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:724)
>>
>>
>> Any suggestion would be helpful. Thanks in advance.
>>
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/d7450d33-b569-493c-8391-e3c68be0cd02%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4070c285-0b91-43db-b9ee-61e674202a83%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: cluster can't recover after upgrade from 1.1.1 to 1.3.2 due to MaxBytesLengthExceededException

[omar]

Thank you Jilles, this really helped. I have actually changed the mapping 
of that field to NO_INDEX to avoid this problem all together. But it is 
nice to know there is a valid solution out there.


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f147bdae-04ec-4d40-9ac7-e11ac1c1339e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[JAVA API] Retrieve all indexes

[Luan Garrido]

Hi all,
First of all I took a look and did not find the answer for my question.
Can anyone teach me how can I find all indexes from a cluster using the 
JAVA API? 
Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/adf8f2c2-311e-4732-8c5e-97835798e573%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to connect elasticsearch instance of docker VMs from two different machine?

[Mark Walkom]

Are they on the same network?
Can you telnet to each of the hosts?

It looks like the two nodes don't have access to each other, hence the no
route to host error.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 15 September 2014 21:02, Anand kumar  wrote:

>
> Hi all,
>
>  I'm trying to connect two ES instances using docker VM and both as
> master nodes, but the instances belong to two different hosts from the same
> network.
>
> My docker commands are as below:
>
> host1 : $ docker run -t -i --name="master-node-01" -p 9200:9200 -p
> 9300:9300
> host2 : $ docker run -t -i --name="master-node-02" -p 9200:9200 -p
> 9300:9300 (as the host were different, am using the same port for both)
>
> And I used the configurations as follows:
>
> host1 : discovery.zen.ping.unicast.hosts: ["host2"]
> host2 : discovery.zen.ping.unicast.hosts: ["host1"]
>
> When i tried to start the host2 instance after starting the host1, I cant
> able connect the host2 with the host1
>
> and tried some other configurations like
>
> host1 : discovery.zen.ping.unicast.hosts: ["host2:9200"]
> host2 : discovery.zen.ping.unicast.hosts: ["host1:9200"]
>
> host1 : discovery.zen.ping.unicast.hosts: ["xxx.xxx.xx.xx:9200"]
> host2 : discovery.zen.ping.unicast.hosts: ["xxx.xxx.xx.xx:9200"]
>
> But I'm getting the following issues, Is it possible to connect the VMs of
> different host through ES
>
> [2014-09-15 09:46:46,920][WARN ][transport.netty  ]
> [ES_MASTER_NODE_1] exception caught on transport layer [[id: 0x2f06682e]],
> closing connection
> java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
> at
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
>
>
> Any suggestion would be helpful. Thanks in advance.
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/d7450d33-b569-493c-8391-e3c68be0cd02%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624ZKmBB_U1yzNfsbyf-cozFiVmPteb1wSp40q8KwkF1ijA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

How to connect elasticsearch instance of docker VMs from two different machine?

[Anand kumar]

Hi all, 

 I'm trying to connect two ES instances using docker VM and both as 
master nodes, but the instances belong to two different hosts from the same 
network.

My docker commands are as below:

host1 : $ docker run -t -i --name="master-node-01" -p 9200:9200 -p 
9300:9300 
host2 : $ docker run -t -i --name="master-node-02" -p 9200:9200 -p 
9300:9300 (as the host were different, am using the same port for both)

And I used the configurations as follows:

host1 : discovery.zen.ping.unicast.hosts: ["host2"]
host2 : discovery.zen.ping.unicast.hosts: ["host1"]

When i tried to start the host2 instance after starting the host1, I cant 
able connect the host2 with the host1

and tried some other configurations like

host1 : discovery.zen.ping.unicast.hosts: ["host2:9200"]
host2 : discovery.zen.ping.unicast.hosts: ["host1:9200"]

host1 : discovery.zen.ping.unicast.hosts: ["xxx.xxx.xx.xx:9200"]
host2 : discovery.zen.ping.unicast.hosts: ["xxx.xxx.xx.xx:9200"]

But I'm getting the following issues, Is it possible to connect the VMs of 
different host through ES

[2014-09-15 09:46:46,920][WARN ][transport.netty  ] 
[ES_MASTER_NODE_1] exception caught on transport layer [[id: 0x2f06682e]], 
closing connection
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at 
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at 
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at 
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at 
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at 
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at 
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at 
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)


Any suggestion would be helpful. Thanks in advance.


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d7450d33-b569-493c-8391-e3c68be0cd02%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Purge the deleted documents on disk

[Michael McCandless]

By default Lucene/ES will only merge away the segment if it has "enough"
deletes, where "enough" defaults to 10% of the segment.

The setting is index.merge.policy.expunge_deletes_allowed ... so you can
change that if you want to.

However I would strongly advise not worrying about this: merging is a very
costly operation, and Lucene naturally merges segments over time, favoring
segments that have more deletions.  Furthermore, deletions are not that
costly because deleted docs are filtered out at a very low level in Lucene,
before expensive query matching/scoring computations even see them.

Only if your index will never change again is it worth forcing merges ...

Mike McCandless

http://blog.mikemccandless.com


On Sun, Sep 14, 2014 at 7:19 PM, Wei  wrote:

> Hi Vineeth,
>
> Thanks a lot for your response.
>
> However I've tried setting max_num_segments to 1 and  only_expunge_deletes
>  to true, using the command
>
> curl -XPOST '
> http://localhost:9200/_optimize?only_expunge_deletes=true&max_num_segments=1
> '
>
> This only cleared some of the deleted documents. e.g, the number of
> deleted documents reduced from 8k to 7k. But cannot merge all of them. I'm
> running es v0.20.5. Is there a way I can force merge all deleted documents?
>
> p.s. I'm upgrading the ES from 0.20.5 to 1.2.1, shards are damaged after
> migration, complaining on the write handler to the .del files. I have a
> feeling this is because of the un-merged deleted documents.
> Thanks
> Wei
>
>
>
> On Friday, September 12, 2014 8:16:25 PM UTC-7, vineeth mohan wrote:
>>
>> Hello Wei ,
>>
>> You can use the in optimize  API - max_num_segments as 1 or
>>  only_expunge_deletes  as true .
>> OPTIMIZE - http://www.elasticsearch.org/guide/en/elasticsearch/
>> reference/current/indices-optimize.html#indices-optimize
>>
>> Thanks
>>  Vineeth
>>
>> On Sat, Sep 13, 2014 at 5:32 AM, Wei  wrote:
>>
>>> Hi all,
>>>
>>> If there's any api to clear all the deleted documents on disk?
>>> I read that
>>> Deleting a document doesn’t immediately remove the document from disk —
>>> it just marks it as deleted.
>>> Elasticsearch will clean up deleted documents in the background as you
>>> continue to index more data.
>>>
>>> By request the index stats, it shows
>>> "primaries": {
>>>
>>>- "docs": {
>>>   - "count": 3268352,
>>>   - "deleted": 71249
>>>} }
>>>
>>> I run refresh, or optimize on my index, But it only cleaned a small
>>> number of deleted documents from disk. Is there any way I can clean all the
>>> deleted docs? to reduce this number to 0?
>>>
>>> Thanks
>>> Wei
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/adc8877e-dbc4-4751-bf4b-05f346e6c497%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/3d566401-c82d-4ec8-bf91-6f4c0a3f2e96%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAD7smRcP0kyz0uDS_GGiyr1R%2BjVeaLxeQxfwPypQFpKLyZ2K0A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to query with JSON when use Java API in ElasticSearch?

[Andrej Rosenheinrich]

Hi,

there are QueryBuilder that can parse JSON, e.g. WrapperQueryBuilder 

 wrapperQuery(java.lang.String source). 
This should do the job.

Greets,
Andrej

PS: Still wondering why javadocs disappeared a while ago. This would be a 
great help if javadocs would be available for each version for download.

Am Montag, 15. September 2014 09:22:45 UTC+2 schrieb 张新星:
>
> Hi all,
>  I'm using ElasticSearch as a search service in Spring Web project. 
> Now I'm using Transport Client to communicate with ES. 
> I'm wondering if there exists a method which can construct a QueryBuilder 
> from a JSON DSL. for example, convert this bool query DSL JSON 
> {
>"query" : {
>"bool" : {
>"must" : { "match" : {"content" : "quick"},
>"should": { "match": {"content" : "lazy"}
> }
>}
> }
>
> to a QueryBuilder. I need this function because I have to receive 
> user's bool string input from web front-side, and parse this bool string to 
> a QueryBuilder.
> However it not suit to use 
> QueryBuilders.boolQuery().must(matchQB).should(shouldQB).must_not(mustNotQB). 
> Because we may need several must or non must keywords.
>
>Thanks for your time! I'm really need some help.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/780122c7-16b2-4f17-9c83-b77a6401e85f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Cluster allocation awareness - opposite

[spezam]

Hi Mark,
precise, there's no replication happening between the nodes in the DC, they 
are just aware of each others.

The gateway is a no-data node that is used by Kibana to query all the other 
nodes in the cluster and return aggregated queries.

Yes, I would like to add a new node (node2) in DC1 (for example) and 
replicate data between node1 and node2 in the same DC.



On Saturday, September 13, 2014 12:40:51 AM UTC+2, Mark Walkom wrote:
>
> It's a little unclear what you are doing.
>
> You indicate you have a single cluster but then no replication happens 
> between the nodes in each DC?
> By gateway do you mean a tribe node?
> And by replicate the local indexes do you want to set replicas = 1 and 
> then have them on this new node in each DC?
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>
>
> On 12 September 2014 22:40, spezam > wrote:
>
>> Hi All,
>> we currently have an Elasticsearch (1.1.1) cluster distributed among DCs
>>
>> * 3 Data nodes in 3 DC
>> * 1 Gateway node
>>
>> each DC index its own data and no shards replica is happening between DC.
>> The gateway lets us to query all the indexes in all the DC.
>>
>> Now, for performance and redundancy we would like to add one data node 
>> per DC and replicate the 'local' indexes.
>>
>> We read about cluster allocation awareness, and thought that was the 
>> perfect solution, until we realize it actually acts opposite of what we 
>> want.
>> If node members are in the same awareness attribute, they would just 
>> 'ignore' each others and don't spread shards or replicate.
>>
>> Is such a deployment possible/does it makes sense?
>>
>> Thanks a million,
>> Matteo
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/3cff8e6e-9b3b-432c-baba-03dd44b3862c%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4100dab3-e194-4834-94cb-6601d98194d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Trouble with search preference=_local

[Kurt Hutchison]

I have ES 1.0.2 and Lucene 4.6. 

I was hoping preference=_local would query on the current node, but it does
not. 

The same node comes up at the top of the list for all shards regardless of
which node I search from, 
it is not the primary node, neither is it the local node. 

I am testing this with the _search_shards API like this: 

curl-XGET localhost:9200/2/_search_shards?preference=_local 

The larger problem I am trying to solve is to restrict searches to a geo, we
are a two geo site 
and don't want searches going across the WAN. 

I have experimented with allocation awareness, also with mixed results,
searches 
always go to the first zone, this might actually be workable since we are a
primary/DR 
setup, but it is not behaving as documented so I am leery of relying on this
behavior. 



--
View this message in context: 
http://elasticsearch-users.115913.n3.nabble.com/Trouble-with-search-preference-local-tp4063439.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1410550675399-4063439.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

Re: Write a plugin to query and aggregate results from multiple shards

[joergpra...@gmail.com]

See the TransportSearchAction, in the doExecute() method, the SearchRequest
is dispatched to several transport actions of the search types.

Assuming you write your own custom action: the shard level request
is ShardSearchRequest. It is easier to add information to SearchRequest,
pass it down, and extract relevant parts from the SearchRequest later. See
ShardSearchRequest constructor for repacking the parameters and delegate it
to a shard.

Maybe it is possible to add info to extrasource.

Jörg

On Mon, Sep 15, 2014 at 8:57 AM, 'Sandeep Ramesh Khanzode' via
elasticsearch  wrote:

> Hi,
>
>
> I am looking through the sources, and I am not sure whether this is
> possible. What I am looking to is the possibility to manipulate the
> SearchRequest object when it reaches the SearchShards level.
> Since I need to update the object with some value that is shard specific.
>
> For this, I was checking the TransportBroadcastOperationAction which
> actually allows to hit multiple shards and we can inject a SearchService.
> However, in the response aggregation, we may have to write our own logic to
> call SearchPhaseController::merge() or something. Not sure if this will be
> a problem when the same code in ElasticSearch changes over releases.
>
> There are also other classes like SearchServiceTransportAction and we can
> also probably extend TransportSearchTypeAction like the other QAF, DFS_QAF,
> QTF, DFS_QTF, etc. However, what I want to know is whether this is standard
> practice and should be done this way? Or is there any other plugin that
> allows me to do this?
>
>
> Thanks,
> Sandeep
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/e1f52da2-bb05-4005-bf88-8031f5440225%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGMr6K81CSWid1%2BoPP_fZ6i1_4J%2B0uNXHuqjCkVB5MXWw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

How to query with JSON when use Java API in ElasticSearch?

[张新星]

Hi all,
 I'm using ElasticSearch as a search service in Spring Web project. Now 
I'm using Transport Client to communicate with ES. 
I'm wondering if there exists a method which can construct a QueryBuilder 
from a JSON DSL. for example, convert this bool query DSL JSON 
{
   "query" : {
   "bool" : {
   "must" : { "match" : {"content" : "quick"},
   "should": { "match": {"content" : "lazy"}
}
   }
}

to a QueryBuilder. I need this function because I have to receive 
user's bool string input from web front-side, and parse this bool string to 
a QueryBuilder.
However it not suit to use 
QueryBuilders.boolQuery().must(matchQB).should(shouldQB).must_not(mustNotQB). 
Because we may need several must or non must keywords.

   Thanks for your time! I'm really need some help.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1a26d1b4-d457-4dbb-b36e-caf469353d55%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.