Re: Run a simple sql group by query in kibana 4

[PMOPMO]

any ideas?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f991dd6e-9a37-4943-a055-39e2e1ed9621%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Looking for a best practice to get all data according to some filters

[Ron Sher]

Again, why not use a very large count size? What are the implications of 
using a very large count?
Regarding performance - it seems doing 1 request with a very large count 
performs better than using scan scroll (with count of 100 using 32 shards)

On Wednesday, December 10, 2014 10:53:50 PM UTC+2, David Pilato wrote:
>
> No I did not say that. Or I did not mean that. Sorry if it was unclear.
> I said: don’t use large sizes:
>
> Never use size:1000 or from:1000. 
>>
>
> You should read this: 
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-scroll.html#scroll-scan
>
> -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com 
> *
> @dadoonet  | @elasticsearchfr 
>  | @scrutmydocs 
> 
>
>
>  
> Le 10 déc. 2014 à 21:16, Ron Sher > a 
> écrit :
>
> So you're saying there's no impact on elasticsearch if I issue a large 
> size? 
> If that's the case then why shouldn't I just call size of 1M if I want to 
> make sure I get everything?
>
> On Wednesday, December 10, 2014 8:22:47 PM UTC+2, David Pilato wrote:
>>
>> Scan/scroll is the best option to extract a huge amount of data.
>> Never use size:1000 or from:1000. 
>>
>> It's not realtime because you basically scroll over a given set of 
>> segments and all new changes that will come in new segments won't be taken 
>> into account during the scroll.
>> Which is good because you won't get inconsistent results.
>>
>> About size, I'd would try and test. It depends on your docs size I 
>> believe.
>> Try with 1 and see how it goes when you increase it. You will may be 
>> discover that getting 10*1 docs is the same as 1*10. :)
>>
>> Best
>>
>> David
>>
>> Le 10 déc. 2014 à 19:09, Ron Sher  a écrit :
>>
>> Hi,
>>
>> I was wondering about best practices to to get all data according to some 
>> filters.
>> The options as I see them are:
>>
>>- Use a very big size that will return all accounts, i.e. use some 
>>value like 1m to make sure I get everything back (even if I need just a 
>> few 
>>hundreds or tens of documents). This is the quickest way, development 
>> wise.
>>- Use paging - using size and from. This requires looping over the 
>>result and the performance gets worse as we advance to later pages. Also, 
>>we need to use preference if we want to get consistent results over the 
>>pages. Also, it's not clear what's the recommended size for each page.
>>- Use scan/scroll - this gives consistent paging but also has several 
>>drawbacks: If I use search_type=scan then it can't be sorted; using 
>>scan/scroll is (maybe) less performant than paging (the documentation 
>> says 
>>it's not for realtime use); again not clear which size is recommended.
>>
>> So you see - many options and not clear which path to take.
>>
>> What do you think?
>>
>> Thanks,
>> Ron
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/764a37c5-1fec-48c4-9c66-7835d8141713%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/838020dc-d2ea-423d-9606-778d807b1a0d%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ac0841ac-4150-435c-a3da-afbf2a4b06a6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is there a way to completely drop incoming documents from indexing based on some criteria?

[Nikolas Everett]

We solve problems like this in two ways:
Adding queueing or concurrent request limits.

Queueing buys retries for free and can absorb temporary shocks. You can
also get things like priority, backlog monitoring, and manual backlog
grooming. I think logstash already supports this, but I don't know it very
well.

Concurrent request limits are more brutal. You just throw away requests to
index if there are too many in flight. You can make it more granular by
giving each incoming application its own pool and limits. We implement
these using a simple server called poolcounter. You can find it by
searching for WMF poolcounterd.

Either way you would have to implement a small application to get these
integrated. Well, maybe someone has already made the queueing one, I don't
know.

Nik
On Dec 13, 2014 11:21 PM, "Konstantin Erman"  wrote:

> I don't crawl the web, just collect rather verbose logs from multiple
> private cloud services and try to keep the size of ES cluster just
> sufficient for comfortable searching those logs. Monitored services are
> under development and occasionally (because of bugs or specifics of the
> source data) they start to send orders of magnitude higher than usual
> torrent of logs. When this happens, very soon ES cluster become
> non-responsive and drops logs from all services, bad behaving or not.
>
> We cannot afford to keep the cluster of the size capable to handle those
> peak loads (and idling most of the time). We rather need some kind of
> Denial of Service attack prevention logic. When some client(s) goes over
> its quota of logs it should be blocked, rather than melting cluster down.
>
> River plugin looks like overkill to me, especially considering deprecation
> of rivers.
>
> On Saturday, December 13, 2014 7:33:05 PM UTC-8, BillyEm wrote:
>>
>> Why are you putting business logic of this type in ES? It belongs in your
>> workflow. At the ES indexer level you will have no idea of the source of
>> truth of the questionable content. Unless you're web crawliing which means
>> you're using the wrong search platform altogether imo.
>>
>> On Friday, December 12, 2014 5:11:05 PM UTC-5, Konstantin Erman wrote:
>>>
>>> I noticed that occasionally I need to shield my ES cluster from some
>>> documents, which are too many or too big or otherwise poison ES.
>>> Usually I can formulate pretty easy query or criteria to detect those
>>> documents and I'm looking for a way to block them from entering the index.
>>>
>>> Is there such pre-indexing filtering mechanism? May be Transforms can be
>>> used for that purpose?
>>>
>>> Thank you!
>>> Konstantin
>>>
>>>
>>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/26556df6-a2a5-495f-bb23-95b5bd0fa63b%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3GuaF%3D9xNyBCtGOpyYZgWYHZKL2i1wR-LdfceV7BV0Og%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is there a way to completely drop incoming documents from indexing based on some criteria?

[Konstantin Erman]

I don't crawl the web, just collect rather verbose logs from multiple 
private cloud services and try to keep the size of ES cluster just 
sufficient for comfortable searching those logs. Monitored services are 
under development and occasionally (because of bugs or specifics of the 
source data) they start to send orders of magnitude higher than usual 
torrent of logs. When this happens, very soon ES cluster become 
non-responsive and drops logs from all services, bad behaving or not. 

We cannot afford to keep the cluster of the size capable to handle those 
peak loads (and idling most of the time). We rather need some kind of 
Denial of Service attack prevention logic. When some client(s) goes over 
its quota of logs it should be blocked, rather than melting cluster down.   

River plugin looks like overkill to me, especially considering deprecation 
of rivers. 

On Saturday, December 13, 2014 7:33:05 PM UTC-8, BillyEm wrote:
>
> Why are you putting business logic of this type in ES? It belongs in your 
> workflow. At the ES indexer level you will have no idea of the source of 
> truth of the questionable content. Unless you're web crawliing which means 
> you're using the wrong search platform altogether imo.
>
> On Friday, December 12, 2014 5:11:05 PM UTC-5, Konstantin Erman wrote:
>>
>> I noticed that occasionally I need to shield my ES cluster from some 
>> documents, which are too many or too big or otherwise poison ES. 
>> Usually I can formulate pretty easy query or criteria to detect those 
>> documents and I'm looking for a way to block them from entering the index. 
>>
>> Is there such pre-indexing filtering mechanism? May be Transforms can be 
>> used for that purpose?
>>
>> Thank you!
>> Konstantin
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/26556df6-a2a5-495f-bb23-95b5bd0fa63b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is there a way to completely drop incoming documents from indexing based on some criteria?

[BillyEm]

it might be the only approach you can come up with Telax. Don't sweat it.


On Saturday, December 13, 2014 9:43:38 AM UTC-5, Telax wrote:
>
> Write a river plugin to receive and filter documents based on whatever 
> criteria you need would be the best approach 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/41b67f66-fc97-49d7-a03a-6f02b510d446%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is there a way to completely drop incoming documents from indexing based on some criteria?

[BillyEm]

Why are you putting business logic of this type in ES? It belongs in your 
workflow. At the ES indexer level you will have no idea of the source of 
truth of the questionable content. Unless you're web crawliing which means 
you're using the wrong search platform altogether imo.

On Friday, December 12, 2014 5:11:05 PM UTC-5, Konstantin Erman wrote:
>
> I noticed that occasionally I need to shield my ES cluster from some 
> documents, which are too many or too big or otherwise poison ES. 
> Usually I can formulate pretty easy query or criteria to detect those 
> documents and I'm looking for a way to block them from entering the index. 
>
> Is there such pre-indexing filtering mechanism? May be Transforms can be 
> used for that purpose?
>
> Thank you!
> Konstantin
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/c5f5b748-a725-4d43-b248-67215e7da576%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch Java version requirement

[Wei Yuan Cho]

Hi

I am using Elasticsearch 1.4.1 and have read from the following links that* 
Java 1.7.0.55* and above are the recommended version to run Elasticsearch 
with.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html
http://www.elasticsearch.org/blog/java-1-7u55-safe-use-elasticsearch-lucene/ 
http://www.elasticsearch.org/blog/elasticsearch-1-2-0-released/

>From the last blog post it has been pointed out that the bug that causes 
index corruption is between 1.7.0.25 to 1.7.0.55. I would like to know 
whether *Java 1.7.0.21* is safe to run Elasticsearch with because I have 
another software/jar that requires me to run that particular version of 
Java.

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/eccfe9c6-0fdb-413b-9298-8aa2ff18c5d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: MultiSearchRequest{Builder} throws NullPointerException in elasticsearch 1.3.2

[David Pilato]

I think a NPE is always bad.
Could you check if this happens with 1.4.1 as well and if so, open an issue?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

> Le 13 déc. 2014 à 18:32, Veerapuram Varadhan  a écrit :
> 
> That was so stupid of me forgetting to set the index and type params.!! 
> Looking at the SearchRequest.java:541 struck a big hammer on my head.!
> 
> Issue solved.  Sorry for the false alarm.
> 
> Thanks,
> 
> V. Varadhan
> 
>> On Saturday, December 13, 2014 10:35:18 PM UTC+5:30, Veerapuram Varadhan 
>> wrote:
>> Hi,
>> 
>> If this is already answered, please point me to the thread.  TIA for that.
>> 
>> Now, the problem:
>> 
>> I am having the following code to test MultiSearchRequest{Buuilder} and it 
>> always throws me NullPointerException (even if I just use matchAllQuery() in 
>> the SearchRequests added to it).
>> 
>> public void performMultiSearchQuery () {
>>  Client client = new TransportClient().addTransportAddress(new 
>> InetSocketTransportAddress("localhost", 9300));
>>  SearchRequestBuilder srb1 = client.prepareSearch()
>>  
>> 
>> .setQuery(QueryBuilders.queryString("sev:1")).setSize(Integer.MAX_VALUE);
>>  SearchRequestBuilder srb2 = client.prepareSearch()
>>  
>> 
>> .setQuery(QueryBuilders.queryString("priority:50")).setSize(Integer.MAX_VALUE);
>> 
>>  MultiSearchResponse sr = client.prepareMultiSearch()
>>  .add(srb1.request())
>>  .add(srb2.request())
>>  .execute().actionGet();
>> 
>>  // You will get all individual responses from 
>> MultiSearchResponse#getResponses()
>>  long nbHits = 0;
>>  for (MultiSearchResponse.Item item : sr.getResponses()) {
>>  SearchResponse response = item.getResponse();
>>  nbHits += response.getHits().getTotalHits();
>>  }
>>  
>>  logger.info("Total Number of Hits: " + nbHits);
>>  client.close();
>> }
>> 
>> And the exception it throws is: 
>> 
>> Exception in thread "main" 
>> org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: 
>> Failed execution
>>  at 
>> org.elasticsearch.action.support.AdapterActionFuture.rethrowExecutionException(AdapterActionFuture.java:90)
>>  at 
>> org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:50)
>>  at esclient.ESClient.buildQuery(ESClient.java:119)
>>  at esclient.ESClient.main(ESClient.java:321)
>> Caused by: java.lang.NullPointerException
>>  at 
>> org.elasticsearch.action.search.SearchRequest.writeTo(SearchRequest.java:541)
>>  at 
>> org.elasticsearch.action.search.MultiSearchRequest.writeTo(MultiSearchRequest.java:271)
>>  at 
>> org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:601)
>>  at 
>> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:206)
>>  at 
>> org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:68)
>>  at 
>> org.elasticsearch.client.transport.support.InternalTransportClient$2.doWithNode(InternalTransportClient.java:108)
>>  at 
>> org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:219)
>>  at 
>> org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:105)
>>  at 
>> org.elasticsearch.client.support.AbstractClient.multiSearch(AbstractClient.java:360)
>>  at 
>> org.elasticsearch.client.transport.TransportClient.multiSearch(TransportClient.java:441)
>>  at 
>> org.elasticsearch.action.search.MultiSearchRequestBuilder.doExecute(MultiSearchRequestBuilder.java:78)
>>  at 
>> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
>>  at 
>> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
>> 
>> I have searched my best and you might notice that the above code is the 
>> sample snippet from ES doc page.  Please help me in figuring out what the 
>> issue is.
>> 
>> P.S:- Individually both those SearchRequestBuilders are working fine.
>> Best,
>> 
>> V. Varadhan
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/b02c98f2-7df3-48e0-a0ba-b1d71967ae69%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscri

Re: MultiSearchRequest{Builder} throws NullPointerException in elasticsearch 1.3.2

[Veerapuram Varadhan]

That was so stupid of me forgetting to set the index and type params.!! 
Looking at the SearchRequest.java:541 struck a big hammer on my head.!

Issue solved.  Sorry for the false alarm.

Thanks,

V. Varadhan

On Saturday, December 13, 2014 10:35:18 PM UTC+5:30, Veerapuram Varadhan 
wrote:
>
> Hi,
>
> If this is already answered, please point me to the thread.  TIA for that.
>
> Now, the problem:
>
> I am having the following code to test MultiSearchRequest{Buuilder} and it 
> always throws me NullPointerException (even if I just use matchAllQuery() 
> in the SearchRequests added to it).
>
> public void performMultiSearchQuery () {
> Client client = new TransportClient().addTransportAddress(new 
> InetSocketTransportAddress("localhost", 9300));
> SearchRequestBuilder srb1 = client.prepareSearch()
> .setQuery(QueryBuilders.queryString("sev:1")).setSize(Integer.MAX_VALUE);
> SearchRequestBuilder srb2 = client.prepareSearch()
>
> .setQuery(QueryBuilders.queryString("priority:50")).setSize(Integer.MAX_VALUE);
>
> MultiSearchResponse sr = client.prepareMultiSearch()
> .add(srb1.request())
> .add(srb2.request())
> .execute().actionGet();
>
> // You will get all individual responses from 
> MultiSearchResponse#getResponses()
> long nbHits = 0;
> for (MultiSearchResponse.Item item : sr.getResponses()) {
>  SearchResponse response = item.getResponse();
>  nbHits += response.getHits().getTotalHits();
> }
>  logger.info("Total Number of Hits: " + nbHits);
> client.close();
> }
>
> And the exception it throws is: 
>
> Exception in thread "main" 
> org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: 
> Failed execution
> at 
> org.elasticsearch.action.support.AdapterActionFuture.rethrowExecutionException(AdapterActionFuture.java:90)
> at 
> org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:50)
> at esclient.ESClient.buildQuery(ESClient.java:119)
> at esclient.ESClient.main(ESClient.java:321)
> Caused by: java.lang.NullPointerException
> at 
> org.elasticsearch.action.search.SearchRequest.writeTo(SearchRequest.java:541)
> at 
> org.elasticsearch.action.search.MultiSearchRequest.writeTo(MultiSearchRequest.java:271)
> at 
> org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:601)
> at 
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:206)
> at 
> org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:68)
> at 
> org.elasticsearch.client.transport.support.InternalTransportClient$2.doWithNode(InternalTransportClient.java:108)
> at 
> org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:219)
> at 
> org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:105)
> at 
> org.elasticsearch.client.support.AbstractClient.multiSearch(AbstractClient.java:360)
> at 
> org.elasticsearch.client.transport.TransportClient.multiSearch(TransportClient.java:441)
> at 
> org.elasticsearch.action.search.MultiSearchRequestBuilder.doExecute(MultiSearchRequestBuilder.java:78)
> at 
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
> at 
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
>
> I have searched my best and you might notice that the above code is the 
> sample snippet from ES doc page.  Please help me in figuring out what the 
> issue is.
>
> P.S:- Individually both those SearchRequestBuilders are working fine.
> Best,
>
> V. Varadhan
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b02c98f2-7df3-48e0-a0ba-b1d71967ae69%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

MultiSearchRequest{Builder} throws NullPointerException in elasticsearch 1.3.2

[Veerapuram Varadhan]

Hi,

If this is already answered, please point me to the thread.  TIA for that.

Now, the problem:

I am having the following code to test MultiSearchRequest{Buuilder} and it 
always throws me NullPointerException (even if I just use matchAllQuery() 
in the SearchRequests added to it).

public void performMultiSearchQuery () {
Client client = new TransportClient().addTransportAddress(new 
InetSocketTransportAddress("localhost", 9300));
SearchRequestBuilder srb1 = client.prepareSearch()
.setQuery(QueryBuilders.queryString("sev:1")).setSize(Integer.MAX_VALUE);
SearchRequestBuilder srb2 = client.prepareSearch()
.setQuery(QueryBuilders.queryString("priority:50")).setSize(Integer.MAX_VALUE);

MultiSearchResponse sr = client.prepareMultiSearch()
.add(srb1.request())
.add(srb2.request())
.execute().actionGet();

// You will get all individual responses from 
MultiSearchResponse#getResponses()
long nbHits = 0;
for (MultiSearchResponse.Item item : sr.getResponses()) {
 SearchResponse response = item.getResponse();
 nbHits += response.getHits().getTotalHits();
}
 logger.info("Total Number of Hits: " + nbHits);
client.close();
}

And the exception it throws is: 

Exception in thread "main" 
org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: 
Failed execution
at 
org.elasticsearch.action.support.AdapterActionFuture.rethrowExecutionException(AdapterActionFuture.java:90)
at 
org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:50)
at esclient.ESClient.buildQuery(ESClient.java:119)
at esclient.ESClient.main(ESClient.java:321)
Caused by: java.lang.NullPointerException
at 
org.elasticsearch.action.search.SearchRequest.writeTo(SearchRequest.java:541)
at 
org.elasticsearch.action.search.MultiSearchRequest.writeTo(MultiSearchRequest.java:271)
at 
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:601)
at 
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:206)
at 
org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:68)
at 
org.elasticsearch.client.transport.support.InternalTransportClient$2.doWithNode(InternalTransportClient.java:108)
at 
org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:219)
at 
org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:105)
at 
org.elasticsearch.client.support.AbstractClient.multiSearch(AbstractClient.java:360)
at 
org.elasticsearch.client.transport.TransportClient.multiSearch(TransportClient.java:441)
at 
org.elasticsearch.action.search.MultiSearchRequestBuilder.doExecute(MultiSearchRequestBuilder.java:78)
at 
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
at 
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)

I have searched my best and you might notice that the above code is the 
sample snippet from ES doc page.  Please help me in figuring out what the 
issue is.

P.S:- Individually both those SearchRequestBuilders are working fine.
Best,

V. Varadhan

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/badbc70d-6c2f-4f63-bb99-dec42addbfec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: analytics on data stored in ES

[Arie]

Hi,,

Consider a non-data master node, this can improve data handling and search 
speed a lot as I understand.


On Friday, December 12, 2014 6:04:46 AM UTC+1, Ramchandra Phadake wrote:
>
> Hi,
>
> We are storing lots of mail messages in ES with multiple fields. 600 
> Millions+ messages across 3 ES nodes.
>
> There is a custom algorithm which works on batch of messages to correlate 
> based on fields & other message semantics. 
> Final result involves groups of messages returned similar to say field 
> collapsing type results. 
>
> Currently we fetch 100K+ messages from ES & apply this logic to return 
> final results to user. The algo can't be modeled using aggregations. 
>
> Obviously this is not scalable approach if say we want to process 100 M 
> messages as part of this processing & return results in few mins.The 
> messages are large & partitioned across few ES nodes. We want to main data 
> locality while processing so as not to download lots of data from ES over 
> network.
>
> Any way to execute some code over shards from within ES, fine if done as 
> part of postFilter as well. What are options available before thinking 
> about Hadoop/Spark using es-hadoop library? 
>
> Solr seems to be having such a plugin hook(experimental) for custom 
> processing. 
> https://cwiki.apache.org/confluence/display/solr/AnalyticsQuery+API
>
> Thanks,
> Ram
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5094ea6e-2b96-4fb2-a2ba-e542db009865%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: To Raid or not to Raid

[joergpra...@gmail.com]

The statement is related to performance and I can't agree with it. You can
easily build a RAID 0 system which has massive I/O throughput performance
and is superior to JBOD, because RAID striping does not slow things down,
it is as always as much as fast than a single drive and in most RAID levels
it is much faster.

In the past, RAID was invented for mirroring cheap and error-prone spindle
disk arrays, while mirrors increase costs but decrease fault probability.

With Elasticsearch, the decision is if you still want to handle disk faults
by drive redundancy (RAID) and all other hardware faults like power outages
by server downtime. This is just a matter of organization and of cost. I
would suggest from my experience: take control over your complete hardware
setup, equip your systems with expensive SAS2 (or even better) controllers
with RAID 0 to reduce cost and maximize performance, and handle all kind of
hardware faults by server downtime, because ES replica level > 0 allows
that.

There is also a simplification of SAN/NAS in the statement but that is a
different discussion. Never use SAN/NAS for ES local gateway.

Jörg

On Fri, Dec 12, 2014 at 7:28 PM, Elvar Böðvarsson  wrote:
>
>
> Second, "Prefer JBODs for data disks over RAID, SAN/NAS", would be ok,
> maybe then to be safe go with 2x replicas, goes well with having 3x nodes
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE9g%2BJFNQdZYH1%3D3pz-b%2Bx0j9cc3M5dLV9rB4gL_SWvWA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Is there a way to completely drop incoming documents from indexing based on some criteria?

[Telax]

Write a river plugin to receive and filter documents based on whatever criteria 
you need would be the best approach 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/cc6e95bc-367f-4f73-bd0a-8fafad8d36c0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.