Re: encountered monitor.jvm warning

[Abid Hussain]

Maybe aggregations are a cause for memory problems. According to the docs, 
we set the fielddata cache property to
indices.fielddata.cache.size: 40%
... hoping this will help to avoid this kind of issues.

Regards,

Abid

Am Mittwoch, 25. März 2015 00:47:09 UTC+1 schrieb Jason Wee:
>
> Few filters should be fine but aggregations... hm 
>
> You could use dump stack trace and/or heap dump if it happen again and 
> start to analyze from there. Try as well, 
>
> http://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html
>  
>
> hth 
>
> jason 
>
> On Tue, Mar 24, 2015 at 11:59 PM, Abid Hussain 
> > wrote: 
> > All settings except from ES_HEAP (set to 10 GB) are defaults, so I 
> actually 
> > am not sure about the new gen setting. 
> > 
> > The host has 80 GB memory in total and 24 CPU cores. All ES indices 
> together 
> > sum up to ~32 GB where the biggest indices are of size ~8 GB. 
> > 
> > We are using queries mostly together with filters and also aggregations. 
> > 
> > We "solved" the problem with a restart of the cluster. Are there any 
> > recommended diagnostics to be performed when this problem occurs next 
> time? 
> > 
> > Regards, 
> > 
> > Abid 
> > 
> > Am Dienstag, 24. März 2015 15:24:43 UTC+1 schrieb Jason Wee: 
> >> 
> >> what is the new gen setting? how much is the system memory in total? 
> >> how many cpu cores in the node? what query do you use? 
> >> 
> >> jason 
> >> 
> >> On Tue, Mar 24, 2015 at 5:26 PM, Abid Hussain 
> >>  wrote: 
> >> > Hi all, 
> >> > 
> >> > we today got (for the first time) warning messages which seem to 
> >> > indicate a 
> >> > memory problem: 
> >> > [2015-03-24 09:08:12,960][WARN ][monitor.jvm  ] [Danger] 
> >> > [gc][young][413224][18109] duration [5m], collections [1]/[5.3m], 
> total 
> >> > [5m]/[16.7m], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young] 
> >> > [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] 
> [149.7mb]->[0b]/[149.7mb]}{[old] 
> >> > [6.9gb]->[3.7gb]/[8.5gb]} 
> >> > [2015-03-24 09:08:12,960][WARN ][monitor.jvm  ] [Danger] 
> >> > [gc][old][413224][104] duration [18.4s], collections [1]/[5.3m], 
> total 
> >> > [18.4s]/[58s], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young] 
> >> > [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] 
> [149.7mb]->[0b]/[149.7mb]}{[old] 
> >> > [6.9gb]->[3.7gb]/[8.5gb]} 
> >> > [2015-03-24 09:08:15,372][WARN ][monitor.jvm  ] [Danger] 
> >> > [gc][young][413225][18110] duration [1.4s], collections [1]/[2.4s], 
> >> > total 
> >> > [1.4s]/[16.7m], memory [3.7gb]->[5gb]/[9.8gb], all_pools {[young] 
> >> > [6.1mb]->[2.7mb]/[1.1gb]}{[survivor] [0b]->[149.7mb]/[149.7mb]}{[old] 
> >> > [3.7gb]->[4.9gb]/[8.5gb]} 
> >> > [2015-03-24 09:08:18,192][WARN ][monitor.jvm  ] [Danger] 
> >> > [gc][young][413227][18111] duration [1.4s], collections [1]/[1.8s], 
> >> > total 
> >> > [1.4s]/[16.7m], memory [5.8gb]->[6.2gb]/[9.8gb], all_pools {[young] 
> >> > [845.4mb]->[1.2mb]/[1.1gb]}{[survivor] 
> >> > [149.7mb]->[149.7mb]/[149.7mb]}{[old] 
> >> > [4.9gb]->[6gb]/[8.5gb]} 
> >> > [2015-03-24 09:08:21,506][WARN ][monitor.jvm  ] [Danger] 
> >> > [gc][young][413229][18112] duration [1.2s], collections [1]/[2.3s], 
> >> > total 
> >> > [1.2s]/[16.7m], memory [7gb]->[7.3gb]/[9.8gb], all_pools {[young] 
> >> > [848.6mb]->[2.1mb]/[1.1gb]}{[survivor] 
> >> > [149.7mb]->[149.7mb]/[149.7mb]}{[old] 
> >> > [6gb]->[7.2gb]/[8.5gb]} 
> >> > 
> >> > We're using ES 1.4.2 as a single node cluster, ES_HEAP is set to 10g, 
> >> > other 
> >> > settings are defaults. From previous posts related to this issue, it 
> is 
> >> > said 
> >> > that field data cache may be a problem. 
> >> > 
> >> > Requesting /_nodes/stats/indices/fielddata says: 
> >> > { 
> >> >"cluster_name": "my_cluster", 
> >> >"nodes": { 
> >> >   "ILUggMfTSvix8Kc0nfNVAw": { 
> >> >  "timestamp": 1427188716203, 
> >> >  "name": "Danger", 
> >> >  "transport_address": "inet[/1

Re: encountered monitor.jvm warning

[Abid Hussain]

All settings except from ES_HEAP (set to 10 GB) are defaults, so I actually 
am not sure about the new gen setting.

The host has 80 GB memory in total and 24 CPU cores. All ES indices 
together sum up to ~32 GB where the biggest indices are of size ~8 GB.

We are using queries mostly together with filters and also aggregations.

We "solved" the problem with a restart of the cluster. Are there any 
recommended diagnostics to be performed when this problem occurs next time?

Regards,

Abid

Am Dienstag, 24. März 2015 15:24:43 UTC+1 schrieb Jason Wee:
>
> what is the new gen setting? how much is the system memory in total? 
> how many cpu cores in the node? what query do you use? 
>
> jason 
>
> On Tue, Mar 24, 2015 at 5:26 PM, Abid Hussain 
> > wrote: 
> > Hi all, 
> > 
> > we today got (for the first time) warning messages which seem to 
> indicate a 
> > memory problem: 
> > [2015-03-24 09:08:12,960][WARN ][monitor.jvm  ] [Danger] 
> > [gc][young][413224][18109] duration [5m], collections [1]/[5.3m], total 
> > [5m]/[16.7m], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young] 
> > [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old] 
> > [6.9gb]->[3.7gb]/[8.5gb]} 
> > [2015-03-24 09:08:12,960][WARN ][monitor.jvm  ] [Danger] 
> > [gc][old][413224][104] duration [18.4s], collections [1]/[5.3m], total 
> > [18.4s]/[58s], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young] 
> > [853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old] 
> > [6.9gb]->[3.7gb]/[8.5gb]} 
> > [2015-03-24 09:08:15,372][WARN ][monitor.jvm  ] [Danger] 
> > [gc][young][413225][18110] duration [1.4s], collections [1]/[2.4s], 
> total 
> > [1.4s]/[16.7m], memory [3.7gb]->[5gb]/[9.8gb], all_pools {[young] 
> > [6.1mb]->[2.7mb]/[1.1gb]}{[survivor] [0b]->[149.7mb]/[149.7mb]}{[old] 
> > [3.7gb]->[4.9gb]/[8.5gb]} 
> > [2015-03-24 09:08:18,192][WARN ][monitor.jvm  ] [Danger] 
> > [gc][young][413227][18111] duration [1.4s], collections [1]/[1.8s], 
> total 
> > [1.4s]/[16.7m], memory [5.8gb]->[6.2gb]/[9.8gb], all_pools {[young] 
> > [845.4mb]->[1.2mb]/[1.1gb]}{[survivor] 
> [149.7mb]->[149.7mb]/[149.7mb]}{[old] 
> > [4.9gb]->[6gb]/[8.5gb]} 
> > [2015-03-24 09:08:21,506][WARN ][monitor.jvm  ] [Danger] 
> > [gc][young][413229][18112] duration [1.2s], collections [1]/[2.3s], 
> total 
> > [1.2s]/[16.7m], memory [7gb]->[7.3gb]/[9.8gb], all_pools {[young] 
> > [848.6mb]->[2.1mb]/[1.1gb]}{[survivor] 
> [149.7mb]->[149.7mb]/[149.7mb]}{[old] 
> > [6gb]->[7.2gb]/[8.5gb]} 
> > 
> > We're using ES 1.4.2 as a single node cluster, ES_HEAP is set to 10g, 
> other 
> > settings are defaults. From previous posts related to this issue, it is 
> said 
> > that field data cache may be a problem. 
> > 
> > Requesting /_nodes/stats/indices/fielddata says: 
> > { 
> >"cluster_name": "my_cluster", 
> >"nodes": { 
> >   "ILUggMfTSvix8Kc0nfNVAw": { 
> >  "timestamp": 1427188716203, 
> >  "name": "Danger", 
> >  "transport_address": "inet[/192.168.110.91:9300]", 
> >  "host": "xxx", 
> >  "ip": [ 
> > "inet[/192.168.110.91:9300]", 
> > "NONE" 
> >  ], 
> >  "indices": { 
> > "fielddata": { 
> >"memory_size_in_bytes": 6484, 
> >"evictions": 0 
> > } 
> >  } 
> >   } 
> >} 
> > } 
> > 
> > Running top results in: 
> > PID USER  PR  NI  VIRT  RES  SHR S   %CPU %MEMTIME+  COMMAND 
> > 12735 root   20   0 15.8g  10g0 S 74 13.2   2485:26 java 
> > 
> > Any ideas what to do? If possible I would rather avoid increasing 
> ES_HEAP as 
> > there isn't that much free memory left on the host. 
> > 
> > Regards, 
> > 
> > Abid 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups 
> > "elasticsearch" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an 
> > email to elasticsearc...@googlegroups.com . 
> > To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4320ec5a-1dc6-4eec-83c2-d9da618bf957%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

encountered monitor.jvm warning

[Abid Hussain]

Hi all,

we today got (for the first time) warning messages which seem to indicate a 
memory problem:
[2015-03-24 09:08:12,960][WARN ][monitor.jvm  ] [Danger] 
[gc][young][413224][18109] duration [5m], collections [1]/[5.3m], total 
[5m]/[16.7m], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young] 
[853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old] 
[6.9gb]->[3.7gb]/[8.5gb]}
[2015-03-24 09:08:12,960][WARN ][monitor.jvm  ] [Danger] 
[gc][old][413224][104] duration [18.4s], collections [1]/[5.3m], total 
[18.4s]/[58s], memory [7.9gb]->[3.7gb]/[9.8gb], all_pools {[young] 
[853.9mb]->[6.1mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old] 
[6.9gb]->[3.7gb]/[8.5gb]}
[2015-03-24 09:08:15,372][WARN ][monitor.jvm  ] [Danger] 
[gc][young][413225][18110] duration [1.4s], collections [1]/[2.4s], total 
[1.4s]/[16.7m], memory [3.7gb]->[5gb]/[9.8gb], all_pools {[young] 
[6.1mb]->[2.7mb]/[1.1gb]}{[survivor] [0b]->[149.7mb]/[149.7mb]}{[old] 
[3.7gb]->[4.9gb]/[8.5gb]}
[2015-03-24 09:08:18,192][WARN ][monitor.jvm  ] [Danger] 
[gc][young][413227][18111] duration [1.4s], collections [1]/[1.8s], total 
[1.4s]/[16.7m], memory [5.8gb]->[6.2gb]/[9.8gb], all_pools {[young] 
[845.4mb]->[1.2mb]/[1.1gb]}{[survivor] 
[149.7mb]->[149.7mb]/[149.7mb]}{[old] [4.9gb]->[6gb]/[8.5gb]}
[2015-03-24 09:08:21,506][WARN ][monitor.jvm  ] [Danger] 
[gc][young][413229][18112] duration [1.2s], collections [1]/[2.3s], total 
[1.2s]/[16.7m], memory [7gb]->[7.3gb]/[9.8gb], all_pools {[young] 
[848.6mb]->[2.1mb]/[1.1gb]}{[survivor] 
[149.7mb]->[149.7mb]/[149.7mb]}{[old] [6gb]->[7.2gb]/[8.5gb]}

We're using ES 1.4.2 as a single node cluster, ES_HEAP is set to 10g, other 
settings are defaults. From previous posts related to this issue, it is 
said that field data cache may be a problem.

Requesting */_nodes/stats/indices/fielddata *says:
{
   "cluster_name": "my_cluster",
   "nodes": {
  "ILUggMfTSvix8Kc0nfNVAw": {
 "timestamp": 1427188716203,
 "name": "Danger",
 "transport_address": "inet[/192.168.110.91:9300]",
 "host": "xxx",
 "ip": [
"inet[/192.168.110.91:9300]",
"NONE"
 ],
 "indices": {
"fielddata": {
   "memory_size_in_bytes": 6484,
   "evictions": 0
}
 }
  }
   }
}

Running top results in:
PID USER  PR  NI  VIRT  RES  SHR S   %CPU %MEMTIME+  COMMAND
12735 root   20   0 15.8g  10g0 S 74 13.2   2485:26 java

Any ideas what to do? If possible I would rather avoid increasing ES_HEAP 
as there isn't that much free memory left on the host.

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Index and find emails using pre- and postfix wildcards

[Abid Hussain]

Hi all,

I'm looking for an easy way to index documents containing an email field 
and searching for them using pre- and postfix wildcard queries (something 
similar to SQL: WHERE email LIKE '%search_term%').

One way may be using the uax_url_email tokenizer but it seems that in this 
case prefix wildcards aren't supported. Is that true?

As an alternative I thought of using a not_analyzed field to store the 
email adress and query them with a something like an ignorecase flag...?

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d1b4d2b3-d5f0-4177-8f84-27a9ae5e81b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Limit large number of threads

[Abid Hussain]

Thanks Jörg, I did a thread dump: 60 % of ~400 threads are in state 
WAITING, 35 % are in state RUNNABLE, the rest is in state TIMED_WAITING, 
none is in state BLOCKED.

So I assume everything is OK - still wondering whats the point of creating 
hundreds of threads as there are "only" 24 cores available on our machine.

Best regards,

Abid

Am Freitag, 20. März 2015 16:47:28 UTC+1 schrieb Jörg Prante:
>
> I think you should check a thread dump created by tools like jstack if you 
> have a high JVM thread count in state BLOCKED. This might be a pointer that 
> something unusual is going on, but I'm not sure.
>
> Jörg
>
>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e0cef7f2-9f38-4c16-806d-45fbd3dc14a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Limit large number of threads

[Abid Hussain]

We're using 1.4.2 currently. Are there any issues known with this version?

If there is nothing to worry about 400 threads, it's ok for me. I just 
wonder why they never seem to be released.

Am Freitag, 20. März 2015 16:20:03 UTC+1 schrieb Jörg Prante:
>
> If thread counts go out of bounds, it may be a lockup somewhere. What 
> version of ES do you use?
>
> Jörg
>
> On Fri, Mar 20, 2015 at 2:08 PM, Abid Hussain  > wrote:
>
>> Thanks for clarification. 
>>
>> Still I wonder why such a huge amount of thread is created and if this 
>> can lead to issues - especially as it seems to me that they are never 
>> released.
>>
>> Am Freitag, 20. März 2015 11:12:42 UTC+1 schrieb Lukáš Vlček:
>>>
>>> Hi,
>>>
>>> please note that Bigdesk does not show all internal thread pools in ES 
>>> now.
>>>
>>> Regards,
>>> Lukas
>>>
>>> On Fri, Mar 20, 2015 at 10:56 AM, Abid Hussain <
>>> hus...@novacom.mygbiz.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I know I'm not the first one wondering about the number of threads but 
>>>> I didn't find anything really appropriate to my question.
>>>>
>>>> We use ES with the default values for the thread pool sizes, that is 
>>>> actually (according to what bigdesk says):
>>>> * Search 72
>>>> * Index 24
>>>> * Bulk 24
>>>> * Refresh 10
>>>>
>>>> This values sum up to 130. But in total, nearly 400 threads have been 
>>>> created by ES which makes a difference of about 280.
>>>>
>>>> So I wonder which components are using the mentioned 280 threads?
>>>>
>>>> And, is there is recommended way to limit extensive thread creation 
>>>> safely (i.e. without compromising cluster functionality)?
>>>>
>>>> Regards,
>>>>
>>>> Abid
>>>>
>>>> -- 
>>>> You received this message because you are subscribed to the Google 
>>>> Groups "elasticsearch" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send 
>>>> an email to elasticsearc...@googlegroups.com.
>>>> To view this discussion on the web visit https://groups.google.com/d/
>>>> msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%
>>>> 40googlegroups.com 
>>>> <https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e13d3eb5-f90d-45e3-ab24-901e1b0c284a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Limit large number of threads

[Abid Hussain]

Thanks for clarification. 

Still I wonder why such a huge amount of thread is created and if this can 
lead to issues - especially as it seems to me that they are never released.

Am Freitag, 20. März 2015 11:12:42 UTC+1 schrieb Lukáš Vlček:
>
> Hi,
>
> please note that Bigdesk does not show all internal thread pools in ES now.
>
> Regards,
> Lukas
>
> On Fri, Mar 20, 2015 at 10:56 AM, Abid Hussain  > wrote:
>
>> Hi all,
>>
>> I know I'm not the first one wondering about the number of threads but I 
>> didn't find anything really appropriate to my question.
>>
>> We use ES with the default values for the thread pool sizes, that is 
>> actually (according to what bigdesk says):
>> * Search 72
>> * Index 24
>> * Bulk 24
>> * Refresh 10
>>
>> This values sum up to 130. But in total, nearly 400 threads have been 
>> created by ES which makes a difference of about 280.
>>
>> So I wonder which components are using the mentioned 280 threads?
>>
>> And, is there is recommended way to limit extensive thread creation 
>> safely (i.e. without compromising cluster functionality)?
>>
>> Regards,
>>
>> Abid
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/52a1c4dd-4edd-4395-b6f0-3d825a9dd424%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Limit large number of threads

[Abid Hussain]

Hi all,

I know I'm not the first one wondering about the number of threads but I 
didn't find anything really appropriate to my question.

We use ES with the default values for the thread pool sizes, that is 
actually (according to what bigdesk says):
* Search 72
* Index 24
* Bulk 24
* Refresh 10

This values sum up to 130. But in total, nearly 400 threads have been 
created by ES which makes a difference of about 280.

So I wonder which components are using the mentioned 280 threads?

And, is there is recommended way to limit extensive thread creation safely 
(i.e. without compromising cluster functionality)?

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/cc37ef7c-0eac-4e1c-b084-dc1f26e8915d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Java API TransportClient Threadpool

[Abid Hussain]

Hi all,

I would like to understand how TransportClient works in terms of how it 
uses it's thread pool.

   - As it uses a threadpool I assume that every request is performed in 
   it's own thread - is that correct?
   - If so, does that mean that i.e. it's pool size is 20 and 21 search 
   requests are coming at the same time the 21st one has to wait until one of 
   the 20 running requests are finished?

And, finally: why does it use a thread pool at all? I mean, all the work is 
done on server side and when I think of database clients I would have 
rather expected something like a connection pool.

Best Regards,

Abid


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6bd05293-03f7-417e-9801-a5a6748bc1db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Queue Size limited?

[Abid Hussain]

Yes it happened with jdbc river. Thanks for clarification.

Am Montag, 23. Februar 2015 17:16:09 UTC+1 schrieb Jörg Prante:
>
> Is this with JDBC river? If so, you can increase the number of bulk 
> actions per request, to decrease the concurrent bulk actions, to remedy 
> situations where many rivers are active.
>
> Jörg
>
> On Mon, Feb 23, 2015 at 9:40 AM, Abid Hussain  > wrote:
>
>> Hi all,
>>
>> last restart of our cluster caused a heavy load as a couple of rivers are 
>> triggered which load alltogether about 40 GB of data into the (single-node) 
>> cluster.
>>
>> This lead to massive exceptions like shown in the logs:
>> [6726]: index [my_index], type [my_doc_type], id [2390342], message 
>> [EsRejectedExecutionException[rejected execution (queue capacity 50) on 
>> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@4a94944]]
>>
>> We didn't change any of the threadpools queue size settings (like 
>> threadpool.bulk.queue_size). AFAIK in this case the queue size is 
>> unlimited.
>>
>> So how is it possible that the exception above can occur? Why does the 
>> log message indicate a queue size of 50?
>>
>> Regards,
>>
>> Abid
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/7f095b58-e471-43a7-9d20-d35dabd1a697%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/7f095b58-e471-43a7-9d20-d35dabd1a697%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b7edddf8-a619-4a3d-a25f-8f8a40af3e36%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Queue Size limited?

[Abid Hussain]

Hi all,

last restart of our cluster caused a heavy load as a couple of rivers are 
triggered which load alltogether about 40 GB of data into the (single-node) 
cluster.

This lead to massive exceptions like shown in the logs:
[6726]: index [my_index], type [my_doc_type], id [2390342], message 
[EsRejectedExecutionException[rejected execution (queue capacity 50) on 
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@4a94944]]

We didn't change any of the threadpools queue size settings (like 
threadpool.bulk.queue_size). AFAIK in this case the queue size is unlimited.

So how is it possible that the exception above can occur? Why does the log 
message indicate a queue size of 50?

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7f095b58-e471-43a7-9d20-d35dabd1a697%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How to wait for changes to be comitted?

[Abid Hussain]

Hi all,

In one of our integration test we do the following:

   1. create an index
   2. put some docs in the created index
   3. assert that documents exist in index

Step 3 sometimes fails if we don't put a Thread.sleep(1000) between 2 and 3.

After reading the docs, I think performing a FlushRequest might help:
client.admin().indices().flush(new FlushRequest(indexName)).actionGet();

My question is: are all changes commited when receiving the response of the 
flush request or is flush request performed in an asynchrounus manner?

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4dcb667c-2ab5-4457-85ff-70327e520c43%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: jdbc river keeping connection open

[Abid Hussain]

I noticed in the elasticsearch logs:
[2015-02-16 15:25:20,398][INFO ][river.jdbc.RiverMetrics  ] pipeline 
org.xbib.elasticsearch.plugin.jdbc.RiverPipeline@25042719 complete: river 
jdbc/ctv4ch_junit_abid_interview metrics: 1 rows, 0.23379861173437763 mean, 
(0.0 0.0 0.0), ingest metrics: elapsed 4 seconds, 0.0 bytes bytes, 0.0 
bytes avg, 0 MB/s
[2015-02-16 15:25:20,398][ERROR][river.jdbc.BulkNodeClient] after bulk [1] 
error
java.lang.InterruptedException
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
at 
org.elasticsearch.action.bulk.BulkProcessor.execute(BulkProcessor.java:325)
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.xbib.elasticsearch.plugin.jdbc.client.BulkProcessorHelper.flush(BulkProcessorHelper.java:43)

This seems to occur when the river, which has just been created, is removed 
again by the next test run. So it seems the an InterruptedException results 
in an open connection remaining.

I did the following: before deletion of river; I wait until the previously 
created one did finish (via _river/jdbc//_state. Then I delete.

This seems to work.

Regards,

Abid

Am Montag, 16. Februar 2015 14:43:37 UTC+1 schrieb Abid Hussain:
>
> Are you sure the deletion of a (non-existent) mapping "_river" makes sense?
>>
> The given code results in river deletion. If there is a better way to do 
> so, I would be happy if you could tell me.
>  
>
>> What connection is open? Database connection?
>>
> Yes, a database connection used by river remains open.
>
> Regards,
>
> Abid
>  
>
>> Jörg
>> Am 16.02.2015 12:29 schrieb "Abid Hussain" :
>>
>>> Hi all,
>>>
>>> we have some JUnit based integration tests in which we basically setup a 
>>> clean system, create a river and do some index operations like 
>>> inserting/updating documents. We are not using ElasticSearchIntegrationTest 
>>> but are running our tests against a dedicated cluster.
>>>
>>> In the setup which is performed before every test run we do the 
>>> following:
>>>
>>>1. remove jdbc river from last test run
>>>2. remove index from last test run
>>>3. truncate all tables in database
>>>
>>> We are facing the issue that step 3 sometimes blocks because there is an 
>>> open connection from the river.
>>>
>>> The deletion is performed by this code:
>>> DeleteMappingResponse response = client.admin().indices()
>>>  .prepareDeleteMapping("_river")
>>>  .setType(rivername)
>>>  .execute()
>>>  .actionGet();
>>>
>>> return response.isAcknowledged();
>>>
>>> Is this a known issue or do we have some misunderstanding of how to deal 
>>> with river deletion?
>>>
>>> May it be the case that the table truncation is performed when the river 
>>> deletion isn't fully completed? If so, is there a way to wait until river 
>>> deletion completed?
>>>
>>> Regards,
>>>
>>> Abid
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/00a6d81c-5f72-4c16-a814-dcb771d3111b%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/elasticsearch/00a6d81c-5f72-4c16-a814-dcb771d3111b%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3885a810-218e-44a9-b31d-2fd51a37c86e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: jdbc river keeping connection open

[Abid Hussain]

>
> Are you sure the deletion of a (non-existent) mapping "_river" makes sense?
>
The given code results in river deletion. If there is a better way to do 
so, I would be happy if you could tell me.
 

> What connection is open? Database connection?
>
Yes, a database connection used by river remains open.

Regards,

Abid
 

> Jörg
> Am 16.02.2015 12:29 schrieb "Abid Hussain"  >:
>
>> Hi all,
>>
>> we have some JUnit based integration tests in which we basically setup a 
>> clean system, create a river and do some index operations like 
>> inserting/updating documents. We are not using ElasticSearchIntegrationTest 
>> but are running our tests against a dedicated cluster.
>>
>> In the setup which is performed before every test run we do the following:
>>
>>1. remove jdbc river from last test run
>>2. remove index from last test run
>>3. truncate all tables in database
>>
>> We are facing the issue that step 3 sometimes blocks because there is an 
>> open connection from the river.
>>
>> The deletion is performed by this code:
>> DeleteMappingResponse response = client.admin().indices()
>>  .prepareDeleteMapping("_river")
>>  .setType(rivername)
>>  .execute()
>>  .actionGet();
>>
>> return response.isAcknowledged();
>>
>> Is this a known issue or do we have some misunderstanding of how to deal 
>> with river deletion?
>>
>> May it be the case that the table truncation is performed when the river 
>> deletion isn't fully completed? If so, is there a way to wait until river 
>> deletion completed?
>>
>> Regards,
>>
>> Abid
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/00a6d81c-5f72-4c16-a814-dcb771d3111b%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/00a6d81c-5f72-4c16-a814-dcb771d3111b%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7108a8e1-8695-40e2-a269-4d3b32773dee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jdbc river keeping connection open

[Abid Hussain]

Hi all,

we have some JUnit based integration tests in which we basically setup a 
clean system, create a river and do some index operations like 
inserting/updating documents. We are not using ElasticSearchIntegrationTest 
but are running our tests against a dedicated cluster.

In the setup which is performed before every test run we do the following:

   1. remove jdbc river from last test run
   2. remove index from last test run
   3. truncate all tables in database

We are facing the issue that step 3 sometimes blocks because there is an 
open connection from the river.

The deletion is performed by this code:
DeleteMappingResponse response = client.admin().indices()
 .prepareDeleteMapping("_river")
 .setType(rivername)
 .execute()
 .actionGet();

return response.isAcknowledged();

Is this a known issue or do we have some misunderstanding of how to deal 
with river deletion?

May it be the case that the table truncation is performed when the river 
deletion isn't fully completed? If so, is there a way to wait until river 
deletion completed?

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/00a6d81c-5f72-4c16-a814-dcb771d3111b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: real one shot jdbc river

[Abid Hussain]

So I tried the oneshot strategy with no success. The river is being rerun 
after restart of cluster.

Am Donnerstag, 12. Februar 2015 15:22:27 UTC+1 schrieb Abid Hussain:
>
> Hi all,
>
> we run a jdbc river once per night for a complete recreation of our index.
>
> The river - once it has been created - will run again after a restart of 
> elasticsearch cluster (single node cluster). But what we try to achieve is 
> a one shot river, which isn't being re-run after restart of cluster.
>
> Naturally we can trigger an event (e.g. as a cronjob) which - after the 
> estimated end of river run - deletes the river again. But this results in 
> additional efforts.
>
> The easiest strategy may be do delete the river right away after creating 
> it. But I suppose in this case the running river would be stopped...?
>
> *My question is: what is the best way to create a one shot river which is 
> not being re-run after restart of cluster?*
>
> I noticed the oneshot strategy but didn't find any documentation in 
> https://github.com/jprante/elasticsearch-river-jdbc/wiki/Strategies.
>
> Any help would be appreciated.
>
> Regards,
>
> Abid
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/dea5b9f2-36a5-4f09-b2c0-ca44157a4d9c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

real one shot jdbc river

[Abid Hussain]

Hi all,

we run a jdbc river once per night for a complete recreation of our index.

The river - once it has been created - will run again after a restart of 
elasticsearch cluster (single node cluster). But what we try to achieve is 
a one shot river, which isn't being re-run after restart of cluster.

Naturally we can trigger an event (e.g. as a cronjob) which - after the 
estimated end of river run - deletes the river again. But this results in 
additional efforts.

The easiest strategy may be do delete the river right away after creating 
it. But I suppose in this case the running river would be stopped...?

*My question is: what is the best way to create a one shot river which is 
not being re-run after restart of cluster?*

I noticed the oneshot strategy but didn't find any documentation 
in https://github.com/jprante/elasticsearch-river-jdbc/wiki/Strategies.

Any help would be appreciated.

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d642e1ff-64a1-4ac9-b2b3-bba38b5c07e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: jdbc river won't start

[Abid Hussain]

That was it. I deleted the river using
DELETE /_river/my_river/_meta

but that was wrong, I should have used:
DELETE /_river/my_river/

Thanks!


Am Mittwoch, 11. Februar 2015 15:13:47 UTC+1 schrieb Jörg Prante:
>
> You are sure you executed
>
> curl -XDELETE 'localhost:9200/_river/my_river/'
>
> before creating the new river with same name? I see there is '_version: 
> 12', this means, you did not stop the old river instance with DELETE.
>
> Jörg
>
> On Wed, Feb 11, 2015 at 2:38 PM, Abid Hussain  > wrote:
>
>> Hi,
>>
>> we are running ES on a single node cluster.
>>
>> I'm facing the issue that a river is not being started after being 
>> re-created. In detail, we first delete the index the river refers to, then 
>> delete the river and then create it again.
>>
>> *GET /_river/my_river/_meta*
>> {
>>"_index": "_river",
>>"_type": "my_river",
>>"_id": "_meta",
>>"_version": 12,
>>"found": true,
>>"_source": {
>>   "type": "jdbc",
>>   "jdbc": {
>>  "driver": "com.mysql.jdbc.Driver",
>>  "url": "jdbc:mysql://localhost:3306/my_db",
>>  "user": "user",
>>  "password": "***",
>>  "index": "my_index",
>>  "type": "my_customer",
>>  "sql": "SELECT * FROM my_customer"
>>   }
>>}
>> }
>>
>> Unfortunately, the river is not being started after creating it. When I 
>> request the river state it tells me that it ran last time yesterday:
>> {
>>"state": [
>>   {
>>  "name": "my_river",
>>  "type": "jdbc",
>>  "started": "2015-02-10T01:00:06.997Z",
>>  "last_active_begin": "2015-02-10T01:00:07.030Z",
>>  "last_active_end": "2015-02-10T01:53:34.364Z",
>>  "map": {
>> "lastExecutionStartDate": 1423530007825,
>> "aborted": false,
>> "lastEndDate": 1423533216703,
>> "counter": 1,
>> "lastExecutionEndDate": 1423533214363,
>> "lastStartDate": 1423530007608,
>> "suspended": false
>>  }
>>   }
>>]
>> }
>>
>> The connection properties didn't change since last succesful run.
>>
>> In the logs is nothing said, except the message after deleting the index.
>>
>> Any idea what could be the problem?
>>
>> Regards,
>>
>> Abid
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/5e8ffbc7-c33f-4157-b8fe-a170d38fc1fc%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/5e8ffbc7-c33f-4157-b8fe-a170d38fc1fc%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2b5be310-90cb-474a-82fd-cd233573d78c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jdbc river won't start

[Abid Hussain]

Hi,

we are running ES on a single node cluster.

I'm facing the issue that a river is not being started after being 
re-created. In detail, we first delete the index the river refers to, then 
delete the river and then create it again.

*GET /_river/my_river/_meta*
{
   "_index": "_river",
   "_type": "my_river",
   "_id": "_meta",
   "_version": 12,
   "found": true,
   "_source": {
  "type": "jdbc",
  "jdbc": {
 "driver": "com.mysql.jdbc.Driver",
 "url": "jdbc:mysql://localhost:3306/my_db",
 "user": "user",
 "password": "***",
 "index": "my_index",
 "type": "my_customer",
 "sql": "SELECT * FROM my_customer"
  }
   }
}

Unfortunately, the river is not being started after creating it. When I 
request the river state it tells me that it ran last time yesterday:
{
   "state": [
  {
 "name": "my_river",
 "type": "jdbc",
 "started": "2015-02-10T01:00:06.997Z",
 "last_active_begin": "2015-02-10T01:00:07.030Z",
 "last_active_end": "2015-02-10T01:53:34.364Z",
 "map": {
"lastExecutionStartDate": 1423530007825,
"aborted": false,
"lastEndDate": 1423533216703,
"counter": 1,
"lastExecutionEndDate": 1423533214363,
"lastStartDate": 1423530007608,
"suspended": false
 }
  }
   ]
}

The connection properties didn't change since last succesful run.

In the logs is nothing said, except the message after deleting the index.

Any idea what could be the problem?

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5e8ffbc7-c33f-4157-b8fe-a170d38fc1fc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: log4j: Configuring SMTPAppender

[Abid Hussain]

Just for evaluation purposes, I tried to configure a simple 
logging.properties in $ES_HOME/config but not even this had any success.

log4j.rootLogger=DEBUG, A1

log4j.appender.A1=org.apache.log4j.RollingFileAppender
log4j.appender.A1.File=/home/my_user/elasticsearch.log
log4j.appender.A1.MaxFileSize=4KB
log4j.appender.A1.MaxBackupIndex=5
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=[%d{ISO8601}][%-5p][%-25c] %m%n
log4j.appender.A1.Encoding=UTF-8

No file /home/my_user/elasticsearch.log is being created when starting 
elasticsearch (directory /home/my_user exists) and I have no idea what I'm 
doing wrong... :-(

Regards,

Abid

Am Montag, 9. Februar 2015 15:31:01 UTC+1 schrieb Abid Hussain:
>
> Hi all,
>
> I'd like to configure a smtpappender in logging.yml so that an email is 
> sent whenever a message is logged.
>
> Unfortunately, the yml configuration isn't very well documented on log4j 
> homepage and I wasn't able to find any working example in the www.
>
> I also tried to put a separate logging.properties in $ES_HOME$/config, but 
> didn't get it working.
>
> My last try (not working - no email is being sent):
> es.logger.level: INFO
> rootLogger: ${es.logger.level}, console, file, email
> logger:
>   ...
>
> appender:
>   console:
> 
>   email:
> type: smtp
> smtpHost: my.smtp.host
> from: e...@example.net
> to: ad...@example.net
> subject: ElasticSearch Warning
> layout:
>   type: pattern
>   conversionPattern: .
>
> So, does anybody have a working example to do so?
>
> Regards,
>
> Abid
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d50ebd43-6c92-473c-b5f2-491623b4b093%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

log4j: Configuring SMTPAppender

[Abid Hussain]

Hi all,

I'd like to configure a smtpappender in logging.yml so that an email is 
sent whenever a message is logged.

Unfortunately, the yml configuration isn't very well documented on log4j 
homepage and I wasn't able to find any working example in the www.

I also tried to put a separate logging.properties in $ES_HOME$/config, but 
didn't get it working.

My last try (not working - no email is being sent):
es.logger.level: INFO
rootLogger: ${es.logger.level}, console, file, email
logger:
  ...

appender:
  console:

  email:
type: smtp
smtpHost: my.smtp.host
from: e...@example.net
to: ad...@example.net
subject: ElasticSearch Warning
layout:
  type: pattern
  conversionPattern: .

So, does anybody have a working example to do so?

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/58c92f42-0046-453e-add0-3b726cafa822%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Recommended setup for using TransportClient in Webapp

[Abid Hussain]

Thanks a lot for you quick help.

Am Mittwoch, 4. Februar 2015 10:36:52 UTC+1 schrieb David Pilato:
>
> One single instance.
>
> -- 
> *David Pilato* | *Technical Advocate* | *Elasticsearch.com 
> <http://Elasticsearch.com>*
> @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr 
> <https://twitter.com/elasticsearchfr> | @scrutmydocs 
> <https://twitter.com/scrutmydocs>
>
>
>  
> Le 4 févr. 2015 à 10:12, Abid Hussain  > a écrit :
>
> Hi all,
>
> I wonder what is the recommended setup for using TransportClient in a 
> multi-threaded environment with parallel search requests (i.e. a webapp).
>
> Basically, I see two alternatives:
>
>1. Using a single instance of TransportClient for all incoming search 
>requests
>2. Creating a new instance of TransportClient for each incoming search 
>request
>
>
> What do you think?
>
> Regards,
>
> Abid
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/ba868b91-2ff2-47eb-9ebe-6ce481211a3d%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/elasticsearch/ba868b91-2ff2-47eb-9ebe-6ce481211a3d%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d47da39d-26f4-418a-a5b8-45a4158be04c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Recommended setup for using TransportClient in Webapp

[Abid Hussain]

Hi all,

I wonder what is the recommended setup for using TransportClient in a 
multi-threaded environment with parallel search requests (i.e. a webapp).

Basically, I see two alternatives:

   1. Using a single instance of TransportClient for all incoming search 
   requests
   2. Creating a new instance of TransportClient for each incoming search 
   request


What do you think?

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ba868b91-2ff2-47eb-9ebe-6ce481211a3d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Obtain jdbc river state via java api

[Abid Hussain]

Thanks, so I implement the state request with http get.

Regards,

Abid

Am Montag, 2. Februar 2015 17:50:53 UTC+1 schrieb Jörg Prante:
>
> There is no stateful river concept in Elasticsearch.
>
> Jörg
>
> On Mon, Feb 2, 2015 at 3:57 PM, Abid Hussain  > wrote:
>
>> Hi all,
>>
>> is there a way to obtain state of a certain river (like *GET 
>> /_river/jdbc//_state*) via TransportClient or any other kind 
>> of java api call?
>>
>> If not, I would have to perform a http get request programmatically and 
>> parse json result, I guess...
>>
>> Regards,
>>
>> Abid
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/eda0c348-4938-4900-a822-37aa520da7fe%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/eda0c348-4938-4900-a822-37aa520da7fe%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/cb420cbb-a66f-4472-93ec-fc6b7fd1bc57%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Obtain jdbc river state via java api

[Abid Hussain]

Hi all,

is there a way to obtain state of a certain river (like *GET 
/_river/jdbc//_state*) via TransportClient or any other kind of 
java api call?

If not, I would have to perform a http get request programmatically and 
parse json result, I guess...

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/eda0c348-4938-4900-a822-37aa520da7fe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Connect to standalone node using TransportClient

[Abid Hussain]

Thanks to both, David and Jürgen.

I used Davids solution which works well for know and keep in mind Jürgens 
proposal for production installation.

Best regards,

Abid

Am Donnerstag, 29. Januar 2015 09:05:15 UTC+1 schrieb Abid Hussain:
>
> Hi all,
>
> I'm running a standalone node (by using *node.local: true* in 
> elasticsearch.yml) and want to connect to this node via TransportClient 
> which fails.
>
> Connecting to the node via Sense succeeds.
>
> I didn't change the cluster.name property in elasticsearch.yml.
>
> My Code is:
> Client client = new TransportClient().addTransportAddress(new 
> InetSocketTransportAddress("xxx.xxx.xxx.xxx", 9300));
>
> Result is:
> *Exception in thread "main" 
> org.elasticsearch.client.transport.NoNodeAvailableException: None of the 
> configured nodes are available: []*
>
> Using
> Settings settings = ImmutableSettings.settingsBuilder().put("node.local", 
> "true").build();
> Client client = new TransportClient().addTransportAddress(new 
> InetSocketTransportAddress("xxx.xxx.xxx.xxx", 9300));
> doesn't help (results in same Exception).
>
> Any ideas what I can do? Is it possible at all to use TransportClient for 
> setting up a connection to a standalone node?
>
> Regards,
>
> Abid
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/93d984c6-410a-4099-801f-1da4addd5d8f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Connect to standalone node using TransportClient

[Abid Hussain]

Sorry, I overread the "not" in you post ;-)

Removing *node.local:true* works in terms that I am then able to connect to 
node via TransportClient.

The reason for using *node:local:true *is that *I want to run several 
independent nodes in my network that do not communicate with each other.*

...?

Regards,

Abid

Am Donnerstag, 29. Januar 2015 14:49:03 UTC+1 schrieb David Pilato:
>
> You can also run a Node in your application. But that's another story I 
> guess.
>
> I said "remove node.local".
> Does this work?
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
> Le 29 janv. 2015 à 14:43, Abid Hussain  > a écrit :
>
> Thanks for help. As you can see in the original quesion above, I already 
> tried setting node.local: true.
>
> This works on server side, but I'm not able to connect to the node via 
> TransportClient using the Java API.
>
> My requirements are:
> * run elasticsearch as single node
> * Use Java API to perform requests
>
> And my question is: what is the recommend way to fullfill these 
> requirements?
>
> Regards,
>
> Abid
>
> Am Donnerstag, 29. Januar 2015 14:36:01 UTC+1 schrieb David Pilato:
>>
>> What about not setting node.local: true?
>>
>> --
>> David ;-)
>> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>>
>> Le 29 janv. 2015 à 14:18, Abid Hussain  a 
>> écrit :
>>
>> After doing some research, It seems to me that it is not possible to 
>> connect to a node configured as *node.local: true* using TransportClient.
>>
>> ... which leads me to the question:
>> What is the recommended way to setup a single node instance and to 
>> connect to it via Java API?
>>
>> Regards,
>>
>> Abid
>>
>>
>> Am Donnerstag, 29. Januar 2015 09:05:15 UTC+1 schrieb Abid Hussain:
>>>
>>> Hi all,
>>>
>>> I'm running a standalone node (by using *node.local: true* in 
>>> elasticsearch.yml) and want to connect to this node via TransportClient 
>>> which fails.
>>>
>>> Connecting to the node via Sense succeeds.
>>>
>>> I didn't change the cluster.name property in elasticsearch.yml.
>>>
>>> My Code is:
>>> Client client = new TransportClient().addTransportAddress(new 
>>> InetSocketTransportAddress("xxx.xxx.xxx.xxx", 9300));
>>>
>>> Result is:
>>> *Exception in thread "main" 
>>> org.elasticsearch.client.transport.NoNodeAvailableException: None of the 
>>> configured nodes are available: []*
>>>
>>> Using
>>> Settings settings = 
>>> ImmutableSettings.settingsBuilder().put("node.local", "true").build();
>>> Client client = new TransportClient().addTransportAddress(new 
>>> InetSocketTransportAddress("xxx.xxx.xxx.xxx", 9300));
>>> doesn't help (results in same Exception).
>>>
>>> Any ideas what I can do? Is it possible at all to use TransportClient 
>>> for setting up a connection to a standalone node?
>>>
>>> Regards,
>>>
>>> Abid
>>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/e249b162-3a85-40f6-bc86-65d7a4e3809b%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/e249b162-3a85-40f6-bc86-65d7a4e3809b%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>>  -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/851a2596-b9ef-459d-aab0-9884e46b3870%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/elasticsearch/851a2596-b9ef-459d-aab0-9884e46b3870%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0b3caa38-6c11-45b0-ab6d-d5436d125726%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Connect to standalone node using TransportClient

[Abid Hussain]

Thanks for help. As you can see in the original quesion above, I already 
tried setting node.local: true.

This works on server side, but I'm not able to connect to the node via 
TransportClient using the Java API.

My requirements are:
* run elasticsearch as single node
* Use Java API to perform requests

And my question is: what is the recommend way to fullfill these 
requirements?

Regards,

Abid

Am Donnerstag, 29. Januar 2015 14:36:01 UTC+1 schrieb David Pilato:
>
> What about not setting node.local: true?
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
> Le 29 janv. 2015 à 14:18, Abid Hussain  > a écrit :
>
> After doing some research, It seems to me that it is not possible to 
> connect to a node configured as *node.local: true* using TransportClient.
>
> ... which leads me to the question:
> What is the recommended way to setup a single node instance and to connect 
> to it via Java API?
>
> Regards,
>
> Abid
>
>
> Am Donnerstag, 29. Januar 2015 09:05:15 UTC+1 schrieb Abid Hussain:
>>
>> Hi all,
>>
>> I'm running a standalone node (by using *node.local: true* in 
>> elasticsearch.yml) and want to connect to this node via TransportClient 
>> which fails.
>>
>> Connecting to the node via Sense succeeds.
>>
>> I didn't change the cluster.name property in elasticsearch.yml.
>>
>> My Code is:
>> Client client = new TransportClient().addTransportAddress(new 
>> InetSocketTransportAddress("xxx.xxx.xxx.xxx", 9300));
>>
>> Result is:
>> *Exception in thread "main" 
>> org.elasticsearch.client.transport.NoNodeAvailableException: None of the 
>> configured nodes are available: []*
>>
>> Using
>> Settings settings = ImmutableSettings.settingsBuilder().put("node.local", 
>> "true").build();
>> Client client = new TransportClient().addTransportAddress(new 
>> InetSocketTransportAddress("xxx.xxx.xxx.xxx", 9300));
>> doesn't help (results in same Exception).
>>
>> Any ideas what I can do? Is it possible at all to use TransportClient for 
>> setting up a connection to a standalone node?
>>
>> Regards,
>>
>> Abid
>>
>  -- 
> You received this message because you are subscribed to the Google Groups 
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to elasticsearc...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/e249b162-3a85-40f6-bc86-65d7a4e3809b%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/elasticsearch/e249b162-3a85-40f6-bc86-65d7a4e3809b%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/851a2596-b9ef-459d-aab0-9884e46b3870%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Connect to standalone node using TransportClient

[Abid Hussain]

After doing some research, It seems to me that it is not possible to 
connect to a node configured as *node.local: true* using TransportClient.

... which leads me to the question:
What is the recommended way to setup a single node instance and to connect 
to it via Java API?

Regards,

Abid


Am Donnerstag, 29. Januar 2015 09:05:15 UTC+1 schrieb Abid Hussain:
>
> Hi all,
>
> I'm running a standalone node (by using *node.local: true* in 
> elasticsearch.yml) and want to connect to this node via TransportClient 
> which fails.
>
> Connecting to the node via Sense succeeds.
>
> I didn't change the cluster.name property in elasticsearch.yml.
>
> My Code is:
> Client client = new TransportClient().addTransportAddress(new 
> InetSocketTransportAddress("xxx.xxx.xxx.xxx", 9300));
>
> Result is:
> *Exception in thread "main" 
> org.elasticsearch.client.transport.NoNodeAvailableException: None of the 
> configured nodes are available: []*
>
> Using
> Settings settings = ImmutableSettings.settingsBuilder().put("node.local", 
> "true").build();
> Client client = new TransportClient().addTransportAddress(new 
> InetSocketTransportAddress("xxx.xxx.xxx.xxx", 9300));
> doesn't help (results in same Exception).
>
> Any ideas what I can do? Is it possible at all to use TransportClient for 
> setting up a connection to a standalone node?
>
> Regards,
>
> Abid
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e249b162-3a85-40f6-bc86-65d7a4e3809b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Connect to standalone node using TransportClient

[Abid Hussain]

Hi all,

I'm running a standalone node (by using *node.local: true* in 
elasticsearch.yml) and want to connect to this node via TransportClient 
which fails.

Connecting to the node via Sense succeeds.

I didn't change the cluster.name property in elasticsearch.yml.

My Code is:
Client client = new TransportClient().addTransportAddress(new 
InetSocketTransportAddress("xxx.xxx.xxx.xxx", 9300));

Result is:
*Exception in thread "main" 
org.elasticsearch.client.transport.NoNodeAvailableException: None of the 
configured nodes are available: []*

Using
Settings settings = ImmutableSettings.settingsBuilder().put("node.local", 
"true").build();
Client client = new TransportClient().addTransportAddress(new 
InetSocketTransportAddress("xxx.xxx.xxx.xxx", 9300));
doesn't help (results in same Exception).

Any ideas what I can do? Is it possible at all to use TransportClient for 
setting up a connection to a standalone node?

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8cb2153c-a168-4c0e-bbf4-da75c4e8df39%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How does sorting on _id work?

[Abid Hussain]

... can it be that _id is treated as string? If so, is there any way 
retrieve the max _id field with treating _id as integer?

Am Dienstag, 27. Januar 2015 19:24:41 UTC+1 schrieb Abid Hussain:
>
> Hi all,
>
> I want to determine the doc with max and min _id value. So, when I run 
> this query:
> GET /my_index/order/_search
> {
> "fields": [ "_id" ],
> "sort": [
>{ "_uid": { "order": "desc" } }
> ],
> "size": 1
> }
> I get a result:
> {
>...
>"hits": {
>   ...
>   "hits": [
>  {
> "_index": "my_index",
> "_type": "order",
> "_id": "99",
> "_score": null,
> "sort": [
>"order#99"
> ]
>  }
>   ]
>}
> }
>
> There is definitevely a doc with _id value 11132106 in index which I would 
> have expected as result.
>
> And, when I run the same search with *order asc* I get a result with 
> _id 100 which is higher than 99...?
>
> What am I doing wrong?
>
> Regards,
>
> Abid
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d3e540ef-8e40-4f38-b655-db9da9b64946%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How does sorting on _id work?

[Abid Hussain]

Hi all,

I want to determine the doc with max and min _id value. So, when I run this 
query:
GET /my_index/order/_search
{
"fields": [ "_id" ],
"sort": [
   { "_uid": { "order": "desc" } }
],
"size": 1
}
I get a result:
{
   ...
   "hits": {
  ...
  "hits": [
 {
"_index": "my_index",
"_type": "order",
"_id": "99",
"_score": null,
"sort": [
   "order#99"
]
 }
  ]
   }
}

There is definitevely a doc with _id value 11132106 in index which I would 
have expected as result.

And, when I run the same search with *order asc* I get a result with 
_id 100 which is higher than 99...?

What am I doing wrong?

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/92429b89-0741-40cc-9d76-8e1a36e5c1f5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: jdbc river re-indexing after each start of server?

[Abid Hussain]

Thanks for you quick help!

Am Montag, 26. Januar 2015 13:15:26 UTC+1 schrieb Jörg Prante:
>
> You should delete the river instance after usage. Otherwise, the river is 
> executed each time the node starts.
>
> Jörg
>
> On Mon, Jan 26, 2015 at 1:09 PM, Abid Hussain  > wrote:
>
>> Hi all,
>>
>> I configured a river for one-time indexing like:
>> PUT /_river/my_river/_meta
>> {
>>   "type": "jdbc",
>>   "jdbc": {
>>  "driver": "com.mysql.jdbc.Driver",
>>  "url": "jdbc:mysql://192.168.110.180:3306/my_db",
>>  "user": "user",
>>  "password": "password",
>>  "index": "my_idx",
>>  "type": "order",
>>  "sql": "SELECT o.order_id AS _id, o.order_name FROM orders o"
>>   }
>> }
>>
>>
>> After performing the PUT, the data is being indexed. When finished, I 
>> stop the server and start it again. The CPU load now is remarkably high. 
>> Looking at  /_river/jdbc/*/_state it tells me that *last_active_end* is 
>> null.
>>
>> So, does the restart of elasicsearch trigger a complete reindexing? If 
>> so, how can I avoid this - e.g by using timestamps?
>>
>> Regards,
>>
>> Abid
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/b006c70b-93f7-48a9-b423-136d288ef7f3%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/b006c70b-93f7-48a9-b423-136d288ef7f3%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/0be9e039-87e2-44fa-8952-67711e83cfe8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jdbc river re-indexing after each start of server?

[Abid Hussain]

Hi all,

I configured a river for one-time indexing like:
PUT /_river/my_river/_meta
{
  "type": "jdbc",
  "jdbc": {
 "driver": "com.mysql.jdbc.Driver",
 "url": "jdbc:mysql://192.168.110.180:3306/my_db",
 "user": "user",
 "password": "password",
 "index": "my_idx",
 "type": "order",
 "sql": "SELECT o.order_id AS _id, o.order_name FROM orders o"
  }
}


After performing the PUT, the data is being indexed. When finished, I stop 
the server and start it again. The CPU load now is remarkably high. Looking 
at  /_river/jdbc/*/_state it tells me that *last_active_end* is null.

So, does the restart of elasicsearch trigger a complete reindexing? If so, 
how can I avoid this - e.g by using timestamps?

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b006c70b-93f7-48a9-b423-136d288ef7f3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Keeping jdbc river in sync with DB

[Abid Hussain]

Thanks for you quick reply.

You say *There is no synchronization between DB rows and JSON docs.*

So how is the index being updated? I guess that new entries are being added 
to index. *Does the river also recognize differences between single columns 
and their corresponding field in index and merge them into index?*

Regards,

Abid

Am Mittwoch, 21. Januar 2015 17:02:27 UTC+1 schrieb Jörg Prante:
>
> Yes, you can set up the JDBC plugin with a 'schedule' parameter so that it 
> is started every minute.
>
> See the doc at
>
> https://github.com/jprante/elasticsearch-river-jdbc
>
> Note, if you delete rows in DB, this is not per se detected by JDBC 
> plugin. There is no synchronization between DB rows and JSON docs. You can 
> only work with documents from the rows that your SQL statement discovers. 
> JSON doc deletions require some additional effort.
>
> Jörg
>
> On Wed, Jan 21, 2015 at 4:56 PM, Abid Hussain  > wrote:
>
>> Hi all,
>>
>> I'm trying to figure out how to configure jdbc river in production 
>> environment.
>>
>> We want to index a DB with appr. 10% write queries and 90% read queries 
>> and need to keep ES index in sync with DB.
>>
>> So I wonder how two configure river to keep Index in near-realtime Sync 
>> with DB.
>>
>> I'm thinking about pulling the DB changes every minute. Is jdbc-river 
>> capable of updating it's already indexed data and is this a good strategy?
>>
>> Regards,
>>
>> Abid
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/99857c16-1ba3-4267-94c8-7538971f366b%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/99857c16-1ba3-4267-94c8-7538971f366b%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/227dada3-0ea0-442a-80c5-e8a59b0462fa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

JDBC river in production environment

[Abid Hussain]

Hi all,

I'm quite new to ES and wonder how/if jdbc river is ready for indexing 
RDBMS in production enviromnent as in GitHub 
 it 
says about River:
*about to be deprecated by Elasticsearch core team*

So is it a good idea to use jdbc river or is there a better strategy?

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/a9bc1c06-a829-488e-8c33-1d99e63fa19b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Keeping jdbc river in sync with DB

[Abid Hussain]

Hi all,

I'm trying to figure out how to configure jdbc river in production 
environment.

We want to index a DB with appr. 10% write queries and 90% read queries and 
need to keep ES index in sync with DB.

So I wonder how two configure river to keep Index in near-realtime Sync 
with DB.

I'm thinking about pulling the DB changes every minute. Is jdbc-river 
capable of updating it's already indexed data and is this a good strategy?

Regards,

Abid

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/99857c16-1ba3-4267-94c8-7538971f366b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.