Re: How to write a custom river
So I changed on minor thing, which was the package a zip assembly. I had
failed to notice this until I checked out and built your project. I hadn't
looked at the pom figuring it was a coding issue :)
You can find the code here:
https://github.com/robottaway/simple-es-river
Then I created the assembly using "mvn package", which put a zip file in
./target/releases/example-river-1.0-SNAPSHOT.zip THAT IS WHAT YOU NEED TO
MAKE THIS WORK.
I see what Luca was talking about, noticed you updated your REST example
changing "example_river" to "river".
Next I installed this in Elasticsearch 1.1.1 I have running local.
robs-mbp-3:elasticsearch-1.1.1 rob$ bin/plugin --url
file:///Users/rob/development/git/example-river/target/releases/example-river-1.0-SNAPSHOT.zip
--install blarggh
-> Installing blarggh...
Trying
file:/Users/rob/development/git/example-river/target/releases/example-river-1.0-SNAPSHOT.zip...
Downloading
DONE
Then
robs-mbp-3:elasticsearch-1.1.1 rob$ curl -X PUT -H "Content-Type:
application/json" http://localhost:9200/_river/example_river/_meta -d '{
> "type": "river",
> "river":{
> "blah":"blah"
> }
> }'
{"_index":"_river","_type":"example_river","_id":"_meta","_version":1,"created":true
as you can see works just fine. Hit me up if you have any trouble.
cheers,
Rob
On Saturday, May 3, 2014 7:10:51 PM UTC-7, Rob Ottaway wrote:
>
> Wow I am (mostly) a Java developer by trade and I really steered you
> wrong. Sorry! You are definitely registering the module, chock up my poor
> answer to a looong day at work.
>
> I think far as I can see you have everything you need in place. You
> register the plugin as "river" and set the type in your REST POST to
> "river" which is right though Luca thinks otherwise. Maybe he knows
> something I don't, I'm not batting exactly 1000 here. Let me check your
> code out and install it locally, do a little dev QA.
>
> -Rob
>
>
>
> On Friday, May 2, 2014 1:59:34 PM UTC-7, Joshua Chan wrote:
>>
>> I should admit I am not a Java developer by trade, so I'm trying pretty
>> hard here.
>>
>> When I call .registerRiver, I am passing the module, not the river.
>> RiverImpl.Type is just the string "river".
>>
>> public void onModule(RiversModule module) {
>> module.registerRiver(RiverImpl.TYPE, ModuleImpl.class);
>> }
>>
>> If this isn't what you mean, please dial me in a bit more. Guice is a
>> foreign to me too.
>>
>>
>>
>> On Friday, May 2, 2014 12:34:47 PM UTC-5, Rob Ottaway wrote:
>>>
>>> Oh sorry module is fine, it's the call to module.registerRiver which is
>>> being passed the River itself and not the River Guice module. Try that
>>> change.
>>>
>>>
>>>
>>>
>>> On Fri, May 2, 2014 at 10:33 AM, Rob Ottaway wrote:
>>>
>>>> I think he means in your Guice module. You are registering the WRONG
>>>> thing ;)
>>>>
>>>>
>>>> On Fri, May 2, 2014 at 6:49 AM, Joshua Chan wrote:
>>>>
>>>>> I've tried this too with no luck
>>>>>
>>>>> http://localhost:9200/_river/example_river/_meta
>>>>> {
>>>>> "type": "river",
>>>>> "river":{
>>>>> "blah":"blah"
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>> On Friday, May 2, 2014 3:31:23 AM UTC-5, Luca Cavanna wrote:
>>>>>
>>>>>> Hi Joshua,
>>>>>> the package is not an issue if you are using the default one for your
>>>>>> classes. Looking deeper, the type of the river that you try to register
>>>>>> with your rest call doesn't match the type of the river you registered
>>>>>> in
>>>>>> the plugin when you did module.registerRiver(type, riverclass).
>>>>>>
>>>>>> Cheers
>>>>>> Luca
>>>>>>
>>>>>> On Friday, May 2, 2014 6:08:28 AM UTC+2, Rob Ottaway wrote:
>>>>>>
>>>>>>> I should have sent you the following earlier rather than a non-river
>>>>>>> plugin:
>>>>&g
Re: How to write a custom river
Wow I am (mostly) a Java developer by trade and I really steered you wrong.
Sorry! You are definitely registering the module, chock up my poor answer
to a looong day at work.
I think far as I can see you have everything you need in place. You
register the plugin as "river" and set the type in your REST POST to
"river" which is right though Luca thinks otherwise. Maybe he knows
something I don't, I'm not batting exactly 1000 here. Let me check your
code out and install it locally, do a little dev QA.
-Rob
On Friday, May 2, 2014 1:59:34 PM UTC-7, Joshua Chan wrote:
>
> I should admit I am not a Java developer by trade, so I'm trying pretty
> hard here.
>
> When I call .registerRiver, I am passing the module, not the river.
> RiverImpl.Type is just the string "river".
>
> public void onModule(RiversModule module) {
> module.registerRiver(RiverImpl.TYPE, ModuleImpl.class);
> }
>
> If this isn't what you mean, please dial me in a bit more. Guice is a
> foreign to me too.
>
>
>
> On Friday, May 2, 2014 12:34:47 PM UTC-5, Rob Ottaway wrote:
>>
>> Oh sorry module is fine, it's the call to module.registerRiver which is
>> being passed the River itself and not the River Guice module. Try that
>> change.
>>
>>
>>
>>
>> On Fri, May 2, 2014 at 10:33 AM, Rob Ottaway wrote:
>>
>>> I think he means in your Guice module. You are registering the WRONG
>>> thing ;)
>>>
>>>
>>> On Fri, May 2, 2014 at 6:49 AM, Joshua Chan wrote:
>>>
>>>> I've tried this too with no luck
>>>>
>>>> http://localhost:9200/_river/example_river/_meta
>>>> {
>>>> "type": "river",
>>>> "river":{
>>>> "blah":"blah"
>>>> }
>>>> }
>>>>
>>>>
>>>> On Friday, May 2, 2014 3:31:23 AM UTC-5, Luca Cavanna wrote:
>>>>
>>>>> Hi Joshua,
>>>>> the package is not an issue if you are using the default one for your
>>>>> classes. Looking deeper, the type of the river that you try to register
>>>>> with your rest call doesn't match the type of the river you registered in
>>>>> the plugin when you did module.registerRiver(type, riverclass).
>>>>>
>>>>> Cheers
>>>>> Luca
>>>>>
>>>>> On Friday, May 2, 2014 6:08:28 AM UTC+2, Rob Ottaway wrote:
>>>>>
>>>>>> I should have sent you the following earlier rather than a non-river
>>>>>> plugin:
>>>>>>
>>>>>> the plugin:
>>>>>> https://github.com/elasticsearch/elasticsearch-
>>>>>> river-rabbitmq/blob/master/src/main/java/org/
>>>>>> elasticsearch/plugin/river/rabbitmq/RabbitmqRiverPlugin.java
>>>>>>
>>>>>> The river implementation:
>>>>>> https://github.com/elasticsearch/elasticsearch-
>>>>>> river-rabbitmq/blob/master/src/main/java/org/
>>>>>> elasticsearch/river/rabbitmq/RabbitmqRiver.java
>>>>>>
>>>>>> The module:
>>>>>> https://github.com/elasticsearch/elasticsearch-
>>>>>> river-rabbitmq/blob/master/src/main/java/org/
>>>>>> elasticsearch/river/rabbitmq/RabbitmqRiverModule.java
>>>>>>
>>>>>> Looks like you are registering the river implementation rather than
>>>>>> the river module hence the not working. Had to look at an example I know
>>>>>> works to figure it out.
>>>>>>
>>>>>> -Rob
>>>>>>
>>>>>>
>>>>>> On Thu, May 1, 2014 at 8:04 PM, Joshua Chan
>>>>>> wrote:
>>>>>>
>>>>>>> So, that's what I did, but no love... I checked in the latest.
>>>>>>>
>>>>>>> -Josh
>>>>>>>
>>>>>>>
>>>>>>> On Thursday, May 1, 2014 9:49:11 PM UTC-5, Rob Ottaway wrote:
>>>>>>>>
>>>>>>>> Look at this plugin for help:
>>>>>>>>
>>>>>>>> https://github.com/elasticsearch/elasticsearch-cloud-aws/
>>>>>>>> blob/master/src/main/resources/es-plugin.properties
>>>>>>>>
>>>>>>>> Yes it needs to be the FQN.
>>>>>>&g
Re: Handling updates from multiple sources
I agree wholly that this is a pretty normal use case to want to be able to bulk index now and again while maintaining an index with updates in (near)realtime. I question your proposed solutione. If the "live updating job" does not feed you whole complete documents from the system or truth (db right?) then you aren't going to be able to use versioned documents. Versioned docs is how you solve this type of situation in Elasticsearch. You would do better to alter your "live updates" to collect all the doc from db (it has to be updated there anyhow right?) for live update index ops and index while using versions to solve the race condition with the bulk job. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAB-KG0uteQhcBYpL7H-yb2W-oxfDBFEEjqs-%3DbDgvsHfMgs5JQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: How to write a custom river
Oh sorry module is fine, it's the call to module.registerRiver which is
being passed the River itself and not the River Guice module. Try that
change.
On Fri, May 2, 2014 at 10:33 AM, Rob Ottaway wrote:
> I think he means in your Guice module. You are registering the WRONG thing
> ;)
>
>
> On Fri, May 2, 2014 at 6:49 AM, Joshua Chan > wrote:
>
>> I've tried this too with no luck
>>
>> http://localhost:9200/_river/example_river/_meta
>> {
>> "type": "river",
>> "river":{
>> "blah":"blah"
>> }
>> }
>>
>>
>> On Friday, May 2, 2014 3:31:23 AM UTC-5, Luca Cavanna wrote:
>>
>>> Hi Joshua,
>>> the package is not an issue if you are using the default one for your
>>> classes. Looking deeper, the type of the river that you try to register
>>> with your rest call doesn't match the type of the river you registered in
>>> the plugin when you did module.registerRiver(type, riverclass).
>>>
>>> Cheers
>>> Luca
>>>
>>> On Friday, May 2, 2014 6:08:28 AM UTC+2, Rob Ottaway wrote:
>>>
>>>> I should have sent you the following earlier rather than a non-river
>>>> plugin:
>>>>
>>>> the plugin:
>>>> https://github.com/elasticsearch/elasticsearch-
>>>> river-rabbitmq/blob/master/src/main/java/org/
>>>> elasticsearch/plugin/river/rabbitmq/RabbitmqRiverPlugin.java
>>>>
>>>> The river implementation:
>>>> https://github.com/elasticsearch/elasticsearch-
>>>> river-rabbitmq/blob/master/src/main/java/org/
>>>> elasticsearch/river/rabbitmq/RabbitmqRiver.java
>>>>
>>>> The module:
>>>> https://github.com/elasticsearch/elasticsearch-
>>>> river-rabbitmq/blob/master/src/main/java/org/
>>>> elasticsearch/river/rabbitmq/RabbitmqRiverModule.java
>>>>
>>>> Looks like you are registering the river implementation rather than the
>>>> river module hence the not working. Had to look at an example I know works
>>>> to figure it out.
>>>>
>>>> -Rob
>>>>
>>>>
>>>> On Thu, May 1, 2014 at 8:04 PM, Joshua Chan wrote:
>>>>
>>>>> So, that's what I did, but no love... I checked in the latest.
>>>>>
>>>>> -Josh
>>>>>
>>>>>
>>>>> On Thursday, May 1, 2014 9:49:11 PM UTC-5, Rob Ottaway wrote:
>>>>>>
>>>>>> Look at this plugin for help:
>>>>>>
>>>>>> https://github.com/elasticsearch/elasticsearch-cloud-aws/
>>>>>> blob/master/src/main/resources/es-plugin.properties
>>>>>>
>>>>>> Yes it needs to be the FQN.
>>>>>>
>>>>>> On Thursday, May 1, 2014 5:47:31 PM UTC-7, Joshua Chan wrote:
>>>>>>>
>>>>>>> Thanks Rob. Someone else also told me the plugin property should be
>>>>>>> the fully qualified name. I didn't declare a package, so I guess I'm
>>>>>>> using
>>>>>>> the default package, and I thought I had the namespacing right since
>>>>>>> IntelliJ corrected the class name when I wrote it.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>>
>>>>>>> -Josh
>>>>>>>
>>>>>>>
>>>>>>> On Thursday, May 1, 2014 5:23:25 PM UTC-5, Rob Ottaway wrote:
>>>>>>>>
>>>>>>>> Look at this file in your BB repo:
>>>>>>>>
>>>>>>>> https://bitbucket.org/futurechan/example-river/src/fd23648c3
>>>>>>>> e7cc42fd2286d4134e80ecd7e98f802/src/main/resources/es-
>>>>>>>> plugin.properties?at=master
>>>>>>>>
>>>>>>>> cheers
>>>>>>>>
>>>>>>>> On Thursday, May 1, 2014 3:21:59 PM UTC-7, Rob Ottaway wrote:
>>>>>>>>>
>>>>>>>>> This strikes me as odd:
>>>>>>>>>
>>>>>>>>> java.lang.ClassNotFoundException: example_river
>>>>>>>>>
>>>>>>>>> Assume you didn't map the string example_river to the actual class
>>>>>>>>> name properly?
>&
Re: How to write a custom river
I think he means in your Guice module. You are registering the WRONG thing
;)
On Fri, May 2, 2014 at 6:49 AM, Joshua Chan
wrote:
> I've tried this too with no luck
>
> http://localhost:9200/_river/example_river/_meta
> {
> "type": "river",
> "river":{
> "blah":"blah"
> }
> }
>
>
> On Friday, May 2, 2014 3:31:23 AM UTC-5, Luca Cavanna wrote:
>
>> Hi Joshua,
>> the package is not an issue if you are using the default one for your
>> classes. Looking deeper, the type of the river that you try to register
>> with your rest call doesn't match the type of the river you registered in
>> the plugin when you did module.registerRiver(type, riverclass).
>>
>> Cheers
>> Luca
>>
>> On Friday, May 2, 2014 6:08:28 AM UTC+2, Rob Ottaway wrote:
>>
>>> I should have sent you the following earlier rather than a non-river
>>> plugin:
>>>
>>> the plugin:
>>> https://github.com/elasticsearch/elasticsearch-
>>> river-rabbitmq/blob/master/src/main/java/org/elasticsearch/plugin/river/
>>> rabbitmq/RabbitmqRiverPlugin.java
>>>
>>> The river implementation:
>>> https://github.com/elasticsearch/elasticsearch-
>>> river-rabbitmq/blob/master/src/main/java/org/
>>> elasticsearch/river/rabbitmq/RabbitmqRiver.java
>>>
>>> The module:
>>> https://github.com/elasticsearch/elasticsearch-
>>> river-rabbitmq/blob/master/src/main/java/org/
>>> elasticsearch/river/rabbitmq/RabbitmqRiverModule.java
>>>
>>> Looks like you are registering the river implementation rather than the
>>> river module hence the not working. Had to look at an example I know works
>>> to figure it out.
>>>
>>> -Rob
>>>
>>>
>>> On Thu, May 1, 2014 at 8:04 PM, Joshua Chan wrote:
>>>
>>>> So, that's what I did, but no love... I checked in the latest.
>>>>
>>>> -Josh
>>>>
>>>>
>>>> On Thursday, May 1, 2014 9:49:11 PM UTC-5, Rob Ottaway wrote:
>>>>>
>>>>> Look at this plugin for help:
>>>>>
>>>>> https://github.com/elasticsearch/elasticsearch-cloud-aws/
>>>>> blob/master/src/main/resources/es-plugin.properties
>>>>>
>>>>> Yes it needs to be the FQN.
>>>>>
>>>>> On Thursday, May 1, 2014 5:47:31 PM UTC-7, Joshua Chan wrote:
>>>>>>
>>>>>> Thanks Rob. Someone else also told me the plugin property should be
>>>>>> the fully qualified name. I didn't declare a package, so I guess I'm
>>>>>> using
>>>>>> the default package, and I thought I had the namespacing right since
>>>>>> IntelliJ corrected the class name when I wrote it.
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>>
>>>>>> -Josh
>>>>>>
>>>>>>
>>>>>> On Thursday, May 1, 2014 5:23:25 PM UTC-5, Rob Ottaway wrote:
>>>>>>>
>>>>>>> Look at this file in your BB repo:
>>>>>>>
>>>>>>> https://bitbucket.org/futurechan/example-river/src/fd23648c3
>>>>>>> e7cc42fd2286d4134e80ecd7e98f802/src/main/resources/es-
>>>>>>> plugin.properties?at=master
>>>>>>>
>>>>>>> cheers
>>>>>>>
>>>>>>> On Thursday, May 1, 2014 3:21:59 PM UTC-7, Rob Ottaway wrote:
>>>>>>>>
>>>>>>>> This strikes me as odd:
>>>>>>>>
>>>>>>>> java.lang.ClassNotFoundException: example_river
>>>>>>>>
>>>>>>>> Assume you didn't map the string example_river to the actual class
>>>>>>>> name properly?
>>>>>>>>
>>>>>>>> -Rob
>>>>>>>>
>>>>>>>> On Thursday, May 1, 2014 11:40:52 AM UTC-7, Joshua Chan wrote:
>>>>>>>>>
>>>>>>>>> I'm making my first go at writing a river. (Here's the source
>>>>>>>>> code: https://bitbucket.org/futurechan/example-river/src)
>>>>>>>>>
>>>>>>>>> I followed this tutorial
>>>>>>>>> http://blog.trifork.com/2013/01/10/how-to-write-an-elasticse
>>
Re: How to write a custom river
I should have sent you the following earlier rather than a non-river plugin:
the plugin:
https://github.com/elasticsearch/elasticsearch-river-rabbitmq/blob/master/src/main/java/org/elasticsearch/plugin/river/rabbitmq/RabbitmqRiverPlugin.java
The river implementation:
https://github.com/elasticsearch/elasticsearch-river-rabbitmq/blob/master/src/main/java/org/elasticsearch/river/rabbitmq/RabbitmqRiver.java
The module:
https://github.com/elasticsearch/elasticsearch-river-rabbitmq/blob/master/src/main/java/org/elasticsearch/river/rabbitmq/RabbitmqRiverModule.java
Looks like you are registering the river implementation rather than the
river module hence the not working. Had to look at an example I know works
to figure it out.
-Rob
On Thu, May 1, 2014 at 8:04 PM, Joshua Chan
wrote:
> So, that's what I did, but no love... I checked in the latest.
>
> -Josh
>
>
> On Thursday, May 1, 2014 9:49:11 PM UTC-5, Rob Ottaway wrote:
>>
>> Look at this plugin for help:
>>
>> https://github.com/elasticsearch/elasticsearch-cloud-aws/blob/master/src/
>> main/resources/es-plugin.properties
>>
>> Yes it needs to be the FQN.
>>
>> On Thursday, May 1, 2014 5:47:31 PM UTC-7, Joshua Chan wrote:
>>>
>>> Thanks Rob. Someone else also told me the plugin property should be the
>>> fully qualified name. I didn't declare a package, so I guess I'm using the
>>> default package, and I thought I had the namespacing right since IntelliJ
>>> corrected the class name when I wrote it.
>>>
>>> Thoughts?
>>>
>>>
>>> -Josh
>>>
>>>
>>> On Thursday, May 1, 2014 5:23:25 PM UTC-5, Rob Ottaway wrote:
>>>>
>>>> Look at this file in your BB repo:
>>>>
>>>> https://bitbucket.org/futurechan/example-river/src/
>>>> fd23648c3e7cc42fd2286d4134e80ecd7e98f802/src/main/resources/
>>>> es-plugin.properties?at=master
>>>>
>>>> cheers
>>>>
>>>> On Thursday, May 1, 2014 3:21:59 PM UTC-7, Rob Ottaway wrote:
>>>>>
>>>>> This strikes me as odd:
>>>>>
>>>>> java.lang.ClassNotFoundException: example_river
>>>>>
>>>>> Assume you didn't map the string example_river to the actual class
>>>>> name properly?
>>>>>
>>>>> -Rob
>>>>>
>>>>> On Thursday, May 1, 2014 11:40:52 AM UTC-7, Joshua Chan wrote:
>>>>>>
>>>>>> I'm making my first go at writing a river. (Here's the source code:
>>>>>> https://bitbucket.org/futurechan/example-river/src)
>>>>>>
>>>>>> I followed this tutorial
>>>>>> http://blog.trifork.com/2013/01/10/how-to-write-an-
>>>>>> elasticsearch-river-plugin/
>>>>>>
>>>>>> and compared it to this existing river
>>>>>> https://github.com/jprante/elasticsearch-river-jdbc
>>>>>>
>>>>>> but I haven't had much luck.
>>>>>>
>>>>>> To deploy the river, I created a folder called example-river under
>>>>>> plugins, dropped my jar in that folder, and restarted the node.
>>>>>> Everything
>>>>>> starts up fine.
>>>>>>
>>>>>> I have also tried bin/plugin --url file:///path/to/plugin --install
>>>>>> example-river, which seems to work, but it unpacks my jar. So, I
>>>>>> tried zipping it first and then installing, which works and does not
>>>>>> unpack
>>>>>> my jar, but it didn't help.
>>>>>>
>>>>>> When I issue this PUT request:
>>>>>>
>>>>>> http://localhost:9200/_river/example_river/_meta
>>>>>> {
>>>>>> "type": "example_river",
>>>>>> "example_river":{
>>>>>> "blah":"blah"
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> I get this exception:
>>>>>>
>>>>>> [2014-04-20 22:28:46,538][DEBUG][river ] [Gloom] creating river
>>>>>> [example_river][example_river]
>>>>>> [2014-04-20 22:28:46,543][WARN ][river ] [Gloom] failed to create river
>>>>>> [example_river][example_river]
>>>>>> org.elasticsearch.common.settings.NoClassSettingsException: Failed to
>>>>>> l
Re: How to write a custom river
Look at this plugin for help:
https://github.com/elasticsearch/elasticsearch-cloud-aws/blob/master/src/main/resources/es-plugin.properties
Yes it needs to be the FQN.
On Thursday, May 1, 2014 5:47:31 PM UTC-7, Joshua Chan wrote:
>
> Thanks Rob. Someone else also told me the plugin property should be the
> fully qualified name. I didn't declare a package, so I guess I'm using the
> default package, and I thought I had the namespacing right since IntelliJ
> corrected the class name when I wrote it.
>
> Thoughts?
>
>
> -Josh
>
>
> On Thursday, May 1, 2014 5:23:25 PM UTC-5, Rob Ottaway wrote:
>>
>> Look at this file in your BB repo:
>>
>>
>> https://bitbucket.org/futurechan/example-river/src/fd23648c3e7cc42fd2286d4134e80ecd7e98f802/src/main/resources/es-plugin.properties?at=master
>>
>> cheers
>>
>> On Thursday, May 1, 2014 3:21:59 PM UTC-7, Rob Ottaway wrote:
>>>
>>> This strikes me as odd:
>>>
>>> java.lang.ClassNotFoundException: example_river
>>>
>>> Assume you didn't map the string example_river to the actual class name
>>> properly?
>>>
>>> -Rob
>>>
>>> On Thursday, May 1, 2014 11:40:52 AM UTC-7, Joshua Chan wrote:
>>>>
>>>> I'm making my first go at writing a river. (Here's the source code:
>>>> https://bitbucket.org/futurechan/example-river/src)
>>>>
>>>> I followed this tutorial
>>>>
>>>> http://blog.trifork.com/2013/01/10/how-to-write-an-elasticsearch-river-plugin/
>>>>
>>>> and compared it to this existing river
>>>> https://github.com/jprante/elasticsearch-river-jdbc
>>>>
>>>> but I haven't had much luck.
>>>>
>>>> To deploy the river, I created a folder called example-river under
>>>> plugins, dropped my jar in that folder, and restarted the node. Everything
>>>> starts up fine.
>>>>
>>>> I have also tried bin/plugin --url file:///path/to/plugin --install
>>>> example-river, which seems to work, but it unpacks my jar. So, I tried
>>>> zipping it first and then installing, which works and does not unpack my
>>>> jar, but it didn't help.
>>>>
>>>> When I issue this PUT request:
>>>>
>>>> http://localhost:9200/_river/example_river/_meta
>>>> {
>>>> "type": "example_river",
>>>> "example_river":{
>>>> "blah":"blah"
>>>> }
>>>> }
>>>>
>>>> I get this exception:
>>>>
>>>> [2014-04-20 22:28:46,538][DEBUG][river ] [Gloom] creating river
>>>> [example_river][example_river]
>>>> [2014-04-20 22:28:46,543][WARN ][river ] [Gloom] failed to create river
>>>> [example_river][example_river]
>>>> org.elasticsearch.common.settings.NoClassSettingsException: Failed to load
>>>> class with value [example_river] at
>>>> org.elasticsearch.river.RiverModule.loadTypeModule(RiverModule.java:87) at
>>>> org.elasticsearch.river.RiverModule.spawnModules(RiverModule.java:58) at
>>>> org.elasticsearch.common.inject.ModulesBuilder.add(ModulesBuilder.java:44)
>>>> at
>>>> org.elasticsearch.river.RiversService.createRiver(RiversService.java:137)
>>>> at
>>>> org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:275)
>>>> at
>>>> org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:269)
>>>> at
>>>> org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.jav
>>>> a:93) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at
>>>> java.lang.Thread.run(Unknown Source) Caused by:
>>>> java.lang.ClassNotFoundException: example_river at
>>>> java.net.URLClassLoader$1.run(Unknown Source) at
>>>> java.net.URLClassLoader$1.run(Unknown Source) at
>>>> java.security.AccessController.doPrivileged(Native Method) at
>>>> java.net.URLClassLoader.findClass(Unknown Source) at
>>>> java.lang.ClassLoader.loadClass(Unknown Source) at
>>>> sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at
>>>> java.lang.ClassLoader.loadClass(Unknown Source) at
>>>> org.elasticsearch.river.RiverModule.loadTypeModule(RiverModule.java:73)
>>>> ... 9 more
>>>>
>>>> Can someone point out what I am missing?
>>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fce8a8f1-b73a-406a-b591-17664369a5f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Handling updates from multiple sources
I missed that the later doc would only be partial. What is the reason to
use the partial doc? That really complicates things.
Filling in missing fields is going to be a very large headache. You'll
probably kill performance trying to do it too. Likely it'll be so complex
it will present a lot more trouble.
I think if you can better present the overall use cases you will get better
insight into how to work this out.
On Thursday, May 1, 2014 4:51:03 PM UTC-7, Michał Zgliczyński wrote:
>
> Hi,
> Thank you for your response. I have looked through this blog post:
> http://www.elasticsearch.org/blog/elasticsearch-versioning-support/
> It looks as if external versioning would be the way to go. Have the
> timestamps act as version numbers and let ES only pick the document with
> the newest version as the correct document. However, with the situation I
> have presented above, ES will fail. A quote from the post:
> "With version_type set to external, Elasticsearch will store the version
> number as given and will not increment it. Also, instead of checking for an
> exact match, Elasticsearch will only return a version collision error if
> the version currently stored is greater or equal to the one in the indexing
> command. This effectively means “only store this information if no one else
> has supplied the same or a more recent version in the meantime”.
> Concretely, the above request will succeed if the stored version number is
> smaller than 526. 526 and above will cause the request to fail."
>
> In my example, we would have that situation. A partial doc with a larger
> version number(later timestamp) is already stored in ES and we get the
> complete document with a smaller timestamp. In this situation we would like
> to merge these 2 documents in a way that, we have all of the fields from
> the partial doc and the other fields(not currently specified in the ES
> document) to be filled from the complete document.
>
> Thanks!
> Michal Zgliczynski
>
> W dniu czwartek, 1 maja 2014 14:58:31 UTC-7 użytkownik Rob Ottaway napisał:
>>
>> Have you looked at using versioning?
>>
>>
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html#index-versioning
>>
>> cheers,
>> Rob
>>
>> On Thursday, May 1, 2014 2:47:39 PM UTC-7, Michał Zgliczyński wrote:
>>>
>>> Hi,
>>>
>>> I am building a system in which I will have two sources of updates:
>>> 1) Bulk updating from the source of truth(db) <- Always inserting
>>> documents(complete docs)
>>> 2) Live updates <- Adding insert and update (complete and incomplete
>>> docs)
>>>
>>> Also, lets assume that each insert/update has a timestamp, which we
>>> belive in (not ES timestamp).
>>>
>>> The idea is to have a complete, up to date index once the bulk updating
>>> finishes. To achieve this I need to guarantee that I will have the correct
>>> data. This would work mostly well, if everything we would do upserts and
>>> the inserts/updates coming into ES have a strictly increasing timestamp.
>>> But one could imagine that this is a possibly problematic situation,
>>> when:
>>>
>>> 1) We are performing bulk indexing,
>>> a) we read an object from the db
>>> b) process it
>>> c) send it to ES.
>>> 2) We have an update on the same object, after step (a) and before if
>>> makes to ES in the bulk updating - phase(c). That is, ES gets an update
>>> with new data and only after that we get the insert with the entire
>>> document from the source of truth with older data. Hence, in ES we have a
>>> document with a newer timestamp, than the newly added one phase(c).
>>>
>>> My theoretical solution: For each operation, have the timestamp for that
>>> change (timestamp from the system that made the change, not from Elastic
>>> Search). Lets say that all of the operations that we will perform are
>>> upserts.
>>> Then once we get an insert or an update (lets call it doc), we have to
>>> perform the following script (pseudo mvel) inside ES.
>>> {
>>> if (doc.timestamp > ctx.source.timestamp) {
>>> // doc is newer than what was in ES
>>> upsert(doc); // update the index with all of the info from the new
>>> doc
>>> } else {
>>> // there is already a document in ES with a newer timestamp, note,
>>> this may be an incomplete document (an update)
>>> __fill the missing fields in the document in ES with values from
>>> doc__
>>>
Re: How to write a custom river
Look at this file in your BB repo:
https://bitbucket.org/futurechan/example-river/src/fd23648c3e7cc42fd2286d4134e80ecd7e98f802/src/main/resources/es-plugin.properties?at=master
cheers
On Thursday, May 1, 2014 3:21:59 PM UTC-7, Rob Ottaway wrote:
>
> This strikes me as odd:
>
> java.lang.ClassNotFoundException: example_river
>
> Assume you didn't map the string example_river to the actual class name
> properly?
>
> -Rob
>
> On Thursday, May 1, 2014 11:40:52 AM UTC-7, Joshua Chan wrote:
>>
>> I'm making my first go at writing a river. (Here's the source code:
>> https://bitbucket.org/futurechan/example-river/src)
>>
>> I followed this tutorial
>>
>> http://blog.trifork.com/2013/01/10/how-to-write-an-elasticsearch-river-plugin/
>>
>> and compared it to this existing river
>> https://github.com/jprante/elasticsearch-river-jdbc
>>
>> but I haven't had much luck.
>>
>> To deploy the river, I created a folder called example-river under
>> plugins, dropped my jar in that folder, and restarted the node. Everything
>> starts up fine.
>>
>> I have also tried bin/plugin --url file:///path/to/plugin --install
>> example-river, which seems to work, but it unpacks my jar. So, I tried
>> zipping it first and then installing, which works and does not unpack my
>> jar, but it didn't help.
>>
>> When I issue this PUT request:
>>
>> http://localhost:9200/_river/example_river/_meta
>> {
>> "type": "example_river",
>> "example_river":{
>> "blah":"blah"
>> }
>> }
>>
>> I get this exception:
>>
>> [2014-04-20 22:28:46,538][DEBUG][river ] [Gloom] creating river
>> [example_river][example_river]
>> [2014-04-20 22:28:46,543][WARN ][river ] [Gloom] failed to create river
>> [example_river][example_river]
>> org.elasticsearch.common.settings.NoClassSettingsException: Failed to load
>> class with value [example_river] at
>> org.elasticsearch.river.RiverModule.loadTypeModule(RiverModule.java:87) at
>> org.elasticsearch.river.RiverModule.spawnModules(RiverModule.java:58) at
>> org.elasticsearch.common.inject.ModulesBuilder.add(ModulesBuilder.java:44)
>> at
>> org.elasticsearch.river.RiversService.createRiver(RiversService.java:137) at
>> org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:275)
>> at
>> org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:269)
>> at
>> org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.jav
>> a:93) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at
>> java.lang.Thread.run(Unknown Source) Caused by:
>> java.lang.ClassNotFoundException: example_river at
>> java.net.URLClassLoader$1.run(Unknown Source) at
>> java.net.URLClassLoader$1.run(Unknown Source) at
>> java.security.AccessController.doPrivileged(Native Method) at
>> java.net.URLClassLoader.findClass(Unknown Source) at
>> java.lang.ClassLoader.loadClass(Unknown Source) at
>> sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at
>> java.lang.ClassLoader.loadClass(Unknown Source) at
>> org.elasticsearch.river.RiverModule.loadTypeModule(RiverModule.java:73) ...
>> 9 more
>>
>> Can someone point out what I am missing?
>>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/505124e5-ff08-4ad0-b135-a2e1d31c68ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: How to write a custom river
This strikes me as odd:
java.lang.ClassNotFoundException: example_river
Assume you didn't map the string example_river to the actual class name
properly?
-Rob
On Thursday, May 1, 2014 11:40:52 AM UTC-7, Joshua Chan wrote:
>
> I'm making my first go at writing a river. (Here's the source code:
> https://bitbucket.org/futurechan/example-river/src)
>
> I followed this tutorial
>
> http://blog.trifork.com/2013/01/10/how-to-write-an-elasticsearch-river-plugin/
>
> and compared it to this existing river
> https://github.com/jprante/elasticsearch-river-jdbc
>
> but I haven't had much luck.
>
> To deploy the river, I created a folder called example-river under
> plugins, dropped my jar in that folder, and restarted the node. Everything
> starts up fine.
>
> I have also tried bin/plugin --url file:///path/to/plugin --install
> example-river, which seems to work, but it unpacks my jar. So, I tried
> zipping it first and then installing, which works and does not unpack my
> jar, but it didn't help.
>
> When I issue this PUT request:
>
> http://localhost:9200/_river/example_river/_meta
> {
> "type": "example_river",
> "example_river":{
> "blah":"blah"
> }
> }
>
> I get this exception:
>
> [2014-04-20 22:28:46,538][DEBUG][river ] [Gloom] creating river
> [example_river][example_river]
> [2014-04-20 22:28:46,543][WARN ][river ] [Gloom] failed to create river
> [example_river][example_river]
> org.elasticsearch.common.settings.NoClassSettingsException: Failed to load
> class with value [example_river] at
> org.elasticsearch.river.RiverModule.loadTypeModule(RiverModule.java:87) at
> org.elasticsearch.river.RiverModule.spawnModules(RiverModule.java:58) at
> org.elasticsearch.common.inject.ModulesBuilder.add(ModulesBuilder.java:44) at
> org.elasticsearch.river.RiversService.createRiver(RiversService.java:137) at
> org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:275)
> at
> org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:269)
> at
> org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.jav
> a:93) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at
> java.lang.Thread.run(Unknown Source) Caused by:
> java.lang.ClassNotFoundException: example_river at
> java.net.URLClassLoader$1.run(Unknown Source) at
> java.net.URLClassLoader$1.run(Unknown Source) at
> java.security.AccessController.doPrivileged(Native Method) at
> java.net.URLClassLoader.findClass(Unknown Source) at
> java.lang.ClassLoader.loadClass(Unknown Source) at
> sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at
> java.lang.ClassLoader.loadClass(Unknown Source) at
> org.elasticsearch.river.RiverModule.loadTypeModule(RiverModule.java:73) ... 9
> more
>
> Can someone point out what I am missing?
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d3df7c6c-c62a-476a-935d-c49636be66d8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Zero Downtime Reindexing
Short of using a river to feed both indexes the same stream of updates updates I doubt that you will find any solution. Good news is why it's tedious as you pointed out, once setup it flows very smoothly. We use RabbitMQ in our case. A possible future feature would be an API within index creation that allows shadowing the indexing of one or more other indexes, without having to go through duplication client side or using a river. When the shadowed index goes away so could the shadowing, or api call could delete the shadow. cheers, Rob On Friday, February 21, 2014 12:11:52 AM UTC-8, Andrew Kane wrote: > > I tried to post a reply yesterday but it looks like it never made it. > > Thank you all for the quick replies. Here's a slightly better explanation > of where I believe the race condition occurs. > > When the scan/scroll starts, the alias is still pointing to the old index, > so updates go to the old index. Let's say you update Document 1. If the > scroll/scan has already passed Document 1, the new index never sees the > update. The three solutions you mentioned Nik are to either: > > 1. Keep track of updates manually [tedious] > 2. Pause the jobs that perform the updates [out of sync] > 3. Send updates to both indexes [also tedious] > > However, none of these seem ideal. > > - Andrew > > On Tuesday, February 18, 2014 8:41:18 PM UTC-8, Andrew Kane wrote: >> >> Hi, >> >> I've followed the documentation for zero-downtime mapping changes and it >> works great. >> http://www.elasticsearch.org/blog/changing-mapping-with-zero-downtime/ >> >> However, there is a (pretty big) race condition with this approach - >> while reindexing, changes may not make it to the new index. I've looked >> all over and haven't found a single solution to address this. The best >> attempt I've seen is to buffer updates, but this is tedious and still >> leaves a race condition (with a smaller window). My initial thoughts were >> to create a write alias that points to the old and new indices and use >> versioning. However, there is no way to write to multiple indices >> atomically. >> >> It seems like this issue should affect most Elasticsearch users (whether >> they realize it or not). Does anyone have a good solution to this? >> >> Thanks, >> Andrew >> >> -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/01041af1-3032-41d0-9b80-c6861aa3d6dc%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: trouble with plugins - phonetic, etc
You would want to compile a list of music industry specific synonyms such
as little -> lil
Metaphone is works around how things "sound" not synonyms. For example
Erika sounds like Erica.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-synonym-tokenfilter.html
Possible there are already some pretty good lists around. You will probably
end up with a cycle of index, test/qa, update synonyms, index, test/qa. Can
take a bit but your application is very domain specific.
cheers,
Rob
On Thursday, May 1, 2014 1:47:34 PM UTC-7, Adrien Grand wrote:
>
> This looks good. Now you just need to create a string field with
> `analyzer: my_analyzer`[1]. Elasticsearch will then transparently search
> based on phonetic similarity when querying the `artist` field.
>
> However, I would recommend on playing with the analyze[2] API. I'm not
> sure that `lil` and `little` would be analyzed to the same token with
> metaphone.
>
> [1]
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#string
> [2]
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-analyze.html
>
>
> On Mon, Apr 28, 2014 at 5:34 PM, Eric Sims
>
> > wrote:
>
>> sorry for a noob question. i'm trying to understand phonetic searches -
>> how to install and use them.
>>
>> perhaps phonetics isn't the right way for my instance.
>>
>> i'm trying to return music artist results for 'lil wayne', but account
>> for the user to type 'little wayne'.
>>
>> i've created and populated an index called /music/artist
>>
>> so i've installed the phonetic plugin and config is like so:
>> - (i created another index (since it won't allow me to put it into
>> /music/artist))
>>
>> PUT /music_admin
>> {
>> "settings" : {
>> "analysis" : {
>> "analyzer" : {
>> "my_analyzer" : {
>> "tokenizer" : "standard",
>> "filter" : ["standard", "lowercase", "my_metaphone"]
>> }
>> },
>> "filter" : {
>> "my_metaphone" : {
>> "type" : "phonetic",
>> "encoder" : "metaphone",
>> "replace" : false
>> }
>> }
>> }
>> }
>> }
>>
>> this feels wrong. i know. i'm confused at this point as to how to use the
>> search. i have a field called 'artist' that i would be searching in.
>>
>> please help!
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/c4b78cef-e10c-4f0b-9b5f-07c7cc8d03f8%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Adrien Grand
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3d1bafb8-496d-44c4-84b1-70be001efcb6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Handling updates from multiple sources
Have you looked at using versioning?
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html#index-versioning
cheers,
Rob
On Thursday, May 1, 2014 2:47:39 PM UTC-7, Michał Zgliczyński wrote:
>
> Hi,
>
> I am building a system in which I will have two sources of updates:
> 1) Bulk updating from the source of truth(db) <- Always inserting
> documents(complete docs)
> 2) Live updates <- Adding insert and update (complete and incomplete docs)
>
> Also, lets assume that each insert/update has a timestamp, which we belive
> in (not ES timestamp).
>
> The idea is to have a complete, up to date index once the bulk updating
> finishes. To achieve this I need to guarantee that I will have the correct
> data. This would work mostly well, if everything we would do upserts and
> the inserts/updates coming into ES have a strictly increasing timestamp.
> But one could imagine that this is a possibly problematic situation, when:
>
> 1) We are performing bulk indexing,
> a) we read an object from the db
> b) process it
> c) send it to ES.
> 2) We have an update on the same object, after step (a) and before if
> makes to ES in the bulk updating - phase(c). That is, ES gets an update
> with new data and only after that we get the insert with the entire
> document from the source of truth with older data. Hence, in ES we have a
> document with a newer timestamp, than the newly added one phase(c).
>
> My theoretical solution: For each operation, have the timestamp for that
> change (timestamp from the system that made the change, not from Elastic
> Search). Lets say that all of the operations that we will perform are
> upserts.
> Then once we get an insert or an update (lets call it doc), we have to
> perform the following script (pseudo mvel) inside ES.
> {
> if (doc.timestamp > ctx.source.timestamp) {
> // doc is newer than what was in ES
> upsert(doc); // update the index with all of the info from the new doc
> } else {
> // there is already a document in ES with a newer timestamp, note,
> this may be an incomplete document (an update)
> __fill the missing fields in the document in ES with values from doc__
> }
> }
>
> My question is:
> 1) Is there a better approach?
> 2) If so, is there a simple approach for doing the ' __fill the missing
> fields in the document in ES with values from doc__' operation/script?
>
> Thanks!
> Michal Zgliczynski
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b988d556-fd38-4f85-9214-2d471558b778%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Custom Queries in Kibana Panels
Adding another shot showing setting up the "selected" queries in the Table (Panel) Settings http://d.pr/i/JDej cheers, Rob On Tuesday, April 29, 2014 2:08:14 PM UTC-7, Chris F. wrote: > > What I'm looking for, is a way to add custom query parameters to some > panels in Kibana. > > For instance, I want to restrict the _type on my date histogram. My user > can still do a * search from the search bar, but I'd like to 'append' the > _type:'something' query to one specific panel... in a way that's hard coded > in the configuration/dashboard.json file, and not something the user can > override. > > Anyone know if that's possible? > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/88bbcac3-018c-4e1d-84a0-2967fbff3156%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Custom Queries in Kibana Panels
Use multiple queries. Make one query that is the _type:"sometype" and pin it. Leave the other unpinned so it's easy to hack in any old value. Then in the panel select the free form query and the pinned one for the type. Here is what it would look like http://d.pr/i/FXHL cheers, Rob On Tuesday, April 29, 2014 2:08:14 PM UTC-7, Chris F. wrote: > > What I'm looking for, is a way to add custom query parameters to some > panels in Kibana. > > For instance, I want to restrict the _type on my date histogram. My user > can still do a * search from the search bar, but I'd like to 'append' the > _type:'something' query to one specific panel... in a way that's hard coded > in the configuration/dashboard.json file, and not something the user can > override. > > Anyone know if that's possible? > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b41f3f79-fa58-45ff-b217-22422bbe4f36%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
EC2 + ZooKeeper Disco: Tips on Simulating Cluster Failures
Hello,
I am doing a bit of work to harden our Elasticsearch cluster configuration.
We've had issues with split brain scenarios in the past. We have clusters
with 3 or 5 nodes where any node can be handling index and search
operations. We have used the min master nodes to try and alleviate issues
but it did not work for us, still split brains.
We are using ES 1.1.0 with the latest ZooKeeper plugin. I have a test
cluster of 3 nodes running on m3.large instances w/ 500 GB EBS volumes. I
have tested a few scenarios so far which have performed as expected:
1. Communication between a node and ZK going down. After short timeout
(~30 seconds) node is eliminated from cluster.
2. The sudden death of a node (master or otherwise) via 'kill -9'.
Rebalancing and election worked out very well here.
3. Stopping a node cleanly, nothing odd here works every time and ZK
makes cluster state updates really fast.
4. Adding new nodes, again quick cluster state updates via ZK.
The last scenario I am interested in is network partitions. In this case I
am trying to sever the communication between two of the nodes and a third.
I have been using iptables to DROP all in/out bound data from one of the 3
nodes in the test cluster to the other 2. I basically make four entries on
the node I want to cease communication with.
After doing so it takes a very long time for the node to finally be evicted
from cluster state. During this time a number of api methods will stop
working, including /_stats and /_nodes but also search will time out on the
node where com was severed. GOOD news is no split brains, bad news is
eviction of the bad node takes a long time.
Any help with explaining what is going on or how I can better test this
sort of scenario is much appreciated.
cheers,
Rob
The following is a bunch of the info on the exceptions I see when things
start to time out finally. Until the node is removed from cluster state
everything works kinda wonky.
The exception on a node trying to talk to the iptable'd node looks like
this:
[2014-04-28 21:15:36,488][WARN ][cluster.service ] [Kukulcan]
failed to reconnect to node [Harold "Happy" Hogan][_xgiPJYmSuecN0--yDB
mlg][zookeeper-test-builders-us-west-1-i-c9f6b095][inet[/10.168.250.15:9300]]{availabilityzone=us-west-1b}
org.elasticsearch.transport.ConnectTransportException: [Harold "Happy"
Hogan][inet[/10.168.250.15:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:773)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:702)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:670)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
at
org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:515)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.elasticsearch.common.netty.channel.ConnectTimeoutException:
connection timed out: /10.168.250.15:9300
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:137)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
I get these messages on the iptable'd node:
[2014-04-28 21:08:37,831][DEBUG][action.admin.indices.stats] [Harold
"Happy" Hogan] [events-2014.04.27][0], node[Cin_0uRIQwubm585lpkYnQ], [P],
s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@bc812]
org.elasticsearch.transport.NodeDisconnectedException: [Herr
Kleiser][inet[/10.176.41.55:9300]][indices/stats/s] disconnected
[2014-04-28 21:08:37,832][DEBUG][action.admin.indices.stats] [Harold
"Happy" Hogan] [events-2014.04.28][0], node[Cin_0uRIQwubm585lpkYnQ], [P],
s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@7c82439e]
org.elasticsearch.transport.NodeDisconnectedException: [Herr
Kleiser][inet[/10.176.41.55:9300]][indices/stats/s] disconnected
[2014-04-28 21:08:37,831][DEBUG][action.admin.indices.stats] [Harold
"Happy" Hogan] [_river][0], node[Cin_0uRIQwubm585lpkYnQ], [R], s[STARTED]:
Failed to execute
[org.elasticsearch.action.
