Hello Ben, Maybe it works if you uninstall the plugin from one node at a time and do a rolling restart (sticking to 1.3.2), then do the upgrade with another rolling restart, then install the plugin back again with yet another rolling restart?
I would understand if you said "no way I do 3 restarts!" :) But maybe this will help in future: https://github.com/jprante/elasticsearch-plugin-deploy Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg <benber...@gmail.com> wrote: > Hello, > I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I > upgraded the first node,removed and reinstalled latest versions of plugins > - two non-site plugins are river-twitter (version 2.4.1) and jdbc (version > 1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the > errors below in logs on upgraded node and node does not join cluster. If i > downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin > version 1.3.0.4 the node properly joins the cluster. > Errors from logs: > [2015-01-06 21:29:39,970][INFO ][node ] [bi-es1] > version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z] > [2015-01-06 21:29:39,971][INFO ][node ] [bi-es1] > initializing ... > [2015-01-06 21:29:40,009][INFO ][plugins ] [bi-es1] > loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk, > paramedic, HQ, whatson] > [2015-01-06 21:29:45,587][INFO ][node ] [bi-es1] > initialized > [2015-01-06 21:29:45,588][INFO ][node ] [bi-es1] > starting ... > [2015-01-06 21:29:45,960][INFO ][transport ] [bi-es1] > bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/ > 192.168.83.231:9300]} > [2015-01-06 21:29:45,982][INFO ][discovery ] [bi-es1] > bi-cluster1/VyTzSKWnQAS9ZCk971fJEg > [2015-01-06 21:29:51,906][WARN ][transport.netty ] [bi-es1] > Message not fully read (request) for [28805963] and action > [discovery/zen/join/validate], resetting > [2015-01-06 21:29:51,915][INFO ][discovery.zen ] [bi-es1] > failed to send join request to master > [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > reason > [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > nested: > RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > nested: ElasticsearchIllegalArgumentException[No custom index metadata > factory registered for type [rivers]]; ] > [2015-01-06 21:29:57,028][WARN ][transport.netty ] [bi-es1] > Message not fully read (request) for [28807127] and action > [discovery/zen/join/validate], resetting > [2015-01-06 21:29:57,036][INFO ][discovery.zen ] [bi-es1] > failed to send join request to master > [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > reason > [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > nested: > RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > nested: ElasticsearchIllegalArgumentException[No custom index metadata > factory registered for type [rivers]]; ] > [2015-01-06 21:30:02,245][WARN ][transport.netty ] [bi-es1] > Message not fully read (request) for [28808254] and action > [discovery/zen/join/validate], resetting > [2015-01-06 21:30:02,252][INFO ][discovery.zen ] [bi-es1] > failed to send join request to master > [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > reason > [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > nested: > RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > nested: ElasticsearchIllegalArgumentException[No custom index metadata > factory registered for type [rivers]]; ] > [2015-01-06 21:30:07,576][WARN ][transport.netty ] [bi-es1] > Message not fully read (request) for [28809377] and action > [discovery/zen/join/validate], resetting > [2015-01-06 21:30:07,583][INFO ][discovery.zen ] [bi-es1] > failed to send join request to master > [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > reason > [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > nested: > RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > nested: ElasticsearchIllegalArgumentException[No custom index metadata > factory registered for type [rivers]]; ] > [2015-01-06 21:30:12,689][WARN ][transport.netty ] [bi-es1] > Message not fully read (request) for [28810050] and action > [discovery/zen/join/validate], resetting > [2015-01-06 21:30:12,696][INFO ][discovery.zen ] [bi-es1] > failed to send join request to master > [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > reason > [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > nested: > RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > nested: ElasticsearchIllegalArgumentException[No custom index metadata > factory registered for type [rivers]]; ] > [2015-01-06 21:30:15,989][WARN ][discovery ] [bi-es1] > waited for 30s and no initial state was set by the discovery > [2015-01-06 21:30:16,002][INFO ][http ] [bi-es1] > bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/ > 192.168.83.231:9200]} > [2015-01-06 21:30:16,003][INFO ][node ] [bi-es1] > started > [2015-01-06 21:30:18,277][WARN ][transport.netty ] [bi-es1] > Message not fully read (request) for [28811164] and action > [discovery/zen/join/validate], resetting > [2015-01-06 21:30:18,292][INFO ][discovery.zen ] [bi-es1] > failed to send join request to master > [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > reason > [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > nested: > RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > nested: ElasticsearchIllegalArgumentException[No custom index metadata > factory registered for type [rivers]]; ] > [2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no > known master node, scheduling a retry > > I have tried removing all plugins from the upgraded node and that results > in same errors. I have tried closing the _river index and removing all > defined rivers (wasn't hopeful that would work but gave it a shot). > > I am not sure what else to try - thoughts? ideas? > > I am safely downgraded back to 1.3.2 and node rejoined and things seem > good except still on 1.3.2. Would prefer not to do a full cluster shutdown > to upgrade all nodes and jdbc plugin but that seems maybe it would solve > this? > > I have had issues in past with upgrades/jdbc river plugin (really love the > plugin though) so my brain is focusing on that and might not be the issue? > I used same process on a test cluster that I just realized did not have the > jdbc plugin installed on it and that went smooth. > > Any help would be greatly appreciated! > Thanks > Ben > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
