Re: Can't stop a snapshot running on my cluster

Yes, by "full cluster restart" I meant shutting down all nodes and then starting them up again, which means downtime. However, after thinking about the issue over the long weekend, I wrote a simple utility that cleans up snapshots without need to restart the cluster - https://github.com/imotov/

Re: Can't stop a snapshot running on my cluster

Well it's the only way I can do it without downtime. Unless of course by "full cluster restart" you mean restarting one node at a time? On Sat, May 24, 2014 at 7:51 PM, Igor Motov wrote: > If your cluster is setup correctly (with proper value set > for discovery.zen.minimum_master_nodes) they s

Re: Can't stop a snapshot running on my cluster

If your cluster is setup correctly (with proper value set for discovery.zen.minimum_master_nodes) they shouldn't. But if you are running without discovery.zen.minimum_master_nodes set, they might indeed form a new cluster. Obviously some shards might end up in one cluster and not in the other a

Re: Can't stop a snapshot running on my cluster

Ok. While you're here, one other question I would like answered: I have 10 nodes in a cluster. I want to break out three nodes into a different cluster as a kind of backup to test out this full cluster restart. Would it be safe to just block the other three nodes from connecting to the main cluste

Re: Can't stop a snapshot running on my cluster

It was caused by this bug - https://github.com/elasticsearch/elasticsearch/issues/5958 The only recovery option right now is full cluster restart. On Saturday, May 24, 2014 2:30:06 PM UTC-4, Andrew Vos wrote: > > Right ok here's the cluster state > https://gist.github.com/AndrewVos/29de3c6735b

Re: Can't stop a snapshot running on my cluster

I meant the output of the cluster state command: curl -XGET 'http://localhost:9200/_cluster/state' It might be large and will contain information about your cluster that you might not want t

Re: Can't stop a snapshot running on my cluster

Right ok here's the cluster state https://gist.github.com/AndrewVos/29de3c6735bbd7808a81 On Sat, May 24, 2014 at 7:18 PM, Andrew Vos wrote: > 1.0.0. What do you mean by state exactly? > > > On Sat, May 24, 2014 at 6:33 PM, Igor Motov wrote: > >> Which version of elsticsearch are you using? Can

Re: Can't stop a snapshot running on my cluster

1.0.0. What do you mean by state exactly? On Sat, May 24, 2014 at 6:33 PM, Igor Motov wrote: > Which version of elsticsearch are you using? Can you send me the current > cluster state? > > On Saturday, May 24, 2014 10:17:43 AM UTC-4, Andrew Vos wrote: >> >> A few days ago I started a snapshot,

Re: Can't stop a snapshot running on my cluster

Which version of elsticsearch are you using? Can you send me the current cluster state? On Saturday, May 24, 2014 10:17:43 AM UTC-4, Andrew Vos wrote: > > A few days ago I started a snapshot, but instead of using a shared network > I used the local filesystem. Because my root partition only had

Can't stop a snapshot running on my cluster

A few days ago I started a snapshot, but instead of using a shared network I used the local filesystem. Because my root partition only had 8gb (and this is where I stored the snapshots) the partition got filled up and three of my seven elasticsearch boxes crashed almost instantly. I've since cr