Stephen, I doubt that we should be able to have different configuration values of some cluster aware properties on cluster nodes (all such values must be the same). This will lead to unstable behaviour of the whole cluster. In the opposite, build a cluster which contains different Ignites node versions - is ok.
Coming back to your original question, sure you can check JMX metrics (like getKeysToRebalanceLeft()) but I think it's not a full guarantee that the rebalance of the whole cluster finished. Probably, the more appropriate thing is listening [1] events like EVT_CACHE_REBALANCE_STOPPED. https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/events/EventType.html#EVT_CACHE_REBALANCE_STOPPED On Fri, 28 Jun 2019 at 12:20, Stephen Darlington <stephen.darling...@gridgain.com> wrote: > > Thanks for your response. > > Certainly getting a future indicating when the process finishes would be a > lot cleaner, but I was looking in CacheMetrics where there are a bunch of > methods around rebalancing, and many of those appear to be exported by JMX. > Would waiting until getKeysToRebalanceLeft() returns to zero be a good proxy, > for example? You’d have to iterate over all the caches. > > I was thinking of updating the configuration of the cluster, for example > changing thread pool sizes, or JVM memory configurations, even adding a cache > if done on the server side. There’s a bunch of things you can’t change > (versions of Ignite, load balancing configuration, etc) but it felt like the > kind of thing that people must be doing, whether with something like Ansible > or Kubernetes. > > Regards, > Stephen > > On 27 Jun 2019, at 19:01, Maxim Muzafarov <maxmu...@gmail.com> wrote: > > Stephen, > > AFAIK there is no such public option to wait for all the cluster > caches are fully rebalanced. Please, take a look here [1]. This is an > internal method which returns the future (will be completed when cache > group rebalance is finished) and it's not exposed to any Ignites > public API, so I doubt your goal is achievable out of the box. But I > think this method can be accessed via some custom Ignite plugin. > > Also, I thought that the rolling upgrade is a part of GridGain and not > supported by pure Apache Ignite, Am I missing something? > What the automation tool do you use for such purpose? > > [1] > https://github.com/Mmuzaf/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCachePreloader.java#L128 > > On Thu, 27 Jun 2019 at 18:18, Stephen Darlington > <stephen.darling...@gridgain.com> wrote: > > > Hi, > > I’m looking to be able to automate a rolling update of Ignite, that is, take > nodes down one at a time until the whole cluster has the new configuration. > > I have my caches configured with at least one backup. What’s the easiest way > of checking that the cluster has finished rebalancing all the caches before > moving onto the next node? > > I can see the information in CacheMetrics and it seems to be exported to JMX, > but it feels like there should be a way to see this in a command line tool. > Not figured out how to see it in “control” or “ignitevisor”. Am I missing > something? > > Thanks in advance. > > Regards, > Stephen > > >