Re: Cluster auto activation design proposal
> > > > > > > and > > > > > > > > > has > > > > > > > > > > > > data > > > > > > > > > > > > > in > > > > > > > > > > > > > > > it. > > > > > > > > > > > > > > > > > -> > > > > > > > > > > > > > > > > > >BLT #1 created. > > > > > > > > > > > > > > > > > >2. We shutdown the grid. Then divide > it > > > into > > > > > two > > > > > > > > > parts: > > > > > > > > > > > > > > Part1_grid > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > >Part2_grid. > > > > > > > > > > > > > > > > > >3. We start and activate Part1_grid . > > > > Topology > > > > > > has > > > > > > > > > > changed > > > > > > > > > > > > -> > > > > > > > > > > > > > > > BLT#2 > > > > > > > > > > > > > > > > > >created. > > > > > > > > > > > > > > > > > >After that we shutdown that > Part1_grid. > > > > > > > > > > > > > > > > > >4. We start and activate Part2_grid. > > > > Topology > > > > > > also > > > > > > > > has > > > > > > > > > > > > changed > > > > > > >
Re: Cluster auto activation design proposal
; > > > > > > Dmitriy, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It was my misunderstanding, I believe that > > > setter > > > > > is > > > > > > > not > > > > > > > > > > enough > > > > > > > > > > > > and > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > need > > > > > > > > > > > > > > > > > a full-fledged entity. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We should also be able to check if BLTs are > > > > > > compatible. > > > > > > > > > > > Interface > > > > > > > > > > > > > > looks > > > > > > > > > > > > > > > > > like this and use case for this > functionality > > > is > > > > > > > > described > > > > > > > > > > > below. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > interface BaselineTopology { > > > > > > > > > > > > > > > > >Collection nodes(); > > > > > > > > > > > > > > > > >boolean isCompatibleWith( > BaselineTopology > > > > blt); > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Let's consider the following scenario: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >1. We have a grid with N nodes: it is > up, > > > > active > > > > > > and > > > > > > > > has > > > > > > > > > > > data > > > > > > > > > > > > in > > > > > > > > > > > > > > it. > > > > > > > > > > > > > > > > -> > > > > > > > > > > > > > > > > >BLT #1 created. > > > > > > > > > > > > > > > > >2. We shutdown the grid. Then divide it > > into > > > > two > > > > > > > > parts: > > > > > > > > > > > > > Part1_grid > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > >Part2_grid. > > > > > > > > > > > > > > > > >3. We start and activate Part1_grid . > > > Topology > > > > > has > > > > > > > > > changed > > > > > > > > > > > -> > > > > > > > > > > > > > > BLT#2 > > > > > > > > > > > > > > > > >created. > > > > > > > > > > > > > > > > >After that we shutdown that Part1_grid. > > > > > > > > > > > > > > > > >4. We start and activate Part2_grid. > > > Topology > > > > > also > > > > > > > has > > > > > > > > > > > changed > > > > > > > > > > > > > -> > > > > > > > > > > > > > > > > BLT#3 > > > > > > > > > > > > > > > > >created. > > > > > > > > > > > > > > > > >5. Then we start Part1_grid and it's > nodes > > > try > > > > > to > > > > > > > join > > > > > > > > > > > > > Part2_grid. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Cluster auto activation design proposal
it is up, > > > active > > > > > and > > > > > > > has > > > > > > > > > > data > > > > > > > > > > > in > > > > > > > > > > > > > it. > > > > > > > > > > > > > > > -> > > > > > > > > > > > > > > > >BLT #1 created. > > > > > > > > > > > > > > > >2. We shutdown the grid. Then divide it > into > > > two > > > > > > > parts: > > > > > > > > > > > > Part1_grid > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > >Part2_grid. > > > > > > > > > > > > > > > >3. We start and activate Part1_grid . > > Topology > > > > has > > > > > > > > changed > > > > > > > > > > -> > > > > > > > > > > > > > BLT#2 > > > > > > > > > > > > > > > >created. > > > > > > > > > > > > > > > >After that we shutdown that Part1_grid. > > > > > > > > > > > > > > > >4. We start and activate Part2_grid. > > Topology > > > > also > > > > > > has > > > > > > > > > > changed > > > > > > > > > > > > -> > > > > > > > > > > > > > > > BLT#3 > > > > > > > > > > > > > > > >created. > > > > > > > > > > > > > > > >5. Then we start Part1_grid and it's nodes > > try > > > > to > > > > > > join > > > > > > > > > > > > Part2_grid. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If join is successful we have an undefined > > state > > > of > > > > > the > > > > > > > > > > resulting > > > > > > > > > > > > > grid: > > > > > > > > > > > > > > > > values for the same key may (and will) differ > > > > between > > > > > > > grid > > > > > > > > > > parts. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > So to prevent this we should keep nodes with > > > BLT#2 > > > > > from > > > > > > > > > joining > > > > > > > > > > > the >
Re: Cluster auto activation design proposal
> > > > the > > > > > > > > > > > > > grid > > > > > > > > > > > > > > > with BLT#3. And we should fail nodes with an > > error > > > > > > message. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Aug 23, 2017 at 5:47 AM, Dmitriy > > Setrakyan > > > < > > > > > > > > > > > > > > dsetrak...@apache.org> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergey, I am still confused. What is the > > > > > > BaselineTopology > > > > > > > > > > > interface > > > > > > > > > > > > > in > > > > > > > > > > > > > > > your > > > > > > > > > > > > > > > > example? I thought that you agreed with me > that > > > we > > > > > > simply > > > > > > > > > need > > > > > > > > > > a > > > > > > > > > > > > > setter > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > activation nodes, no? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 22, 2017 at 4:47 AM, Sergey > > Chugunov > > > < > > > > > > > > > > > > > > > > sergey.chugu...@gmail.com> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As I understand you use the term > > > > > > > "minimalActivationNodes" > > > > > > > > > as > > > > > > > > > > a > > > > > > > > > > > > > > synonym > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > > BaselineTopology concept. > > > > > > > > > > > > > > > > > In that case I agree with you that we can > > > replace > > > > > > both > > > > > > > > > > > > "establish*" > > > > > > > > > > > > > > > > methods > > > > > > > > > > > > > > > > > with a simple setter method (see below in > > > > summary). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Summing up the whole discussion I see the > > > > > > functionality > > > > > > > > as > > > > > > > > > > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > New concept BaselineTopology is introduced. > > The > > > > > main > > > > > > > > > features > > > > > > > > > > > it > > > > > > > > > > > > > > > enables > > > > > > > > > > > > > > > > > are: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >1. automatic activation of cluster; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >2. easy management of cluster topology > > > changes > > > > > > > > (planned > > > > > > > > > > > nodes > > > > > > > > > > > > > > > > >maintenance, adding new nodes etc); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >3. eliminating of rebalancing traffic on > > > > > > short-term > > > > > > > > node > > > > > > > > > > > > > failures. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Use cases to create BLT: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >1. user starts up new cluster of desired > > > > number > > > > > of > > > > > > > > nodes > > > > > > > > > > and > > > > > > > > > > > > > > > activates > > > > > > > > > > > > > > > > >it using existing API. BLT is created > with > > > all > > > > > > nodes > > > > > > > > > > > presented > > > > > > > > > > > > > in > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >cluster at the moment of activation, no > > API > > > is > > > > > > > needed; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >2. user prepares BLT using web-console > or > > > > visor > > > > > > CMD > > > > > > > > > tools > > > > > > > > > > > and > > > > > > > > > > > > > sets > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > >the cluster. New API setter is needed: > > > > > > > > > > > > > > > > >Ignite.activation().setBaselin > > > > > > > > > eTopology(BaselineTopology > > > > > > > > > > > > blt); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >3. user provides via static > configuration > > a > > > > list > > > > > > of > > > > > > > > > nodes > > > > > > > > > > > that > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > >expected to be in the cluster. > > > > > > > > > > > > > > > > >User starts nodes one by one; when all > > > > > > preconfigured > > > > > > > > > nodes > > > > > > > > > > > are > > > > > > > > > > > > > > > started > > > > > > > > > > > > > > > > >cluster is activated and BLT is created. > > > > > > > > > > > > > > > > >As list of nodes may be huge it is > > provided > > > > via > > > > > > > > separate > > > > > > > > > > > file > > > > > > > > > > > > to > > > > > > > > > > > > > > > avoid > > > > > > > > > > > > > > > > >flooding main configuration. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Igniters, does this description match with > > your > > > > > > > > > understanding > > > > > > > > > > > of > > > > > > > > > > > > > > > > > functionality? If it does I'll create a set > > of > > > > > > tickets > > > > > > > > and > > > > > > > > > > > start > > > > > > > > > > > > > > > working > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy > > > > Setrakyan > > > > > < > > > > > > > > > > > > > > > > dsetrak...@apache.org> > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I still do not see why anyone would > > > explicitly > > > > > call > > > > > > > > > these 2 > > > > > > > > > > > > > > methods: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation:: > > > > > establishBaselineTopology();* > > > > > > > > > > > > > > > > > > *Ignite::activation:: > > > > establishBaselineTopology( > > > > > > > > > > > > BaselineTopology > > > > > > > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For example, if a web console, or some > > other > > > > > admin > > > > > > > > > process, > > > > > > > > > > > > want > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > automatically set currently started nodes > > as > > > > the > > > > > > > > baseline > > > > > > > > > > > > > topology, > > > > > > > > > > > > > > > > > > shouldn't they just call a setter for > > > > > > > > > > minimalActivationNodes? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey > > > > > Dmitriev < > > > > > > > > > > > > > > > > > admitr...@gridgain.com> > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > API is proposed in the head of the > thread > > > by > > > > > > > Sergey, > > > > > > > > > as I > > > > > > > > > > > > > > > understood: > > > > > > > > > > > > > > > > > > > __ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > API for BaselineTopology manipulation > may > > > > look > > > > > > like > > > > > > > > > this: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation:: > > > > > > establishBaselineTopology();* > > > > > > > > > > > > > > > > > > > *Ignite::activation:: > > > > > establishBaselineTopology( > > > > > > > > > > > > BaselineTopology > > > > > > > > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Both methods will establish BT and > > activate > > > > > > cluster > > > > > > > > > once > > > > > > > > > > it > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > established. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The first one allows user to establish > BT > > > > using > > > > > > > > current > > > > > > > > > > > > > topology. > > > > > > > > > > > > > > > If > > > > > > > > > > > > > > > > > any > > > > > > > > > > > > > > > > > > > changes happen to the topology during > > > > > > establishing > > > > > > > > > > process, > > > > > > > > > > > > > user > > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > notified and allowed to proceed or > abort > > > the > > > > > > > > procedure. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Second method allows to use some > > > > > > > > > monitoring'n'management > > > > > > > > > > > > tools > > > > > > > > > > > > > > like > > > > > > > > > > > > > > > > > > > WebConsole where user can prepare a > list > > of > > > > > > nodes, > > > > > > > > > using > > > > > > > > > > > them > > > > > > > > > > > > > > > create > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > BT > > > > > > > > > > > > > > > > > > > and send to the cluster a command to > > > finally > > > > > > > > establish > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From high level BaselineTopology entity > > > > > contains > > > > > > > only > > > > > > > > > > > > > collection > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > nodes: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *BaselineTopology {* > > > > > > > > > > > > > > > > > > > * Collection nodes;* > > > > > > > > > > > > > > > > > > > *}* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *TopologyNode* here contains > information > > > > about > > > > > > > node - > > > > > > > > > its > > > > > > > > > > > > > > > consistent > > > > > > > > > > > > > > > > id > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > set of user attributes used to > calculate > > > > > affinity > > > > > > > > > > function. > > > > > > > > > > > > > > > > > > > __ > > > __ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > View this message in context: > > > > > > > http://apache-ignite- > > > > > > > > > > > > > > > > > > > developers.2346864.n4.nabble. > > > > > > > > > > com/Cluster-auto-activation- > > > > > > > > > > > > > > > > > > > design-proposal-tp20295p21066.html > > > > > > > > > > > > > > > > > > > Sent from the Apache Ignite Developers > > > > mailing > > > > > > list > > > > > > > > > > archive > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > Nabble.com. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Cluster auto activation design proposal
; > > > > -> > > > > > > > > > > > > > >BLT #1 created. > > > > > > > > > > > > > >2. We shutdown the grid. Then divide it into > two > > > > > parts: > > > > > > > > > > Part1_grid > > > > > > > > > > > > and > > > > > > > > > > > > > >Part2_grid. > > > > > > > > > > > > > >3. We start and activate Part1_grid . Topology > > has > > > > > > changed > > > > > > > > -> > > > > > > > > > > > BLT#2 > > > > > > > > > > > > > >created. > > > > > > > > > > > > > >After that we shutdown that Part1_grid. > > > > > > > > > > > > > >4. We start and activate Part2_grid. Topology > > also > > > > has > > > > > > > > changed > > > > > > > > > > -> > > > > > > > > > > > > > BLT#3 > > > > > > > > > > > > > >created. > > > > > > > > > > > > > >5. Then we start Part1_grid and it's nodes try > > to > > > > join > > > > > > > > > > Part2_grid. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If join is successful we have an undefined state > of > > > the > > > > > > > > resulting > > > > > > > > > > > grid: > > > > > > > > > > > > > > values for the same key may (and will) differ > > between > > > > > grid > > > > > > > > parts. > > > > > > > > > > > > > > > > > > > > > > > > > > > > So to prevent this we should keep nodes with > BLT#2 > > > from > > > > > > > joining > > > > > > > > > the > > > > > > > > > > > > grid > > > > > > > > > > > > > > with BLT#3. And we should fail nodes with an > error > > > > > message. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Aug 23, 2017 at 5:47 AM, Dmitriy > Setrakyan > > < > > > > > > > > > > > > > dsetrak...@apache.org> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergey, I am still confused. What is the > > > > > BaselineTopology > > > > > > > > > > interface > > > > > > > > > > > > in > > > > > > > > > > > > > > your > > > > > > > > > > > > > > > example? I thought that you agreed with me that > > we > > > > > simply > > > > > > > > need > > > > > > > > > a > > > > > > > > > > > > setter > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > activation nodes, no? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 22, 2017 at 4:47 AM, Sergey > Chugunov > > < > > > > > > > > > > > > > > > sergey.chugu...@gmail.com> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As I understand you use the term > > > > > > "minimalActivationNodes" > > > > > > > > as > > > > > > > > > a > > > > > > > > > > > > > synonym > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > BaselineTopology concept. > > > > > > > > > > > > > > > > In that case I agree with you that we can > > replace > > > > > both > > > > > > > > > > > "establish*" > > > > > > > > > > > > > > > methods > > > > > > > > > > > > > > > > with a simple setter method (see below in > > > summary). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Summing up the whole discussion I see the > > > > > functionality > > > > > > > as > > > > > > > > > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > New concept BaselineTopology is introduced. > The > > > > main > > > > > > > > features > > > > > > > > > > it > > > > > > > > > > > > > > enables > > > > > > > > > > > > > > > > are: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >1. automatic activation of cluster; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >2. easy management of cluster topology > > changes > > > > > > > (planned > > > > > > > > > > nodes > > > > > > > > > > > > > > > >maintenance, adding new nodes etc); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >3. eliminating of rebalancing traffic on > > > > > short-term > > > > > > > node > > > > > > > > > > > > failures. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Use cases to create BLT: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >1. user starts up new cluster of desired > > > number > > > > of > > > > > > > nodes > > > > > > > > > and > > > > > > > > > > > > > > activates > > > > > > > > > > > > > > > >it using existing API. BLT is created with > > all > > > > > nodes > > > > > > > > > > presented > > > > > > > > > > > > in > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > >cluster at the moment of activation, no > API > > is > > > > > > needed; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >2. user prepares BLT using web-console or > > > visor > > > > > CMD > > > > > > > > tools > > > > > > > > > > and > > > > > > > > > > > > sets > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > >the cluster. New API setter is needed: > > > > > > > > > > > > > > > >Ignite.activation().setBaselin > > > > > > > > eTopology(BaselineTopology > > > > > > > > > > > blt); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >3. user provides via static configuration > a > > > list > > > > > of > > > > > > > > nodes > > > > > > > > > > that > > > > > > > > > > > > are > > > > > > > > > > > > > > > >expected to be in the cluster. > > > > > > > > > > > > > > > >User starts nodes one by one; when all > > > > > preconfigured > > > > > > > > nodes > > > > > > > > > > are > > > > > > > > > > > > > > started > > > > > > > > > > > > > > > >cluster is activated and BLT is created. > > > > > > > > > > > > > > > >As list of nodes may be huge it is > provided > > > via > > > > > > > separate > > > > > > > > > > file > > > > > > > > > > > to > > > > > > > > > > > > > > avoid > > > > > > > > > > > > > > > >flooding main configuration. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Igniters, does this description match with > your > > > > > > > > understanding > > > > > > > > > > of > > > > > > > > > > > > > > > > functionality? If it does I'll create a set > of > > > > > tickets > > > > > > > and > > > > > > > > > > start > > > > > > > > > > > > > > working > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy > > > Setrakyan > > > > < > > > > > > > > > > > > > > > dsetrak...@apache.org> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I still do not see why anyone would > > explicitly > > > > call > > > > > > > > these 2 > > > > > > > > > > > > > methods: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation:: > > > > establishBaselineTopology();* > > > > > > > > > > > > > > > > > *Ignite::activation:: > > > establishBaselineTopology( > > > > > > > > > > > BaselineTopology > > > > > > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For example, if a web console, or some > other > > > > admin > > > > > > > > process, > > > > > > > > > > > want > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > automatically set currently started nodes > as > > > the > > > > > > > baseline > > > > > > > > > > > > topology, > > > > > > > > > > > > > > > > > shouldn't they just call a setter for > > > > > > > > > minimalActivationNodes? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey > > > > Dmitriev < > > > > > > > > > > > > > > > > admitr...@gridgain.com> > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > API is proposed in the head of the thread > > by > > > > > > Sergey, > > > > > > > > as I > > > > > > > > > > > > > > understood: > > > > > > > > > > > > > > > > > > __ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > API for BaselineTopology manipulation may > > > look > > > > > like > > > > > > > > this: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation:: > > > > > establishBaselineTopology();* > > > > > > > > > > > > > > > > > > *Ignite::activation:: > > > > establishBaselineTopology( > > > > > > > > > > > BaselineTopology > > > > > > > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Both methods will establish BT and > activate > > > > > cluster > > > > > > > > once > > > > > > > > > it > > > > > > > > > > > is > > > > > > > > > > > > > > > > > established. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The first one allows user to establish BT > > > using > > > > > > > current > > > > > > > > > > > > topology. > > > > > > > > > > > > > > If > > > > > > > > > > > > > > > > any > > > > > > > > > > > > > > > > > > changes happen to the topology during > > > > > establishing > > > > > > > > > process, > > > > > > > > > > > > user > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > notified and allowed to proceed or abort > > the > > > > > > > procedure. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Second method allows to use some > > > > > > > > monitoring'n'management > > > > > > > > > > > tools > > > > > > > > > > > > > like > > > > > > > > > > > > > > > > > > WebConsole where user can prepare a list > of > > > > > nodes, > > > > > > > > using > > > > > > > > > > them > > > > > > > > > > > > > > create > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > BT > > > > > > > > > > > > > > > > > > and send to the cluster a command to > > finally > > > > > > > establish > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From high level BaselineTopology entity > > > > contains > > > > > > only > > > > > > > > > > > > collection > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > nodes: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *BaselineTopology {* > > > > > > > > > > > > > > > > > > * Collection nodes;* > > > > > > > > > > > > > > > > > > *}* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *TopologyNode* here contains information > > > about > > > > > > node - > > > > > > > > its > > > > > > > > > > > > > > consistent > > > > > > > > > > > > > > > id > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > set of user attributes used to calculate > > > > affinity > > > > > > > > > function. > > > > > > > > > > > > > > > > > > __ > > __ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > View this message in context: > > > > > > http://apache-ignite- > > > > > > > > > > > > > > > > > > developers.2346864.n4.nabble. > > > > > > > > > com/Cluster-auto-activation- > > > > > > > > > > > > > > > > > > design-proposal-tp20295p21066.html > > > > > > > > > > > > > > > > > > Sent from the Apache Ignite Developers > > > mailing > > > > > list > > > > > > > > > archive > > > > > > > > > > > at > > > > > > > > > > > > > > > > > Nabble.com. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Cluster auto activation design proposal
e should keep nodes with BLT#2 > > from > > > > > > joining > > > > > > > > the > > > > > > > > > > > grid > > > > > > > > > > > > > with BLT#3. And we should fail nodes with an error > > > > message. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Aug 23, 2017 at 5:47 AM, Dmitriy Setrakyan > < > > > > > > > > > > > > dsetrak...@apache.org> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Sergey, I am still confused. What is the > > > > BaselineTopology > > > > > > > > > interface > > > > > > > > > > > in > > > > > > > > > > > > > your > > > > > > > > > > > > > > example? I thought that you agreed with me that > we > > > > simply > > > > > > > need > > > > > > > > a > > > > > > > > > > > setter > > > > > > > > > > > > > for > > > > > > > > > > > > > > activation nodes, no? > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 22, 2017 at 4:47 AM, Sergey Chugunov > < > > > > > > > > > > > > > > sergey.chugu...@gmail.com> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As I understand you use the term > > > > > "minimalActivationNodes" > > > > > > > as > > > > > > > > a > > > > > > > > > > > > synonym > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > BaselineTopology concept. > > > > > > > > > > > > > > > In that case I agree with you that we can > replace > > > > both > > > > > > > > > > "establish*" > > > > > > > > > > > > > > methods > > > > > > > > > > > > > > > with a simple setter method (see below in > > summary). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Summing up the whole discussion I see the > > > > functionality > > > > > > as > > > > > > > > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > New concept BaselineTopology is introduced. The > > > main > > > > > > > features > > > > > > > > > it > > > > > > > > > > > > > enables > > > > > > > > > > > > > > > are: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >1. automatic activation of cluster; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >2. easy management of cluster topology > changes > > > > > > (planned > > > > > > > > > nodes > > > > > > > > > > > > > > >maintenance, adding new nodes etc); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >3. eliminating of rebalancing traffic on > > > > short-term > > > > > > node > > > > > > > > > > > failures. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Use cases to create BLT: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >1. user starts up new cluster of desired > > number > > > of > > > > > > nodes > > > > > > > > and > > > > > > > > > > > > > activates > > > > > > > > > > > > > > >it using existing API. BLT is created with > all > > > > nodes > > > > > > > > > presented > > > > > > > > > > > in > > > > > > > > > > > > > the > > > > > > > > > > > > > > >cluster at the moment of activation, no API > is > > > > > needed; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >2. user prepares BLT using web-console or > > visor > > > > CMD > > > > > > > tools > > > > > > > > > and > > > > > > > > > > > sets > > > > > > > > > > > > > it > > > > > > > > > > > > > > to > > > > > > > > > > > > > > >the cluster. New API setter is needed: > > > > > > > > > > > > > > >Ignite.activation().setBaselin > > > > > > > eTopology(BaselineTopology > > > > > > > > > > blt); > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >3. user provides via static configuration a > > list > > > > of > > > > > > > nodes > > > > > > > > > that > > > > > > > > > > > are > > > > > > > > > > > > > > >expected to be in the cluster. > > > > > > > > > > > > > > >User starts nodes one by one; when all > > > > preconfigured > > > > > > > nodes > > > > > > > > > are > > > > > > > > > > > > > started > > > > > > > > > > > > > > >cluster is activated and BLT is created. > > > > > > > > > > > > > > >As list of nodes may be huge it is provided > > via > > > > > > separate > > > > > > > > > file > > > > > > > > > > to > > > > > > > > > > > > > avoid > > > > > > > > > > > > > > >flooding main configuration. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Igniters, does this description match with your > > > > > > > understanding > > > > > > > > > of > > > > > > > > > > > > > > > functionality? If it does I'll create a set of > > > > tickets > > > > > > and > > > > > > > > > start > > > > > > > > > > > > > working > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy > > Setrakyan > > > < > > > > > > > > > > > > > > dsetrak...@apache.org> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I still do not see why anyone would > explicitly > > > call > > > > > > > these 2 > > > > > > > > > > > > methods: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation:: > > > establishBaselineTopology();* > > > > > > > > > > > > > > > > *Ignite::activation:: > > establishBaselineTopology( > > > > > > > > > > BaselineTopology > > > > > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For example, if a web console, or some other > > > admin > > > > > > > process, > > > > > > > > > > want > > > > > > > > > > > to > > > > > > > > > > > > > > > > automatically set currently started nodes as > > the > > > > > > baseline > > > > > > > > > > > topology, > > > > > > > > > > > > > > > > shouldn't they just call a setter for > > > > > > > > minimalActivationNodes? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey > > > Dmitriev < > > > > > > > > > > > > > > > admitr...@gridgain.com> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > API is proposed in the head of the thread > by > > > > > Sergey, > > > > > > > as I > > > > > > > > > > > > > understood: > > > > > > > > > > > > > > > > > __ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > API for BaselineTopology manipulation may > > look > > > > like > > > > > > > this: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation:: > > > > establishBaselineTopology();* > > > > > > > > > > > > > > > > > *Ignite::activation:: > > > establishBaselineTopology( > > > > > > > > > > BaselineTopology > > > > > > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Both methods will establish BT and activate > > > > cluster > > > > > > > once > > > > > > > > it > > > > > > > > > > is > > > > > > > > > > > > > > > > established. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The first one allows user to establish BT > > using > > > > > > current > > > > > > > > > > > topology. > > > > > > > > > > > > > If > > > > > > > > > > > > > > > any > > > > > > > > > > > > > > > > > changes happen to the topology during > > > > establishing > > > > > > > > process, > > > > > > > > > > > user > > > > > > > > > > > > > will > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > notified and allowed to proceed or abort > the > > > > > > procedure. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Second method allows to use some > > > > > > > monitoring'n'management > > > > > > > > > > tools > > > > > > > > > > > > like > > > > > > > > > > > > > > > > > WebConsole where user can prepare a list of > > > > nodes, > > > > > > > using > > > > > > > > > them > > > > > > > > > > > > > create > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > BT > > > > > > > > > > > > > > > > > and send to the cluster a command to > finally > > > > > > establish > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From high level BaselineTopology entity > > > contains > > > > > only > > > > > > > > > > > collection > > > > > > > > > > > > of > > > > > > > > > > > > > > > > nodes: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *BaselineTopology {* > > > > > > > > > > > > > > > > > * Collection nodes;* > > > > > > > > > > > > > > > > > *}* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *TopologyNode* here contains information > > about > > > > > node - > > > > > > > its > > > > > > > > > > > > > consistent > > > > > > > > > > > > > > id > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > set of user attributes used to calculate > > > affinity > > > > > > > > function. > > > > > > > > > > > > > > > > > __ > __ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > View this message in context: > > > > > http://apache-ignite- > > > > > > > > > > > > > > > > > developers.2346864.n4.nabble. > > > > > > > > com/Cluster-auto-activation- > > > > > > > > > > > > > > > > > design-proposal-tp20295p21066.html > > > > > > > > > > > > > > > > > Sent from the Apache Ignite Developers > > mailing > > > > list > > > > > > > > archive > > > > > > > > > > at > > > > > > > > > > > > > > > > Nabble.com. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Cluster auto activation design proposal
> > > > > sergey.chugu...@gmail.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > > > > > > > > > > > > > > > As I understand you use the term > > > > "minimalActivationNodes" > > > > > > as > > > > > > > a > > > > > > > > > > > synonym > > > > > > > > > > > > > for > > > > > > > > > > > > > > BaselineTopology concept. > > > > > > > > > > > > > > In that case I agree with you that we can replace > > > both > > > > > > > > > "establish*" > > > > > > > > > > > > > methods > > > > > > > > > > > > > > with a simple setter method (see below in > summary). > > > > > > > > > > > > > > > > > > > > > > > > > > > > Summing up the whole discussion I see the > > > functionality > > > > > as > > > > > > > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > New concept BaselineTopology is introduced. The > > main > > > > > > features > > > > > > > > it > > > > > > > > > > > > enables > > > > > > > > > > > > > > are: > > > > > > > > > > > > > > > > > > > > > > > > > > > >1. automatic activation of cluster; > > > > > > > > > > > > > > > > > > > > > > > > > > > >2. easy management of cluster topology changes > > > > > (planned > > > > > > > > nodes > > > > > > > > > > > > > >maintenance, adding new nodes etc); > > > > > > > > > > > > > > > > > > > > > > > > > > > >3. eliminating of rebalancing traffic on > > > short-term > > > > > node > > > > > > > > > > failures. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Use cases to create BLT: > > > > > > > > > > > > > > > > > > > > > > > > > > > >1. user starts up new cluster of desired > number > > of > > > > > nodes > > > > > > > and > > > > > > > > > > > > activates > > > > > > > > > > > > > >it using existing API. BLT is created with all > > > nodes > > > > > > > > presented > > > > > > > > > > in > > > > > > > > > > > > the > > > > > > > > > > > > > >cluster at the moment of activation, no API is > > > > needed; > > > > > > > > > > > > > > > > > > > > > > > > > > > >2. user prepares BLT using web-console or > visor > > > CMD > > > > > > tools > > > > > > > > and > > > > > > > > > > sets > > > > > > > > > > > > it > > > > > > > > > > > > > to > > > > > > > > > > > > > >the cluster. New API setter is needed: > > > > > > > > > > > > > >Ignite.activation().setBaselin > > > > > > eTopology(BaselineTopology > > > > > > > > > blt); > > > > > > > > > > > > > > > > > > > > > > > > > > > >3. user provides via static configuration a > list > > > of > > > > > > nodes > > > > > > > > that > > > > > > > > > > are > > > > > > > > > > > > > >expected to be in the cluster. > > > > > > > > > > > > > >User starts nodes one by one; when all > > > preconfigured > > > > > > nodes > > > > > > > > are > > > > > > > > > > > > started > > > > > > > > > > > > > >cluster is activated and BLT is created. > > > > > > > > > > > > > >As list of nodes may be huge it is provided > via > > > > > separate > > > > > > > > file > > > > > > > > > to > > > > > > > > > > > > avoid > > > > > > > > > > > > > >flooding main configuration. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Igniters, does this description match with your > > > > > > understanding > > > > > > > > of > > > > > > > > > > > > > > functionality? If it does I'll create a set of > > > tickets > > > > > and > > > > > > > > start > > > > > > > > > > > > working > > > > > > > > > > > > > on > > > > > > > > > > > > > > implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy > Setrakyan > > < > > > > > > > > > > > > > dsetrak...@apache.org> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I still do not see why anyone would explicitly > > call > > > > > > these 2 > > > > > > > > > > > methods: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation:: > > establishBaselineTopology();* > > > > > > > > > > > > > > > *Ignite::activation:: > establishBaselineTopology( > > > > > > > > > BaselineTopology > > > > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For example, if a web console, or some other > > admin > > > > > > process, > > > > > > > > > want > > > > > > > > > > to > > > > > > > > > > > > > > > automatically set currently started nodes as > the > > > > > baseline > > > > > > > > > > topology, > > > > > > > > > > > > > > > shouldn't they just call a setter for > > > > > > > minimalActivationNodes? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey > > Dmitriev < > > > > > > > > > > > > > > admitr...@gridgain.com> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > API is proposed in the head of the thread by > > > > Sergey, > > > > > > as I > > > > > > > > > > > > understood: > > > > > > > > > > > > > > > > __ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > API for BaselineTopology manipulation may > look > > > like > > > > > > this: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation:: > > > establishBaselineTopology();* > > > > > > > > > > > > > > > > *Ignite::activation:: > > establishBaselineTopology( > > > > > > > > > BaselineTopology > > > > > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Both methods will establish BT and activate > > > cluster > > > > > > once > > > > > > > it > > > > > > > > > is > > > > > > > > > > > > > > > established. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The first one allows user to establish BT > using > > > > > current > > > > > > > > > > topology. > > > > > > > > > > > > If > > > > > > > > > > > > > > any > > > > > > > > > > > > > > > > changes happen to the topology during > > > establishing > > > > > > > process, > > > > > > > > > > user > > > > > > > > > > > > will > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > notified and allowed to proceed or abort the > > > > > procedure. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Second method allows to use some > > > > > > monitoring'n'management > > > > > > > > > tools > > > > > > > > > > > like > > > > > > > > > > > > > > > > WebConsole where user can prepare a list of > > > nodes, > > > > > > using > > > > > > > > them > > > > > > > > > > > > create > > > > > > > > > > > > > a > > > > > > > > > > > > > > BT > > > > > > > > > > > > > > > > and send to the cluster a command to finally > > > > > establish > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From high level BaselineTopology entity > > contains > > > > only > > > > > > > > > > collection > > > > > > > > > > > of > > > > > > > > > > > > > > > nodes: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *BaselineTopology {* > > > > > > > > > > > > > > > > * Collection nodes;* > > > > > > > > > > > > > > > > *}* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *TopologyNode* here contains information > about > > > > node - > > > > > > its > > > > > > > > > > > > consistent > > > > > > > > > > > > > id > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > set of user attributes used to calculate > > affinity > > > > > > > function. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > View this message in context: > > > > http://apache-ignite- > > > > > > > > > > > > > > > > developers.2346864.n4.nabble. > > > > > > > com/Cluster-auto-activation- > > > > > > > > > > > > > > > > design-proposal-tp20295p21066.html > > > > > > > > > > > > > > > > Sent from the Apache Ignite Developers > mailing > > > list > > > > > > > archive > > > > > > > > > at > > > > > > > > > > > > > > > Nabble.com. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Cluster auto activation design proposal
> > > > > > > > > > > > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > > > > > > > > > > > > > > > As I understand you use the term > > > > "minimalActivationNodes" > > > > > > as > > > > > > > a > > > > > > > > > > > synonym > > > > > > > > > > > > > for > > > > > > > > > > > > > > BaselineTopology concept. > > > > > > > > > > > > > > In that case I agree with you that we can replace > > > both > > > > > > > > > "establish*" > > > > > > > > > > > > > methods > > > > > > > > > > > > > > with a simple setter method (see below in > summary). > > > > > > > > > > > > > > > > > > > > > > > > > > > > Summing up the whole discussion I see the > > > functionality > > > > > as > > > > > > > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > New concept BaselineTopology is introduced. The > > main > > > > > > features > > > > > > > > it > > > > > > > > > > > > enables > > > > > > > > > > > > > > are: > > > > > > > > > > > > > > > > > > > > > > > > > > > >1. automatic activation of cluster; > > > > > > > > > > > > > > > > > > > > > > > > > > > >2. easy management of cluster topology changes > > > > > (planned > > > > > > > > nodes > > > > > > > > > > > > > >maintenance, adding new nodes etc); > > > > > > > > > > > > > > > > > > > > > > > > > > > >3. eliminating of rebalancing traffic on > > > short-term > > > > > node > > > > > > > > > > failures. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Use cases to create BLT: > > > > > > > > > > > > > > > > > > > > > > > > > > > >1. user starts up new cluster of desired > number > > of > > > > > nodes > > > > > > > and > > > > > > > > > > > > activates > > > > > > > > > > > > > >it using existing API. BLT is created with all > > > nodes > > > > > > > > presented > > > > > > > > > > in > > > > > > > > > > > > the > > > > > > > > > > > > > >cluster at the moment of activation, no API is > > > > needed; > > > > > > > > > > > > > > > > > > > > > > > > > > > >2. user prepares BLT using web-console or > visor > > > CMD > > > > > > tools > > > > > > > > and > > > > > > > > > > sets > > > > > > > > > > > > it > > > > > > > > > > > > > to > > > > > > > > > > > > > >the cluster. New API setter is needed: > > > > > > > > > > > > > >Ignite.activation().setBaselin > > > > > > eTopology(BaselineTopology > > > > > > > > > blt); > > > > > > > > > > > > > > > > > > > > > > > > > > > >3. user provides via static configuration a > list > > > of > > > > > > nodes > > > > > > > > that > > > > > > > > > > are > > > > > > > > > > > > > >expected to be in the cluster. > > > > > > > > > > > > > >User starts nodes one by one; when all > > > preconfigured > > > > > > nodes > > > > > > > > are > > > > > > > > > > > > started > > > > > > > > > > > > > >cluster is activated and BLT is created. > > > > > > > > > > > > > >As list of nodes may be huge it is provided > via > > > > > separate > > > > > > > > file > > > > > > > > > to > > > > > > > > > > > > avoid > > > > > > > > > > > > > >flooding main configuration. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Igniters, does this description match with your > > > > > > understanding > > > > > > > > of > > > > > > > > > > > > > > functionality? If it does I'll create a set of > > > tickets > > > > > and > > > > > > > > start > > > > > > > > > > > > working > > > > > > > > > > > > > on > > > > > > > > > > > > > > implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy > Setrakyan > > < > > > > > > > > > > > > > dsetrak...@apache.org> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I still do not see why anyone would explicitly > > call > > > > > > these 2 > > > > > > > > > > > methods: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation:: > > establishBaselineTopology();* > > > > > > > > > > > > > > > *Ignite::activation:: > establishBaselineTopology( > > > > > > > > > BaselineTopology > > > > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For example, if a web console, or some other > > admin > > > > > > process, > > > > > > > > > want > > > > > > > > > > to > > > > > > > > > > > > > > > automatically set currently started nodes as > the > > > > > baseline > > > > > > > > > > topology, > > > > > > > > > > > > > > > shouldn't they just call a setter for > > > > > > > minimalActivationNodes? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey > > Dmitriev < > > > > > > > > > > > > > > admitr...@gridgain.com> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > API is proposed in the head of the thread by > > > > Sergey, > > > > > > as I > > > > > > > > > > > > understood: > > > > > > > > > > > > > > > > __ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > API for BaselineTopology manipulation may > look > > > like > > > > > > this: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation:: > > > establishBaselineTopology();* > > > > > > > > > > > > > > > > *Ignite::activation:: > > establishBaselineTopology( > > > > > > > > > BaselineTopology > > > > > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Both methods will establish BT and activate > > > cluster > > > > > > once > > > > > > > it > > > > > > > > > is > > > > > > > > > > > > > > > established. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The first one allows user to establish BT > using > > > > > current > > > > > > > > > > topology. > > > > > > > > > > > > If > > > > > > > > > > > > > > any > > > > > > > > > > > > > > > > changes happen to the topology during > > > establishing > > > > > > > process, > > > > > > > > > > user > > > > > > > > > > > > will > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > notified and allowed to proceed or abort the > > > > > procedure. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Second method allows to use some > > > > > > monitoring'n'management > > > > > > > > > tools > > > > > > > > > > > like > > > > > > > > > > > > > > > > WebConsole where user can prepare a list of > > > nodes, > > > > > > using > > > > > > > > them > > > > > > > > > > > > create > > > > > > > > > > > > > a > > > > > > > > > > > > > > BT > > > > > > > > > > > > > > > > and send to the cluster a command to finally > > > > > establish > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From high level BaselineTopology entity > > contains > > > > only > > > > > > > > > > collection > > > > > > > > > > > of > > > > > > > > > > > > > > > nodes: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *BaselineTopology {* > > > > > > > > > > > > > > > > * Collection nodes;* > > > > > > > > > > > > > > > > *}* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *TopologyNode* here contains information > about > > > > node - > > > > > > its > > > > > > > > > > > > consistent > > > > > > > > > > > > > id > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > set of user attributes used to calculate > > affinity > > > > > > > function. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > View this message in context: > > > > http://apache-ignite- > > > > > > > > > > > > > > > > developers.2346864.n4.nabble. > > > > > > > com/Cluster-auto-activation- > > > > > > > > > > > > > > > > design-proposal-tp20295p21066.html > > > > > > > > > > > > > > > > Sent from the Apache Ignite Developers > mailing > > > list > > > > > > > archive > > > > > > > > > at > > > > > > > > > > > > > > > Nabble.com. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Best regards, Ilya
Re: Cluster auto activation design proposal
> > > > Part2_grid. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If join is successful we have an undefined state of the > > > > > resulting > > > > > > > > grid: > > > > > > > > > > > values for the same key may (and will) differ between > > grid > > > > > parts. > > > > > > > > > > > > > > > > > > > > > > So to prevent this we should keep nodes with BLT#2 from > > > > joining > > > > > > the > > > > > > > > > grid > > > > > > > > > > > with BLT#3. And we should fail nodes with an error > > message. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Aug 23, 2017 at 5:47 AM, Dmitriy Setrakyan < > > > > > > > > > > dsetrak...@apache.org> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Sergey, I am still confused. What is the > > BaselineTopology > > > > > > > interface > > > > > > > > > in > > > > > > > > > > > your > > > > > > > > > > > > example? I thought that you agreed with me that we > > simply > > > > > need > > > > > > a > > > > > > > > > setter > > > > > > > > > > > for > > > > > > > > > > > > activation nodes, no? > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 22, 2017 at 4:47 AM, Sergey Chugunov < > > > > > > > > > > > > sergey.chugu...@gmail.com> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > > > > > > > > > > > > > As I understand you use the term > > > "minimalActivationNodes" > > > > > as > > > > > > a > > > > > > > > > > synonym > > > > > > > > > > > > for > > > > > > > > > > > > > BaselineTopology concept. > > > > > > > > > > > > > In that case I agree with you that we can replace > > both > > > > > > > > "establish*" > > > > > > > > > > > > methods > > > > > > > > > > > > > with a simple setter method (see below in summary). > > > > > > > > > > > > > > > > > > > > > > > > > > Summing up the whole discussion I see the > > functionality > > > > as > > > > > > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > > > New concept BaselineTopology is introduced. The > main > > > > > features > > > > > > > it > > > > > > > > > > > enables > > > > > > > > > > > > > are: > > > > > > > > > > > > > > > > > > > > > > > > > >1. automatic activation of cluster; > > > > > > > > > > > > > > > > > > > > > > > > > >2. easy management of cluster topology changes > > > > (planned > > > > > > > nodes > > > > > > > > > > > > >maintenance, adding new nodes etc); > > > > > > > > > > > > > > > > > > > > > > > > > >3. eliminating of rebalancing traffic on > > short-term > > > > node > > > > > > > > > failures. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Use cases to create BLT: > > > > > > > > > > > > > > > > > > > > > > > > > >1. user starts up new cluster of desired number > of > > > > nodes > > > > > > and > > > > > > > > > > > activates > > > > > > > > > > > > >it using existing API. BLT is created with all > > nodes > > > > > > > presented > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > > >cluster at the moment of activation, no API is > > > needed; > > > > > > > > > > > > > > > > > > > > > > > > > >2. user prepares BLT using web-console or visor > > CMD > > > > > tools > > > > > > > and > > > > > > > > > sets > > > > > > > > > > > it > > > > > > > > > > > > to > > > > > > > > > > > > >the cluster. New API setter is needed: > > > > > > > > > > > > >Ignite.activation().setBaselin > > > > > eTopology(BaselineTopology > > > > > > > > blt); > > > > > > > > > > > > > > > > > > > > > > > > > >3. user provides via static configuration a list > > of > > > > > nodes > > > > > > > that > > > > > > > > > are > > > > > > > > > > > > >expected to be in the cluster. > > > > > > > > > > > > >User starts nodes one by one; when all > > preconfigured > > > > > nodes > > > > > > > are > > > > > > > > > > > started > > > > > > > > > > > > >cluster is activated and BLT is created. > > > > > > > > > > > > >As list of nodes may be huge it is provided via > > > > separate > > > > > > > file > > > > > > > > to > > > > > > > > > > > avoid > > > > > > > > > > > > >flooding main configuration. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Igniters, does this description match with your > > > > > understanding > > > > > > > of > > > > > > > > > > > > > functionality? If it does I'll create a set of > > tickets > > > > and > > > > > > > start > > > > > > > > > > > working > > > > > > > > > > > > on > > > > > > > > > > > > > implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan > < > > > > > > > > > > > > dsetrak...@apache.org> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > I still do not see why anyone would explicitly > call > > > > > these 2 > > > > > > > > > > methods: > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation:: > establishBaselineTopology();* > > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology( > > > > > > > > BaselineTopology > > > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > > > > > For example, if a web console, or some other > admin > > > > > process, > > > > > > > > want > > > > > > > > > to > > > > > > > > > > > > > > automatically set currently started nodes as the > > > > baseline > > > > > > > > > topology, > > > > > > > > > > > > > > shouldn't they just call a setter for > > > > > > minimalActivationNodes? > > > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey > Dmitriev < > > > > > > > > > > > > > admitr...@gridgain.com> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > API is proposed in the head of the thread by > > > Sergey, > > > > > as I > > > > > > > > > > > understood: > > > > > > > > > > > > > > > __ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > API for BaselineTopology manipulation may look > > like > > > > > this: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation:: > > establishBaselineTopology();* > > > > > > > > > > > > > > > *Ignite::activation:: > establishBaselineTopology( > > > > > > > > BaselineTopology > > > > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Both methods will establish BT and activate > > cluster > > > > > once > > > > > > it > > > > > > > > is > > > > > > > > > > > > > > established. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The first one allows user to establish BT using > > > > current > > > > > > > > > topology. > > > > > > > > > > > If > > > > > > > > > > > > > any > > > > > > > > > > > > > > > changes happen to the topology during > > establishing > > > > > > process, > > > > > > > > > user > > > > > > > > > > > will > > > > > > > > > > > > > be > > > > > > > > > > > > > > > notified and allowed to proceed or abort the > > > > procedure. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Second method allows to use some > > > > > monitoring'n'management > > > > > > > > tools > > > > > > > > > > like > > > > > > > > > > > > > > > WebConsole where user can prepare a list of > > nodes, > > > > > using > > > > > > > them > > > > > > > > > > > create > > > > > > > > > > > > a > > > > > > > > > > > > > BT > > > > > > > > > > > > > > > and send to the cluster a command to finally > > > > establish > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From high level BaselineTopology entity > contains > > > only > > > > > > > > > collection > > > > > > > > > > of > > > > > > > > > > > > > > nodes: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *BaselineTopology {* > > > > > > > > > > > > > > > * Collection nodes;* > > > > > > > > > > > > > > > *}* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *TopologyNode* here contains information about > > > node - > > > > > its > > > > > > > > > > > consistent > > > > > > > > > > > > id > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > set of user attributes used to calculate > affinity > > > > > > function. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > View this message in context: > > > http://apache-ignite- > > > > > > > > > > > > > > > developers.2346864.n4.nabble. > > > > > > com/Cluster-auto-activation- > > > > > > > > > > > > > > > design-proposal-tp20295p21066.html > > > > > > > > > > > > > > > Sent from the Apache Ignite Developers mailing > > list > > > > > > archive > > > > > > > > at > > > > > > > > > > > > > > Nabble.com. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Cluster auto activation design proposal
> > > > in > > > > > > > > > > your > > > > > > > > > > > example? I thought that you agreed with me that we > simply > > > > need > > > > > a > > > > > > > > setter > > > > > > > > > > for > > > > > > > > > > > activation nodes, no? > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 22, 2017 at 4:47 AM, Sergey Chugunov < > > > > > > > > > > > sergey.chugu...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > > > > > > > > > > > As I understand you use the term > > "minimalActivationNodes" > > > > as > > > > > a > > > > > > > > > synonym > > > > > > > > > > > for > > > > > > > > > > > > BaselineTopology concept. > > > > > > > > > > > > In that case I agree with you that we can replace > both > > > > > > > "establish*" > > > > > > > > > > > methods > > > > > > > > > > > > with a simple setter method (see below in summary). > > > > > > > > > > > > > > > > > > > > > > > > Summing up the whole discussion I see the > functionality > > > as > > > > > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > New concept BaselineTopology is introduced. The main > > > > features > > > > > > it > > > > > > > > > > enables > > > > > > > > > > > > are: > > > > > > > > > > > > > > > > > > > > > > > >1. automatic activation of cluster; > > > > > > > > > > > > > > > > > > > > > > > >2. easy management of cluster topology changes > > > (planned > > > > > > nodes > > > > > > > > > > > >maintenance, adding new nodes etc); > > > > > > > > > > > > > > > > > > > > > > > >3. eliminating of rebalancing traffic on > short-term > > > node > > > > > > > > failures. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Use cases to create BLT: > > > > > > > > > > > > > > > > > > > > > > > >1. user starts up new cluster of desired number of > > > nodes > > > > > and > > > > > > > > > > activates > > > > > > > > > > > >it using existing API. BLT is created with all > nodes > > > > > > presented > > > > > > > > in > > > > > > > > > > the > > > > > > > > > > > >cluster at the moment of activation, no API is > > needed; > > > > > > > > > > > > > > > > > > > > > > > >2. user prepares BLT using web-console or visor > CMD > > > > tools > > > > > > and > > > > > > > > sets > > > > > > > > > > it > > > > > > > > > > > to > > > > > > > > > > > >the cluster. New API setter is needed: > > > > > > > > > > > >Ignite.activation().setBaselin > > > > eTopology(BaselineTopology > > > > > > > blt); > > > > > > > > > > > > > > > > > > > > > > > >3. user provides via static configuration a list > of > > > > nodes > > > > > > that > > > > > > > > are > > > > > > > > > > > >expected to be in the cluster. > > > > > > > > > > > >User starts nodes one by one; when all > preconfigured > > > > nodes > > > > > > are > > > > > > > > > > started > > > > > > > > > > > >cluster is activated and BLT is created. > > > > > > > > > > > >As list of nodes may be huge it is provided via > > > separate > > > > > > file > > > > > > > to > > > > > > > > > > avoid > > > > > > > > > > > >flooding main configuration. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Igniters, does this description match with your > > > > understanding > > > > > > of > > > > > > > > > > > > functionality? If it does I'll create a set of > tickets > > > and > > > > > > start > > > > > > > > > > working > > > > > > > > > > > on > > > > > > > > > > > > implementation. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan < > > > > > > > > > > > dsetrak...@apache.org> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > I still do not see why anyone would explicitly call > > > > these 2 > > > > > > > > > methods: > > > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology( > > > > > > > BaselineTopology > > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > > > For example, if a web console, or some other admin > > > > process, > > > > > > > want > > > > > > > > to > > > > > > > > > > > > > automatically set currently started nodes as the > > > baseline > > > > > > > > topology, > > > > > > > > > > > > > shouldn't they just call a setter for > > > > > minimalActivationNodes? > > > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev < > > > > > > > > > > > > admitr...@gridgain.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > API is proposed in the head of the thread by > > Sergey, > > > > as I > > > > > > > > > > understood: > > > > > > > > > > > > > > __ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > API for BaselineTopology manipulation may look > like > > > > this: > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation:: > establishBaselineTopology();* > > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology( > > > > > > > BaselineTopology > > > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > > > > > Both methods will establish BT and activate > cluster > > > > once > > > > > it > > > > > > > is > > > > > > > > > > > > > established. > > > > > > > > > > > > > > > > > > > > > > > > > > > > The first one allows user to establish BT using > > > current > > > > > > > > topology. > > > > > > > > > > If > > > > > > > > > > > > any > > > > > > > > > > > > > > changes happen to the topology during > establishing > > > > > process, > > > > > > > > user > > > > > > > > > > will > > > > > > > > > > > > be > > > > > > > > > > > > > > notified and allowed to proceed or abort the > > > procedure. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Second method allows to use some > > > > monitoring'n'management > > > > > > > tools > > > > > > > > > like > > > > > > > > > > > > > > WebConsole where user can prepare a list of > nodes, > > > > using > > > > > > them > > > > > > > > > > create > > > > > > > > > > > a > > > > > > > > > > > > BT > > > > > > > > > > > > > > and send to the cluster a command to finally > > > establish > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > From high level BaselineTopology entity contains > > only > > > > > > > > collection > > > > > > > > > of > > > > > > > > > > > > > nodes: > > > > > > > > > > > > > > > > > > > > > > > > > > > > *BaselineTopology {* > > > > > > > > > > > > > > * Collection nodes;* > > > > > > > > > > > > > > *}* > > > > > > > > > > > > > > > > > > > > > > > > > > > > *TopologyNode* here contains information about > > node - > > > > its > > > > > > > > > > consistent > > > > > > > > > > > id > > > > > > > > > > > > > and > > > > > > > > > > > > > > set of user attributes used to calculate affinity > > > > > function. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > View this message in context: > > http://apache-ignite- > > > > > > > > > > > > > > developers.2346864.n4.nabble. > > > > > com/Cluster-auto-activation- > > > > > > > > > > > > > > design-proposal-tp20295p21066.html > > > > > > > > > > > > > > Sent from the Apache Ignite Developers mailing > list > > > > > archive > > > > > > > at > > > > > > > > > > > > > Nabble.com. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Cluster auto activation design proposal
automatic activation of cluster; > > > > > > > > > > > > > > > > > > > > > >2. easy management of cluster topology changes > > (planned > > > > > nodes > > > > > > > > > > >maintenance, adding new nodes etc); > > > > > > > > > > > > > > > > > > > > > >3. eliminating of rebalancing traffic on short-term > > node > > > > > > > failures. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Use cases to create BLT: > > > > > > > > > > > > > > > > > > > > > >1. user starts up new cluster of desired number of > > nodes > > > > and > > > > > > > > > activates > > > > > > > > > > >it using existing API. BLT is created with all nodes > > > > > presented > > > > > > > in > > > > > > > > > the > > > > > > > > > > >cluster at the moment of activation, no API is > needed; > > > > > > > > > > > > > > > > > > > > > >2. user prepares BLT using web-console or visor CMD > > > tools > > > > > and > > > > > > > sets > > > > > > > > > it > > > > > > > > > > to > > > > > > > > > > >the cluster. New API setter is needed: > > > > > > > > > > >Ignite.activation().setBaselin > > > eTopology(BaselineTopology > > > > > > blt); > > > > > > > > > > > > > > > > > > > > > >3. user provides via static configuration a list of > > > nodes > > > > > that > > > > > > > are > > > > > > > > > > >expected to be in the cluster. > > > > > > > > > > >User starts nodes one by one; when all preconfigured > > > nodes > > > > > are > > > > > > > > > started > > > > > > > > > > >cluster is activated and BLT is created. > > > > > > > > > > >As list of nodes may be huge it is provided via > > separate > > > > > file > > > > > > to > > > > > > > > > avoid > > > > > > > > > > >flooding main configuration. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Igniters, does this description match with your > > > understanding > > > > > of > > > > > > > > > > > functionality? If it does I'll create a set of tickets > > and > > > > > start > > > > > > > > > working > > > > > > > > > > on > > > > > > > > > > > implementation. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan < > > > > > > > > > > dsetrak...@apache.org> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > I still do not see why anyone would explicitly call > > > these 2 > > > > > > > > methods: > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology( > > > > > > BaselineTopology > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > For example, if a web console, or some other admin > > > process, > > > > > > want > > > > > > > to > > > > > > > > > > > > automatically set currently started nodes as the > > baseline > > > > > > > topology, > > > > > > > > > > > > shouldn't they just call a setter for > > > > minimalActivationNodes? > > > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev < > > > > > > > > > > > admitr...@gridgain.com> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > API is proposed in the head of the thread by > Sergey, > > > as I > > > > > > > > > understood: > > > > > > > > > > > > > __ > > > > > > > > > > > > > > > > > > > > > > > > > > > > API for BaselineTopology manipulation may look like > > > this: > > > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology( > > > > > > BaselineTopology > > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > > > Both methods will establish BT and activate cluster > > > once > > > > it > > > > > > is > > > > > > > > > > > > established. > > > > > > > > > > > > > > > > > > > > > > > > > > The first one allows user to establish BT using > > current > > > > > > > topology. > > > > > > > > > If > > > > > > > > > > > any > > > > > > > > > > > > > changes happen to the topology during establishing > > > > process, > > > > > > > user > > > > > > > > > will > > > > > > > > > > > be > > > > > > > > > > > > > notified and allowed to proceed or abort the > > procedure. > > > > > > > > > > > > > > > > > > > > > > > > > > Second method allows to use some > > > monitoring'n'management > > > > > > tools > > > > > > > > like > > > > > > > > > > > > > WebConsole where user can prepare a list of nodes, > > > using > > > > > them > > > > > > > > > create > > > > > > > > > > a > > > > > > > > > > > BT > > > > > > > > > > > > > and send to the cluster a command to finally > > establish > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > From high level BaselineTopology entity contains > only > > > > > > > collection > > > > > > > > of > > > > > > > > > > > > nodes: > > > > > > > > > > > > > > > > > > > > > > > > > > *BaselineTopology {* > > > > > > > > > > > > > * Collection nodes;* > > > > > > > > > > > > > *}* > > > > > > > > > > > > > > > > > > > > > > > > > > *TopologyNode* here contains information about > node - > > > its > > > > > > > > > consistent > > > > > > > > > > id > > > > > > > > > > > > and > > > > > > > > > > > > > set of user attributes used to calculate affinity > > > > function. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > View this message in context: > http://apache-ignite- > > > > > > > > > > > > > developers.2346864.n4.nabble. > > > > com/Cluster-auto-activation- > > > > > > > > > > > > > design-proposal-tp20295p21066.html > > > > > > > > > > > > > Sent from the Apache Ignite Developers mailing list > > > > archive > > > > > > at > > > > > > > > > > > > Nabble.com. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Cluster auto activation design proposal
ld keep nodes with BLT#2 from > joining > > > the > > > > > > grid > > > > > > > > with BLT#3. And we should fail nodes with an error message. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Aug 23, 2017 at 5:47 AM, Dmitriy Setrakyan < > > > > > > > dsetrak...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Sergey, I am still confused. What is the BaselineTopology > > > > interface > > > > > > in > > > > > > > > your > > > > > > > > > example? I thought that you agreed with me that we simply > > need > > > a > > > > > > setter > > > > > > > > for > > > > > > > > > activation nodes, no? > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > On Tue, Aug 22, 2017 at 4:47 AM, Sergey Chugunov < > > > > > > > > > sergey.chugu...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > > > > > > > As I understand you use the term "minimalActivationNodes" > > as > > > a > > > > > > > synonym > > > > > > > > > for > > > > > > > > > > BaselineTopology concept. > > > > > > > > > > In that case I agree with you that we can replace both > > > > > "establish*" > > > > > > > > > methods > > > > > > > > > > with a simple setter method (see below in summary). > > > > > > > > > > > > > > > > > > > > Summing up the whole discussion I see the functionality > as > > > > > > following: > > > > > > > > > > > > > > > > > > > > New concept BaselineTopology is introduced. The main > > features > > > > it > > > > > > > > enables > > > > > > > > > > are: > > > > > > > > > > > > > > > > > > > >1. automatic activation of cluster; > > > > > > > > > > > > > > > > > > > >2. easy management of cluster topology changes > (planned > > > > nodes > > > > > > > > > >maintenance, adding new nodes etc); > > > > > > > > > > > > > > > > > > > >3. eliminating of rebalancing traffic on short-term > node > > > > > > failures. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Use cases to create BLT: > > > > > > > > > > > > > > > > > > > >1. user starts up new cluster of desired number of > nodes > > > and > > > > > > > > activates > > > > > > > > > >it using existing API. BLT is created with all nodes > > > > presented > > > > > > in > > > > > > > > the > > > > > > > > > >cluster at the moment of activation, no API is needed; > > > > > > > > > > > > > > > > > > > >2. user prepares BLT using web-console or visor CMD > > tools > > > > and > > > > > > sets > > > > > > > > it > > > > > > > > > to > > > > > > > > > >the cluster. New API setter is needed: > > > > > > > > > >Ignite.activation().setBaselin > > eTopology(BaselineTopology > > > > > blt); > > > > > > > > > > > > > > > > > > > >3. user provides via static configuration a list of > > nodes > > > > that > > > > > > are > > > > > > > > > >expected to be in the cluster. > > > > > > > > > >User starts nodes one by one; when all preconfigured > > nodes > > > > are > > > > > > > > started > > > > > > > > > >cluster is activated and BLT is created. > > > > > > > > > >As list of nodes may be huge it is provided via > separate > > > > file > > > > > to > > > > > > > > avoid > > > > > > > > > >flooding main configuration. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Igniters, does this description match with your > > understanding > > > > of > > > > > > > > > > functionality? If it does I'll create a set of tickets > and > > > > start > > > > > > > > working > > > > > > > > > on > > > > > > > > > > implementation. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan < > > > > > > > > > dsetrak...@apache.org> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > I still do not see why anyone would explicitly call > > these 2 > > > > > > > methods: > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > > > > > > > *Ignite::activation::establishBaselineTopology( > > > > > BaselineTopology > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > For example, if a web console, or some other admin > > process, > > > > > want > > > > > > to > > > > > > > > > > > automatically set currently started nodes as the > baseline > > > > > > topology, > > > > > > > > > > > shouldn't they just call a setter for > > > minimalActivationNodes? > > > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev < > > > > > > > > > > admitr...@gridgain.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > API is proposed in the head of the thread by Sergey, > > as I > > > > > > > > understood: > > > > > > > > > > > > __ > > > > > > > > > > > > > > > > > > > > > > > > > API for BaselineTopology manipulation may look like > > this: > > > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology( > > > > > BaselineTopology > > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > > > Both methods will establish BT and activate cluster > > once > > > it > > > > > is > > > > > > > > > > > established. > > > > > > > > > > > > > > > > > > > > > > > > The first one allows user to establish BT using > current > > > > > > topology. > > > > > > > > If > > > > > > > > > > any > > > > > > > > > > > > changes happen to the topology during establishing > > > process, > > > > > > user > > > > > > > > will > > > > > > > > > > be > > > > > > > > > > > > notified and allowed to proceed or abort the > procedure. > > > > > > > > > > > > > > > > > > > > > > > > Second method allows to use some > > monitoring'n'management > > > > > tools > > > > > > > like > > > > > > > > > > > > WebConsole where user can prepare a list of nodes, > > using > > > > them > > > > > > > > create > > > > > > > > > a > > > > > > > > > > BT > > > > > > > > > > > > and send to the cluster a command to finally > establish > > > it. > > > > > > > > > > > > > > > > > > > > > > > > From high level BaselineTopology entity contains only > > > > > > collection > > > > > > > of > > > > > > > > > > > nodes: > > > > > > > > > > > > > > > > > > > > > > > > *BaselineTopology {* > > > > > > > > > > > > * Collection nodes;* > > > > > > > > > > > > *}* > > > > > > > > > > > > > > > > > > > > > > > > *TopologyNode* here contains information about node - > > its > > > > > > > > consistent > > > > > > > > > id > > > > > > > > > > > and > > > > > > > > > > > > set of user attributes used to calculate affinity > > > function. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > View this message in context: http://apache-ignite- > > > > > > > > > > > > developers.2346864.n4.nabble. > > > com/Cluster-auto-activation- > > > > > > > > > > > > design-proposal-tp20295p21066.html > > > > > > > > > > > > Sent from the Apache Ignite Developers mailing list > > > archive > > > > > at > > > > > > > > > > > Nabble.com. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Cluster auto activation design proposal
t; > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > > > > > As I understand you use the term "minimalActivationNodes" > as > > a > > > > > > synonym > > > > > > > > for > > > > > > > > > BaselineTopology concept. > > > > > > > > > In that case I agree with you that we can replace both > > > > "establish*" > > > > > > > > methods > > > > > > > > > with a simple setter method (see below in summary). > > > > > > > > > > > > > > > > > > Summing up the whole discussion I see the functionality as > > > > > following: > > > > > > > > > > > > > > > > > > New concept BaselineTopology is introduced. The main > features > > > it > > > > > > > enables > > > > > > > > > are: > > > > > > > > > > > > > > > > > >1. automatic activation of cluster; > > > > > > > > > > > > > > > > > >2. easy management of cluster topology changes (planned > > > nodes > > > > > > > > >maintenance, adding new nodes etc); > > > > > > > > > > > > > > > > > >3. eliminating of rebalancing traffic on short-term node > > > > > failures. > > > > > > > > > > > > > > > > > > > > > > > > > > > Use cases to create BLT: > > > > > > > > > > > > > > > > > >1. user starts up new cluster of desired number of nodes > > and > > > > > > > activates > > > > > > > > >it using existing API. BLT is created with all nodes > > > presented > > > > > in > > > > > > > the > > > > > > > > >cluster at the moment of activation, no API is needed; > > > > > > > > > > > > > > > > > >2. user prepares BLT using web-console or visor CMD > tools > > > and > > > > > sets > > > > > > > it > > > > > > > > to > > > > > > > > >the cluster. New API setter is needed: > > > > > > > > >Ignite.activation().setBaselin > eTopology(BaselineTopology > > > > blt); > > > > > > > > > > > > > > > > > >3. user provides via static configuration a list of > nodes > > > that > > > > > are > > > > > > > > >expected to be in the cluster. > > > > > > > > >User starts nodes one by one; when all preconfigured > nodes > > > are > > > > > > > started > > > > > > > > >cluster is activated and BLT is created. > > > > > > > > >As list of nodes may be huge it is provided via separate > > > file > > > > to > > > > > > > avoid > > > > > > > > >flooding main configuration. > > > > > > > > > > > > > > > > > > > > > > > > > > > Igniters, does this description match with your > understanding > > > of > > > > > > > > > functionality? If it does I'll create a set of tickets and > > > start > > > > > > > working > > > > > > > > on > > > > > > > > > implementation. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan < > > > > > > > > dsetrak...@apache.org> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I still do not see why anyone would explicitly call > these 2 > > > > > > methods: > > > > > > > > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > > > > > > *Ignite::activation::establishBaselineTopology( > > > > BaselineTopology > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > For example, if a web console, or some other admin > process, > > > > want > > > > > to > > > > > > > > > > automatically set currently started nodes as the baseline > > > > > topology, > > > > > > > > > > shouldn't they just call a setter for > > minimalActivationNodes? > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev < > > > > > > > > > admitr...@gridgain.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > API is proposed in the head of the thread by Sergey, > as I > > > > > > > understood: > > > > > > > > > > > __ > > > > > > > > > > > > > > > > > > > > > > API for BaselineTopology manipulation may look like > this: > > > > > > > > > > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > > > > > > > *Ignite::activation::establishBaselineTopology( > > > > BaselineTopology > > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > > > Both methods will establish BT and activate cluster > once > > it > > > > is > > > > > > > > > > established. > > > > > > > > > > > > > > > > > > > > > > The first one allows user to establish BT using current > > > > > topology. > > > > > > > If > > > > > > > > > any > > > > > > > > > > > changes happen to the topology during establishing > > process, > > > > > user > > > > > > > will > > > > > > > > > be > > > > > > > > > > > notified and allowed to proceed or abort the procedure. > > > > > > > > > > > > > > > > > > > > > > Second method allows to use some > monitoring'n'management > > > > tools > > > > > > like > > > > > > > > > > > WebConsole where user can prepare a list of nodes, > using > > > them > > > > > > > create > > > > > > > > a > > > > > > > > > BT > > > > > > > > > > > and send to the cluster a command to finally establish > > it. > > > > > > > > > > > > > > > > > > > > > > From high level BaselineTopology entity contains only > > > > > collection > > > > > > of > > > > > > > > > > nodes: > > > > > > > > > > > > > > > > > > > > > > *BaselineTopology {* > > > > > > > > > > > * Collection nodes;* > > > > > > > > > > > *}* > > > > > > > > > > > > > > > > > > > > > > *TopologyNode* here contains information about node - > its > > > > > > > consistent > > > > > > > > id > > > > > > > > > > and > > > > > > > > > > > set of user attributes used to calculate affinity > > function. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > View this message in context: http://apache-ignite- > > > > > > > > > > > developers.2346864.n4.nabble. > > com/Cluster-auto-activation- > > > > > > > > > > > design-proposal-tp20295p21066.html > > > > > > > > > > > Sent from the Apache Ignite Developers mailing list > > archive > > > > at > > > > > > > > > > Nabble.com. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Cluster auto activation design proposal
> > > > > > > > > > > > > >1. user starts up new cluster of desired number of nodes > and > > > > > > activates > > > > > > > >it using existing API. BLT is created with all nodes > > presented > > > > in > > > > > > the > > > > > > > >cluster at the moment of activation, no API is needed; > > > > > > > > > > > > > > > >2. user prepares BLT using web-console or visor CMD tools > > and > > > > sets > > > > > > it > > > > > > > to > > > > > > > >the cluster. New API setter is needed: > > > > > > > >Ignite.activation().setBaselineTopology(BaselineTopology > > > blt); > > > > > > > > > > > > > > > >3. user provides via static configuration a list of nodes > > that > > > > are > > > > > > > >expected to be in the cluster. > > > > > > > >User starts nodes one by one; when all preconfigured nodes > > are > > > > > > started > > > > > > > >cluster is activated and BLT is created. > > > > > > > >As list of nodes may be huge it is provided via separate > > file > > > to > > > > > > avoid > > > > > > > >flooding main configuration. > > > > > > > > > > > > > > > > > > > > > > > > Igniters, does this description match with your understanding > > of > > > > > > > > functionality? If it does I'll create a set of tickets and > > start > > > > > > working > > > > > > > on > > > > > > > > implementation. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan < > > > > > > > dsetrak...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I still do not see why anyone would explicitly call these 2 > > > > > methods: > > > > > > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > > > > > *Ignite::activation::establishBaselineTopology( > > > BaselineTopology > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > For example, if a web console, or some other admin process, > > > want > > > > to > > > > > > > > > automatically set currently started nodes as the baseline > > > > topology, > > > > > > > > > shouldn't they just call a setter for > minimalActivationNodes? > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev < > > > > > > > > admitr...@gridgain.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > API is proposed in the head of the thread by Sergey, as I > > > > > > understood: > > > > > > > > > > __ > > > > > > > > > > > > > > > > > > > > API for BaselineTopology manipulation may look like this: > > > > > > > > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > > > > > > *Ignite::activation::establishBaselineTopology( > > > BaselineTopology > > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > > > Both methods will establish BT and activate cluster once > it > > > is > > > > > > > > > established. > > > > > > > > > > > > > > > > > > > > The first one allows user to establish BT using current > > > > topology. > > > > > > If > > > > > > > > any > > > > > > > > > > changes happen to the topology during establishing > process, > > > > user > > > > > > will > > > > > > > > be > > > > > > > > > > notified and allowed to proceed or abort the procedure. > > > > > > > > > > > > > > > > > > > > Second method allows to use some monitoring'n'management > > > tools > > > > > like > > > > > > > > > > WebConsole where user can prepare a list of nodes, using > > them > > > > > > create > > > > > > > a > > > > > > > > BT > > > > > > > > > > and send to the cluster a command to finally establish > it. > > > > > > > > > > > > > > > > > > > > From high level BaselineTopology entity contains only > > > > collection > > > > > of > > > > > > > > > nodes: > > > > > > > > > > > > > > > > > > > > *BaselineTopology {* > > > > > > > > > > * Collection nodes;* > > > > > > > > > > *}* > > > > > > > > > > > > > > > > > > > > *TopologyNode* here contains information about node - its > > > > > > consistent > > > > > > > id > > > > > > > > > and > > > > > > > > > > set of user attributes used to calculate affinity > function. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > View this message in context: http://apache-ignite- > > > > > > > > > > developers.2346864.n4.nabble. > com/Cluster-auto-activation- > > > > > > > > > > design-proposal-tp20295p21066.html > > > > > > > > > > Sent from the Apache Ignite Developers mailing list > archive > > > at > > > > > > > > > Nabble.com. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Cluster auto activation design proposal
gt;flooding main configuration. > > > > > > > > > > > > > > > > > > > > > Igniters, does this description match with your understanding > of > > > > > > > functionality? If it does I'll create a set of tickets and > start > > > > > working > > > > > > on > > > > > > > implementation. > > > > > > > > > > > > > > Thanks, > > > > > > > Sergey. > > > > > > > > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan < > > > > > > dsetrak...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > I still do not see why anyone would explicitly call these 2 > > > > methods: > > > > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > > > > *Ignite::activation::establishBaselineTopology( > > BaselineTopology > > > > > > > bltTop);* > > > > > > > > > > > > > > > > For example, if a web console, or some other admin process, > > want > > > to > > > > > > > > automatically set currently started nodes as the baseline > > > topology, > > > > > > > > shouldn't they just call a setter for minimalActivationNodes? > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev < > > > > > > > admitr...@gridgain.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > API is proposed in the head of the thread by Sergey, as I > > > > > understood: > > > > > > > > > __ > > > > > > > > > > > > > > > > > > API for BaselineTopology manipulation may look like this: > > > > > > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > > > > > *Ignite::activation::establishBaselineTopology( > > BaselineTopology > > > > > > > > bltTop);* > > > > > > > > > > > > > > > > > > Both methods will establish BT and activate cluster once it > > is > > > > > > > > established. > > > > > > > > > > > > > > > > > > The first one allows user to establish BT using current > > > topology. > > > > > If > > > > > > > any > > > > > > > > > changes happen to the topology during establishing process, > > > user > > > > > will > > > > > > > be > > > > > > > > > notified and allowed to proceed or abort the procedure. > > > > > > > > > > > > > > > > > > Second method allows to use some monitoring'n'management > > tools > > > > like > > > > > > > > > WebConsole where user can prepare a list of nodes, using > them > > > > > create > > > > > > a > > > > > > > BT > > > > > > > > > and send to the cluster a command to finally establish it. > > > > > > > > > > > > > > > > > > From high level BaselineTopology entity contains only > > > collection > > > > of > > > > > > > > nodes: > > > > > > > > > > > > > > > > > > *BaselineTopology {* > > > > > > > > > * Collection nodes;* > > > > > > > > > *}* > > > > > > > > > > > > > > > > > > *TopologyNode* here contains information about node - its > > > > > consistent > > > > > > id > > > > > > > > and > > > > > > > > > set of user attributes used to calculate affinity function. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > View this message in context: http://apache-ignite- > > > > > > > > > developers.2346864.n4.nabble.com/Cluster-auto-activation- > > > > > > > > > design-proposal-tp20295p21066.html > > > > > > > > > Sent from the Apache Ignite Developers mailing list archive > > at > > > > > > > > Nabble.com. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Cluster auto activation design proposal
e should fail nodes with an error message. > > > > > > > > Thanks, > > > > Sergey. > > > > > > > > > > > > > > > > > > > > On Wed, Aug 23, 2017 at 5:47 AM, Dmitriy Setrakyan < > > > dsetrak...@apache.org> > > > > wrote: > > > > > > > > > Sergey, I am still confused. What is the BaselineTopology interface > > in > > > > your > > > > > example? I thought that you agreed with me that we simply need a > > setter > > > > for > > > > > activation nodes, no? > > > > > > > > > > D. > > > > > > > > > > On Tue, Aug 22, 2017 at 4:47 AM, Sergey Chugunov < > > > > > sergey.chugu...@gmail.com> > > > > > wrote: > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > As I understand you use the term "minimalActivationNodes" as a > > > synonym > > > > > for > > > > > > BaselineTopology concept. > > > > > > In that case I agree with you that we can replace both > "establish*" > > > > > methods > > > > > > with a simple setter method (see below in summary). > > > > > > > > > > > > Summing up the whole discussion I see the functionality as > > following: > > > > > > > > > > > > New concept BaselineTopology is introduced. The main features it > > > > enables > > > > > > are: > > > > > > > > > > > >1. automatic activation of cluster; > > > > > > > > > > > >2. easy management of cluster topology changes (planned nodes > > > > > >maintenance, adding new nodes etc); > > > > > > > > > > > >3. eliminating of rebalancing traffic on short-term node > > failures. > > > > > > > > > > > > > > > > > > Use cases to create BLT: > > > > > > > > > > > >1. user starts up new cluster of desired number of nodes and > > > > activates > > > > > >it using existing API. BLT is created with all nodes presented > > in > > > > the > > > > > >cluster at the moment of activation, no API is needed; > > > > > > > > > > > >2. user prepares BLT using web-console or visor CMD tools and > > sets > > > > it > > > > > to > > > > > >the cluster. New API setter is needed: > > > > > >Ignite.activation().setBaselineTopology(BaselineTopology > blt); > > > > > > > > > > > >3. user provides via static configuration a list of nodes that > > are > > > > > >expected to be in the cluster. > > > > > >User starts nodes one by one; when all preconfigured nodes are > > > > started > > > > > >cluster is activated and BLT is created. > > > > > >As list of nodes may be huge it is provided via separate file > to > > > > avoid > > > > > >flooding main configuration. > > > > > > > > > > > > > > > > > > Igniters, does this description match with your understanding of > > > > > > functionality? If it does I'll create a set of tickets and start > > > > working > > > > > on > > > > > > implementation. > > > > > > > > > > > > Thanks, > > > > > > Sergey. > > > > > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan < > > > > > dsetrak...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > I still do not see why anyone would explicitly call these 2 > > > methods: > > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > > > *Ignite::activation::establishBaselineTopology( > BaselineTopology > > > > > > bltTop);* > > > > > > > > > > > > > > For example, if a web console, or some other admin process, > want > > to > > > > > > > automatically set currently started nodes as the baseline > > topology, > > > > > > > shouldn't they just call a setter for minimalActivationNodes? > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev < > > > > > > admitr...@gridgain.com> > > > > > > > wrote: > > > > > > > > > > > > > > > API is proposed in the head of the thread by Sergey, as I > > > > understood: > > > > > > > > __ > > > > > > > > > > > > > > > > API for BaselineTopology manipulation may look like this: > > > > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > > > > *Ignite::activation::establishBaselineTopology( > BaselineTopology > > > > > > > bltTop);* > > > > > > > > > > > > > > > > Both methods will establish BT and activate cluster once it > is > > > > > > > established. > > > > > > > > > > > > > > > > The first one allows user to establish BT using current > > topology. > > > > If > > > > > > any > > > > > > > > changes happen to the topology during establishing process, > > user > > > > will > > > > > > be > > > > > > > > notified and allowed to proceed or abort the procedure. > > > > > > > > > > > > > > > > Second method allows to use some monitoring'n'management > tools > > > like > > > > > > > > WebConsole where user can prepare a list of nodes, using them > > > > create > > > > > a > > > > > > BT > > > > > > > > and send to the cluster a command to finally establish it. > > > > > > > > > > > > > > > > From high level BaselineTopology entity contains only > > collection > > > of > > > > > > > nodes: > > > > > > > > > > > > > > > > *BaselineTopology {* > > > > > > > > * Collection nodes;* > > > > > > > > *}* > > > > > > > > > > > > > > > > *TopologyNode* here contains information about node - its > > > > consistent > > > > > id > > > > > > > and > > > > > > > > set of user attributes used to calculate affinity function. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > View this message in context: http://apache-ignite- > > > > > > > > developers.2346864.n4.nabble.com/Cluster-auto-activation- > > > > > > > > design-proposal-tp20295p21066.html > > > > > > > > Sent from the Apache Ignite Developers mailing list archive > at > > > > > > > Nabble.com. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Cluster auto activation design proposal
logy changes (planned nodes > > > > >maintenance, adding new nodes etc); > > > > > > > > > >3. eliminating of rebalancing traffic on short-term node > failures. > > > > > > > > > > > > > > > Use cases to create BLT: > > > > > > > > > >1. user starts up new cluster of desired number of nodes and > > > activates > > > > >it using existing API. BLT is created with all nodes presented > in > > > the > > > > >cluster at the moment of activation, no API is needed; > > > > > > > > > >2. user prepares BLT using web-console or visor CMD tools and > sets > > > it > > > > to > > > > >the cluster. New API setter is needed: > > > > >Ignite.activation().setBaselineTopology(BaselineTopology blt); > > > > > > > > > >3. user provides via static configuration a list of nodes that > are > > > > >expected to be in the cluster. > > > > >User starts nodes one by one; when all preconfigured nodes are > > > started > > > > >cluster is activated and BLT is created. > > > > >As list of nodes may be huge it is provided via separate file to > > > avoid > > > > >flooding main configuration. > > > > > > > > > > > > > > > Igniters, does this description match with your understanding of > > > > > functionality? If it does I'll create a set of tickets and start > > > working > > > > on > > > > > implementation. > > > > > > > > > > Thanks, > > > > > Sergey. > > > > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan < > > > > dsetrak...@apache.org> > > > > > wrote: > > > > > > > > > > > I still do not see why anyone would explicitly call these 2 > > methods: > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > > *Ignite::activation::establishBaselineTopology(BaselineTopology > > > > > bltTop);* > > > > > > > > > > > > For example, if a web console, or some other admin process, want > to > > > > > > automatically set currently started nodes as the baseline > topology, > > > > > > shouldn't they just call a setter for minimalActivationNodes? > > > > > > > > > > > > D. > > > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev < > > > > > admitr...@gridgain.com> > > > > > > wrote: > > > > > > > > > > > > > API is proposed in the head of the thread by Sergey, as I > > > understood: > > > > > > > __ > > > > > > > > > > > > > > API for BaselineTopology manipulation may look like this: > > > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > > > *Ignite::activation::establishBaselineTopology(BaselineTopology > > > > > > bltTop);* > > > > > > > > > > > > > > Both methods will establish BT and activate cluster once it is > > > > > > established. > > > > > > > > > > > > > > The first one allows user to establish BT using current > topology. > > > If > > > > > any > > > > > > > changes happen to the topology during establishing process, > user > > > will > > > > > be > > > > > > > notified and allowed to proceed or abort the procedure. > > > > > > > > > > > > > > Second method allows to use some monitoring'n'management tools > > like > > > > > > > WebConsole where user can prepare a list of nodes, using them > > > create > > > > a > > > > > BT > > > > > > > and send to the cluster a command to finally establish it. > > > > > > > > > > > > > > From high level BaselineTopology entity contains only > collection > > of > > > > > > nodes: > > > > > > > > > > > > > > *BaselineTopology {* > > > > > > > * Collection nodes;* > > > > > > > *}* > > > > > > > > > > > > > > *TopologyNode* here contains information about node - its > > > consistent > > > > id > > > > > > and > > > > > > > set of user attributes used to calculate affinity function. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > View this message in context: http://apache-ignite- > > > > > > > developers.2346864.n4.nabble.com/Cluster-auto-activation- > > > > > > > design-proposal-tp20295p21066.html > > > > > > > Sent from the Apache Ignite Developers mailing list archive at > > > > > > Nabble.com. > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Cluster auto activation design proposal
ist of nodes may be huge it is provided via separate file to > > avoid > > > >flooding main configuration. > > > > > > > > > > > > Igniters, does this description match with your understanding of > > > > functionality? If it does I'll create a set of tickets and start > > working > > > on > > > > implementation. > > > > > > > > Thanks, > > > > Sergey. > > > > > > > > > > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan < > > > dsetrak...@apache.org> > > > > wrote: > > > > > > > > > I still do not see why anyone would explicitly call these 2 > methods: > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > *Ignite::activation::establishBaselineTopology(BaselineTopology > > > > bltTop);* > > > > > > > > > > For example, if a web console, or some other admin process, want to > > > > > automatically set currently started nodes as the baseline topology, > > > > > shouldn't they just call a setter for minimalActivationNodes? > > > > > > > > > > D. > > > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev < > > > > admitr...@gridgain.com> > > > > > wrote: > > > > > > > > > > > API is proposed in the head of the thread by Sergey, as I > > understood: > > > > > > __ > > > > > > > > > > > > API for BaselineTopology manipulation may look like this: > > > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > > *Ignite::activation::establishBaselineTopology(BaselineTopology > > > > > bltTop);* > > > > > > > > > > > > Both methods will establish BT and activate cluster once it is > > > > > established. > > > > > > > > > > > > The first one allows user to establish BT using current topology. > > If > > > > any > > > > > > changes happen to the topology during establishing process, user > > will > > > > be > > > > > > notified and allowed to proceed or abort the procedure. > > > > > > > > > > > > Second method allows to use some monitoring'n'management tools > like > > > > > > WebConsole where user can prepare a list of nodes, using them > > create > > > a > > > > BT > > > > > > and send to the cluster a command to finally establish it. > > > > > > > > > > > > From high level BaselineTopology entity contains only collection > of > > > > > nodes: > > > > > > > > > > > > *BaselineTopology {* > > > > > > * Collection nodes;* > > > > > > *}* > > > > > > > > > > > > *TopologyNode* here contains information about node - its > > consistent > > > id > > > > > and > > > > > > set of user attributes used to calculate affinity function. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > View this message in context: http://apache-ignite- > > > > > > developers.2346864.n4.nabble.com/Cluster-auto-activation- > > > > > > design-proposal-tp20295p21066.html > > > > > > Sent from the Apache Ignite Developers mailing list archive at > > > > > Nabble.com. > > > > > > > > > > > > > > > > > > > > >
Re: Cluster auto activation design proposal
shouldn't they just call a setter for minimalActivationNodes? > > > > > > > > D. > > > > > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev < > > > admitr...@gridgain.com> > > > > wrote: > > > > > > > > > API is proposed in the head of the thread by Sergey, as I > understood: > > > > > __ > > > > > > > > > > API for BaselineTopology manipulation may look like this: > > > > > > > > > > *Ignite::activation::establishBaselineTopology();* > > > > > *Ignite::activation::establishBaselineTopology(BaselineTopology > > > > bltTop);* > > > > > > > > > > Both methods will establish BT and activate cluster once it is > > > > established. > > > > > > > > > > The first one allows user to establish BT using current topology. > If > > > any > > > > > changes happen to the topology during establishing process, user > will > > > be > > > > > notified and allowed to proceed or abort the procedure. > > > > > > > > > > Second method allows to use some monitoring'n'management tools like > > > > > WebConsole where user can prepare a list of nodes, using them > create > > a > > > BT > > > > > and send to the cluster a command to finally establish it. > > > > > > > > > > From high level BaselineTopology entity contains only collection of > > > > nodes: > > > > > > > > > > *BaselineTopology {* > > > > > * Collection nodes;* > > > > > *}* > > > > > > > > > > *TopologyNode* here contains information about node - its > consistent > > id > > > > and > > > > > set of user attributes used to calculate affinity function. > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > View this message in context: http://apache-ignite- > > > > > developers.2346864.n4.nabble.com/Cluster-auto-activation- > > > > > design-proposal-tp20295p21066.html > > > > > Sent from the Apache Ignite Developers mailing list archive at > > > > Nabble.com. > > > > > > > > > > > > > > >
Re: Cluster auto activation design proposal
; > established. > > > > > > > > The first one allows user to establish BT using current topology. If > > any > > > > changes happen to the topology during establishing process, user will > > be > > > > notified and allowed to proceed or abort the procedure. > > > > > > > > Second method allows to use some monitoring'n'management tools like > > > > WebConsole where user can prepare a list of nodes, using them create > a > > BT > > > > and send to the cluster a command to finally establish it. > > > > > > > > From high level BaselineTopology entity contains only collection of > > > nodes: > > > > > > > > *BaselineTopology {* > > > > * Collection nodes;* > > > > *}* > > > > > > > > *TopologyNode* here contains information about node - its consistent > id > > > and > > > > set of user attributes used to calculate affinity function. > > > > > > > > > > > > > > > > > > > > -- > > > > View this message in context: http://apache-ignite- > > > > developers.2346864.n4.nabble.com/Cluster-auto-activation- > > > > design-proposal-tp20295p21066.html > > > > Sent from the Apache Ignite Developers mailing list archive at > > > Nabble.com. > > > > > > > > > >
Re: Cluster auto activation design proposal
Sergey, I am still confused. What is the BaselineTopology interface in your example? I thought that you agreed with me that we simply need a setter for activation nodes, no? D. On Tue, Aug 22, 2017 at 4:47 AM, Sergey Chugunov <sergey.chugu...@gmail.com> wrote: > Dmitriy, > > As I understand you use the term "minimalActivationNodes" as a synonym for > BaselineTopology concept. > In that case I agree with you that we can replace both "establish*" methods > with a simple setter method (see below in summary). > > Summing up the whole discussion I see the functionality as following: > > New concept BaselineTopology is introduced. The main features it enables > are: > >1. automatic activation of cluster; > >2. easy management of cluster topology changes (planned nodes >maintenance, adding new nodes etc); > >3. eliminating of rebalancing traffic on short-term node failures. > > > Use cases to create BLT: > >1. user starts up new cluster of desired number of nodes and activates >it using existing API. BLT is created with all nodes presented in the >cluster at the moment of activation, no API is needed; > >2. user prepares BLT using web-console or visor CMD tools and sets it to >the cluster. New API setter is needed: >Ignite.activation().setBaselineTopology(BaselineTopology blt); > >3. user provides via static configuration a list of nodes that are >expected to be in the cluster. >User starts nodes one by one; when all preconfigured nodes are started >cluster is activated and BLT is created. >As list of nodes may be huge it is provided via separate file to avoid >flooding main configuration. > > > Igniters, does this description match with your understanding of > functionality? If it does I'll create a set of tickets and start working on > implementation. > > Thanks, > Sergey. > > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan <dsetrak...@apache.org> > wrote: > > > I still do not see why anyone would explicitly call these 2 methods: > > > > *Ignite::activation::establishBaselineTopology();* > > *Ignite::activation::establishBaselineTopology(BaselineTopology > bltTop);* > > > > For example, if a web console, or some other admin process, want to > > automatically set currently started nodes as the baseline topology, > > shouldn't they just call a setter for minimalActivationNodes? > > > > D. > > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev < > admitr...@gridgain.com> > > wrote: > > > > > API is proposed in the head of the thread by Sergey, as I understood: > > > __ > > > > > > API for BaselineTopology manipulation may look like this: > > > > > > *Ignite::activation::establishBaselineTopology();* > > > *Ignite::activation::establishBaselineTopology(BaselineTopology > > bltTop);* > > > > > > Both methods will establish BT and activate cluster once it is > > established. > > > > > > The first one allows user to establish BT using current topology. If > any > > > changes happen to the topology during establishing process, user will > be > > > notified and allowed to proceed or abort the procedure. > > > > > > Second method allows to use some monitoring'n'management tools like > > > WebConsole where user can prepare a list of nodes, using them create a > BT > > > and send to the cluster a command to finally establish it. > > > > > > From high level BaselineTopology entity contains only collection of > > nodes: > > > > > > *BaselineTopology {* > > > * Collection nodes;* > > > *}* > > > > > > *TopologyNode* here contains information about node - its consistent id > > and > > > set of user attributes used to calculate affinity function. > > > > > > > > > > > > > > > -- > > > View this message in context: http://apache-ignite- > > > developers.2346864.n4.nabble.com/Cluster-auto-activation- > > > design-proposal-tp20295p21066.html > > > Sent from the Apache Ignite Developers mailing list archive at > > Nabble.com. > > > > > >
Re: Cluster auto activation design proposal
Dmitriy, As I understand you use the term "minimalActivationNodes" as a synonym for BaselineTopology concept. In that case I agree with you that we can replace both "establish*" methods with a simple setter method (see below in summary). Summing up the whole discussion I see the functionality as following: New concept BaselineTopology is introduced. The main features it enables are: 1. automatic activation of cluster; 2. easy management of cluster topology changes (planned nodes maintenance, adding new nodes etc); 3. eliminating of rebalancing traffic on short-term node failures. Use cases to create BLT: 1. user starts up new cluster of desired number of nodes and activates it using existing API. BLT is created with all nodes presented in the cluster at the moment of activation, no API is needed; 2. user prepares BLT using web-console or visor CMD tools and sets it to the cluster. New API setter is needed: Ignite.activation().setBaselineTopology(BaselineTopology blt); 3. user provides via static configuration a list of nodes that are expected to be in the cluster. User starts nodes one by one; when all preconfigured nodes are started cluster is activated and BLT is created. As list of nodes may be huge it is provided via separate file to avoid flooding main configuration. Igniters, does this description match with your understanding of functionality? If it does I'll create a set of tickets and start working on implementation. Thanks, Sergey. On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan <dsetrak...@apache.org> wrote: > I still do not see why anyone would explicitly call these 2 methods: > > *Ignite::activation::establishBaselineTopology();* > *Ignite::activation::establishBaselineTopology(BaselineTopology bltTop);* > > For example, if a web console, or some other admin process, want to > automatically set currently started nodes as the baseline topology, > shouldn't they just call a setter for minimalActivationNodes? > > D. > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <admitr...@gridgain.com> > wrote: > > > API is proposed in the head of the thread by Sergey, as I understood: > > __ > > > > API for BaselineTopology manipulation may look like this: > > > > *Ignite::activation::establishBaselineTopology();* > > *Ignite::activation::establishBaselineTopology(BaselineTopology > bltTop);* > > > > Both methods will establish BT and activate cluster once it is > established. > > > > The first one allows user to establish BT using current topology. If any > > changes happen to the topology during establishing process, user will be > > notified and allowed to proceed or abort the procedure. > > > > Second method allows to use some monitoring'n'management tools like > > WebConsole where user can prepare a list of nodes, using them create a BT > > and send to the cluster a command to finally establish it. > > > > From high level BaselineTopology entity contains only collection of > nodes: > > > > *BaselineTopology {* > > * Collection nodes;* > > *}* > > > > *TopologyNode* here contains information about node - its consistent id > and > > set of user attributes used to calculate affinity function. > > > > > > > > > > -- > > View this message in context: http://apache-ignite- > > developers.2346864.n4.nabble.com/Cluster-auto-activation- > > design-proposal-tp20295p21066.html > > Sent from the Apache Ignite Developers mailing list archive at > Nabble.com. > > >
Re: Cluster auto activation design proposal
I still do not see why anyone would explicitly call these 2 methods: *Ignite::activation::establishBaselineTopology();* *Ignite::activation::establishBaselineTopology(BaselineTopology bltTop);* For example, if a web console, or some other admin process, want to automatically set currently started nodes as the baseline topology, shouldn't they just call a setter for minimalActivationNodes? D. On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <admitr...@gridgain.com> wrote: > API is proposed in the head of the thread by Sergey, as I understood: > __ > > API for BaselineTopology manipulation may look like this: > > *Ignite::activation::establishBaselineTopology();* > *Ignite::activation::establishBaselineTopology(BaselineTopology bltTop);* > > Both methods will establish BT and activate cluster once it is established. > > The first one allows user to establish BT using current topology. If any > changes happen to the topology during establishing process, user will be > notified and allowed to proceed or abort the procedure. > > Second method allows to use some monitoring'n'management tools like > WebConsole where user can prepare a list of nodes, using them create a BT > and send to the cluster a command to finally establish it. > > From high level BaselineTopology entity contains only collection of nodes: > > *BaselineTopology {* > * Collection nodes;* > *}* > > *TopologyNode* here contains information about node - its consistent id and > set of user attributes used to calculate affinity function. > > > > > -- > View this message in context: http://apache-ignite- > developers.2346864.n4.nabble.com/Cluster-auto-activation- > design-proposal-tp20295p21066.html > Sent from the Apache Ignite Developers mailing list archive at Nabble.com. >
Re: Cluster auto activation design proposal
API is proposed in the head of the thread by Sergey, as I understood: __ API for BaselineTopology manipulation may look like this: *Ignite::activation::establishBaselineTopology();* *Ignite::activation::establishBaselineTopology(BaselineTopology bltTop);* Both methods will establish BT and activate cluster once it is established. The first one allows user to establish BT using current topology. If any changes happen to the topology during establishing process, user will be notified and allowed to proceed or abort the procedure. Second method allows to use some monitoring'n'management tools like WebConsole where user can prepare a list of nodes, using them create a BT and send to the cluster a command to finally establish it. >From high level BaselineTopology entity contains only collection of nodes: *BaselineTopology {* * Collection nodes;* *}* *TopologyNode* here contains information about node - its consistent id and set of user attributes used to calculate affinity function. -- View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/Cluster-auto-activation-design-proposal-tp20295p21066.html Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
Re: Cluster auto activation design proposal
On Thu, Aug 17, 2017 at 7:34 AM, Sergey Chugunovwrote: > Dmitriy, > > As automatic creation of Baseline Topology is just another use case about > the whole concept, lets discuss it in this thread and forget about parallel > one. > > *initialActivationNodes* for desired configuration of Baseline Topology > sounds good to me, I would like to hear from other Igniters. > I don't think this sounds right. It should be either "inititialActiveNodes" or "initialNodes". I am also open to other suggestions. > > However other options to create Baseline Topology were proposed: > >- Prepare it manually with WebConsole or visor-console (no >initialActivationNodes configuration is involved). >- Create it automatically on the first manual activation (again no >initial configuration is involved). > > Thus we still need Baseline Topology entity not only in documentation. > In either case, wouldn't it involve creation of initialActiveNodes implicitly? > > Also please take into account that at some point in time user may want to > recreate BLT (e.g. to add nodes to existing cluster). User will have to > interact with some entity; so we need it not only internally but on public > API. > What would the public API look like and who would user interact with it?
Re: Cluster auto activation design proposal
Dmitriy, As automatic creation of Baseline Topology is just another use case about the whole concept, lets discuss it in this thread and forget about parallel one. *initialActivationNodes* for desired configuration of Baseline Topology sounds good to me, I would like to hear from other Igniters. However other options to create Baseline Topology were proposed: - Prepare it manually with WebConsole or visor-console (no initialActivationNodes configuration is involved). - Create it automatically on the first manual activation (again no initial configuration is involved). Thus we still need Baseline Topology entity not only in documentation. Also please take into account that at some point in time user may want to recreate BLT (e.g. to add nodes to existing cluster). User will have to interact with some entity; so we need it not only internally but on public API. Thanks, Sergey. On Sat, Aug 12, 2017 at 4:07 AM, Dmitriy Setrakyanwrote: > Sergey, > > As it is becoming clear from another thread, the only configuration users > will need to provide is the initial set of nodes. Everything else will be > handled by Ignite automatically. > > In this case, the name Baseline Topology will appear only in documentation, > in which case the name is OK (that is if I am understanding the design > correctly). > > However, the list initial set of nodes can be called > initialActivationNodes. > > Makes sense? > > D. > > On Fri, Aug 11, 2017 at 2:46 AM, Sergey Chugunov < > sergey.chugu...@gmail.com> > wrote: > > > More options to name the concept (one may put "node set" or "topology" at > > the end - for me these are interchangeable): > > > > > >1. essential > >2. basic > >3. completed > >4. prepared > >5. solid > >6. enduring > > > > To me the name must stress the fact that "node set" or "topology" > contains > > all nodes user is going to work with for a long run. > > It is what user plans to work with and base the work on. > > > > Any thoughts on this? > > > > Thanks, > > Sergey. > > > > On Thu, Aug 10, 2017 at 5:41 PM, Sergey Chugunov < > > sergey.chugu...@gmail.com> > > wrote: > > > > > Going down "node set" road: > > > > > > -fixed node set > > > -established node set > > > -base node set > > > > > > On Thu, Aug 10, 2017 at 5:23 PM, Dmitriy Setrakyan < > > dsetrak...@apache.org> > > > wrote: > > > > > >> Can we brainstorm on the names again, I am not sure we have a > consensus > > on > > >> the name "baseline topology". This will be included in Ignite > > >> configuration, so the name has to be clear. > > >> > > >> Some of the proposals were: > > >> > > >> - baseline topology > > >> - minimal node set > > >> - node restart set > > >> - minimal topology > > >> > > >> Any other suggestions? > > >> > > >> D. > > >> > > >> On Thu, Aug 10, 2017 at 2:13 AM, Alexey Goncharuk < > > >> alexey.goncha...@gmail.com> wrote: > > >> > > >> > Denis, > > >> > > > >> > This should be handled by the BT triggers. If I have 3 backups > > >> configured, > > >> > I actually won't care if my cluster will live 6 hours without an > > >> additional > > >> > backup. If for a partition there is only one backup left - a new BT > > >> should > > >> > be triggered automatically. > > >> > > > >> > 2017-08-10 0:33 GMT+03:00 Denis Magda : > > >> > > > >> > > Sergey, > > >> > > > > >> > > That’s the only concern I have: > > >> > > > > >> > > * 5. User takes out nodes from cluster (e.g. for maintenance > > >> purposes): > > >> > no > > >> > > rebalance happens until user recreates BLT on new cluster > > topology.* > > >> > > > > >> > > What if a node is crashed (or some other kind of outage) in the > > >> middle of > > >> > > the night and the user has to be sure that survived nodes will > > >> rearrange > > >> > > and rebalancing partitions? > > >> > > > > >> > > — > > >> > > Denis > > >> > > > > >> > > > > >> > > > On Aug 4, 2017, at 9:21 AM, Sergey Chugunov < > > >> sergey.chugu...@gmail.com > > >> > > > > >> > > wrote: > > >> > > > > > >> > > > Folks, > > >> > > > > > >> > > > I've summarized all results from our discussion so far on wiki > > page: > > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/ > > >> > > Automatic+activation+design+-+draft > > >> > > > > > >> > > > I hope I reflected the most important details and going to add > API > > >> > > > suggestions for all use cases soon. > > >> > > > > > >> > > > Feel free to give feedback here or in comments under the page. > > >> > > > > > >> > > > Thanks, > > >> > > > Sergey. > > >> > > > > > >> > > > On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov < > > >> > akuznet...@apache.org> > > >> > > > wrote: > > >> > > > > > >> > > >> Hi, > > >> > > >> > > >> > > 1. User creates new BLT using WebConsole or other tool and > > >> "applies" > > >> > > it > > >> > > >> to brand-new cluster. > > >> > > >> > > >> > > >> Good idea, but we also should implement *command-line utility* > > for > > >> the > >
Re: Cluster auto activation design proposal
Sergey, As it is becoming clear from another thread, the only configuration users will need to provide is the initial set of nodes. Everything else will be handled by Ignite automatically. In this case, the name Baseline Topology will appear only in documentation, in which case the name is OK (that is if I am understanding the design correctly). However, the list initial set of nodes can be called initialActivationNodes. Makes sense? D. On Fri, Aug 11, 2017 at 2:46 AM, Sergey Chugunovwrote: > More options to name the concept (one may put "node set" or "topology" at > the end - for me these are interchangeable): > > >1. essential >2. basic >3. completed >4. prepared >5. solid >6. enduring > > To me the name must stress the fact that "node set" or "topology" contains > all nodes user is going to work with for a long run. > It is what user plans to work with and base the work on. > > Any thoughts on this? > > Thanks, > Sergey. > > On Thu, Aug 10, 2017 at 5:41 PM, Sergey Chugunov < > sergey.chugu...@gmail.com> > wrote: > > > Going down "node set" road: > > > > -fixed node set > > -established node set > > -base node set > > > > On Thu, Aug 10, 2017 at 5:23 PM, Dmitriy Setrakyan < > dsetrak...@apache.org> > > wrote: > > > >> Can we brainstorm on the names again, I am not sure we have a consensus > on > >> the name "baseline topology". This will be included in Ignite > >> configuration, so the name has to be clear. > >> > >> Some of the proposals were: > >> > >> - baseline topology > >> - minimal node set > >> - node restart set > >> - minimal topology > >> > >> Any other suggestions? > >> > >> D. > >> > >> On Thu, Aug 10, 2017 at 2:13 AM, Alexey Goncharuk < > >> alexey.goncha...@gmail.com> wrote: > >> > >> > Denis, > >> > > >> > This should be handled by the BT triggers. If I have 3 backups > >> configured, > >> > I actually won't care if my cluster will live 6 hours without an > >> additional > >> > backup. If for a partition there is only one backup left - a new BT > >> should > >> > be triggered automatically. > >> > > >> > 2017-08-10 0:33 GMT+03:00 Denis Magda : > >> > > >> > > Sergey, > >> > > > >> > > That’s the only concern I have: > >> > > > >> > > * 5. User takes out nodes from cluster (e.g. for maintenance > >> purposes): > >> > no > >> > > rebalance happens until user recreates BLT on new cluster > topology.* > >> > > > >> > > What if a node is crashed (or some other kind of outage) in the > >> middle of > >> > > the night and the user has to be sure that survived nodes will > >> rearrange > >> > > and rebalancing partitions? > >> > > > >> > > — > >> > > Denis > >> > > > >> > > > >> > > > On Aug 4, 2017, at 9:21 AM, Sergey Chugunov < > >> sergey.chugu...@gmail.com > >> > > > >> > > wrote: > >> > > > > >> > > > Folks, > >> > > > > >> > > > I've summarized all results from our discussion so far on wiki > page: > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/ > >> > > Automatic+activation+design+-+draft > >> > > > > >> > > > I hope I reflected the most important details and going to add API > >> > > > suggestions for all use cases soon. > >> > > > > >> > > > Feel free to give feedback here or in comments under the page. > >> > > > > >> > > > Thanks, > >> > > > Sergey. > >> > > > > >> > > > On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov < > >> > akuznet...@apache.org> > >> > > > wrote: > >> > > > > >> > > >> Hi, > >> > > >> > >> > > 1. User creates new BLT using WebConsole or other tool and > >> "applies" > >> > > it > >> > > >> to brand-new cluster. > >> > > >> > >> > > >> Good idea, but we also should implement *command-line utility* > for > >> the > >> > > same > >> > > >> use case. > >> > > >> > >> > > >> -- > >> > > >> Alexey Kuznetsov > >> > > >> > >> > > > >> > > > >> > > >> > > > > >
Re: Cluster auto activation design proposal
More options to name the concept (one may put "node set" or "topology" at the end - for me these are interchangeable): 1. essential 2. basic 3. completed 4. prepared 5. solid 6. enduring To me the name must stress the fact that "node set" or "topology" contains all nodes user is going to work with for a long run. It is what user plans to work with and base the work on. Any thoughts on this? Thanks, Sergey. On Thu, Aug 10, 2017 at 5:41 PM, Sergey Chugunovwrote: > Going down "node set" road: > > -fixed node set > -established node set > -base node set > > On Thu, Aug 10, 2017 at 5:23 PM, Dmitriy Setrakyan > wrote: > >> Can we brainstorm on the names again, I am not sure we have a consensus on >> the name "baseline topology". This will be included in Ignite >> configuration, so the name has to be clear. >> >> Some of the proposals were: >> >> - baseline topology >> - minimal node set >> - node restart set >> - minimal topology >> >> Any other suggestions? >> >> D. >> >> On Thu, Aug 10, 2017 at 2:13 AM, Alexey Goncharuk < >> alexey.goncha...@gmail.com> wrote: >> >> > Denis, >> > >> > This should be handled by the BT triggers. If I have 3 backups >> configured, >> > I actually won't care if my cluster will live 6 hours without an >> additional >> > backup. If for a partition there is only one backup left - a new BT >> should >> > be triggered automatically. >> > >> > 2017-08-10 0:33 GMT+03:00 Denis Magda : >> > >> > > Sergey, >> > > >> > > That’s the only concern I have: >> > > >> > > * 5. User takes out nodes from cluster (e.g. for maintenance >> purposes): >> > no >> > > rebalance happens until user recreates BLT on new cluster topology.* >> > > >> > > What if a node is crashed (or some other kind of outage) in the >> middle of >> > > the night and the user has to be sure that survived nodes will >> rearrange >> > > and rebalancing partitions? >> > > >> > > — >> > > Denis >> > > >> > > >> > > > On Aug 4, 2017, at 9:21 AM, Sergey Chugunov < >> sergey.chugu...@gmail.com >> > > >> > > wrote: >> > > > >> > > > Folks, >> > > > >> > > > I've summarized all results from our discussion so far on wiki page: >> > > > https://cwiki.apache.org/confluence/display/IGNITE/ >> > > Automatic+activation+design+-+draft >> > > > >> > > > I hope I reflected the most important details and going to add API >> > > > suggestions for all use cases soon. >> > > > >> > > > Feel free to give feedback here or in comments under the page. >> > > > >> > > > Thanks, >> > > > Sergey. >> > > > >> > > > On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov < >> > akuznet...@apache.org> >> > > > wrote: >> > > > >> > > >> Hi, >> > > >> >> > > 1. User creates new BLT using WebConsole or other tool and >> "applies" >> > > it >> > > >> to brand-new cluster. >> > > >> >> > > >> Good idea, but we also should implement *command-line utility* for >> the >> > > same >> > > >> use case. >> > > >> >> > > >> -- >> > > >> Alexey Kuznetsov >> > > >> >> > > >> > > >> > >> > >
Re: Cluster auto activation design proposal
Going down "node set" road: -fixed node set -established node set -base node set On Thu, Aug 10, 2017 at 5:23 PM, Dmitriy Setrakyanwrote: > Can we brainstorm on the names again, I am not sure we have a consensus on > the name "baseline topology". This will be included in Ignite > configuration, so the name has to be clear. > > Some of the proposals were: > > - baseline topology > - minimal node set > - node restart set > - minimal topology > > Any other suggestions? > > D. > > On Thu, Aug 10, 2017 at 2:13 AM, Alexey Goncharuk < > alexey.goncha...@gmail.com> wrote: > > > Denis, > > > > This should be handled by the BT triggers. If I have 3 backups > configured, > > I actually won't care if my cluster will live 6 hours without an > additional > > backup. If for a partition there is only one backup left - a new BT > should > > be triggered automatically. > > > > 2017-08-10 0:33 GMT+03:00 Denis Magda : > > > > > Sergey, > > > > > > That’s the only concern I have: > > > > > > * 5. User takes out nodes from cluster (e.g. for maintenance purposes): > > no > > > rebalance happens until user recreates BLT on new cluster topology.* > > > > > > What if a node is crashed (or some other kind of outage) in the middle > of > > > the night and the user has to be sure that survived nodes will > rearrange > > > and rebalancing partitions? > > > > > > — > > > Denis > > > > > > > > > > On Aug 4, 2017, at 9:21 AM, Sergey Chugunov < > sergey.chugu...@gmail.com > > > > > > wrote: > > > > > > > > Folks, > > > > > > > > I've summarized all results from our discussion so far on wiki page: > > > > https://cwiki.apache.org/confluence/display/IGNITE/ > > > Automatic+activation+design+-+draft > > > > > > > > I hope I reflected the most important details and going to add API > > > > suggestions for all use cases soon. > > > > > > > > Feel free to give feedback here or in comments under the page. > > > > > > > > Thanks, > > > > Sergey. > > > > > > > > On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov < > > akuznet...@apache.org> > > > > wrote: > > > > > > > >> Hi, > > > >> > > > 1. User creates new BLT using WebConsole or other tool and > "applies" > > > it > > > >> to brand-new cluster. > > > >> > > > >> Good idea, but we also should implement *command-line utility* for > the > > > same > > > >> use case. > > > >> > > > >> -- > > > >> Alexey Kuznetsov > > > >> > > > > > > > > >
Re: Cluster auto activation design proposal
Can we brainstorm on the names again, I am not sure we have a consensus on the name "baseline topology". This will be included in Ignite configuration, so the name has to be clear. Some of the proposals were: - baseline topology - minimal node set - node restart set - minimal topology Any other suggestions? D. On Thu, Aug 10, 2017 at 2:13 AM, Alexey Goncharuk < alexey.goncha...@gmail.com> wrote: > Denis, > > This should be handled by the BT triggers. If I have 3 backups configured, > I actually won't care if my cluster will live 6 hours without an additional > backup. If for a partition there is only one backup left - a new BT should > be triggered automatically. > > 2017-08-10 0:33 GMT+03:00 Denis Magda: > > > Sergey, > > > > That’s the only concern I have: > > > > * 5. User takes out nodes from cluster (e.g. for maintenance purposes): > no > > rebalance happens until user recreates BLT on new cluster topology.* > > > > What if a node is crashed (or some other kind of outage) in the middle of > > the night and the user has to be sure that survived nodes will rearrange > > and rebalancing partitions? > > > > — > > Denis > > > > > > > On Aug 4, 2017, at 9:21 AM, Sergey Chugunov > > > wrote: > > > > > > Folks, > > > > > > I've summarized all results from our discussion so far on wiki page: > > > https://cwiki.apache.org/confluence/display/IGNITE/ > > Automatic+activation+design+-+draft > > > > > > I hope I reflected the most important details and going to add API > > > suggestions for all use cases soon. > > > > > > Feel free to give feedback here or in comments under the page. > > > > > > Thanks, > > > Sergey. > > > > > > On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov < > akuznet...@apache.org> > > > wrote: > > > > > >> Hi, > > >> > > 1. User creates new BLT using WebConsole or other tool and "applies" > > it > > >> to brand-new cluster. > > >> > > >> Good idea, but we also should implement *command-line utility* for the > > same > > >> use case. > > >> > > >> -- > > >> Alexey Kuznetsov > > >> > > > > >
Re: Cluster auto activation design proposal
Denis, This should be handled by the BT triggers. If I have 3 backups configured, I actually won't care if my cluster will live 6 hours without an additional backup. If for a partition there is only one backup left - a new BT should be triggered automatically. 2017-08-10 0:33 GMT+03:00 Denis Magda: > Sergey, > > That’s the only concern I have: > > * 5. User takes out nodes from cluster (e.g. for maintenance purposes): no > rebalance happens until user recreates BLT on new cluster topology.* > > What if a node is crashed (or some other kind of outage) in the middle of > the night and the user has to be sure that survived nodes will rearrange > and rebalancing partitions? > > — > Denis > > > > On Aug 4, 2017, at 9:21 AM, Sergey Chugunov > wrote: > > > > Folks, > > > > I've summarized all results from our discussion so far on wiki page: > > https://cwiki.apache.org/confluence/display/IGNITE/ > Automatic+activation+design+-+draft > > > > I hope I reflected the most important details and going to add API > > suggestions for all use cases soon. > > > > Feel free to give feedback here or in comments under the page. > > > > Thanks, > > Sergey. > > > > On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov > > wrote: > > > >> Hi, > >> > 1. User creates new BLT using WebConsole or other tool and "applies" > it > >> to brand-new cluster. > >> > >> Good idea, but we also should implement *command-line utility* for the > same > >> use case. > >> > >> -- > >> Alexey Kuznetsov > >> > >
Re: Cluster auto activation design proposal
Sergey, That’s the only concern I have: * 5. User takes out nodes from cluster (e.g. for maintenance purposes): no rebalance happens until user recreates BLT on new cluster topology.* What if a node is crashed (or some other kind of outage) in the middle of the night and the user has to be sure that survived nodes will rearrange and rebalancing partitions? — Denis > On Aug 4, 2017, at 9:21 AM, Sergey Chugunovwrote: > > Folks, > > I've summarized all results from our discussion so far on wiki page: > https://cwiki.apache.org/confluence/display/IGNITE/Automatic+activation+design+-+draft > > I hope I reflected the most important details and going to add API > suggestions for all use cases soon. > > Feel free to give feedback here or in comments under the page. > > Thanks, > Sergey. > > On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov > wrote: > >> Hi, >> 1. User creates new BLT using WebConsole or other tool and "applies" it >> to brand-new cluster. >> >> Good idea, but we also should implement *command-line utility* for the same >> use case. >> >> -- >> Alexey Kuznetsov >>
Re: Cluster auto activation design proposal
Folks, I've summarized all results from our discussion so far on wiki page: https://cwiki.apache.org/confluence/display/IGNITE/Automatic+activation+design+-+draft I hope I reflected the most important details and going to add API suggestions for all use cases soon. Feel free to give feedback here or in comments under the page. Thanks, Sergey. On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsovwrote: > Hi, > > >>1. User creates new BLT using WebConsole or other tool and "applies" it > to brand-new cluster. > > Good idea, but we also should implement *command-line utility* for the same > use case. > > -- > Alexey Kuznetsov >
Re: Cluster auto activation design proposal
I also would like to provide more use cases of how BLT is supposed to work (let me call it this way until we come up with a better one): 1. User creates new BLT using WebConsole or other tool and "applies" it to brand-new cluster. 2. User starts up brand-new cluster with desired amount of nodes and activates it. At the moment of activation BLT is created with all server non-daemon nodes presented in the cluster. 3. User starts up a cluster with previously prepared BLT -> when set of nodes in the cluster matches with BLT cluster gets automatically activated. 4. User has an up-and-running active cluster and starts a few more nodes. They join the cluster but no partitions are assigned to them. User recreates BLT on new cluster topology -> partitions are assigned to new nodes. 5. User takes out nodes from cluster (e.g. for maintenance purposes): no rebalance happens until user recreates BLT on new cluster topology. 6. If some parameters reach critical levels (e.g. number of backups for a partition is too low) coordinator automatically recreates BLT and thus triggers rebalancing. I hope these use cases will help to clarify purposes of the proposed feature. On Thu, Aug 3, 2017 at 4:08 PM, Alexey Goncharukwrote: > My understanding of Baseline Topology is the set of nodes which are > *expected* to be in the cluster. > Let me go a little bit further because BT (or whatever name we choose) may > and will solve more issues than just auto-activation: > > 1) More graceful control over rebalancing than just rebalance delay. If a > server is shut down for maintenance and there are enough backup nodes in > the cluster, there is no need to rebalance. > 2) Guarantee that there will be no conflicting key-value mappings due to > incorrect cluster activation. For example, consider a scenario when there > was a cluster of 10 nodes, then the cluster was shut down, started first 5 > nodes, activated, made some updates, shut down 5 nodes, start up other 5 > nodes, activate, make some updates, start up first 5 nodes. Currently, > there is no way to determine that there was an incompatible topology change > which leads to data inconsistency. > 3) When a cluster is shutting down node-by-node, we must track a node which > has 'seen' a partition last time and not activate the cluster until all > nodes are present. Otherwise, again, we may activate too early and see > outdated values. > > I do not want to add any 'faster' hacks here because they will only make > the issue above appear more likely. Besides, BT should be available in 2.2 > anyway, so no need to rush with hacks. > > --AG > > 2017-08-03 15:09 GMT+03:00 Yakov Zhdanov : > > > >Obvious connotation of "minimal set" is a set that cannot be decreased. > > > > >But lets consider the following case: user has a cluster of 50 nodes and > > >decides to switch off 3 nodes for maintenance for a while. Ok, user just > > >does it and then recreates this "minimal node set" to only 47 nodes. > > > > >So initial minimal node set was decreased - something counter-intuitive > to > > >me and may cause confusion as well. > > > > That was my point. If I have 50 nodes and 3 backups I can restart on 48, > 49 > > and 50 without data loss. In case of 48 and 49 after cluster gets > activated > > missing backups are assigned and rebalancing starts. > > > > --Yakov > > >
Re: Cluster auto activation design proposal
My understanding of Baseline Topology is the set of nodes which are *expected* to be in the cluster. Let me go a little bit further because BT (or whatever name we choose) may and will solve more issues than just auto-activation: 1) More graceful control over rebalancing than just rebalance delay. If a server is shut down for maintenance and there are enough backup nodes in the cluster, there is no need to rebalance. 2) Guarantee that there will be no conflicting key-value mappings due to incorrect cluster activation. For example, consider a scenario when there was a cluster of 10 nodes, then the cluster was shut down, started first 5 nodes, activated, made some updates, shut down 5 nodes, start up other 5 nodes, activate, make some updates, start up first 5 nodes. Currently, there is no way to determine that there was an incompatible topology change which leads to data inconsistency. 3) When a cluster is shutting down node-by-node, we must track a node which has 'seen' a partition last time and not activate the cluster until all nodes are present. Otherwise, again, we may activate too early and see outdated values. I do not want to add any 'faster' hacks here because they will only make the issue above appear more likely. Besides, BT should be available in 2.2 anyway, so no need to rush with hacks. --AG 2017-08-03 15:09 GMT+03:00 Yakov Zhdanov: > >Obvious connotation of "minimal set" is a set that cannot be decreased. > > >But lets consider the following case: user has a cluster of 50 nodes and > >decides to switch off 3 nodes for maintenance for a while. Ok, user just > >does it and then recreates this "minimal node set" to only 47 nodes. > > >So initial minimal node set was decreased - something counter-intuitive to > >me and may cause confusion as well. > > That was my point. If I have 50 nodes and 3 backups I can restart on 48, 49 > and 50 without data loss. In case of 48 and 49 after cluster gets activated > missing backups are assigned and rebalancing starts. > > --Yakov >
Re: Cluster auto activation design proposal
>Obvious connotation of "minimal set" is a set that cannot be decreased. >But lets consider the following case: user has a cluster of 50 nodes and >decides to switch off 3 nodes for maintenance for a while. Ok, user just >does it and then recreates this "minimal node set" to only 47 nodes. >So initial minimal node set was decreased - something counter-intuitive to >me and may cause confusion as well. That was my point. If I have 50 nodes and 3 backups I can restart on 48, 49 and 50 without data loss. In case of 48 and 49 after cluster gets activated missing backups are assigned and rebalancing starts. --Yakov
Re: Cluster auto activation design proposal
>I think it is not just restarts, this set of nodes is minimally required for the cluster to function, no? I don't think so. Cluster can function if there is no data loss. --Yakov
Re: Cluster auto activation design proposal
Igniters, what about Target Node Set? Complete Node Set? As when we reach this topology, we can activate cluster. чт, 3 авг. 2017 г. в 12:58, Sergey Chugunov: > Dmitriy, > > Obvious connotation of "minimal set" is a set that cannot be decreased. > > But lets consider the following case: user has a cluster of 50 nodes and > decides to switch off 3 nodes for maintenance for a while. Ok, user just > does it and then recreates this "minimal node set" to only 47 nodes. > > So initial minimal node set was decreased - something counter-intuitive to > me and may cause confusion as well. > > > On Thu, Aug 3, 2017 at 12:37 PM, wrote: > > > Yakov, > > > > I think it is not just restarts, this set of nodes is minimally required > > for the cluster to function, no? > > > > D. > > > > On Aug 3, 2017, 11:23 AM, at 11:23 AM, Yakov Zhdanov < > yzhda...@apache.org> > > wrote: > > >Ю> How about naming it "minimal node set" or "required node set"? > > > > > >Required for what? I would add restart if there are no confusion. > > > > > >--Yakov > > >
Re: Cluster auto activation design proposal
Dmitriy, Obvious connotation of "minimal set" is a set that cannot be decreased. But lets consider the following case: user has a cluster of 50 nodes and decides to switch off 3 nodes for maintenance for a while. Ok, user just does it and then recreates this "minimal node set" to only 47 nodes. So initial minimal node set was decreased - something counter-intuitive to me and may cause confusion as well. On Thu, Aug 3, 2017 at 12:37 PM,wrote: > Yakov, > > I think it is not just restarts, this set of nodes is minimally required > for the cluster to function, no? > > D. > > On Aug 3, 2017, 11:23 AM, at 11:23 AM, Yakov Zhdanov > wrote: > >Ю> How about naming it "minimal node set" or "required node set"? > > > >Required for what? I would add restart if there are no confusion. > > > >--Yakov >
Re: Cluster auto activation design proposal
Yakov, I think it is not just restarts, this set of nodes is minimally required for the cluster to function, no? D. On Aug 3, 2017, 11:23 AM, at 11:23 AM, Yakov Zhdanovwrote: >Ю> How about naming it "minimal node set" or "required node set"? > >Required for what? I would add restart if there are no confusion. > >--Yakov
Re: Cluster auto activation design proposal
>From my standpoint name for the concept should emphasize that nodes from the set constitute a target topology - the place where user wants to be. If we go in a "node set" way, what about FixedNodeSet or BaseNodeSet? "restart node set" also is a bit confusing because this concept works not only to restart but to manage adding and removing nodes to/from cluster. E.g. cluster admin decides to add ten more nodes to existing cluster: he/she starts them one by one, nodes join the cluster but don't receive any data as they are not in FixedNodeSet yet. Then admin issues "change fixed node set" command or adds them to the set in some other way and nodes become operational. As one can see, no restarts are involved in the process. Thanks, Sergey. On Thu, Aug 3, 2017 at 12:23 PM, Yakov Zhdanovwrote: > Ю> How about naming it "minimal node set" or "required node set"? > > Required for what? I would add restart if there are no confusion. > > --Yakov >
Re: Cluster auto activation design proposal
Ю> How about naming it "minimal node set" or "required node set"? Required for what? I would add restart if there are no confusion. --Yakov
Re: Cluster auto activation design proposal
How about naming it "minimal node set" or "required node set"? D. On Aug 3, 2017, 11:15 AM, at 11:15 AM, Yakov Zhdanovwrote: >> * Based on some sort of policies when the actual cluster topology >differs >too much from the baseline or when some critical condition happens >(e.g., >when there are no more backups for a partition) > >Good point, Alex! I would even go further. If cluster is active and >under >load and nodes continue joining and leaving then we can have several >BT's >that are possible to restart on - the main condition is to have all the >up >to date data partitions. I.e. if you have 4 servers and 3 backups most >probably you can have all the data with 2, 3 and, of course, 4 nodes. >Makes >sense? > >I would also think of different name. Topology (for me) also implies >the >version, but here only nodes carrying data are important. How about >"restart nodes set"? > >--Yakov
Re: Cluster auto activation design proposal
> * Based on some sort of policies when the actual cluster topology differs too much from the baseline or when some critical condition happens (e.g., when there are no more backups for a partition) Good point, Alex! I would even go further. If cluster is active and under load and nodes continue joining and leaving then we can have several BT's that are possible to restart on - the main condition is to have all the up to date data partitions. I.e. if you have 4 servers and 3 backups most probably you can have all the data with 2, 3 and, of course, 4 nodes. Makes sense? I would also think of different name. Topology (for me) also implies the version, but here only nodes carrying data are important. How about "restart nodes set"? --Yakov
Re: Cluster auto activation design proposal
First of all, this sounds a bit too scientific and will likely be misunderstood and misused. Is there a chance to give it a better name and a simpler use case and description? Secondly, I do agree that it should be mandatory to update the BT. For example, what if I want to add several nodes to the cluster for scalability reasons? As Alexey G. suggested, perhaps we can have some policies that will update the baseline topology automatically and will properly and clearly report it to the user in logs. D. On Wed, Aug 2, 2017 at 9:51 AM, Alexey Goncharukwrote: > I think we should be able to change the BT in the runtime, and a user > should have several ways to do this: > > * programmatically via the API suggested by Sergey > * Using management tools (console visor on Web Console) > * Based on some sort of policies when the actual cluster topology differs > too much from the baseline or when some critical condition happens (e.g., > when there are no more backups for a partition) > > > > 2017-08-01 22:21 GMT+03:00 Denis Magda : > > > Sergey, > > > > Is it assumed that the baseline topology can be updated in runtime? Like, > > initially I had a cluster of 10 nodes but in a couple of weeks it was > > expanded to 15 nodes. How the baseline topology should be updated in this > > way? Will it happen automatically? > > > > — > > Denis > > > > > On Aug 1, 2017, at 7:51 AM, Sergey Chugunov > > > wrote: > > > > > > Hello Ignite developers, > > > > > > I would like to start a discussion about design of important > improvement > > > enabling automatic activation of cluster with durable store turned on > > [1]. > > > Also it will help us to solve an issue with data divergence (e.g. this > > may > > > happen when half of the cluster goes down and updates are applied to > > > another half, and than online and offline parts of the cluster switch). > > > > > > The idea is to introduce a *BaselineTopology *concept. Simplifying it > is > > > just a collection of nodes that are expected to be in the cluster. > > > > > > User establishes BaselineTopology (BT) on a cluster of desired > > > configuration (I mean here number of nodes in the first place), after > > that > > > this topology is persisted. > > > > > > Once established BT represents a "frozen state" of topology which means > > > that affinity function uses it instead of actual topology. As a result > no > > > rebalancing can happen until BT is reestablished. > > > > > > Having BT established it is easy to implement automatic activation: > when > > > nodes of starting cluster join it one by one, a special listener may > > > trigger cluster activation when composition of nodes matches with the > one > > > described by BaselineTopology. > > > > > > API for BaselineTopology manipulation may look like this: > > > > > > *Ignite::activation::establishBaselineTopology();* > > > *Ignite::activation::establishBaselineTopology(BaselineTopology > > bltTop);* > > > > > > Both methods will establish BT and activate cluster once it is > > established. > > > > > > The first one allows user to establish BT using current topology. If > any > > > changes happen to the topology during establishing process, user will > be > > > notified and allowed to proceed or abort the procedure. > > > > > > Second method allows to use some monitoring'n'management tools like > > > WebConsole where user can prepare a list of nodes, using them create a > BT > > > and send to the cluster a command to finally establish it. > > > > > > From high level BaselineTopology entity contains only collection of > > nodes: > > > > > > *BaselineTopology {* > > > * Collection nodes;* > > > *}* > > > > > > *TopologyNode* here contains information about node - its consistent id > > and > > > set of user attributes used to calculate affinity function. > > > > > > In order to support data divergence prevention some kind of versioning > > must > > > be added to BT entity to refuse joining new node but we can clarify it > > > later. > > > > > > Please provide your feedback/thoughts and ask any questions about > > suggested > > > improvement. > > > > > > Thanks, > > > Sergey. > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5851 > > > > >
Re: Cluster auto activation design proposal
I think we should be able to change the BT in the runtime, and a user should have several ways to do this: * programmatically via the API suggested by Sergey * Using management tools (console visor on Web Console) * Based on some sort of policies when the actual cluster topology differs too much from the baseline or when some critical condition happens (e.g., when there are no more backups for a partition) 2017-08-01 22:21 GMT+03:00 Denis Magda: > Sergey, > > Is it assumed that the baseline topology can be updated in runtime? Like, > initially I had a cluster of 10 nodes but in a couple of weeks it was > expanded to 15 nodes. How the baseline topology should be updated in this > way? Will it happen automatically? > > — > Denis > > > On Aug 1, 2017, at 7:51 AM, Sergey Chugunov > wrote: > > > > Hello Ignite developers, > > > > I would like to start a discussion about design of important improvement > > enabling automatic activation of cluster with durable store turned on > [1]. > > Also it will help us to solve an issue with data divergence (e.g. this > may > > happen when half of the cluster goes down and updates are applied to > > another half, and than online and offline parts of the cluster switch). > > > > The idea is to introduce a *BaselineTopology *concept. Simplifying it is > > just a collection of nodes that are expected to be in the cluster. > > > > User establishes BaselineTopology (BT) on a cluster of desired > > configuration (I mean here number of nodes in the first place), after > that > > this topology is persisted. > > > > Once established BT represents a "frozen state" of topology which means > > that affinity function uses it instead of actual topology. As a result no > > rebalancing can happen until BT is reestablished. > > > > Having BT established it is easy to implement automatic activation: when > > nodes of starting cluster join it one by one, a special listener may > > trigger cluster activation when composition of nodes matches with the one > > described by BaselineTopology. > > > > API for BaselineTopology manipulation may look like this: > > > > *Ignite::activation::establishBaselineTopology();* > > *Ignite::activation::establishBaselineTopology(BaselineTopology > bltTop);* > > > > Both methods will establish BT and activate cluster once it is > established. > > > > The first one allows user to establish BT using current topology. If any > > changes happen to the topology during establishing process, user will be > > notified and allowed to proceed or abort the procedure. > > > > Second method allows to use some monitoring'n'management tools like > > WebConsole where user can prepare a list of nodes, using them create a BT > > and send to the cluster a command to finally establish it. > > > > From high level BaselineTopology entity contains only collection of > nodes: > > > > *BaselineTopology {* > > * Collection nodes;* > > *}* > > > > *TopologyNode* here contains information about node - its consistent id > and > > set of user attributes used to calculate affinity function. > > > > In order to support data divergence prevention some kind of versioning > must > > be added to BT entity to refuse joining new node but we can clarify it > > later. > > > > Please provide your feedback/thoughts and ask any questions about > suggested > > improvement. > > > > Thanks, > > Sergey. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5851 > >
Re: Cluster auto activation design proposal
Sergey, Is it assumed that the baseline topology can be updated in runtime? Like, initially I had a cluster of 10 nodes but in a couple of weeks it was expanded to 15 nodes. How the baseline topology should be updated in this way? Will it happen automatically? — Denis > On Aug 1, 2017, at 7:51 AM, Sergey Chugunovwrote: > > Hello Ignite developers, > > I would like to start a discussion about design of important improvement > enabling automatic activation of cluster with durable store turned on [1]. > Also it will help us to solve an issue with data divergence (e.g. this may > happen when half of the cluster goes down and updates are applied to > another half, and than online and offline parts of the cluster switch). > > The idea is to introduce a *BaselineTopology *concept. Simplifying it is > just a collection of nodes that are expected to be in the cluster. > > User establishes BaselineTopology (BT) on a cluster of desired > configuration (I mean here number of nodes in the first place), after that > this topology is persisted. > > Once established BT represents a "frozen state" of topology which means > that affinity function uses it instead of actual topology. As a result no > rebalancing can happen until BT is reestablished. > > Having BT established it is easy to implement automatic activation: when > nodes of starting cluster join it one by one, a special listener may > trigger cluster activation when composition of nodes matches with the one > described by BaselineTopology. > > API for BaselineTopology manipulation may look like this: > > *Ignite::activation::establishBaselineTopology();* > *Ignite::activation::establishBaselineTopology(BaselineTopology bltTop);* > > Both methods will establish BT and activate cluster once it is established. > > The first one allows user to establish BT using current topology. If any > changes happen to the topology during establishing process, user will be > notified and allowed to proceed or abort the procedure. > > Second method allows to use some monitoring'n'management tools like > WebConsole where user can prepare a list of nodes, using them create a BT > and send to the cluster a command to finally establish it. > > From high level BaselineTopology entity contains only collection of nodes: > > *BaselineTopology {* > * Collection nodes;* > *}* > > *TopologyNode* here contains information about node - its consistent id and > set of user attributes used to calculate affinity function. > > In order to support data divergence prevention some kind of versioning must > be added to BT entity to refuse joining new node but we can clarify it > later. > > Please provide your feedback/thoughts and ask any questions about suggested > improvement. > > Thanks, > Sergey. > > [1] https://issues.apache.org/jira/browse/IGNITE-5851
Cluster auto activation design proposal
Hello Ignite developers, I would like to start a discussion about design of important improvement enabling automatic activation of cluster with durable store turned on [1]. Also it will help us to solve an issue with data divergence (e.g. this may happen when half of the cluster goes down and updates are applied to another half, and than online and offline parts of the cluster switch). The idea is to introduce a *BaselineTopology *concept. Simplifying it is just a collection of nodes that are expected to be in the cluster. User establishes BaselineTopology (BT) on a cluster of desired configuration (I mean here number of nodes in the first place), after that this topology is persisted. Once established BT represents a "frozen state" of topology which means that affinity function uses it instead of actual topology. As a result no rebalancing can happen until BT is reestablished. Having BT established it is easy to implement automatic activation: when nodes of starting cluster join it one by one, a special listener may trigger cluster activation when composition of nodes matches with the one described by BaselineTopology. API for BaselineTopology manipulation may look like this: *Ignite::activation::establishBaselineTopology();* *Ignite::activation::establishBaselineTopology(BaselineTopology bltTop);* Both methods will establish BT and activate cluster once it is established. The first one allows user to establish BT using current topology. If any changes happen to the topology during establishing process, user will be notified and allowed to proceed or abort the procedure. Second method allows to use some monitoring'n'management tools like WebConsole where user can prepare a list of nodes, using them create a BT and send to the cluster a command to finally establish it. >From high level BaselineTopology entity contains only collection of nodes: *BaselineTopology {* * Collection nodes;* *}* *TopologyNode* here contains information about node - its consistent id and set of user attributes used to calculate affinity function. In order to support data divergence prevention some kind of versioning must be added to BT entity to refuse joining new node but we can clarify it later. Please provide your feedback/thoughts and ask any questions about suggested improvement. Thanks, Sergey. [1] https://issues.apache.org/jira/browse/IGNITE-5851