Re: Task failover in ignite
Prasad, prasadbhalerao1983 wrote > Are you saying that when a primary node dies the former backup node > becomes > new primary for ALL backup partitions present on it and only primary > partitions are moved in rebalancing process? Not for all partitions, but only for those for which primary copy existed on the recently failed node. For example, you have nodes A, B and C. Partition #1 has primary copy on A and backup on B. Partition #2 has primary copy on C and backup on B. If A dies, B becomes primary node for partition #1, and then rebalancing creates a new backup for it on C. However, nothing changes for partition #2, because A didn't own any of its copies. prasadbhalerao1983 wrote > Partition to node mapping is defined by affinity function which uses > Rendezvous Hashing. How does it make sure that back up partitions on a > node > becomes primary partitions on same node? That's how the function is implemented. If you're interested in more details, I would suggest you to go through the code of RendezvousAffinityFunction. -Val -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Task failover in ignite
Hi Val, Thank you for the explanation. Are you saying that when a primary node dies the former backup node becomes new primary for ALL backup partitions present on it and only primary partitions are moved in rebalancing process? Partition to node mapping is defined by affinity function which uses Rendezvous Hashing. How does it make sure that back up partitions on a node becomes primary partitions on same node? Regards, Prasad On Tue, Oct 2, 2018, 3:16 AM vkulichenko wrote: > Prasad, > > When a primary node for a partition dies, former backup node for this > partition becomes new primary. Therefore there is no need to wait for > rebalancing in this case, data is already there. By default job will be > automatically remapped to that node, but with 'withNoFailover()' you'll > have > to retry manually. > > In addition, affinityRun/Call acquires a partition lock for a duration of > computation to make sure data is not moved until it's completed. I.e. if > there is a new node that becomes a new owner for this partition, data for > this partition will not be evicted from previous primary until all > collocated computations are done. > > -Val > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Task failover in ignite
Prasad, When a primary node for a partition dies, former backup node for this partition becomes new primary. Therefore there is no need to wait for rebalancing in this case, data is already there. By default job will be automatically remapped to that node, but with 'withNoFailover()' you'll have to retry manually. In addition, affinityRun/Call acquires a partition lock for a duration of computation to make sure data is not moved until it's completed. I.e. if there is a new node that becomes a new owner for this partition, data for this partition will not be evicted from previous primary until all collocated computations are done. -Val -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Task failover in ignite
Hi, Ignite doc says "at least once guarantee". If I submit the task using just "ignite.compute().affinityRun()", then ignite will try to execute this task on backup node if the primary node goes down. Am I correct? What I want is if the primary data node which is executing the task goes down then the task should be executed on its back node. Does ignite immediately start rebalancing when a node goes down? I am trying to understand how does ignite re-executes affinity task on backup or new primary node when primary goes down? Does ignite wait for rebalancing to complete and then resubmits the affinity task to new primary node? Or does ignite resubmits the task to backup node and waits for task to complete then does the rebalancing? In case of node failure does backup node becomes new primary for backup partitions or it is decided after partition re-exchange process? How does it decides which node will become new primary for backup partitions so that minimum data exchange will happen? Thanks, Prasad On Sat, Sep 29, 2018 at 8:29 AM Prasad Bhalerao < prasadbhalerao1...@gmail.com> wrote: > Hi, > > Ignite doc says "at least once guarantee". > > If I sumbit the task using just > "ignite.compute().withNoFailover().affinityRun()", > then ignite will try to execute this task on backup node if the primary > node goes down. > > Does ignite immediately start rebalancing when a node goes down? > > I am trying to understand how does ignite re-executes affinity task on > backup or new primary node when primary goes down? > > Does ignite wait for rebalancing to complete and then resubmits the > affinity task to new primary node? > > Or does ignite resubmits the task to backup node and waits for task to > complete then does the rebalancing? > > In case of node failure does backup node becomes new primary for backup > partitions or it is decided after partition reexchange process? > > How does it decides which node will become new primary for backup > partitions so that minimum data exchange will happen? > > Thanks, > Prasad > > On Sat, Sep 29, 2018, 2:31 AM vkulichenko > wrote: > >> Prasad, >> >> Since you're using withNoFailover(), failover will never happen and the >> task >> will just fail with an exception on client side if primary nodes dies. >> It's >> up to your code to retry in this case. >> >> When you retry, the task will be mapped to the new primary, which is >> former >> backup and therefore has all the data. No need to wait for rebalancing. >> >> In general, affinityRun/Call guarantees that all data is available locally >> during task execution. If that's not possible for any reason, an exception >> is thrown. >> >> -Val >> >> >> >> >> >> -- >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >
Re: Task failover in ignite
Hi, Ignite doc says "at least once guarantee". If I sumbit the task using just "ignite.compute().withNoFailover().affinityRun()", then ignite will try to execute this task on backup node if the primary node goes down. Does ignite immediately start rebalancing when a node goes down? I am trying to understand how does ignite re-executes affinity task on backup or new primary node when primary goes down? Does ignite wait for rebalancing to complete and then resubmits the affinity task to new primary node? Or does ignite resubmits the task to backup node and waits for task to complete then does the rebalancing? In case of node failure does backup node becomes new primary for backup partitions or it is decided after partition reexchange process? How does it decides which node will become new primary for backup partitions so that minimum data exchange will happen? Thanks, Prasad On Sat, Sep 29, 2018, 2:31 AM vkulichenko wrote: > Prasad, > > Since you're using withNoFailover(), failover will never happen and the > task > will just fail with an exception on client side if primary nodes dies. It's > up to your code to retry in this case. > > When you retry, the task will be mapped to the new primary, which is former > backup and therefore has all the data. No need to wait for rebalancing. > > In general, affinityRun/Call guarantees that all data is available locally > during task execution. If that's not possible for any reason, an exception > is thrown. > > -Val > > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Task failover in ignite
Prasad, Since you're using withNoFailover(), failover will never happen and the task will just fail with an exception on client side if primary nodes dies. It's up to your code to retry in this case. When you retry, the task will be mapped to the new primary, which is former backup and therefore has all the data. No need to wait for rebalancing. In general, affinityRun/Call guarantees that all data is available locally during task execution. If that's not possible for any reason, an exception is thrown. -Val -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Task failover in ignite
Hi, I have created multiple ignite runnable task by extending IgniteRunnable nad IgniteCallable interface. I submitting these tasks to the primary data node using "ignite.compute().withNoFailover().affinityRun()" method. I have not set max failover attempt, so I think default max failover attempt will be 5. I have kept the back count as 1. What happens when the primary node executing this task goes down? In this case does ignite move this task to the backup node and execute it on back node? When the node goes down ignite starts rebalancing the cluster. In this case how this task will be executed? Does ignite waits for rebalancing process to complete and then execute this task? Can some one please explain this in detail? Thanks, Prasad