I read the data which is in avro format using spark sql and load it to cache from spark program. I build the Ignite key-store inside spark executors. When due to some reason a task fails, the ignite node stops and when the task re-starts, the new node joins back.I see slowness from here onwards. I was thinking cause of rebalancing this is becoming slow. I can look at tuning the rebalancing too. Let me know if you have any suggestions.
On Thu, Feb 1, 2018 at 1:54 PM, Evgenii Zhuravlev <[email protected]> wrote: > Ranjit, > > How do you load data to the cache? > > Evgenii > > 2018-02-01 11:18 GMT+03:00 Ranjit Sahu <[email protected]>: > >> Hi Val, >> >> Not always but out of 10, we see at least once the issue. Whats happening >> is when one node crashes\stops the new node joins . The loading process >> restarts but what ever was happening in few minutes (3-5) goes to 2-3 >> hours. >> >> Thanks, >> Ranjit >> >> On Wed, Jan 31, 2018 at 3:12 AM, vkulichenko < >> [email protected]> wrote: >> >>> Ranjit, >>> >>> Is it a really frequent event for node to crash in the middle of loading >>> process? If so, then I think you should fix that instead of working >>> around >>> by disabling rebalancing. Such configuration definitely has a lot >>> drawbacks >>> and therefore can cause issues. >>> >>> -Val >>> >>> >>> >>> -- >>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >>> >> >> >
