Denis, Those ticket are rather complex, and so I don't know when I'll be able to start working on them.
On Fri, Mar 30, 2018 at 11:45 PM, Denis Magda <dma...@apache.org> wrote: > Ilya, > > Just came across the IEP put together by you: > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > 16%3A+Optimization+of+rebalancing > > Excellent explanation, thanks for aggregating everything there. > > Two tickets below don't have a fixed version assigned: > https://issues.apache.org/jira/browse/IGNITE-8020 > https://issues.apache.org/jira/browse/IGNITE-7935 > > Do you plan to work on them in 2.6 time frame, right? > > -- > Denis > > On Tue, Mar 27, 2018 at 9:29 AM, Denis Magda <dma...@apache.org> wrote: > > > Ilya, granted you all the required permissions. Please let me know if you > > still have troubles with the wiki. > > > > -- > > Denis > > > > On Tue, Mar 27, 2018 at 8:56 AM, Ilya Lantukh <ilant...@gridgain.com> > > wrote: > > > >> Unfortunately, I don't have permission to create page for IEP on wiki. > >> Denis, can you grant it? My username is ilantukh. > >> > >> On Mon, Mar 26, 2018 at 8:04 PM, Anton Vinogradov <a...@apache.org> > wrote: > >> > >> > >> It is impossible to disable WAL only for certain partitions without > >> > >> completely overhauling design of Ignite storage mechanism. Right > now > >> we > >> > can > >> > >> afford only to change WAL mode per cache group. > >> > > >> > Cache group rebalancing is a one cache rebalancing, and then this > cache > >> > ("cache group") can be presented as a set of virtual caches. > >> > So, there is no issues for initial rebalancing. > >> > Lets disable WAL on initial rebalancing. > >> > > >> > 2018-03-26 16:46 GMT+03:00 Ilya Lantukh <ilant...@gridgain.com>: > >> > > >> > > Dmitry, > >> > > It is impossible to disable WAL only for certain partitions without > >> > > completely overhauling design of Ignite storage mechanism. Right now > >> we > >> > can > >> > > afford only to change WAL mode per cache group. > >> > > > >> > > The idea is to disable WAL when node doesn't have any partition in > >> OWNING > >> > > state, which means it doesn't have any consistent data and won't be > >> able > >> > to > >> > > restore from WAL anyway. I don't see any potential use for WAL on > such > >> > > node, but we can keep a configurable parameter indicating can we > >> > > automatically disable WAL in such case or not. > >> > > > >> > > On Fri, Mar 23, 2018 at 10:40 PM, Dmitry Pavlov < > >> dpavlov....@gmail.com> > >> > > wrote: > >> > > > >> > > > Denis, as I understood, there is and idea to exclude only > rebalanced > >> > > > partition(s) data. All other data will go to the WAL. > >> > > > > >> > > > Ilya, please correct me if I'm wrong. > >> > > > > >> > > > пт, 23 мар. 2018 г. в 22:15, Denis Magda <dma...@apache.org>: > >> > > > > >> > > > > Ilya, > >> > > > > > >> > > > > That's a decent boost (5-20%) even having WAL enabled. Not sure > >> that > >> > we > >> > > > > should stake on the WAL "off" mode here because if the whole > >> cluster > >> > > goes > >> > > > > down, it's then the data consistency is questionable. As an > >> > architect, > >> > > I > >> > > > > wouldn't disable WAL for the sake of rebalancing; it's too > risky. > >> > > > > > >> > > > > If you agree, then let's create the IEP. This way it will be > >> easier > >> > to > >> > > > > track this endeavor. BTW, are you already ready to release any > >> > > > > optimizations in 2.5 that is being discussed in a separate > thread? > >> > > > > > >> > > > > -- > >> > > > > Denis > >> > > > > > >> > > > > > >> > > > > > >> > > > > On Fri, Mar 23, 2018 at 6:37 AM, Ilya Lantukh < > >> ilant...@gridgain.com > >> > > > >> > > > > wrote: > >> > > > > > >> > > > > > Denis, > >> > > > > > > >> > > > > > > - Don't you want to aggregate the tickets under an IEP? > >> > > > > > Yes, I think so. > >> > > > > > > >> > > > > > > - Does it mean we're going to update our B+Tree > >> implementation? > >> > Any > >> > > > > ideas > >> > > > > > how risky it is? > >> > > > > > One of tickets that I created ( > >> > > > > > https://issues.apache.org/jira/browse/IGNITE-7935) involves > >> B+Tree > >> > > > > > modification, but I am not planning to do it in the nearest > >> future. > >> > > It > >> > > > > > shouldn't affect existing tree operations, only introduce new > >> ones > >> > > > > (putAll, > >> > > > > > invokeAll, removeAll). > >> > > > > > > >> > > > > > > - Any chance you had a prototype that shows performance > >> > > optimizations > >> > > > > the > >> > > > > > approach you are suggesting to take? > >> > > > > > I have a prototype for simplest improvements ( > >> > > > https://issues.apache.org/ > >> > > > > > jira/browse/IGNITE-8019 & https://issues.apache.org/ > >> > > > > > jira/browse/IGNITE-8018) > >> > > > > > - together they increase throughput by 5-20%, depending on > >> > > > configuration > >> > > > > > and environment. Also, I've tested different WAL modes - > >> switching > >> > > from > >> > > > > > LOG_ONLY to NONE gives over 100% boost - this is what I expect > >> from > >> > > > > > https://issues.apache.org/jira/browse/IGNITE-8017. > >> > > > > > > >> > > > > > On Thu, Mar 22, 2018 at 9:48 PM, Denis Magda < > dma...@apache.org > >> > > >> > > > wrote: > >> > > > > > > >> > > > > > > Ilya, > >> > > > > > > > >> > > > > > > That's outstanding research and summary. Thanks for spending > >> your > >> > > > time > >> > > > > on > >> > > > > > > this. > >> > > > > > > > >> > > > > > > Not sure I have enough expertise to challenge your approach, > >> but > >> > it > >> > > > > > sounds > >> > > > > > > 100% reasonable to me. As side notes: > >> > > > > > > > >> > > > > > > - Don't you want to aggregate the tickets under an IEP? > >> > > > > > > - Does it mean we're going to update our B+Tree > >> > implementation? > >> > > > Any > >> > > > > > > ideas how risky it is? > >> > > > > > > - Any chance you had a prototype that shows performance > >> > > > > optimizations > >> > > > > > of > >> > > > > > > the approach you are suggesting to take? > >> > > > > > > > >> > > > > > > -- > >> > > > > > > Denis > >> > > > > > > > >> > > > > > > On Thu, Mar 22, 2018 at 8:38 AM, Ilya Lantukh < > >> > > ilant...@gridgain.com > >> > > > > > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > Igniters, > >> > > > > > > > > >> > > > > > > > I've spent some time analyzing performance of rebalancing > >> > > process. > >> > > > > The > >> > > > > > > > initial goal was to understand, what limits it's > throughput, > >> > > > because > >> > > > > it > >> > > > > > > is > >> > > > > > > > significantly slower than network and storage device can > >> > > > > theoretically > >> > > > > > > > handle. > >> > > > > > > > > >> > > > > > > > Turns out, our current implementation has a number of > issues > >> > > caused > >> > > > > by > >> > > > > > a > >> > > > > > > > single fundamental problem. > >> > > > > > > > > >> > > > > > > > During rebalance data is sent in batches called > >> > > > > > > > GridDhtPartitionSupplyMessages. Batch size is > configurable, > >> > > > default > >> > > > > > > value > >> > > > > > > > is 512KB, which could mean thousands of key-value pairs. > >> > However, > >> > > > we > >> > > > > > > don't > >> > > > > > > > take any advantage over this fact and process each entry > >> > > > > independently: > >> > > > > > > > - checkpointReadLock is acquired multiple times for every > >> > entry, > >> > > > > > leading > >> > > > > > > to > >> > > > > > > > unnecessary contention - this is clearly a bug; > >> > > > > > > > - for each entry we write (and fsync, if configuration > >> assumes > >> > > it) > >> > > > a > >> > > > > > > > separate WAL record - so, if batch contains N entries, we > >> might > >> > > end > >> > > > > up > >> > > > > > > > doing N fsyncs; > >> > > > > > > > - adding every entry into CacheDataStore also happens > >> > completely > >> > > > > > > > independently. It means, we will traverse and modify each > >> index > >> > > > tree > >> > > > > N > >> > > > > > > > times, we will allocate space in FreeList N times and we > >> will > >> > > have > >> > > > to > >> > > > > > > > additionally store in WAL O(N*log(N)) page delta records. > >> > > > > > > > > >> > > > > > > > I've created a few tickets in JIRA with very different > >> levels > >> > of > >> > > > > scale > >> > > > > > > and > >> > > > > > > > complexity. > >> > > > > > > > > >> > > > > > > > Ways to reduce impact of independent processing: > >> > > > > > > > - https://issues.apache.org/jira/browse/IGNITE-8019 - > >> > > > aforementioned > >> > > > > > > bug, > >> > > > > > > > causing contention on checkpointReadLock; > >> > > > > > > > - https://issues.apache.org/jira/browse/IGNITE-8018 - > >> > > inefficiency > >> > > > > in > >> > > > > > > > GridCacheMapEntry implementation; > >> > > > > > > > - https://issues.apache.org/jira/browse/IGNITE-8017 - > >> > > > automatically > >> > > > > > > > disable > >> > > > > > > > WAL during preloading. > >> > > > > > > > > >> > > > > > > > Ways to solve problem on more global level: > >> > > > > > > > - https://issues.apache.org/jira/browse/IGNITE-7935 - a > >> ticket > >> > > to > >> > > > > > > > introduce > >> > > > > > > > batch modification; > >> > > > > > > > - https://issues.apache.org/jira/browse/IGNITE-8020 - > >> complete > >> > > > > > redesign > >> > > > > > > of > >> > > > > > > > rebalancing process for persistent caches, based on file > >> > > transfer. > >> > > > > > > > > >> > > > > > > > Everyone is welcome to criticize above ideas, suggest new > >> ones > >> > or > >> > > > > > > > participate in implementation. > >> > > > > > > > > >> > > > > > > > -- > >> > > > > > > > Best regards, > >> > > > > > > > Ilya > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > -- > >> > > > > > Best regards, > >> > > > > > Ilya > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > > -- > >> > > Best regards, > >> > > Ilya > >> > > > >> > > >> > >> > >> > >> -- > >> Best regards, > >> Ilya > >> > > > > > -- Best regards, Ilya
