Anton, Good points!
Can you please copy the design points made in this discussion to the Jira ticket? D. On Wed, Aug 5, 2015 at 2:57 AM, Anton Vinogradov <avinogra...@gridgain.com> wrote: > Parallel Demand pool seems to be ready. > Working on parallel Supply pool. > > General idea is to use systemPool to handle demand requests. Amount ot > parallel workers will be limited indirectly by Demand poll's parallelism > limits. > But overload still can happens in this case. Will write load tests. > > Next idea is to go through whole partition and when > s.messageSize() >= cctx.config().getRebalanceBatchSize() > send supply batch, store current iterator, swapLsnr and etc. and return > from supply thread. > When same node will make next request then supplyPool just continue > iterating and give next or last batch for requested partition. > > Also speed can be increased if SupplyPool will prepare N batches on first > demand request and will generate additional on each new request. > > On Thu, Jul 30, 2015 at 9:41 PM, Dmitriy Setrakyan <dsetrak...@apache.org> > wrote: > > > Anton, > > > > Very good point. Valentin is already doing per-partition topic in his > work > > for Continuous Queries: > > https://issues.apache.org/jira/browse/IGNITE-104 > > > > Perhaps it makes sense for both of you to sync up. > > > > Also, as a side note, whenever starting a dev list discussion about a > > ticket, it always helps to include a link to the dev list thread into > Jira > > (I already did it for IGNITE-1093). > > > > D. > > > > On Thu, Jul 30, 2015 at 11:31 AM, Anton Vinogradov < > > avinogra...@gridgain.com > > > wrote: > > > > > Hello, > > > I think that speed of rebalancing can be increased by parallel > handling > > > of supply messages. > > > Main Idea is to use per-partition topics. > > > > > > At the start of DemandPool we can create topics for every partition > and > > > handle each partition's supply messages at separated listener. In this > > case > > > only one thread will request lock for partition it have to relocate. > > > > > > Also we can do the same at SupplyPool, and make supply messages in > > > parallel way. > > > > > > Amount of simultaneous messages should be limited to prevent > > overloading. > > > Current idea is to request N partitions at start and request one next > > > partition when one of prevous relocated. > > > > > > Per-partition listeners can be created this way: > > > for (int p = 0; p <= maxP; p++) { > > > cctx.io().addOrderedHandler(topic(p), new CI2<UUID, > > > GridDhtPartitionSupplyMessage>() { > > > @Override public void apply(UUID nodeId, > > > GridDhtPartitionSupplyMessage msg) { > > > handleSupplyMessage(new SupplyMessage(nodeId, msg)); > > > } > > > }); > > > } > > > > > > Thoughts? > > > > > >
