Say, that's an idea. However, I think the complex option is a recipe for confusions.
IMO, we have to finish the basics clearly. Otherwise, everything can be fizzled out, like Fault Tolerance and Superstep API. On Wed, May 28, 2014 at 7:42 PM, Tommaso Teofili <[email protected]> wrote: > 2014-05-28 12:21 GMT+02:00 Edward J. Yoon <[email protected]>: > >> Even though we use the disk or spilling queue, we need to add whole >> messages into the bundle object (memory). So, I said, they are >> pointless. The bundle should be spillable. Receive-side queue is also >> the same. >> > > +1 > > >> >> If there's no objections, I'll remove the spilling queue. >> > > I'm not sure that's the right move, in the latest months we've been adding > and removing things back and forth in this area. > For example beforehand we didn't bundle everything together and so the > spilling queue made sense (if I recall correctly), again I would prefer to > have everything to be configurable, so by default for example we have no > spilling queue and bundle message but we may configure things in order to > use the spilling queue and to eventually send single messages rather than > the whole bundle. > > Regards, > Tommaso > > >> >> >> On Tue, May 13, 2014 at 4:55 PM, Andronidis Anastasios >> <[email protected]> wrote: >> > I am +1. That plan is going to make things much more straight to other >> developers too. >> > >> > Anastasis >> > >> > On 13 Μαϊ 2014, at 7:27 π.μ., Edward J. Yoon <[email protected]> >> wrote: >> > >> >> Just opened https://issues.apache.org/jira/browse/HAMA-903 >> >> >> >> On Tue, May 13, 2014 at 10:51 AM, Edward J. Yoon <[email protected]> >> wrote: >> >>> The AbstractMessageManager.loopBackMessages() method unbundle messages >> >>> into localQueueForNextIteration. Finally, in clearOutgoingMessages() >> >>> method, localQueue is switched by localQueueForNextIteration. >> >>> >> >>> My goal is simplify this procedure. This is huge overhead. >> >>> >> >>> Basically, BSPMessageBundle provides iterator access to the messages >> >>> contained in the bundle. So, we can skip whole conversion from bundle >> >>> to queue. >> >>> >> >>>> Is it going to use rpc? >> >>> >> >>> I won't touch RPC mechanism. Only queue implementations will be >> changed. >> >>> >> >>> On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin <[email protected]> >> wrote: >> >>>> Is it going to use rpc? >> >>>> >> >>>> Will it still use the interface, for instance, MessageManager.java? >> >>>> >> >>>> Just to check if any point for integrating with the current ongoing >> >>>> refactoring process. >> >>>> >> >>>> If possible, perhaps decoupling io part and rpc from interface would >> >>>> somehow simplify the integration progress. >> >>>> >> >>>> >> >>>> On 12 May 2014 09:01, Edward J. Yoon <[email protected]> wrote: >> >>>>> The old design of outgoing/incoming message queues is readable but it >> >>>>> has some problems, and the most performance and memory issues are >> >>>>> dependent upon this part. >> >>>>> >> >>>>> 1) To send a messages to destination Peer, we serialize, compress, >> and >> >>>>> bundle the messages. So, using disk or spilling queue for the >> outgoing >> >>>>> messages is pointless and cause of degradation. This issue SOLVED by >> >>>>> HAMA-853. We'll need to add disk-based bundle in the future. >> >>>>> >> >>>>> 2) Receive-side queue is also the same. Instead of unbundling (and >> >>>>> deserializing, decompressing) bundles into {memory, disk, or >> spilling} >> >>>>> queue, we should use bundles in efficient and asynchronous way. >> >>>>> >> >>>>> If you agree with this, I'll start to refactor the whole queue >> system. >> >>>>> >> >>>>> If you have any other ideas e.g., asynchronous message >> >>>>> synchronization, Pls let me know. >> >>>>> >> >>>>> Thanks. >> >>>>> >> >>>>> -- >> >>>>> Best Regards, Edward J. Yoon >> >>>>> CEO at DataSayer Co., Ltd. >> >>> >> >>> >> >>> >> >>> -- >> >>> Best Regards, Edward J. Yoon >> >>> CEO at DataSayer Co., Ltd. >> >> >> >> >> >> >> >> -- >> >> Best Regards, Edward J. Yoon >> >> CEO at DataSayer Co., Ltd. >> > >> >> >> >> -- >> Best Regards, Edward J. Yoon >> CEO at DataSayer Co., Ltd. >> -- Best Regards, Edward J. Yoon CEO at DataSayer Co., Ltd.
