Even though we use the disk or spilling queue, we need to add whole messages into the bundle object (memory). So, I said, they are pointless. The bundle should be spillable. Receive-side queue is also the same.
If there's no objections, I'll remove the spilling queue. On Tue, May 13, 2014 at 4:55 PM, Andronidis Anastasios <[email protected]> wrote: > I am +1. That plan is going to make things much more straight to other > developers too. > > Anastasis > > On 13 Μαϊ 2014, at 7:27 π.μ., Edward J. Yoon <[email protected]> wrote: > >> Just opened https://issues.apache.org/jira/browse/HAMA-903 >> >> On Tue, May 13, 2014 at 10:51 AM, Edward J. Yoon <[email protected]> >> wrote: >>> The AbstractMessageManager.loopBackMessages() method unbundle messages >>> into localQueueForNextIteration. Finally, in clearOutgoingMessages() >>> method, localQueue is switched by localQueueForNextIteration. >>> >>> My goal is simplify this procedure. This is huge overhead. >>> >>> Basically, BSPMessageBundle provides iterator access to the messages >>> contained in the bundle. So, we can skip whole conversion from bundle >>> to queue. >>> >>>> Is it going to use rpc? >>> >>> I won't touch RPC mechanism. Only queue implementations will be changed. >>> >>> On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin <[email protected]> >>> wrote: >>>> Is it going to use rpc? >>>> >>>> Will it still use the interface, for instance, MessageManager.java? >>>> >>>> Just to check if any point for integrating with the current ongoing >>>> refactoring process. >>>> >>>> If possible, perhaps decoupling io part and rpc from interface would >>>> somehow simplify the integration progress. >>>> >>>> >>>> On 12 May 2014 09:01, Edward J. Yoon <[email protected]> wrote: >>>>> The old design of outgoing/incoming message queues is readable but it >>>>> has some problems, and the most performance and memory issues are >>>>> dependent upon this part. >>>>> >>>>> 1) To send a messages to destination Peer, we serialize, compress, and >>>>> bundle the messages. So, using disk or spilling queue for the outgoing >>>>> messages is pointless and cause of degradation. This issue SOLVED by >>>>> HAMA-853. We'll need to add disk-based bundle in the future. >>>>> >>>>> 2) Receive-side queue is also the same. Instead of unbundling (and >>>>> deserializing, decompressing) bundles into {memory, disk, or spilling} >>>>> queue, we should use bundles in efficient and asynchronous way. >>>>> >>>>> If you agree with this, I'll start to refactor the whole queue system. >>>>> >>>>> If you have any other ideas e.g., asynchronous message >>>>> synchronization, Pls let me know. >>>>> >>>>> Thanks. >>>>> >>>>> -- >>>>> Best Regards, Edward J. Yoon >>>>> CEO at DataSayer Co., Ltd. >>> >>> >>> >>> -- >>> Best Regards, Edward J. Yoon >>> CEO at DataSayer Co., Ltd. >> >> >> >> -- >> Best Regards, Edward J. Yoon >> CEO at DataSayer Co., Ltd. > -- Best Regards, Edward J. Yoon CEO at DataSayer Co., Ltd.
