I meant "easily seen" as "not exposed" and we are easy to miss to consider.
2018년 2월 1일 (목) 오전 8:36, Jungtaek Lim <kabh...@gmail.com>님이 작성: > I'm not clear whether you're saying message transfer for each bolt took > 200ms, or summation of 4 or 5 times network transfer latencies were 200 ms. > > Why I say I'm not clear is that if it's latter and there's 5 times network > transfers, it is ideal latency in theory, since 1Gbps is 125MBps (1000Mbps, > not 1024Mbps) and 5M/125MBps = 40ms per each transfer. (I'd rather suspect > how it was possible in this case.) > > Even there's 4 times network transfers, we may need to take this to > account: the latency calculation above is in theory, and there're many > overheads other than messaging which is not easily seen, so the latency may > not be from messaging overhead. > > If it took 200 ms for each transfer that can be a thing to talk about. > Please let me know which is your case. > > Thanks, > Jungtaek Lim (HeartSaVioR) > > 2018년 2월 1일 (목) 오전 8:11, Wuyang Zhang <wuy...@winlab.rutgers.edu>님이 작성: > >> I am playing with Apache Storm for a real-time image processing >> application which requires ultra low latency. In the topology definition, a >> single spout will emit raw images(5MB) in every 1s and a few bolts will >> process them. The processing latency of each bolt is acceptable and the >> overall computing delay can be around 150ms. >> >> *However, I find that the message passing delay between workers on the >> different nodes is really high. The overall such delay on the 5 successive >> bolts is around 200ms.* To calculate this delay, I subtract all the task >> latencies from the end-to-end latency. Moreover, I implement a timer bolt >> and other processing bolts will register in this timer bolt to record the >> timestamp before starting the real processing. By comparing the timestamps >> of the bolts, I find the delay between each bolt is high as I previously >> noticed. >> >> To analyze the source of this high additional delay, I firstly reduce the >> sending interval to 1s and thus there should be no queuing delay due to the >> high computing overheads. Also, from the Storm UI, I find none bolt is in >> high CPU utilization. >> >> Then, I checked the network delay. I am using a 1Gbps network testbed and >> test the network by RTT and bandwidth. The network latency should not be >> that high to send a 5MB image. >> >> Finally, I am thinking about the buffer delay. I find each thread >> maintains its own sending buffer and transfer the data to the worker's >> sending buffer. I am not sure how long it takes before the receiver bolt >> can get this sending message. As suggested by the community, I increase the >> sender/receiver buffer size to 16384, modify STORM_NETTY_MESSAGE_BATCH_SIZE >> to 32768. However, it did not help. >> >> *My question is that how to remove/reduce the messaging overheads between >> bolts?(inter workers)* It is possible to synchronize the communication >> between bolts and have the receiver got the sending messages immediately >> without any delay? >> >> ᐧ >> >
