performance problem of streaming data over TCP
Hi, friends, I am implementing a protocol on top of 'asyncore.dispatcher' to send streaming multimedia data over TCP socket. However, I found that the throughput of my current implementation is surprisingly low. Below is a snippet of my code with a note that: the packet sent over the socket is of variable length, of which the first a couple bytes indicates the length of the packet. I implemented a 'var_str_to_int' function to get the 'length' and the offset of the payload. ### def handle_read(self): self.socket.setblocking(1) while (True) : data = self.assemble_msg() if (not data) : self.socket.setblocking(0) return (length,index) = var_str_to_int(data) self._dispatch_cmd(data[index:]) if (not self.recv_buffer) : break self.socket.setblocking(0) def assemble_msg(self) : if (self.recv_buffer) : data = self.recv_buffer else : data = self.socket.recv(self.BUFFER_SIZE) if (len(data) == 0) : return None length,index = var_str_to_int(data) while (length +index > len(data)) : data += self.socket.recv(self.BUFFER_SIZE) if (length + index == len(data)) : self.recv_buffer = '' else : self.recv_buffer = data[length+index:] data = data[:length+index] return data ### I found it took around 7 seconds to receive a packet of 40KB sent from localhost. The throughput is even not enough to support any streaming multimedia file for live playback in a LAN. I read some threads in the group. It sounds like the problem might have to do with the TCP option, setting TCP_NODELAY probably could solve the problem. But I tried the following on both a Linux and a windows XP machine. Neither of them worked: ## self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) ## Any comments or suggests is highly appreciated. Thanks, Changhao -- http://mail.python.org/mailman/listinfo/python-list
Re: performance problem of streaming data over TCP
I figured out the reason. It was because of asyncore.dispatcher's inefficient implementation of messange sending. Instead of using its 'push' method. I directly call the underlying 'socket.send' and got the problem solved. Sorry about the spam. Thanks Changhao wrote: > Hi, friends, > > I am implementing a protocol on top of 'asyncore.dispatcher' to send > streaming multimedia data over TCP socket. However, I found that the > throughput of my current implementation is surprisingly low. > > Below is a snippet of my code with a note that: the packet sent over > the socket is of variable length, of which the first a couple bytes > indicates the length of the packet. I implemented a 'var_str_to_int' > function to get the 'length' and the offset of the payload. > > ### > def handle_read(self): > self.socket.setblocking(1) > while (True) : > > >data = > self.assemble_msg() > if (not data) : > self.socket.setblocking(0) > return > (length,index) = var_str_to_int(data) > self._dispatch_cmd(data[index:]) > if (not self.recv_buffer) : >break > self.socket.setblocking(0) > > def assemble_msg(self) : > if (self.recv_buffer) : > data = self.recv_buffer > else : > data = self.socket.recv(self.BUFFER_SIZE) > if (len(data) == 0) : > return None > length,index = var_str_to_int(data) > while (length +index > len(data)) : > data += self.socket.recv(self.BUFFER_SIZE) > if (length + index == len(data)) : > self.recv_buffer = '' > else : > self.recv_buffer = data[length+index:] > data = data[:length+index] > return data > ### > > I found it took around 7 seconds to receive a packet of 40KB sent > from localhost. The throughput is even not enough to support any > streaming multimedia file for live playback in a LAN. > > I read some threads in the group. It sounds like the problem might > have to do with the TCP option, setting TCP_NODELAY probably could solve > the problem. But I tried the following on both a Linux and a windows XP > machine. Neither of them worked: > > ## > self.socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) > ## > > Any comments or suggests is highly appreciated. > > Thanks, > Changhao -- http://mail.python.org/mailman/listinfo/python-list
