Re: Using logging module to log into flash drive
On Jun 9, 7:57 pm, Carl Banks pavlovevide...@gmail.com wrote: On Jun 9, 8:57 am, kretel krzysztof.re...@googlemail.com wrote: Hi All, I am trying to implement the following functionality: 1. log messages to the flash drive 2. if the flash drive is not available, switch handler to the BufferringHandler and log into buffer, 3. once the flash drive is plugged in and available store the logs from BufferHandler into that flash drive and switch the handler into RotateFileHandler. Which approach would you suggest to use while implementing this functionality? One that come into my mind is to have one process or thread to check periodically if the flashdrive is available, and have a flag that will indicate that we can store the logs into the flash drive. But I don't particularly like this approach. Would you do it different way? Any suggestions are appreciated. I'd refactor the steps this way: 1. log messages to a buffer 2. periodically flush the buffer to the flash drive, if it's available The periodically part could be accomplished with a thread or scheduled delays, however suits your application. It might not be a final if you need to log messages promptly, but that's how I'd begin. Carl Banks Hi Carl, Thanks for the advice. I haven't think about it this way, but it looks like that might be the starting point. Thanks, Krzysztof -- http://mail.python.org/mailman/listinfo/python-list
Re: Using logging module to log into flash drive
On Jun 9, 6:10 pm, A. Cavallo a.cava...@mailsnare.com wrote: Hi, the problem screams for a separate thread. I was thinking about that, as mentioned in the first post. Although, I was wonder if there is another way to tackle the problem. Anyway there's a TimedRotatingFileHandler handler in the logging package: you can derive from it and change the emit/doRollover pair to hold the records until a device is not ready. Hm, that might be the way to go. Will have a try. If anyone has other thoughts, please share them. Thanks, Krzysztof -- http://mail.python.org/mailman/listinfo/python-list
Re: Using logging module to log into flash drive
Anyway there's a TimedRotatingFileHandler handler in the logging package: you can derive from it and change the emit/doRollover pair to hold the records until a device is not ready. Hm, that might be the way to go. Will have a try. I had another look at the logging package. The class TimedRotatingFileHandler within the initialisation method uses the FileHandler.__init__. Whereas the FileHandler.__init__ method opens the stream in the 'append' mode (mode='a'). If the flash drive is not available while you start the script, which equals that you can't open a file in the append mode, then the TimedRotatingFileHandler (or FileHandler) fails. In my opinion the only way to go through this is to use two different handlers and once the flash drive is available then start to use the FileHandler. Still not sure if I want to use separate thread for checking the flash drive availability. Probably I need to build it this way. Regards Krzysztof -- http://mail.python.org/mailman/listinfo/python-list
Re: ANN: Python 2.6 Quick Reference available
On Feb 12, 12:54 am, pyt...@bdurham.com wrote: Richard, An excellent tool. Great job!!! Thank you for sharing this with the Python community. Regards, Malcolm Many thanks Richard and Josh. I've just started my adventure with Python, and this document will help me a lot. Cheers, K -- http://mail.python.org/mailman/listinfo/python-list
Re: TracShell 0.1 released
On 12 Feb, 14:06, J Kenneth King ja...@agentultra.com wrote: I tend to work a lot with Trac for project management and have always found the browser interface to be a productivity killer. I always wanted a simple command-line interface to Trac, but having never found one I found a little free time and got off my laurels to make one. TracShell 0.1 is an early release, but it works. So far you can only query and view tickets, but planned updates include the obvious ability to create and edit tickets. Future plans will allow browsing of comments, change histories, attachments, and so forth. Please consider it really beta. The code needs a little tidying up around the edges. If you find any bugs, please report them and I'll fix them ASAP. Ideas, suggestions, and contributions are welcome. http://code.google.com/p/tracshell/ Cheers. Nice work. Maybe it would be worth building at some point vim or emacs interface. Krzysztof -- http://mail.python.org/mailman/listinfo/python-list
Network packets processor advice
Hi, I am wrting a network packet processor. The processor listens on a specific port for incomming UDP or TCP packets. When the packet arrives it has to parse it, store in DB and if this succeed it has to acknowledge the packet to the client. Now the problem is that I want to have a control over possible failures of the system. For instance, if the DB is gone, the processor should stop working and restart when DB is back to live. So the questions are: 1. Which approach is the best for building such software? For instance using threads? 2. I can use the 'monit' tool for monitorig processes. However, not sure yet how to start a python script with a specific process id. Is it possible at all? 3. Any other suggestions? Cheers, K -- http://mail.python.org/mailman/listinfo/python-list
Re: Skipping bytes while reading a binary file?
On Feb 5, 11:51 pm, Lionel lionel.ke...@gmail.com wrote:
On Feb 5, 3:35 pm, Lionel lionel.ke...@gmail.com wrote:
On Feb 5, 2:56 pm, Lionel lionel.ke...@gmail.com wrote:
On Feb 5, 2:48 pm, MRAB goo...@mrabarnett.plus.com wrote:
Lionel wrote:
Hello,
I have data stored in binary files. Some of these files are
huge...upwards of 2 gigs or more. They consist of 32-bit float
complex
numbers where the first 32 bits of the file is the real component,
the
second 32bits is the imaginary, the 3rd 32-bits is the real component
of the second number, etc.
I'd like to be able to read in just the real components, load them
into a numpy.ndarray, then load the imaginary coponents and load them
into a numpy.ndarray. I need the real and imaginary components
stored
in seperate arrays, they cannot be in a single array of complex
numbers except for temporarily. I'm trying to avoid temporary
storage,
though, because of the size of the files.
I'm currently reading the file scanline-by-scanline to extract rows
of
complex numbers which I then loop over and load into the real/
imaginary arrays as follows:
self._realData = numpy.empty((Rows, Columns), dtype =
numpy.float32)
self._imaginaryData = numpy.empty((Rows, Columns), dtype =
numpy.float32)
floatData = array.array('f')
for CurrentRow in range(Rows):
floatData.fromfile(DataFH, (Columns*2))
position = 0
for CurrentColumn in range(Columns):
self._realData[CurrentRow, CurrentColumn] =
floatData[position]
self._imaginaryData[CurrentRow, CurrentColumn] =
floatData[position+1]
position = position + 2
The above code works but is much too slow. If I comment out the body
of the for CurrentColumn in range(Columns) loop, the performance is
perfectly adequate i.e. function call overhead associated with the
fromfile(...) call is not very bad at all. What seems to be most
time-consuming are the simple assignment statements in the
CurrentColumn for-loop.
[snip]
Try array slicing. floatData[0::2] will return the real parts and
floatData[1::2] will return the imaginary parts. You'll have to read up
how to assign to a slice of the numpy array (it might be
self._realData[CurrentRow] = real_parts or self._realData[CurrentRow,
:] = real_parts).
BTW, it's not the function call overhead of fromfile() which takes the
time, but actually reading data from the file.
Very nice! I like that! I'll post the improvement (if any).
L- Hide quoted text -
- Show quoted text -
Okay, the following:
self._realData[CurrentRow] = floatData[0::2]
self._imaginaryData[CurrentRow] = floatData[1::2]
gives a 3.5x improvement in execution speed over the original that I
posted. That's much better. Thank you for the suggestion.
L- Hide quoted text -
- Show quoted text -
Correction: improvement is around 7-8x.
I had similar issues while Slicing Network packets (TCP/UDP) on a real
time basis.
I was using 're' and found it a lot more time and resource consuming,
than 'normal' string slicing as suggested by MRAB.
K
--
http://mail.python.org/mailman/listinfo/python-list
Re: Problem with writing fast UDP server
On Nov 21, 6:55 pm, Greg Copeland [EMAIL PROTECTED] wrote: On Nov 21, 11:05 am, Krzysztof Retel [EMAIL PROTECTED] wrote: On Nov 21, 4:48 pm, Peter Pearson [EMAIL PROTECTED] wrote: On Fri, 21 Nov 2008 08:14:19 -0800 (PST), Krzysztof Retel wrote: I am not sure what do you mean by CPU-bound? How can I find out if I run it on CPU-bound? CPU-bound is the state in which performance is limited by the availability of processor cycles. On a Unix box, you might run the top utility and look to see whether the %CPU figure indicates 100% CPU use. Alternatively, you might have a tool for plotting use of system resources. -- To email me, substitute nowhere-spamcop, invalid-net. Thanks. I run it without CPU-bound With clearer eyes, I did confirm my math above is correct. I don't have a networking reference to provide. You'll likely have some good results via Google. :) If you are not CPU bound, you are likely IO-bound. That means you computer is waiting for IO to complete - likely on the sending side. In this case, it likely means you have reached your ethernet bandwidth limits available to your computer. Since you didn't correct me when I assumed you're running 10Mb ethernet, I'll continue to assume that's a safe assumption. So, assuming you are running on 10Mb ethernet, try converting your application to use TCP. I'd bet, unless you have requirements which prevent its use, you'll suddenly have enough bandwidth (in this case, frames) to achieve your desired results. This is untested and off the top of my head but it should get you pointed in the right direction pretty quickly. Make the following changes to the server: sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) to sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) Make this: print Waiting for first packet to arrive..., sock.recvfrom(BUFSIZE) look like: print Waiting for first packet to arrive..., cliSock = sock.accept() Change your calls to sock.recvfrom(BUFSIZE) to cliSock.recv(BUFSIZE). Notice the change to cliSock. Keep in mind TCP is stream based, not datagram based so you may need to add additional logic to determine data boundaries for re-assemble of your data on the receiving end. There are several strategies to address that, but for now I'll gloss it over. As someone else pointed out above, change your calls to time.clock() to time.time(). On your client, make the following changes. sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) to sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.connect( (remotehost,port) ) nbytes = sock.sendto(data, (remotehost,port)) to nbytes = sock.send(data) Now, rerun your tests on your network. I expect you'll be faster now because TCP can be pretty smart about buffering. Let's say you write 16, 90B blocks to the socket. If they are timely enough, it is possible all of those will be shipped across ethernet as a single frame. So what took 16 frames via UDP can now *potentially* be done in a single ethernet frame (assuming 1500MTU). I say potentially because the exact behaviour is OS/stack and NIC-driver specific and is often tunable to boot. Likewise, on the client end, what previously required 15 calls to recvfrom, each returning 90B, can *potentially* be completed in a single call to recv, returning 1440B. Remember, fewer frames means less protocol overhead which makes more bandwidth available to your applications. When sending 90B datagrams, you're waisting over 48% of your available bandwidth because of protocol overhead (actually a lot more because I'm not accounting for UDP headers). Because of the differences between UDP and TCP, unlike your original UDP implementation which can receive from multiple clients, the TCP implementation can only receive from a single client. If you need to receive from multiple clients concurrently, look at python's select module to take up the slack. Hopefully you'll be up and running. Please report back your findings. I'm curious as to your results. I've been out of online for a while. Anyway, clarifing few things: - I am running on IO-bound - we have 10/100/1000MB ethernet, and 10/100MB switches, routers, servers, - the MTU is default 1500 I know what you are saying regarding TCP. I was using it in another project. However this project needs to be done using UDP and can't be changed :( Was testing today multiple approaches to client. Kept one similar to the above one, rewrote one using threads and found another issue. The speed is pretty acceptable, but there is an issue with sending 1.000.000 packets per client (it does it within around 1.5min). It runs from a client machine to the server machine, both on the same network. So, when sending milion packets only around 50%-70% are send over. On the client machine it looks like all the packets where transmitted however tcpdump running on the server shows that only 50-70% went through. I
Re: Problem with writing fast UDP server
On Nov 21, 3:52 am, Jean-Paul Calderone [EMAIL PROTECTED] wrote: Start the server before the client. If you want to try this program out on POSIX, make sure you change the time.clock() calls to time.time() calls instead, otherwise the results aren't very meaningful. I gave this a try on an AMD64 3200+ running a 32 bit Linux installation. Here's the results I got on the server: Jean-Paul, thanks very much for the code snippets. I have tried this out with number of tests. Both approaches are working fine when I run them on the same machine. When I run the client (twisted version) on another machine and the udp server (non twisted) on server machine, I got some strange behaviour. The client send 255 messages and than it cause an error: socket.error: (11, 'Resource temporarily unavailable') Any idea what could be wrong? -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with writing fast UDP server
On Nov 21, 5:49 am, Greg Copeland [EMAIL PROTECTED] wrote: On Nov 20, 9:03 am, Krzysztof Retel [EMAIL PROTECTED] wrote: Hi guys, I am struggling writing fast UDP server. It has to handle around 1 UDP packets per second. I started building that with non blocking socket and threads. Unfortunately my approach does not work at all. I wrote a simple case test: client and server. The client sends 2200 packets within 0.137447118759 secs. The tcpdump received 2189 packets, which is not bad at all. But the server only handles 700 -- 870 packets, when it is non- blocking, and only 670 – 700 received with blocking sockets. The client and the server are working within the same local network and tcpdump shows pretty correct amount of packets received. I included a bit of the code of the UDP server. class PacketReceive(threading.Thread): def __init__(self, tname, socket, queue): self._tname = tname self._socket = socket self._queue = queue threading.Thread.__init__(self, name=self._tname) def run(self): print 'Started thread: ', self.getName() cnt = 1 cnt_msgs = 0 while True: try: data = self._socket.recv(512) msg = data cnt_msgs += 1 total += 1 # self._queue.put(msg) print 'thread: %s, cnt_msgs: %d' % (self.getName(), cnt_msgs) except: pass I was also using Queue, but this didn't help neither. Any idea what I am doing wrong? I was reading that Python socket modules was causing some delays with TCP server. They recomended to set up socket option for nondelays: sock.setsockopt(SOL_TCP, TCP_NODELAY, 1) . I couldn't find any similar option for UDP type sockets. Is there anything I have to change in socket options to make it working faster? Why the server can't process all incomming packets? Is there a bug in the socket layer? btw. I am using Python 2.5 on Ubuntu 8.10. Cheers K First and foremost, you are not being realistic here. Attempting to squeeze 10,000 packets per second out of 10Mb/s (assumed) Ethernet is not realistic. The maximum theoretical limit is 14,880 frames per second, and that assumes each frame is only 84 bytes per frame, making it useless for data transport. Using your numbers, each frame requires (90B + 84B) 174B, which works out to be a theoretical maximum of ~7200 frames per second. These are obviously some rough numbers but I believe you get the point. It's late here, so I'll double check my numbers tomorrow. In your case, you would not want to use TCP_NODELAY, even if you were to use TCP, as it would actually limit your throughput. UDP does not have such an option because each datagram is an ethernet frame - which is not true for TCP as TCP is a stream. In this case, use of TCP may significantly reduce the number of frames required for transport - assuming TCP_NODELAY is NOT used. If you want to increase your throughput, use larger datagrams. If you are on a reliable connection, which we can safely assume since you are currently using UDP, use of TCP without the use of TCP_NODELAY may yield better performance because of its buffering strategy. Assuming you are using 10Mb ethernet, you are nearing its frame- saturation limits. If you are using 100Mb ethernet, you'll obviously have a lot more elbow room but not nearly as much as one would hope because 100Mb is only possible when frames which are completely filled. It's been a while since I last looked at 100Mb numbers, but it's not likely most people will see numbers near its theoretical limits simply because that number has so many caveats associated with it - and small frames are its nemesis. Since you are using very small datagrams, you are wasting a lot of potential throughput. And if you have other computers on your network, the situation is made yet more difficult. Additionally, many switches and/or routes also have bandwidth limits which may or may not pose a wall for your application. And to make matters worse, you are allocating lots of buffers (4K) to send/receive 90 bytes of data, creating yet more work for your computer. Options to try: See how TCP measures up for you Attempt to place multiple data objects within a single datagram, thereby optimizing available ethernet bandwidth You didn't say if you are CPU-bound, but you are creating a tuple and appending it to a list on every datagram. You may find allocating smaller buffers and optimizing your history accounting may help if you're CPU-bound. Don't forget, localhost does not suffer from frame limits - it's basically testing your memory/bus speed If this is for local use only, considering using a different IPC mechanism - unix domain sockets or memory mapped files Greg, thanks very much for your reply. I am not sure what do
Re: Problem with writing fast UDP server
On Nov 21, 4:48 pm, Peter Pearson [EMAIL PROTECTED] wrote: On Fri, 21 Nov 2008 08:14:19 -0800 (PST), Krzysztof Retel wrote: I am not sure what do you mean by CPU-bound? How can I find out if I run it on CPU-bound? CPU-bound is the state in which performance is limited by the availability of processor cycles. On a Unix box, you might run the top utility and look to see whether the %CPU figure indicates 100% CPU use. Alternatively, you might have a tool for plotting use of system resources. -- To email me, substitute nowhere-spamcop, invalid-net. Thanks. I run it without CPU-bound -- http://mail.python.org/mailman/listinfo/python-list
Problem with writing fast UDP server
Hi guys, I am struggling writing fast UDP server. It has to handle around 1 UDP packets per second. I started building that with non blocking socket and threads. Unfortunately my approach does not work at all. I wrote a simple case test: client and server. The client sends 2200 packets within 0.137447118759 secs. The tcpdump received 2189 packets, which is not bad at all. But the server only handles 700 -- 870 packets, when it is non- blocking, and only 670 – 700 received with blocking sockets. The client and the server are working within the same local network and tcpdump shows pretty correct amount of packets received. I included a bit of the code of the UDP server. class PacketReceive(threading.Thread): def __init__(self, tname, socket, queue): self._tname = tname self._socket = socket self._queue = queue threading.Thread.__init__(self, name=self._tname) def run(self): print 'Started thread: ', self.getName() cnt = 1 cnt_msgs = 0 while True: try: data = self._socket.recv(512) msg = data cnt_msgs += 1 total += 1 # self._queue.put(msg) print 'thread: %s, cnt_msgs: %d' % (self.getName(), cnt_msgs) except: pass I was also using Queue, but this didn't help neither. Any idea what I am doing wrong? I was reading that Python socket modules was causing some delays with TCP server. They recomended to set up socket option for nondelays: sock.setsockopt(SOL_TCP, TCP_NODELAY, 1) . I couldn't find any similar option for UDP type sockets. Is there anything I have to change in socket options to make it working faster? Why the server can't process all incomming packets? Is there a bug in the socket layer? btw. I am using Python 2.5 on Ubuntu 8.10. Cheers K -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with writing fast UDP server
On Nov 20, 3:34 pm, Hrvoje Niksic [EMAIL PROTECTED] wrote: Krzysztof Retel [EMAIL PROTECTED] writes: But the server only handles 700 -- 870 packets, when it is non- blocking, and only 670 – 700 received with blocking sockets. What are your other threads doing? Have you tried the same code without any threading? I have only this one thread, which I can run couple of times. I tried without a threading and was the same result, not all packets were processed. -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with writing fast UDP server
On Nov 20, 4:00 pm, [EMAIL PROTECTED] wrote: On 20 Nov, 16:03, Krzysztof Retel [EMAIL PROTECTED] wrote: Hi guys, I am struggling writing fast UDP server. It has to handle around 1 UDP packets per second. I started building that with non blocking socket and threads. Unfortunately my approach does not work at all. I wrote a simple case test: client and server. The client sends 2200 packets within 0.137447118759 secs. The tcpdump received 2189 packets, which is not bad at all. But the server only handles 700 -- 870 packets, when it is non- blocking, and only 670 – 700 received with blocking sockets. The client and the server are working within the same local network and tcpdump shows pretty correct amount of packets received. I included a bit of the code of the UDP server. class PacketReceive(threading.Thread): def __init__(self, tname, socket, queue): self._tname = tname self._socket = socket self._queue = queue threading.Thread.__init__(self, name=self._tname) def run(self): print 'Started thread: ', self.getName() cnt = 1 cnt_msgs = 0 while True: try: data = self._socket.recv(512) msg = data cnt_msgs += 1 total += 1 # self._queue.put(msg) print 'thread: %s, cnt_msgs: %d' % (self.getName(), cnt_msgs) except: pass I was also using Queue, but this didn't help neither. Any idea what I am doing wrong? I was reading that Python socket modules was causing some delays with TCP server. They recomended to set up socket option for nondelays: sock.setsockopt(SOL_TCP, TCP_NODELAY, 1) . I couldn't find any similar option for UDP type sockets. Is there anything I have to change in socket options to make it working faster? Why the server can't process all incomming packets? Is there a bug in the socket layer? btw. I am using Python 2.5 on Ubuntu 8.10. Cheers K Stupid question: did you try removing the print (e.g. printing once every 100 messages) ? :) Of course I did Nothing has changed I wonder if there is a kind of setting for socket to allow no delays? -- http://mail.python.org/mailman/listinfo/python-list
