Re: Newbie prob: How to write a file with 3 threads?
est wrote: > I'd like to say VERY VERY VERY thank you for your detailed > information, that's a lot encourage for a beginner. In fact I am > writing a multi thread download ultility, and I found that very > hard for me. You don't need any system threads for multiple download threads. Since network connections are buffered and much slower than CPU cycles, you can also use event based mechanisms. (I've written 30+ connection servers like this already.) The general pseudocode strategy is while True: waitForEvents() event = getEvent() processEvent(event) # in here, you inspect the event, # e. g. from which connection it is, # and get or write data The only drawback to event based programming is that processor cycle "rationing" is decided in your code and not by the OS, which can make difficulties with really long calculations. But if these problems arise you usually spawn a seperate worker thread :) > Can you recommand any sample code where I can start > with? If you choose event based programming, definitely have a look at Twisted. It lets you create multiple connection clients and servers in few code lines without synchronisation hassle. http://twistedmatrix.com/projects/core/documentation/howto/ http://twistedmatrix.com/projects/core/documentation/howto/clients.html (BTW, don't get scared by "Deferred"s, I never explicitly used them in my code ... though they often work in the background) Regards, Björn -- BOFH excuse #376: Budget cuts forced us to sell all the power cords for the servers. -- http://mail.python.org/mailman/listinfo/python-list
Re: Newbie prob: How to write a file with 3 threads?
On May 8, 2:13 pm, Dennis Lee Bieber <[EMAIL PROTECTED]> wrote: > On 7 May 2007 18:22:07 -0700, est <[EMAIL PROTECTED]> declaimed > the following in comp.lang.python: > > > > > I guess I will write multiple files, one thread one file, and after > > all threads terminates, I combine the files. That's a cheaper > > solution, I think. > > Given your first description: > > > I need to write a file using 3 threads simutaniously, e.g. Thread 1 > > write the first byte of test.bin with an "a", second thread write the > > second byte "b", third thread write the third byte "c". Anyone could > > give a little example on how to do that? > > ... any other solution may not have worked anyway. That is, if you > really expect the three threads to interleave their output as > abcabcabcabc. If threading, you have no assurance that any single thread > will follow any other during a task switch. It all depends upon where a > task switch takes place. > > But then, your example isn't too clear of what you really are > producing for output. If, instead of "bytes", you meant that each thread > was writing logging information, the solution would be to use the > logging module -- so far as I know, the logging module /does/ perform > the needed access locking. > > OTOH, if you really mean to have three separate byte producers, > interleaving output, the only safe way would be to have /each/ have an > output Queue, and a fourth thread doing the writing using a loop of the > form: > > while True: > a = aQueue.get() > fout.write(a) > b = bQueue.get() > fout.write(b) > c = cQueue.get() > fout.write(c) > > Using the separate queues means that the writer WILL wait until the > next producer in the interleave has produced its data. Of course, this > structure has the drawback that all producers must produce the same > amount of data -- otherwise it blocks forever on the queue from the > producer has stopped generating data. > -- > WulfraedDennis Lee Bieber KD6MOG > [EMAIL PROTECTED] [EMAIL PROTECTED] > HTTP://wlfraed.home.netcom.com/ > (Bestiaria Support Staff: [EMAIL PROTECTED]) > HTTP://www.bestiaria.com/ I'd like to say VERY VERY VERY thank you for your detailed information, that's a lot encourage for a beginner. In fact I am writing a multi thread download ultility, and I found that very hard for me. Can you recommand any sample code where I can start with? I hope I can catch up with everyone here, I'll try my best learning python. Thank you again. -- http://mail.python.org/mailman/listinfo/python-list
Re: Newbie prob: How to write a file with 3 threads?
On May 8, 1:29 am, Dennis Lee Bieber <[EMAIL PROTECTED]> wrote: > On 7 May 2007 00:13:46 -0700, est <[EMAIL PROTECTED]> declaimed > the following in comp.lang.python: > > > > > I'll try Queue.Queue, thank you. I didn't expect that multithread > > write a file is so troublesome > > Multiple writers to a file, threaded or not (ie, multiple processes, > hosts on network, etc.), is always a pain unless one can ensure that all > writers use a common locking mode to prevent overlapping calls. If all > the writers are internal to one program, one can implement internal > locks -- if external one needs OS support for multi-process locking (VMS > common event flag clusters, for example). Easier to dedicate one > (internal) writer and use the naturally locked Queue to control access. > -- > WulfraedDennis Lee Bieber KD6MOG > [EMAIL PROTECTED] [EMAIL PROTECTED] > HTTP://wlfraed.home.netcom.com/ > (Bestiaria Support Staff: [EMAIL PROTECTED]) > HTTP://www.bestiaria.com/ I guess I will write multiple files, one thread one file, and after all threads terminates, I combine the files. That's a cheaper solution, I think. -- http://mail.python.org/mailman/listinfo/python-list
Re: Newbie prob: How to write a file with 3 threads?
est wrote: > On May 7, 5:12 am, MRAB <[EMAIL PROTECTED]> wrote: > >> On May 6, 9:51 am, Marc 'BlackJack' Rintsch <[EMAIL PROTECTED]> wrote:> In >> <[EMAIL PROTECTED]>, est wrote: >> I need to write a file using 3 threads simutaniously, e.g. Thread 1 write the first byte of test.bin with an "a", second thread write the second byte "b", third thread write the third byte "c". Anyone could give a little example on how to do that? >>> Simplest solution is: don't do that. Write from one thread and send the >>> date from the other threads via a `Queue.Queue` to the writing thread. >>> Send the number of the thread with the data so the writer thread knows in >>> which order the data has to be written. >>> >> [snip] >> Or have a `Queue.Queue` for each source thread and make the writer >> thread read from each in turn. >> > > > I'll try Queue.Queue, thank you. I didn't expect that multithread > write a file is so troublesome > As a general rule, *ALL* multithread operations are at least that troublesome, and most are far more so. Pessimistically-yours, Gary Herron -- http://mail.python.org/mailman/listinfo/python-list
Re: Newbie prob: How to write a file with 3 threads?
On May 7, 5:12 am, MRAB <[EMAIL PROTECTED]> wrote: > On May 6, 9:51 am, Marc 'BlackJack' Rintsch <[EMAIL PROTECTED]> wrote:> In > <[EMAIL PROTECTED]>, est wrote: > > > I need to write a file using 3 threads simutaniously, e.g. Thread 1 > > > write the first byte of test.bin with an "a", second thread write the > > > second byte "b", third thread write the third byte "c". Anyone could > > > give a little example on how to do that? > > > Simplest solution is: don't do that. Write from one thread and send the > > date from the other threads via a `Queue.Queue` to the writing thread. > > Send the number of the thread with the data so the writer thread knows in > > which order the data has to be written. > > [snip] > Or have a `Queue.Queue` for each source thread and make the writer > thread read from each in turn. I'll try Queue.Queue, thank you. I didn't expect that multithread write a file is so troublesome -- http://mail.python.org/mailman/listinfo/python-list
Re: Newbie prob: How to write a file with 3 threads?
On May 6, 9:51 am, Marc 'BlackJack' Rintsch <[EMAIL PROTECTED]> wrote: > In <[EMAIL PROTECTED]>, est wrote: > > I need to write a file using 3 threads simutaniously, e.g. Thread 1 > > write the first byte of test.bin with an "a", second thread write the > > second byte "b", third thread write the third byte "c". Anyone could > > give a little example on how to do that? > > Simplest solution is: don't do that. Write from one thread and send the > date from the other threads via a `Queue.Queue` to the writing thread. > Send the number of the thread with the data so the writer thread knows in > which order the data has to be written. > [snip] Or have a `Queue.Queue` for each source thread and make the writer thread read from each in turn. -- http://mail.python.org/mailman/listinfo/python-list
Re: Newbie prob: How to write a file with 3 threads?
Is it not possible to acomplish this with a token-based algorithm? With 3 Threads: A, B and C Thread A start with Token. Thread with the token writes the byte and send the token to the next Or python has any issues to a file object shared among a few threads? []'s Flavio On 5/6/07, Marc 'BlackJack' Rintsch <[EMAIL PROTECTED]> wrote: In <[EMAIL PROTECTED]>, est wrote: > I need to write a file using 3 threads simutaniously, e.g. Thread 1 > write the first byte of test.bin with an "a", second thread write the > second byte "b", third thread write the third byte "c". Anyone could > give a little example on how to do that? Simplest solution is: don't do that. Write from one thread and send the date from the other threads via a `Queue.Queue` to the writing thread. Send the number of the thread with the data so the writer thread knows in which order the data has to be written. > I have my code, but it makes python intepreter crash everytime on my > Vista. Show minimal (non-)working code, tell us the exception plus traceback and explain "crash". Ciao, Marc 'BlackJack' Rintsch -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Newbie prob: How to write a file with 3 threads?
In <[EMAIL PROTECTED]>, est wrote: > I need to write a file using 3 threads simutaniously, e.g. Thread 1 > write the first byte of test.bin with an "a", second thread write the > second byte "b", third thread write the third byte "c". Anyone could > give a little example on how to do that? Simplest solution is: don't do that. Write from one thread and send the date from the other threads via a `Queue.Queue` to the writing thread. Send the number of the thread with the data so the writer thread knows in which order the data has to be written. > I have my code, but it makes python intepreter crash everytime on my > Vista. Show minimal (non-)working code, tell us the exception plus traceback and explain "crash". Ciao, Marc 'BlackJack' Rintsch -- http://mail.python.org/mailman/listinfo/python-list
Newbie prob: How to write a file with 3 threads?
I need to write a file using 3 threads simutaniously, e.g. Thread 1 write the first byte of test.bin with an "a", second thread write the second byte "b", third thread write the third byte "c". Anyone could give a little example on how to do that? I have my code, but it makes python intepreter crash everytime on my Vista. -- http://mail.python.org/mailman/listinfo/python-list
