Re: simultaneous copy to multiple media
I need a program that simultaneously can copy a single file (1 GB) from my pc to multiple USB-harddrives. Sounds like a pretty simple simple python script: --- #!/bin/env python def spew(source_file_name, output_file_names, block_size = 8*1024): source = file(source_file_name, rb) output_files = [file(s, wb) for s in output_file_names] s = source.read(block_size) while s: for outfile in output_files: outfile.write(s) s = source.read(block_size) for outfile in output_files: outfile.close() source.close() if __name__ == '__main__': from sys import argv if len(argv) 2: spew(argv[1], argv[2:]) else: #print some usage/help print %s source_file dest_file1 [dest_file2 [dest_file3 [...]]] % argv[0] --- Just call it with bash cd /mnt bash python spew.py file1.txt usb1/file1.txt usb2/file2.txt for as many destinations as you want. Adjust the blocksize as you see fit: spew(argv[1], argv[2:], 1024*1024) or whatever your favorite blocksize is. It's not terribly graceful about error-checking, or prompting if any of the files exist previously, or if it can't write to any of the output files because for any reason (full disk, write-protected disk, destination path doesn't exist, etc). Thus, if there's a problem, you'll get a pretty backtrace that will allow you to solve all your problems ;) -tkc -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
[EMAIL PROTECTED] wrote: I need a program that simultaneously can copy a single file (1 GB) from my pc to multiple USB-harddrives. Why not just use: copy c:\file.ext u:\file.exe in one shell and in another copy c:\file.ext v:\file.exe where u: and v: are the USB drives? There is usually not much gain on USB when on Windows, especially in case of a single file which fits into memory (and the file cache) to do it 'simultaneously'. This subject was discussed here already in the past: http://mail.python.org/pipermail/python-list/2005-March/271985.html Claudio Grondi I have searched the internet and only found one program that does this on http://mastermind.com.pl/multicopy/ but this link doesnt work anymore somebody that can help me, is there any other programs out there. -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
On Mon, 21 Mar 2005 00:17:05 +0100, rumours say that Heiko Wundram [EMAIL PROTECTED] might have written: Well, 430 MB/s is only for USB 2.0. It's 480 Mb/s (megabit or mebibit, I am not sure... :), so it maxes at about 60 MB/s (or MiB/s) for all devices on the same controller. -- TZOTZIOY, I speak England very best. Be strict when sending and tolerant when receiving. (from RFC1958) I really should keep that in mind when talking with people, actually... -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
Try this: http://mastermind.com.pl/multicopy/ This is small tool I've wrote, that does use large memory buffers with asynchronous I/O to copy file. Thank you! This (with a drawback of blocking the entire system) does it! ( dzieñ dobry i dziêkujê za t± konstruktywn± odpowied¼ na moje pytanie ) At least I know, that it works and you explain how it works, so I am now quite sure, that it will be also possible to have a Python tool which does the same thing. This way the discussion going towards driver hacking for the USB access was in my case just the wrong way, but anyway, thanks also for that to the contributors, because it sharpened my understanding of USB. Here my current test results on my Windows 2000 system with a 2.8 GHz P4 CPU and two 5 1/4 IDE-USB 2.0 casings (I have got from http://www.computeruniverse.net/ a German online shop, article no.: 90052509) by the way: does someone know an IDE-USB case able to perform data transfer faster than the max. 20 MByte/s I experience with one of my another IDE-USB 5 1/4 casings? ? ) : Single file copy: USB-1 to USB-2 = 11 MB/s, max. 70% CPU USB-2 to USB-1 = 11 MB/s, max. 70% CPU E-IDE to USB-1 = 12 MB/s, max. 70% CPU (first time) E-IDE to USB-1 = 13 MB/s, max. 70% CPU (second time same file) E-IDE to E-IDE = max. 45 MB/s (two physical drives) Simultaneous file copy: USB-1 to USB-2 = 5 MB/s USB-2 to USB-1 = 7 MB/s --- 12 MB/s, max. 80% CPU Simultaneous file copy using http://mastermind.com.pl/multicopy/release/1.0.0/multicopy.exe : E-IDE to USB-1 = 10 MB/s E-IDE to USB-2 = 10 MB/s --- 20 MB/s, unknown CPU load (1) (1) work with the PC during copying not possible, system hangs from 5 to 15 seconds between e.g. displaying current system time (with Windows clock). From my point of view this thread has reached its end (I have a solution I can live with), except if someone would like to contribute or point to a better multicopy.exe which does not block the system or best to contribute or point to a _Python_ script or module which is able to find out the optimal buffer size for copying on the current system (and the best way of copying files), so that after this information is saved in an .INI file the tool is best adopted to the system it works on. I am quite sure, that there is a perfect ready-to-use solution out there (up to now I just only failed to find one by Googling, e.g. the tee.exe provided by http://unxutils.sourceforge.net/ doesn't work with large files at all), so please don't hesitate to provide it, so, that I don't need to reinvent the wheel. Claudio Jacek Trzmiel [EMAIL PROTECTED] schrieb im Newsbeitrag news:[EMAIL PROTECTED] Claudio Grondi wrote: I am on a Widows 2000 box using the NTFS file system. Both up to now suggested approaches as - tee.exe (where I used the http://david.tribble.com/dos/tee.exe DOS-port with redirecting stdout to NULL) and - parallel copy (hoping caching does the job) are by far slower than the consecutive copy: single copy speed 12-15 MByte/s which gives effectively 6-7 MByte/s, tee.exe or twice copy in parallel 1-3 MByte/s. Any other suggestions except writing an own optimised version of tee.exe Try this: http://mastermind.com.pl/multicopy/ This is small tool I've wrote, that does use large memory buffers with asynchronous I/O to copy file. Following command: multicopy c:\testfile d:\testfile e:\testfile f:\testfile will copy c:\testfile to d, e and f disks. With four separate IDE disks I can copy file at about 30MB/s, which means 120MB/s total I/O. You can give it a try, but I don't know if it will work fast with USB drives. HTH, sc0rp. -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
This is small tool I've wrote, that does use large memory buffers with asynchronous I/O to copy file. Claudio Grondi wrote: Thank you! This (with a drawback of blocking the entire system) does it! ( dzie dobry i dzikuj za t konstruktywn odpowied na moje pytanie ) :) From my point of view this thread has reached its end (I have a solution I can live with), except if someone would like to contribute or point to a better multicopy.exe which does not block the system Symptoms (high cpu usage, unresponsive system) look similar to situation when you try to read/write as fast as possible from/to IDE drive running in PIO mode. I think that it's either USB driver problem, or inherent design flaw in USB (anyone?). Anyway, I've added buffersize and sleeptime options to multicopy, so you may try to throttle it down. Download it here: http://mastermind.com.pl/multicopy/ HTH, sc0rp. -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
On Mon, 21 Mar 2005 17:53:34 +0100, Jacek Trzmiel [EMAIL PROTECTED] wrote: This is small tool I've wrote, that does use large memory buffers with asynchronous I/O to copy file. Claudio Grondi wrote: Thank you! This (with a drawback of blocking the entire system) does it! ( dzieñ dobry i dziêkujê za t± konstruktywn± odpowied¼ na moje pytanie ) :) From my point of view this thread has reached its end (I have a solution I can live with), except if someone would like to contribute or point to a better multicopy.exe which does not block the system Symptoms (high cpu usage, unresponsive system) look similar to situation when you try to read/write as fast as possible from/to IDE drive running in PIO mode. I think that it's either USB driver problem, or inherent design flaw in USB (anyone?). Anyway, I've added buffersize and sleeptime options to multicopy, so you may try to throttle it down. Download it here: http://mastermind.com.pl/multicopy/ What if some disks could benefit from running ahead a few buffers while others are hanging slowed by e.g. allocation and seeking activity? ISTM there could be a benefit to keeping a multibuffer readahead window of the source stream going? (I didn't look at your code, maybe you do this? Also maybe a particular OS might do this for you so that using several open source streams coming from the same datafile would automatically share system readahead buffer info if reads stayed within a few buffers of each other. Do you us multiple open readonly files as source streams? Or are OS file systems so dumb they don't notice shareability of temporarily memory-resident readonly file data buffers? I guess it would vary, and you could lose or gain by single or multifile source strategy, depending ;-) Regards, Bengt Richter -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
On Sun, 20 Mar 2005 11:41:10 -, Claudio Grondi [EMAIL PROTECTED] wrote: I would like to save time copying the same file (6 GByte) to various different target storage media connected to the system via USB. Is there a (Python or other) tool able to help me to do this, so that I don't need to copy the source file first to the first media, then to the second, etc.? If you are using *nix, maybe you can use the `tee` command[1] and redirect the file to different places. For example, cat somefile | tee file1 | tee file2 file3 I haven't tried it out but it should work. [1]: http://unixhelp.ed.ac.uk/CGI/man-cgi?tee HTH, -- Swaroop C H Blog: http://www.swaroopch.info Book: http://www.byteofpython.info -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
Claudio Grondi wrote: I would like to save time copying the same file (6 GByte) to various different target storage media connected to the system via USB. Is there a (Python or other) tool able to help me to do this, so that I don't need to copy the source file first to the first media, then to the second, etc.? Use 'tee' command from shell: tee file /dest1/file /dest2/file Or run two copies in parallel (one of them probably will read file from cache): cp file /dest1/file cp file /dest2/file -- Jacek. -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
On 2005-03-20, Claudio Grondi [EMAIL PROTECTED] wrote: Is there maybe a way to use a direct DMA transfer to multiple target destinations (I copy to drives connected via USB ports) ? Sure, but you'll have to throw away your OS, and write a program that runs on the bare metal. Can someone point me in the right direction? I think they already did. -- Grant Edwards grante Yow! I am NOT a nut at visi.com -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
means your message, that you think, that the consecutive copy is the fastest possible method if using Windows 2000? Claudio Grant Edwards [EMAIL PROTECTED] schrieb im Newsbeitrag news:[EMAIL PROTECTED] On 2005-03-20, Claudio Grondi [EMAIL PROTECTED] wrote: Is there maybe a way to use a direct DMA transfer to multiple target destinations (I copy to drives connected via USB ports) ? Sure, but you'll have to throw away your OS, and write a program that runs on the bare metal. Can someone point me in the right direction? I think they already did. -- Grant Edwards grante Yow! I am NOT a nut at visi.com -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
On Sunday 20 March 2005 17:16, Claudio Grondi wrote: Is there maybe a way to use a direct DMA transfer to multiple target destinations (I copy to drives connected via USB ports) ? Think about what USB stands for. Then reconsider whether you'll ever have the chance of writing truly simultaneously to several devices connected via USB... And then, as an extra exercise, think about why it takes so long when several different jobs are done in parallel, writing to devices connected via USB. Bus design... Don't they teach anything at uni these days? ;) -- --- Heiko. pgpMCGCKFMjC0.pgp Description: PGP signature -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
Swaroop C H wrote: [...] If you are using *nix, maybe you can use the `tee` command[1] and redirect the file to different places. For example, cat somefile | tee file1 | tee file2 file3 On Unixes I know, only 1 process is needed: infile tee outfile1 outfile2 outfile3 ... /dev/null It does work! ** Rolf I haven't tried it out but it should work. [1]: http://unixhelp.ed.ac.uk/CGI/man-cgi?tee HTH, -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
I don't know any deep details about USB, except that I _know_ that it is a serial bus, but considering following: 1) I can read/write 45 MByte/s from harddrive to harddrive on the E-IDE bus (theoretically 100 MByte/s), so the speed of the harddrive I read/write from/to is probably the bottleneck. 2) if I understand it right, an USB controller is connected to to the PCI bus and there can be many separate USB controller on one PC 3) the theoreticall speed of USB (430 MByte/s?) is much higher as the experienced 15 MByte/s, probably due to slow controller on the side of the external storage media, so maybe even on one USB line I will have the chance to use it full capacity connecting many slow devices (I can't imagine, that USB goes down with the speed to the speed of the slowest component connected, does it?) 4) with 45 MByte/s I could theoretically simultaneously supply three separate data streams to three USB controller writing with 15 MByte/s each out to the external storage media. 5) watching the low CPU usage while writing to USB shows me, that CPU-time is not a problem. I still see a chance of writing truly simultaneously to several devices connected via USB. Please let me know if I am wrong in any of the points (1-5) or if my conclusion is based on wrong assumptions or if I use a wrong way of infering. By the way: I was not thought about bus design at university, so what I know about it is comes from self-study :-). Claudio P.S. Your message appears in my Outlook Express as one having no content, but an attachment, so I can't read it directly. Is there maybe a way to use a direct DMA transfer to multiple target destinations (I copy to drives connected via USB ports) ? Think about what USB stands for. Then reconsider whether you'll ever have the chance of writing truly simultaneously to several devices connected via USB... And then, as an extra exercise, think about why it takes so long when several different jobs are done in parallel, writing to devices connected via USB. Bus design... Don't they teach anything at uni these days? ;) -- --- Heiko. -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
2) if I understand it right, an USB controller is connected to to the PCI bus and there can be many separate USB controller on one PC True. 3) the theoreticall speed of USB (430 MByte/s?) is much higher as the experienced 15 MByte/s, probably due to slow controller on the side of the external storage media, so maybe even on one USB line I will have the chance to use it full capacity connecting many slow devices (I can't imagine, that USB goes down with the speed to the speed of the slowest component connected, does it?) Nope. Or yes. Here comes into play why Heiko said think of what USB stands for. While the devices appear to be responsive concurrently (the bus arbitration and multiplexing/demultiplexing is tranparent to the user), it still is a serial bus, so at a given point in time you can only write to _one_ device. And if that device is limited in its speed, you're doomed. The only thing I could imagine is that if you send small bursts that don't make the write cache full, it _might_ be possible to switch fast between the devices and thus write the data faster, as the devices can try and write the data to disk while the others are feeded. But I'm on no sure ground here with that assumption, and it certainly would depend on how the usb-ata-controller deals with that. You said yourself that you only had 15MB/sec, so it doesn't look to good. To make this work would certainly be a deep lowlever driver hack and nowhere in the scope of python per se. 4) with 45 MByte/s I could theoretically simultaneously supply three separate data streams to three USB controller writing with 15 MByte/s each out to the external storage media. That might be the case if you use three real separate controllers. But that will depend on how the OS deals with these, if you really achieve the theoretical throughoutput. 5) watching the low CPU usage while writing to USB shows me, that CPU-time is not a problem. That is certainly true - the bottleneck are the buses. -- Regards, Diez B. Roggisch -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
Nope. Or yes. Here comes into play why Heiko said think of what USB stands for. While the devices appear to be responsive concurrently (the bus arbitration and multiplexing/demultiplexing is tranparent to the user), it still is a serial bus, so at a given point in time you can only write to _one_ device. And if that device is limited in its speed, you're doomed. The only thing I could imagine is that if you send small bursts that don't make the write cache full, it _might_ be possible to switch fast between the devices and thus write the data faster, as the devices can try and write the data to disk while the others are feeded. I had second thoughts of this, and have to admit that I don't make too much senes here. So ignore that. But I'm on no sure ground here with that assumption, and it certainly would depend on how the usb-ata-controller deals with that. You said yourself that you only had 15MB/sec, so it doesn't look to good. To make this work would certainly be a deep lowlever driver hack and nowhere in the scope of python per se. -- Regards, Diez B. Roggisch -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
Am Sonntag, 20. März 2005 23:16 schrieb Claudio Grondi: 2) if I understand it right, an USB controller is connected to to the PCI bus and there can be many separate USB controller on one PC Yes, there may be more than one USB-controller in a PC, but this doesn't matter, they are all connected to the same bus (the PCI bus). This is a parallel bus, but also a bus which can only serve a single endpoint at a time. The frequency is normally 66 Mhz, so you get a theoretical 66*32 Mb/s over the PCI bus. But, this is only theoretical usage. Consider the EIDE-controller trying to write data into DMA memory which it has read from disk. It has to do the following to communicate with memory: 1) signal the DMA chip (which is also a device on the PCI bus) that it has data. This implies grabbing the PCI bus, making sure noone else is talking at the moment. 2) pass the data, 32 bit at a time, to the DMA chip, which places this data into main memory. 3) you may not grab the bus too long, as other devices might also want to communicate with some other device (e.g. the PIC, to signal an interrupt to the CPU). Thus, you have to free the bus after some time, and you have to forcibly give free the bus when some other device signals an interrupt. This also means starting transmission again after the bus has been successfully reacquired. 4) finally, after communication with the DMA-chip is done, signal the PIC via an interrupt that you wish the CPU to be signaled that it should do a software interrupt, so that the OS interrupt handler can be scheduled and can take the data from the DMA region (and do something with it). Now, even when the bus is fast, setting it up all still takes time, so you never get the theoretical throughput you might wish to see. Accessing the USB controller does the game the other way around; the OS uses the DMA-chip to stream the data onto the PCI bus for the controller to transmit. So, overall, adding more USB controllers to your PC only adds extra throughput until you reach the top limit of PCI (when the controllers start stealing bandwith from each other). 3) the theoreticall speed of USB (430 MByte/s?) is much higher as the experienced 15 MByte/s, probably due to slow controller on the side of the external storage media, so maybe even on one USB line I will have the chance to use it full capacity connecting many slow devices (I can't imagine, that USB goes down with the speed to the speed of the slowest component connected, does it?) Well, 430 MB/s is only for USB 2.0. AFAIK, most devices (esp. storage devices) are still only USB 1.1 compliant, which makes this rate go down to a mere 40 MB/s or something close. 4) with 45 MByte/s I could theoretically simultaneously supply three separate data streams to three USB controller writing with 15 MByte/s each out to the external storage media. No, you can't. As I said in response to point 2, it doesn't matter what kind of bus you use, when the bus isn't meant to do real broadcasts or can actually talk to several devices simultaneously because all wires are duplicated a corresponding number of times, there's always overhead when switching between the devices. The bus must be acquired for communication with one device (so that the others won't simply go blabbering over the communication), then the data must be streamed, then the bus released again. This process takes time, and AFAIK, USB is pretty expensive (compared to data transmission speed) here, as USB guarantees that a packet reaches the endpoint when its transmitted over the wire (in the sense that it won't get lost because someone else was also talking). Now, when more devices come into play, this overhead becomes significant, and takes away quite some bandwith. For example, Ethernet takes another way. Ethernet simply says: when I have data which is transmitted while someone else was talking, throw away the resulting mess (a collision, remember those cool blinking lights on hubs?). Someone will retransmit it if it was important. 5) watching the low CPU usage while writing to USB shows me, that CPU-time is not a problem. No, it most certainly isn't. That's why modern computers always use a DMA-chip to do the heavy work of dealing with the hardware on external busses... ;) P.S. Your message appears in my Outlook Express as one having no content, but an attachment, so I can't read it directly. Err, that's pretty strange. I can read my KMail written emails in pretty much any client that I have access to... So I guess this is a problem with Outlook Express not understanding MIME correctly, and not a problem on my side... ;) And, don't worry, I won't switch to windows sucks-mode now. ;) -- --- Heiko. pgp6Uj3A40bZ4.pgp Description: PGP signature -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
Claudio Grondi wrote: I am on a Widows 2000 box using the NTFS file system. Both up to now suggested approaches as - tee.exe (where I used the http://david.tribble.com/dos/tee.exe DOS-port with redirecting stdout to NULL) and - parallel copy (hoping caching does the job) are by far slower than the consecutive copy: single copy speed 12-15 MByte/s which gives effectively 6-7 MByte/s, tee.exe or twice copy in parallel 1-3 MByte/s. Any other suggestions except writing an own optimised version of tee.exe Try this: http://mastermind.com.pl/multicopy/ This is small tool I've wrote, that does use large memory buffers with asynchronous I/O to copy file. Following command: multicopy c:\testfile d:\testfile e:\testfile f:\testfile will copy c:\testfile to d, e and f disks. With four separate IDE disks I can copy file at about 30MB/s, which means 120MB/s total I/O. You can give it a try, but I don't know if it will work fast with USB drives. HTH, sc0rp. -- http://mail.python.org/mailman/listinfo/python-list
Re: simultaneous copy to multiple media
Well, 430 MB/s is only for USB 2.0. AFAIK, most devices (esp. storage devices) are still only USB 1.1 compliant, which makes this rate go down to a mere 40 MB/s or something close. I think it is 430 MBit(!), so about 50MB per second. The old usb 1.1 has 12MBits. So I don't think that Claudio could observe 15MByte/s with a 1.1 bus 4) with 45 MByte/s I could theoretically simultaneously supply three separate data streams to three USB controller writing with 15 MByte/s each out to the external storage media. No, you can't. As I said in response to point 2, it doesn't matter what kind of bus you use, when the bus isn't meant to do real broadcasts or can actually talk to several devices simultaneously because all wires are duplicated a corresponding number of times, there's always overhead when switching between the devices. Well, if the bus is faster, it can (and does) use multiplexing. All busses do that, apart from crossbar switches which have n:n point2point connectivity. The bus must be acquired for communication with one device (so that the others won't simply go blabbering over the communication), then the data must be streamed, then the bus released again. This process takes time, and AFAIK, USB is pretty expensive (compared to data transmission speed) here, as USB guarantees that a packet reaches the endpoint when its transmitted over the wire (in the sense that it won't get lost because someone else was also talking). Now, when more devices come into play, this overhead becomes significant, and takes away quite some bandwith. USB uses a host-based bus arbitration scheme where only the controller decides who can talk when. So there is very limited bus arbitration overhead. But of course this also means that the host has grant each attached device bus time every now and then, as the device itself has no chance of signalling data availability itself. So - that adds again some overhead :) In the end, I'm not sure what throughoutput is actually possible. As I said to Claudio in a private mail: Apart from multithreaded writing to the bus and _hoping_ that things speed up one can't do much - at least not in python, and not without deep driver fiddling or even writing drivers oneself. -- Regards, Diez B. Roggisch -- http://mail.python.org/mailman/listinfo/python-list
