Re: Copying memstick image to a USB (flash/thumb) drive
In message 5153a2fd.8020...@sneakertech.com, you wrote: Why exactly is the bs=10240 is there? Wouldn't the default of 512 do just as well? Modern systems can read and write far more than 512 bytes per operation. Sticking with 512 would work perfectly fine, but you'd be imposing an unnecessary bottleneck and the copy would be a lot slower overall. Whether 10K is optimal or not depends on the exact hardware you're messing with (it looks pretty low to me, I'd suggest more like 1M). I agree. And 10 kibibytes is probably quite a bit less than optimal. I have filed the following PR: http://www.freebsd.org/cgi/query-pr.cgi?pr=177431 Thank you for your reply. Regards, rfg ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Copying memstick image to a USB (flash/thumb) drive
I have filed the following PR: http://www.freebsd.org/cgi/query-pr.cgi?pr=177431 Er, don't take my word for law: I have *no* idea if 1M is a good idea for most systems, I'm not even sure if it's optimal for mine. I did a single test with three random values at different orders of magnitude and picked the fastest. I do think that 10k is probably way under the right value, but someone should do proper testing on a variety of hardware before changing all the docs. As for the conv=sync option, I'm not convinced it's necessary either way. I've dd'd zillions of images to various media over the years and have never specified a conv parameter and I've never had problems. I don't think modern systems really care what the end is padded with (provided nothing is corrupt of course). Someone with more experience would need to chime in on this. __ it has a certain smooth-brained appeal ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Copying memstick image to a USB (flash/thumb) drive
On Thu, 28 Mar 2013 04:27:43 -0400 Quartz qua...@sneakertech.com wrote: I have filed the following PR: http://www.freebsd.org/cgi/query-pr.cgi?pr=177431 Er, don't take my word for law: I have *no* idea if 1M is a good idea for most systems, I'm not even sure if it's optimal for mine. I did a single test with three random values at different orders of magnitude and picked the fastest. I do think that 10k is probably way under the right value, but someone should do proper testing on a variety of hardware before changing all the docs. The 1M will work fine, it's way bigger than any physical write. In theory the performance should max out when the block size matches the maximum physical write size of the controller (often 64K), but that assumes zero read latency on the data feed so in practice larger block sizes help, but except for things like tape they don't help much once you pass the device/controller max write block size. -- Steve O'Hara-Smith st...@sohara.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Copying memstick image to a USB (flash/thumb) drive
In message 5153feff.4090...@sneakertech.com, you wrote: I have filed the following PR: http://www.freebsd.org/cgi/query-pr.cgi?pr=177431 Er, don't take my word for law: I didn't. I won't. I have *no* idea if 1M is a good idea Any size which is an exact multiple of the physical block size for the target device should provide performance which is as good as it gets. I googled around and read various comments. Some of these kinds of devices have a physical block size of 64KiB. Some have 128KiB. Some have 256KiB. Some have 1MiB. For all of these devices, seting blocksize to 1MiB will provide optimal performance with at worst only a _relatively_ tiny waste of space. As for the conv=sync option, I'm not convinced it's necessary either way. Neither am I, but I would rather have it there, than not and take chances. It won't hurt anything, and it appears that it _may_ perhaps help. I don't think modern systems really care what the end is padded with That wasn't my concern. My concern is that I personally do not know what the officially defined semantics are in cases where dd is asked to copy data in a specific input block size _and_ the actual data available from the input device doesn't perfectly fill up that last block. It is possible, I would guess, that dd may notice the EOF occuring before it has filled up an entire input buffer, and then just quit at that point, _without_ writing the partial last block to the output device. It seems to me that conv=sync is cheap insurance against this possibility. I have always been a belt and suspenders kind of guy. Regards, rfg ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Copying memstick image to a USB (flash/thumb) drive
On 03/28/13 10:32, Ronald F. Guilmette wrote: In message 5153feff.4090...@sneakertech.com, you wrote: I have filed the following PR: http://www.freebsd.org/cgi/query-pr.cgi?pr=177431 Er, don't take my word for law: I didn't. I won't. I have *no* idea if 1M is a good idea Any size which is an exact multiple of the physical block size for the target device should provide performance which is as good as it gets. I googled around and read various comments. Some of these kinds of devices have a physical block size of 64KiB. Some have 128KiB. Some have 256KiB. Some have 1MiB. For all of these devices, seting blocksize to 1MiB will provide optimal performance with at worst only a _relatively_ tiny waste of space. As for the conv=sync option, I'm not convinced it's necessary either way. Neither am I, but I would rather have it there, than not and take chances. It won't hurt anything, and it appears that it _may_ perhaps help. I don't think modern systems really care what the end is padded with That wasn't my concern. My concern is that I personally do not know what the officially defined semantics are in cases where dd is asked to copy data in a specific input block size _and_ the actual data available from the input device doesn't perfectly fill up that last block. It is possible, I would guess, that dd may notice the EOF occuring before it has filled up an entire input buffer, and then just quit at that point, _without_ writing the partial last block to the output device. It will attempt to write a short block. E.g. arthur@fileserver[2] ls -l t.pdf -rw-rw-r-- 1 arthur arthur 233812 Feb 18 12:26 t.pdf arthur@fileserver[2] dd if=t.pdf of=/dev/null bs=1k 228+1 records in 228+1 records out 233812 bytes transferred in 0.036731 secs (6365521 bytes/sec) Those +1 records are the final short block. It seems to me that conv=sync is cheap insurance against this possibility. It's used as an insurance against output devices which have a fixed (or in the case of tape, a minimum) block size. If the short block is not an exact integer multiple of the device block size then the final write will fail. conv=sync and a bs (or obs) that's a strict multiple of block size prevents the problem. That's exactly your use case. I have always been a belt and suspenders kind of guy. Show me the experienced sysadmin who isn't. Invariably learnt the hard way. :-} ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Copying memstick image to a USB (flash/thumb) drive
In message 51543b7a.4030...@qeng-ho.org, Arthur Chance free...@qeng-ho.org wrote: On 03/28/13 10:32, Ronald F. Guilmette wrote: It is possible, I would guess, that dd may notice the EOF occuring before it has filled up an entire input buffer, and then just quit at that point, _without_ writing the partial last block to the output device. It will attempt to write a short block. E.g. arthur@fileserver[2] ls -l t.pdf -rw-rw-r-- 1 arthur arthur 233812 Feb 18 12:26 t.pdf arthur@fileserver[2] dd if=t.pdf of=/dev/null bs=1k 228+1 records in 228+1 records out 233812 bytes transferred in 0.036731 secs (6365521 bytes/sec) Those +1 records are the final short block. OK. Thanks. That's definitely good to know. I wonder if this behavior is documented in the man page. (It certainly should be, if it isn't already.) It seems to me that conv=sync is cheap insurance against this possibility. It's used as an insurance against output devices which have a fixed (or in the case of tape, a minimum) block size. If the short block is not an exact integer multiple of the device block size then the final write will fail. Right. I understand. conv=sync and a bs (or obs) that's a strict multiple of block size prevents the problem. That's exactly your use case. Right. Regards, rfg ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Copying memstick image to a USB (flash/thumb) drive
I've never used any FreeBSD memstick image before, but now I have reason to do so. I'm reading the instructions for creating a bootable memstick that are located on this page: http://www.freebsd.org/releases/9.1R/announce.html which include the following example of how to perform the copy: # dd if=FreeBSD-9.1-RELEASE-amd64-memstick.img of=/dev/da0 bs=10240 conv=sync Question: Why exactly is conv=sync is there? Also, in practice what does it actually do? (I've used dd plenty in my lifetime, but never found any reason to use conv= at all. I always thought that it was... mostly... just an archaic leftover from the days when some big iron used EBCDIC that needed to get converted to ASCII or vise versa.) The dd man page describes the sync type of conversion thusly: Pad every input block to the input buffer size. Spaces are used for pad bytes if a block oriented conversion value is specified, otherwise NUL bytes are used. Ummm... ok. WTF is a block oriented conversion value? How would I know if I had specified one? Question: Why exactly is the bs=10240 is there? Wouldn't the default of 512 do just as well? Regards, rfg ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Copying memstick image to a USB (flash/thumb) drive
In the last episode (Mar 27), Ronald F. Guilmette said: I've never used any FreeBSD memstick image before, but now I have reason to do so. I'm reading the instructions for creating a bootable memstick that are located on this page: http://www.freebsd.org/releases/9.1R/announce.html which include the following example of how to perform the copy: # dd if=FreeBSD-9.1-RELEASE-amd64-memstick.img of=/dev/da0 bs=10240 conv=sync Question: Why exactly is conv=sync is there? Question: Why exactly is the bs=10240 is there? Wouldn't the default of 512 do just as well? It looks like someone just copied a dd commandline from somewhere else, maybe something to do with tar files (since tar defaults to a 10k blocksize when writing to tape). conv=sync isn't needed since the source file is already a multiple of the target device blocksize (512 bytes), and bs=64k would be much faster when writing to cheap flash devices like USB sticks since they don't have a write cache and individual writes are slowish. -- Dan Nelson dnel...@allantgroup.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Copying memstick image to a USB (flash/thumb) drive
On 28/03/2013 8:10 AM, Ronald F. Guilmette wrote: Question: Why exactly is conv=sync is there? I found this on http://www.mkssoftware.com/docs/man1/dd.1.asp If you specified conv=sync and this input block is smaller than the specified input block size, dd pads it to the specified size with null bytes. When you also specify a block or unblock conversion, dd uses spaces instead of null bytes. So the last block of output gets padded with 0x00 at the end to fill the input block size (from bs=...) Question: Why exactly is the bs=10240 is there? Wouldn't the default of 512 do just as well? It would, but then you would have FILESIZE/512 reads and writes instead of FILESIZE/10240 reads and writes - 20 times more. The end result is the same, but the large bs makes the operation go faster. I routinely use bs=655360 simply because it is a big number divisible by 512, which I can easily remember. 512000 would do just as well, though... Danny ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Copying memstick image to a USB (flash/thumb) drive
Why exactly is the bs=10240 is there? Wouldn't the default of 512 do just as well? Modern systems can read and write far more than 512 bytes per operation. Sticking with 512 would work perfectly fine, but you'd be imposing an unnecessary bottleneck and the copy would be a lot slower overall. Whether 10K is optimal or not depends on the exact hardware you're messing with (it looks pretty low to me, I'd suggest more like 1M). __ it has a certain smooth-brained appeal ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
