Re: Narrowing down problem source
Hi, Richard Owlett wrote: > Block count: 1941504 > ... > 1941504+0 records in > 1941504+0 records out This is ok. > Block count: 2284052 > ... > 1941504+0 records in > 1941504+0 records out This is not ok. The variable "block" was computed correctly, but program "dd" did not retrieve enough blocks. (I wonder whether it's a bug or a feature that it does not report an error. At least this behavior is reproducible.) It seems that the system does not tell "dd" the block count of the new DVD but the count of the previous one. What do you get from this command echo "Device block count: $(expr $(cat /sys/class/block/sr0/size) / 4)" inserted in the script after the line echo "Byte count: $(expr $blocks '*' 2048)" (The /sys filesystem is an inexhaustible source of wonders.) Have a nice day :) Thomas
Re: Narrowing down problem source
Thomas Schmitt wrote: Hi, Richard Owlett wrote: Block count: 1941504 ... 1941504+0 records in 1941504+0 records out This is ok. Block count: 2284052 ... 1941504+0 records in 1941504+0 records out This is not ok. The variable "block" was computed correctly, but program "dd" did not retrieve enough blocks. (I wonder whether it's a bug or a feature that it does not report an error. At least this behavior is reproducible.) It seems that the system does not tell "dd" the block count of the new DVD but the count of the previous one. What do you get from this command echo "Device block count: $(expr $(cat /sys/class/block/sr0/size) / 4)" inserted in the script after the line echo "Byte count: $(expr $blocks '*' 2048)" (The /sys filesystem is an inexhaustible source of wonders.) "Device block count" <> "blocks" HOWEVER, resulting *.iso for DVD1 and DVD2 are the same size even though input sizes were different. Output of script command below: Script started on Mon 21 Sep 2015 08:44:20 AM CDT N=0 F="JessieDVD" E='.iso' while true do let N=N+1 echo "Insert medium, wait until blinking ends, press Enter key (or Ctrl+C to end)" read dummyvar blocks=$(expr $(/sbin/isosize /dev/sr0) / 2048) echo "Block count: "$blocks echo "Byte count: $(expr $blocks '*' 2048)" DEVICEBLKCOUNT=$(expr $(cat /sys/class/block/sr0/size) / 4) echo "Device block count: "$DEVICEBLKCOUNT FILENAME=$F$N$E echo "The output file will be $FILENAME" echo "" dd if=/dev/sr0 bs=2048 count=$DEVICEBLKCOUNT of=$FILENAME echo "" echo "next lines are output of ls and isosize commnds" ls -l $FILENAME /sbin/isosize $FILENAME echo "about to repeat loop" echo "" done ]0;richard@debian-8: ~richard@debian-8:~$ ]0;richard@debian-8: ~richard@debian-8:~$ N=0 ]0;richard@debian-8: ~richard@debian-8:~$ ]0;richard@debian-8: ~richard@debian-8:~$ F="JessieDVD" ]0;richard@debian-8: ~richard@debian-8:~$ ]0;richard@debian-8: ~richard@debian-8:~$ E='.iso' ]0;richard@debian-8: ~richard@debian-8:~$ ]0;richard@debian-8: ~richard@debian-8:~$ while true do let N=N+1 echo "Insert medium, wait until blinking ends, press Enter key (or Ctrl+C to end)" read dummyvar blocks=$(expr $(/sbin/isosize /dev/sr0) / 2048) echo "Block count: "$blocks echo "Byte count: $(expr $blocks '*' 2048)" DEVICEBLKCOUNT=$(expr $(cat /sys/class/block/sr0/size) / 4) echo "Device block count: "$DEVICEBLKCOUNT FILENAME=$F$N$E echo "The output file will be $FILENAME" echo "" dd if=/dev/sr0 bs=2048 count=$DEVICEBLKCOUNT of=$FILENAME echo "" echo "next lines are output of ls and isosize commnds" ls -l $FILENAME /sbin/isosize $FILENAME echo "about to repeat loop" echo "" done Insert medium, wait until blinking ends, press Enter key (or Ctrl+C to end) Block count: 1941504 Byte count: 3976200192 Device block count: 1941504 The output file will be JessieDVD1.iso 1941504+0 records in 1941504+0 records out 3976200192 bytes (4.0 GB) copied, 762.909 s, 5.2 MB/s next lines are output of ls and isosize commnds -rw-r--r-- 1 richard richard 3976200192 Sep 21 08:57 JessieDVD1.iso 3976200192 about to repeat loop Insert medium, wait until blinking ends, press Enter key (or Ctrl+C to end) Block count: 2284052 Byte count: 4677738496 Device block count: 2284064 The output file will be JessieDVD2.iso 1941504+0 records in 1941504+0 records out 3976200192 bytes (4.0 GB) copied, 2655.4 s, 1.5 MB/s next lines are output of ls and isosize commnds -rw-r--r-- 1 richard richard 3976200192 Sep 21 09:43 JessieDVD2.iso 4677738496 about to repeat loop Insert medium, wait until blinking ends, press Enter key (or Ctrl+C to end) ^C ]0;richard@debian-8: ~richard@debian-8:~$ exit Script done on Mon 21 Sep 2015 09:43:50 AM CDT
Re: Narrowing down problem source
Hi, Richard Owlett wrote: > Block count: 2284052 > Device block count: 2284064 This matches. The DVD offers more than the ISO claims. (Just rounded up to a multiple of 16 blocks, in this case.) > 1941504+0 records in > 1941504+0 records out And this is wrong. Why does dd not obey but also not report error ? /bin/dd is from coreutils ... google ... http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=src/dd.c;h=e64729477101f471a31adcb6582d03d2274499cf;hb=HEAD Looks like read(2) gets an End-Of-File and returns 0. (In line 1075, causing the iread() call in line 2127 to get return value 0, which causes end of the copy loop in line 2133, hopping to line 2230. From then on it is treated as successful.) read(2) belongs to libc. The actual work is supposed to happen in the kernel. I will not dive in there. Not yet. Well, if the normal i/o is swimming belly up, lets try whether the drive is willing to hand out the blocks which it announced to the kernel. Replace the dd run by xorriso -indev /dev/sr0 \ -check_media data_to="$FILENAME" -- xorriso will find out the size of the ISO on its own. It will not use system function read(2) but ioctl(SG_IO) and SCSI command READ10. In the end it will report something like Media checks :lba , size , quality Media region : 0 , 13472 , + good Media region : 13472 , 32 , + slow Media region : 13504 , 958590 , + good ... because it is more for finding bad spots than for retrieving image files from medium. Now i am curious how far it gets. Maybe you see some SCSI error replies. If it does not yield the expected number of bytes, please copy as much of the last messages as possible and send them to me in private. Have a nice day :) Thomas
Re: Narrowing down problem source
Thomas Schmitt wrote:
Hi,
My later comments of run-to-run inconsistencies with dd and your recommended
tests were all using my home directory.
In this case any dd run with short result file should issue
some error message and/or there should be some hardware
error message in the system logs.
I captured a test run with the script command.
NOTE dd copies exactly same number of records for differently
sized DVDs :{
Result was:
[snip]
]0;richard@debian-8: ~richard@debian-8:~$
]0;richard@debian-8: ~richard@debian-8:~$ N=0
]0;richard@debian-8: ~richard@debian-8:~$
]0;richard@debian-8: ~richard@debian-8:~$ F="JessieDVD"
]0;richard@debian-8: ~richard@debian-8:~$
]0;richard@debian-8: ~richard@debian-8:~$ E='.iso'
]0;richard@debian-8: ~richard@debian-8:~$
]0;richard@debian-8: ~richard@debian-8:~$ while true
do
let N=N+1
echo "Insert medium, wait until blinking ends, press Enter key (or Ctrl+C
to end)"
read dummyvar
blocks=$(expr $(/sbin/isosize /dev/sr0) / 2048)
echo "Block count: "$blocks
echo "Byte count: $(expr $blocks '*' 2048)"
FILENAME=$F$N$E
echo "The output file will be $FILENAME"
echo ""
dd if=/dev/sr0 bs=2048 count=$blocks of=$FILENAME
echo ""
echo "next lines are output of ls and isosize commnds"
ls -l $FILENAME
/sbin/isosize $FILENAME
echo "about to repeat loop"
echo ""
done
Insert medium, wait until blinking ends, press Enter key (or
Ctrl+C to end)
Block count: 1941504
Byte count: 3976200192
The output file will be JessieDVD1.iso
1941504+0 records in
1941504+0 records out
3976200192 bytes (4.0 GB) copied, 876.109 s, 4.5 MB/s
next lines are output of ls and isosize commnds
-rw-r--r-- 1 richard richard 3976200192 Sep 20 13:39 JessieDVD1.iso
3976200192
about to repeat loop
Insert medium, wait until blinking ends, press Enter key (or
Ctrl+C to end)
Block count: 2284052
Byte count: 4677738496
The output file will be JessieDVD2.iso
1941504+0 records in
1941504+0 records out
3976200192 bytes (4.0 GB) copied, 733.727 s, 5.4 MB/s
next lines are output of ls and isosize commnds
-rw-r--r-- 1 richard richard 3976200192 Sep 20 13:52 JessieDVD2.iso
4677738496
about to repeat loop
Insert medium, wait until blinking ends, press Enter key (or
Ctrl+C to end)
^C
]0;richard@debian-8: ~richard@debian-8:~$ exit
Script done on Sat 20 Sep 2014 01:53:32 PM CDT
Re: Narrowing down problem source
Hi, Steve McIntyre wrote: > > > ... FAT32 filesystem ... i wrote: > > Richard: Did you copy to a different directory ? Richard Owlett wrote: > When reusing a file name I had used filemanager to send file to trash and > then emptied trash before reusing. The question was more about whether you had your successful dd runs on a filesystem which can take files larger than 4 GiB, wheras the unsuccessful ones were on FAT. Different filesystems would have to be mounted under different directories. So staying within the same directory during all tries would make clear that failure or success does not stick with a particular filesystem type. > > No bugs shared. > Why would creative minds need to "share" ;) Our bugs look completely different than theirs. Ours are for free - theirs are proprietary. Total cost of ownership for an oldfashioned frugal GNU/Linux bug is far less than for a Windows 10 glitch which has to last for the whole lifetime of a toddler born in 2013. Have a nice day :) Thomas
Re: Narrowing down problem source - was [Re: Extracting directories from an ISO image, command line tool?]
Steve McIntyre wrote: Richard Owlett wrote: I started over again. I switched to a laptop dedicated to things that may not work as expected. For some of my projects I regularly wipe the disk and do a complete reinstall of Debian. Currently I'm running Debian 8.0 with Mate as DE. No updates. I used blocks=$(expr $(/sbin/isosize /dev/sr0) / 2048) echo "Byte count: $(expr $blocks '*' 2048)" dd if=/dev/sr0 bs=2048 count=$blocks of=dvd8_2.iso as a pattern for each DVD. _Some_ passed your test of ls -l /media/richard/Lexar/dvd8_2.iso /sbin/isosize /media/richard/Lexar/dvd8_2.iso Example: ls -l dvd8_9.iso /sbin/isosize dvd8_9.iso resulted in richard@debian:~$ ls -l dvd8_9.iso -rw-r--r-- 1 richard richard 3976200192 Sep 16 11:50 dvd8_9.iso richard@debian:~$ richard@debian:~$ /sbin/isosize dvd8_9.iso 4460664832 I'll tell you what's most likely wrong - you're storing your extracted ISO images on a FAT32 filesystem which doesn't support file sizes larger than 4GiB. That's why DVD #1 works, it's explicitly reduced in size to fit on a 4GB USB stick too. That was true at the very beginning. Latest runs were under the home directory of my Debian 8 install.
Re: Narrowing down problem source
Thomas Schmitt wrote: Hi, Richard Owlett wrote: I did a rerun. The only apparent change in test conditions was I used a different instance of Mate Terminal. ... 4460664832 bytes (4.5 GB) copied, 798.488 s, 5.6 MB/s ... richard@debian:~$ /sbin/isosize rerundvd9.iso 4460664832 Did the dd runs with short "ls -l" result tell the "isosize" result, nevertheless ? I think I did. I'll revise my procedure to use the script command to keep better records. I've a gut feeling that Mate Terminal isn't redoing blocks=$(expr $(/sbin/isosize /dev/sr0) / 2048) every time. Command interpretation is supposed to depend on the shell interpreter (e.g. bash, dash, ksh) and not on the terminal emulator (e.g. xterm, gnome-terminal, Konsole). What should have been identical runs did not have identical results -- part of reason I'll make extensive use of script command in the next round. On the first hand we have to concentrate on the messages from the dd runs which deliver too few bytes in the .iso files. After a dd run turned out to copy not enough bytes, look into the system log whether any error message about "sr0" has been freshly recorded. Steve McIntyre: I'll tell you what's most likely wrong - you're storing your extracted ISO images on a FAT32 filesystem which doesn't support file sizes larger than 4GiB. But why did the retry succeed ? Richard: Did you copy to a different directory ? When reusing a file name I had used filemanager to send file to trash and then emptied trash before reusing. Nagging memories of a similar problem with either Win95 or Win98. We fiercely deny any similarity with said systems. No bugs shared. Why would creative minds need to "share" ;) Have a nice day :) Thomas
Re: Narrowing down problem source
Thomas Schmitt wrote: Hi, Steve McIntyre wrote: ... FAT32 filesystem ... i wrote: Richard: Did you copy to a different directory ? Richard Owlett wrote: When reusing a file name I had used filemanager to send file to trash and then emptied trash before reusing. The question was more about whether you had your successful dd runs on a filesystem which can take files larger than 4 GiB, wheras the unsuccessful ones were on FAT. Different filesystems would have to be mounted under different directories. So staying within the same directory during all tries would make clear that failure or success does not stick with a particular filesystem type. My initial xorriso problems were FAT32 related. My later comments of run-to-run inconsistencies with dd and your recommended tests were all using my home directory.
Re: Narrowing down problem source
Hi, > My later comments of run-to-run inconsistencies with dd and your recommended > tests were all using my home directory. In this case any dd run with short result file should issue some error message and/or there should be some hardware error message in the system logs. Have a nice day :) Thomas
Re: Narrowing down problem source - was [Re: Extracting directories from an ISO image, command line tool?]
Richard Owlett wrote: > >I started over again. I switched to a laptop dedicated to things >that may not work as expected. For some of my projects I >regularly wipe the disk and do a complete reinstall of Debian. >Currently I'm running Debian 8.0 with Mate as DE. No updates. > >I used > blocks=$(expr $(/sbin/isosize /dev/sr0) / 2048) > echo "Byte count: $(expr $blocks '*' 2048)" > dd if=/dev/sr0 bs=2048 count=$blocks of=dvd8_2.iso >as a pattern for each DVD. > >_Some_ passed your test of > ls -l /media/richard/Lexar/dvd8_2.iso > /sbin/isosize /media/richard/Lexar/dvd8_2.iso > >Example: > ls -l dvd8_9.iso > /sbin/isosize dvd8_9.iso > >resulted in >richard@debian:~$ ls -l dvd8_9.iso >-rw-r--r-- 1 richard richard 3976200192 Sep 16 11:50 dvd8_9.iso >richard@debian:~$ >richard@debian:~$ /sbin/isosize dvd8_9.iso >4460664832 I'll tell you what's most likely wrong - you're storing your extracted ISO images on a FAT32 filesystem which doesn't support file sizes larger than 4GiB. That's why DVD #1 works, it's explicitly reduced in size to fit on a 4GB USB stick too. -- Steve McIntyre, Cambridge, UK.st...@einval.com "Further comment on how I feel about IBM will appear once I've worked out whether they're being malicious or incompetent. Capital letters are forecast." Matthew Garrett, http://www.livejournal.com/users/mjg59/30675.html
Narrowing down problem source - was [Re: Extracting directories from an ISO image, command line tool?]
Richard Owlett wrote: Thomas Schmitt wrote: Hi, Richard Owlett wrote: richard@jessie:~$ ls -l /media/richard/Lexar/dvd8_2.iso -rw-r--r-- 1 richard richard 3976200192 Sep 9 07:36 /media/richard/Lexar/dvd8_2.iso richard@jessie:~$ /sbin/isosize /media/richard/Lexar/dvd8_2.iso 4677738496 You will need to get a new dvd8_2.iso of at least 4677738496 bytes size. I've got some links "somewhere" for copying DVDs to files such that checksums can be verified. Suspect that info may useful. Depending on the DVD medium type you have to expect trailing garbage after the end of the ISO. This has to be taken into respect when copying, diffing, or md5summing. I would use for copying from DVD to disk: blocks=$(expr $(/sbin/isosize /dev/sr0) / 2048) echo "Byte count: $(expr $blocks '*' 2048)" dd if=/dev/sr0 bs=2048 count=$blocks of=dvd8_2.iso (It would be nice if Debian's MD5SUMS would not only tell MD5 and image name but also the image size, or if there was a SIZES file.) Error messages from dd or dvd8_2.iso turning out smaller than the announced byte count indicate failure of copying. It appears that all of my dvd8_*.iso (except possibly dvd8_1.iso) were defective. Using instructions above to create dvd8_2.iso apparently works. Don't know what I did wrong last time around. I started over again. I switched to a laptop dedicated to things that may not work as expected. For some of my projects I regularly wipe the disk and do a complete reinstall of Debian. Currently I'm running Debian 8.0 with Mate as DE. No updates. I used blocks=$(expr $(/sbin/isosize /dev/sr0) / 2048) echo "Byte count: $(expr $blocks '*' 2048)" dd if=/dev/sr0 bs=2048 count=$blocks of=dvd8_2.iso as a pattern for each DVD. _Some_ passed your test of ls -l /media/richard/Lexar/dvd8_2.iso /sbin/isosize /media/richard/Lexar/dvd8_2.iso Example: ls -l dvd8_9.iso /sbin/isosize dvd8_9.iso resulted in richard@debian:~$ ls -l dvd8_9.iso -rw-r--r-- 1 richard richard 3976200192 Sep 16 11:50 dvd8_9.iso richard@debian:~$ richard@debian:~$ /sbin/isosize dvd8_9.iso 4460664832 ** 3976200192 <> 4460664832 I did a rerun. The only apparent change in test conditions was I used a different instance of Mate Terminal. blocks=$(expr $(/sbin/isosize /dev/sr0) / 2048) echo "Byte count: $(expr $blocks '*' 2048)" dd if=/dev/sr0 bs=2048 count=$blocks of=rerundvd9.iso ls -l rerundvd9.iso /sbin/isosize rerundvd9.iso result of rerun richard@debian:~$ blocks=$(expr $(/sbin/isosize /dev/sr0) / 2048) richard@debian:~$ richard@debian:~$ echo "Byte count: $(expr $blocks '*' 2048)" Byte count: 4460664832 richard@debian:~$ richard@debian:~$ dd if=/dev/sr0 bs=2048 count=$blocks of=rerundvd9.iso 2178059+0 records in 2178059+0 records out 4460664832 bytes (4.5 GB) copied, 798.488 s, 5.6 MB/s richard@debian:~$ ls -l rerundvd9.iso -rw-r--r-- 1 richard richard 4460664832 Sep 17 08:26 rerundvd9.iso richard@debian:~$ richard@debian:~$ /sbin/isosize rerundvd9.iso 4460664832 I've a gut feeling that Mate Terminal isn't redoing blocks=$(expr $(/sbin/isosize /dev/sr0) / 2048) every time. Nagging memories of a similar problem with either Win95 or Win98. Might not get to do suitable test until tomorrow. Later.
Re: Narrowing down problem source
Hi, Richard Owlett wrote: > I did a rerun. The only apparent change in test conditions was I used a > different instance of Mate Terminal. > ... > 4460664832 bytes (4.5 GB) copied, 798.488 s, 5.6 MB/s > ... > richard@debian:~$ /sbin/isosize rerundvd9.iso > 4460664832 Did the dd runs with short "ls -l" result tell the "isosize" result, nevertheless ? > I've a gut feeling that Mate Terminal isn't redoing > blocks=$(expr $(/sbin/isosize /dev/sr0) / 2048) > every time. Command interpretation is supposed to depend on the shell interpreter (e.g. bash, dash, ksh) and not on the terminal emulator (e.g. xterm, gnome-terminal, Konsole). On the first hand we have to concentrate on the messages from the dd runs which deliver too few bytes in the .iso files. After a dd run turned out to copy not enough bytes, look into the system log whether any error message about "sr0" has been freshly recorded. Steve McIntyre: > I'll tell you what's most likely wrong - you're storing your extracted > ISO images on a FAT32 filesystem which doesn't support file sizes > larger than 4GiB. But why did the retry succeed ? Richard: Did you copy to a different directory ? > Nagging memories of a similar problem with either Win95 or Win98. We fiercely deny any similarity with said systems. No bugs shared. Have a nice day :) Thomas
