Re: drive names and UUIDs, was Re: Intresting dd fsck grub uuid fstab action

[Joel Rees]

On Sun, Jun 4, 2017 at 12:59 AM, Pascal Hambourg  wrote:
> Le 03/06/2017 à 17:48, Gene Heskett a écrit :
>>
>>
>> I don't believe that will work.  dd runs on the raw device, not to an
>> artificially created "partition".
>
>
> dd runs on any type of device, including partitions.
>

But it copies the raw data.

In the context of this discussion, it makes absolutely no sense to
have it twiddling any of the data it copies, much less any of the data
that refers to what is being copied.

(Sorry about the misfire, Pascal.)

-- 
Joel Rees

One of these days I'll get someone to pay me
to design a language that combines the best of Forth and C.
Then I'll be able to leap wide instruction sets with a single #ifdef,
run faster than a speeding infinite loop with a #define,
and stop all integer size bugs with a bare cast.

More of my delusions:
http://reiisi.blogspot.com/2017/05/do-not-pay-modern-danegeld-ransomware.html
http://reiisi.blogspot.jp/p/novels-i-am-writing.html

Re: drive names and UUIDs, was Re: Intresting dd fsck grub uuid fstab action

[Joel Rees]

(Google or something is screwing up the threading. My apologies if I
mess it up further.)

On Sun, Jun 4, 2017 at 7:30 AM, Fungi4All  wrote:
>
>> From: deb...@lionunicorn.co.uk
>>
>>> Ι was waiting to see if anyone else found something like this significant
>> and willing to contribute some wisdom
>> No wisdom here, I'm afraid.
>
> just evolution of the unix-dna

It would definitely be evolution. And it would be something that should be
relegated to an explicitly specified option (maybe like character
encoding stuff), if at all.

And it would be orders of magnitude more complex than what dd now
does. That's part of the reason we have (g)parted and other similar
tools. (And, if you are using LVM, LVM has its own tools.)

>>[...]
>>> Also, I believe that when dd is used to copy something from disk to disk
>>> it should provide an option of whether to
>>> produce a new uuid or retain the original (backup, not a concurrent
>>> system).
>
>> Here you're asking for the impossible. dd is blind to what it's
>> copying at that level. It can fiddle with something it calls "records"
>> (which stinks of IBM FB↔VB conversion) and that's about it.
>
>
> I actually like dd the more I learn about it but what I was suggesting was
> to have
> an option to change the uuid to a new random one after it is done copying.

If at all, an option, but it really is out of dd's scope. dd is not parted.

> I understand (think) that dd does not even care about the format of the fs
> it copies
> or that of what it copies to, just blocks of space, where to start and where
> to finish.

Very true.

> So if a 10gb NTFS partition is copied to a 20gb EXT4 partition, the target
> will be an
> ntfs 20gb partition.

No, the target will, depending on what you specify, be a 10gb file in
the ext4 partition or a 10gb NTFS partition (overwriting the ext4 file
system completely) and a 10gb gap of unused disk space that the MBR
says goes with the partition which is formatted as a 10gb NTFS
partition, but the NTFS partition really doesn't know anything about
(unless you tell it about it afterwards by expanding the partition
using NTFS tools.)

IIUC.

>  So I suspect it does formatting in there too,

dd is not parted, nor is it an NTFS partition editor.

> otherwise the
> partition would have been left half ntfs half ext4.

The MBR partition is left half used by the NTFS file system and half
unused. The unused part may have some useless bits of ext4 left
behind, but it has nothing like a file system in it.

> The kernel I assume as soon as dd is done picks up the new set of uuids and
> updates the table.  So if dd does not do it it leaves the system in a mesh.

Not that I am aware. Not unless you tell the kernel to do so.

>[...]


-- 
Joel Rees

One of these days I'll get someone to pay me
to design a language that combines the best of Forth and C.
Then I'll be able to leap wide instruction sets with a single #ifdef,
run faster than a speeding infinite loop with a #define,
and stop all integer size bugs with a bare cast.

More of my delusions:
http://reiisi.blogspot.com/2017/05/do-not-pay-modern-danegeld-ransomware.html
http://reiisi.blogspot.jp/p/novels-i-am-writing.html

Re: drive names and UUIDs, was Re: Intresting dd fsck grub uuid fstab action

[Fungi4All]

From: deb...@lionunicorn.co.uk

> Ι was waiting to see if anyone else found something like this significant and 
> willing to contribute some wisdom
No wisdom here, I'm afraid.

just evolution of the unix-dna

> I suspect the experiment would be simple.
> Let's say we make a new partition on a disk with a debian installed system 
> and use dd to clone that system in
> the new partition (I have no idea whether doing this in the same disk or a 
> second one makes a difference),
> Let's say that dd copies uuid to the new partition so we have two partitions 
> with the same uuid in the same system.
> Intentionally we make the mistake and log in to the original system and 
> update-grub with the conflict.

AIUI there's a race condition here, perhaps even several. The correct
MBR should be read as normal by the BIOS, but grub then searches
by UUID for the kernel/ramdisk, and the kernel searches by UUID for
the root filesystem, and we don't know how it chooses between them.

I think this bit of info gets one level deeper to source of problems.

Do they (a) take the UUID being searched for and look at each disk/
partition's UUID for a match, or (b) read all the available UUIDs into
a table (overwriting any duplicates by the new entry) and then take
the UUID being searched for and look it up in the table? These
strategies give opposite results.

I suspect at some point, when 2 matching uuids come up for different
disks or parts of the same disk, the partition is left out as unmountable.
In my case it would mix and match the partitions of one system with
some from its clone. How do I know this? I had labeled the partitions
as their sda/sdb numbers and the system booted with 5 partitions
showing me the mix match as unmounted and when I'd attempt to mount
them it would say no no.

> Then relogin and create new uuids for the clone, edit ftstab and check 
> original, then update-grub.
> Will it boot having the first root or the clone root? Then look at grub.cfg 
> of the original to see if the uuids for each
> entry are matching or there is a mismatch.
> I went through the grub manual and didn't come up with an answer. It smells 
> like a tiny bug or room for improvement.

I think you'd have to study both grub and the kernel's code to
find an answer, and their developers would say "Just don't do it,
it's not defined, it may change tomorrow".

if time=23:59 do yesterday :)

> Also, I believe that when dd is used to copy something from disk to disk it 
> should provide an option of whether to
> produce a new uuid or retain the original (backup, not a concurrent system).

Here you're asking for the impossible. dd is blind to what it's
copying at that level. It can fiddle with something it calls "records"
(which stinks of IBM FB↔VB conversion) and that's about it.

I actually like dd the more I learn about it but what I was suggesting was to 
have
an option to change the uuid to a new random one after it is done copying.
I understand (think) that dd does not even care about the format of the fs it 
copies
or that of what it copies to, just blocks of space, where to start and where to 
finish.
So if a 10gb NTFS partition is copied to a 20gb EXT4 partition, the target will 
be an
ntfs 20gb partition. So I suspect it does formatting in there too, otherwise the
partition would have been left half ntfs half ext4.
The kernel I assume as soon as dd is done picks up the new set of uuids and
updates the table. So if dd does not do it it leaves the system in a mesh.

For copying filesystems around, there are commands for doing that.
Years ago I used cpio -vdamp but I've seen better commands (cp -a ?)
in threads here more recently. These have the advantage that you
prepare the empty filesystem the way you want it before you copy
into it.

I thought that dd is more accurate and quicker with the exception of little
data in a huge partition, where it copies a huge amount of empty space.

Cheers,
David.

cheers back

Re: drive names and UUIDs, was Re: Intresting dd fsck grub uuid fstab action

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Sat, Jun 03, 2017 at 05:59:06PM +0200, Pascal Hambourg wrote:
> Le 03/06/2017 à 17:48, Gene Heskett a écrit :
> >
> >I don't believe that will work.  dd runs on the raw device, not to an
> >artificially created "partition".
> 
> dd runs on any type of device, including partitions.

Exactly. Or to nit the pick a tad more: dd copies a raw byte stream:
that can be a device, a partition, a chunk thereof or even a file.
Whatever the OS can represent as a byte stream.

Cheers
- -- t
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlkzC0wACgkQBcgs9XrR2kYuPQCaAn0s5QjN7FgsIxsmm+B7H4vx
hpwAn33UTBG8fyWPPmE1hFtXZ6D4FKYb
=q5IS
-END PGP SIGNATURE-

Re: drive names and UUIDs, was Re: Intresting dd fsck grub uuid fstab action

[Pascal Hambourg]

Le 03/06/2017 à 18:04, David Wright a écrit :


AIUI there's a race condition here, perhaps even several. The correct
MBR should be read as normal by the BIOS, but grub then searches
by UUID for the kernel/ramdisk, and the kernel searches by UUID for
the root filesystem, and  we don't know how it chooses between them.


Note that the kernel does not know anything about filesystem UUIDs 
(stored in filesystem metadata). It only knows about partition UUIDs 
(stored in partition tables). The part that uses UUIDs to locate the 
root filesystem is the initrd/initramfs userland scripts, with the help 
of libblkid/udev.



Do they (a) take the UUID being searched for and look at each disk/
partition's UUID for a match, or (b) read all the available UUIDs into
a table (overwriting any duplicates by the new entry) and then take
the UUID being searched for and look it up in the table? These
strategies give opposite results.


I think GRUB does the former and libblkid/udev does the latter.

Re: drive names and UUIDs, was Re: Intresting dd fsck grub uuid fstab action

[David Wright]

On Sat 03 Jun 2017 at 11:02:54 (-0400), Fungi4All wrote:

> Ι was waiting to see if anyone else found something like this significant and 
> willing to contribute some wisdom

No wisdom here, I'm afraid.

> I suspect the experiment would be simple.
> Let's say we make a new partition on a disk with a debian installed system 
> and use dd to clone that system in
> the new partition (I have no idea whether doing this in the same disk or a 
> second one makes a difference),
> Let's say that dd copies uuid to the new partition so we have two partitions 
> with the same uuid in the same system.
> Intentionally we make the mistake and log in to the original system and 
> update-grub with the conflict.

AIUI there's a race condition here, perhaps even several. The correct
MBR should be read as normal by the BIOS, but grub then searches
by UUID for the kernel/ramdisk, and the kernel searches by UUID for
the root filesystem, and  we don't know how it chooses between them.

Do they (a) take the UUID being searched for and look at each disk/
partition's UUID for a match, or (b) read all the available UUIDs into
a table (overwriting any duplicates by the new entry) and then take
the UUID being searched for and look it up in the table? These
strategies give opposite results.

> Then relogin and create new uuids for the clone, edit ftstab and check 
> original, then update-grub.
> Will it boot having the first root or the clone root? Then look at grub.cfg 
> of the original to see if the uuids for each
> entry are matching or there is a mismatch.
> I went through the grub manual and didn't come up with an answer. It smells 
> like a tiny bug or room for improvement.

I think you'd have to study both grub and the kernel's code to
find an answer, and their developers would say "Just don't do it,
it's not defined, it may change tomorrow".

> Also, I believe that when dd is used to copy something from disk to disk it 
> should provide an option of whether to
> produce a new uuid or retain the original (backup, not a concurrent system).

Here you're asking for the impossible. dd is blind to what it's
copying at that level. It can fiddle with something it calls "records"
(which stinks of IBM FB↔VB conversion) and that's about it.

For copying filesystems around, there are commands for doing that.
Years ago I used cpio -vdamp but I've seen better commands (cp -a ?)
in threads here more recently. These have the advantage that you
prepare the empty filesystem the way you want it before you copy
into it.

Cheers,
David.

Re: drive names and UUIDs, was Re: Intresting dd fsck grub uuid fstab action

[Pascal Hambourg]

Le 03/06/2017 à 17:48, Gene Heskett a écrit :


I don't believe that will work.  dd runs on the raw device, not to an
artificially created "partition".


dd runs on any type of device, including partitions.

Re: drive names and UUIDs, was Re: Intresting dd fsck grub uuid fstab action

[Gene Heskett]

On Saturday 03 June 2017 11:02:54 Fungi4All wrote:

>  Original Message 
> From: deb...@lionunicorn.co.uk
> To: debian-user@lists.debian.org
>
> On Thu 01 Jun 2017 at 12:24:28 (-0400), Fungi4All wrote:
> > Why don't just skip all this that we are in perfect agreement with
> > and go to the juicy part. After all uuids are unique and fstab are
> > all correct, updating-grub would mix match uuids in writing its
> > grub.cfg
> > Two uuids on the same entry! Over and over again till I edited
> > it out to the correct ones and it all worked. Why does everyone
> > choses to skip on this issue and keeps explaining me over and over
> > what I have well understood by now?
>
> Well, we don't have a lot of evidence. But we do have a tiny bit:
>
> Ι was waiting to see if anyone else found something like this
> significant and willing to contribute some wisdom I suspect the
> experiment would be simple.
> Let's say we make a new partition on a disk with a debian installed
> system and use dd to clone that system in the new partition (I have no
> idea whether doing this in the same disk or a second one makes a
> difference),

I don't believe that will work.  dd runs on the raw device, not to an 
artificially created "partition".

rsync OTOH, can run on a partition to partition basis.  And that should 
give you different blkid's.

> Let's say that dd copies uuid to the new partition so we 
> have two partitions with the same uuid in the same system.
> Intentionally we make the mistake and log in to the original system
> and update-grub with the conflict. Then relogin and create new uuids
> for the clone, edit ftstab and check original, then update-grub. Will
> it boot having the first root or the clone root? Then look at grub.cfg
> of the original to see if the uuids for each entry are matching or
> there is a mismatch.
> I went through the grub manual and didn't come up with an answer. It
> smells like a tiny bug or room for improvement. Also, I believe that
> when dd is used to copy something from disk to disk it should provide
> an option of whether to produce a new uuid or retain the original
> (backup, not a concurrent system).
>
> if [ x$feature_platform_search_hint = xy ]; then
> search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos3
> --hint-efi=hd0,msdos3 --hint-baremetal=ahci0,msdos3
> --hint='hd0,msdos3' UUID on sda3 XXX else
> search --no-floppy --fs-uuid --set=root UUID on sda3 YYY
> fi


Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page 

Re: drive names and UUIDs, was Re: Intresting dd fsck grub uuid fstab action

[Fungi4All]

 Original Message 
From: deb...@lionunicorn.co.uk
To: debian-user@lists.debian.org

On Thu 01 Jun 2017 at 12:24:28 (-0400), Fungi4All wrote:

> Why don't just skip all this that we are in perfect agreement with and go to 
> the juicy part.
> After all uuids are unique and fstab are all correct, updating-grub would mix 
> match uuids in writing
> its grub.cfg
> Two uuids on the same entry! Over and over again till I edited it out to 
> the correct ones and it all worked.
> Why does everyone choses to skip on this issue and keeps explaining me over 
> and over
> what I have well understood by now?

Well, we don't have a lot of evidence. But we do have a tiny bit:

Ι was waiting to see if anyone else found something like this significant and 
willing to contribute some wisdom
I suspect the experiment would be simple.
Let's say we make a new partition on a disk with a debian installed system and 
use dd to clone that system in
the new partition (I have no idea whether doing this in the same disk or a 
second one makes a difference),
Let's say that dd copies uuid to the new partition so we have two partitions 
with the same uuid in the same system.
Intentionally we make the mistake and log in to the original system and 
update-grub with the conflict.
Then relogin and create new uuids for the clone, edit ftstab and check 
original, then update-grub.
Will it boot having the first root or the clone root? Then look at grub.cfg of 
the original to see if the uuids for each
entry are matching or there is a mismatch.
I went through the grub manual and didn't come up with an answer. It smells 
like a tiny bug or room for improvement.
Also, I believe that when dd is used to copy something from disk to disk it 
should provide an option of whether to
produce a new uuid or retain the original (backup, not a concurrent system).

if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos3 
--hint-efi=hd0,msdos3 --hint-baremetal=ahci0,msdos3 --hint='hd0,msdos3' UUID on 
sda3 XXX
else
search --no-floppy --fs-uuid --set=root UUID on sda3 YYY
fi

Re: drive names and UUIDs, was Re: Intresting dd fsck grub uuid fstab action

[David Wright]

On Thu 01 Jun 2017 at 12:24:28 (-0400), Fungi4All wrote:

> Why don't just skip all this that we are in perfect agreement with and go to 
> the juicy part.
> After all uuids are unique and fstab are all correct, updating-grub would mix 
> match uuids in writing
> its grub.cfg
> Two uuids on the same entry! Over and over again till I edited it out to 
> the correct ones and it all worked.
> Why does everyone choses to skip on this issue and keeps explaining me over 
> and over
> what I have well understood by now?

Well, we don't have a lot of evidence. But we do have a tiny bit:

if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos3 
--hint-efi=hd0,msdos3 --hint-baremetal=ahci0,msdos3 --hint='hd0,msdos3' UUID on 
sda3 XXX
else
search --no-floppy --fs-uuid --set=root UUID on sda3 YYY
fi
insmod png

Not a lot of context, and also edited and unindented, so the best
I can do to try to find what wrote this is to grep "platform_search"
and that comes up with /usr/share/grub/grub-mkconfig_lib in grub-common.

Within this file are these lines (starting around l.169):

  # If there's a filesystem UUID that GRUB is capable of identifying, use it;
  # otherwise set root as per value in device.map.
  fs_hint="`"${grub_probe}" --device $@ --target=compatibility_hint`"
  if [ "x$fs_hint" != x ]; then
echo "set root='$fs_hint'"
  fi
  if fs_uuid="`"${grub_probe}" --device $@ --target=fs_uuid 2> /dev/null`" ; 
then
hints="`"${grub_probe}" --device $@ --target=hints_string 2> /dev/null`" || 
hints=
echo "if [ x\$feature_platform_search_hint = xy ]; then"
echo "  search --no-floppy --fs-uuid --set=root ${hints} ${fs_uuid}"
echo "else"
echo "  search --no-floppy --fs-uuid --set=root ${fs_uuid}"
echo "fi"
  fi
  IFS="$old_ifs"

I'll let the shell experts come up with ideas about how ${fs_uuid}
produces two different substitutions within three lines.
I have made one assumption, that these lines from grub.cfg are from
sections 00 throught 10.

Once you reach section 30, things are different. Although I still
would not expect to see the disagreement in UUIDs quoted above,
I wouldn't be surprised if the UUIDs in the lines   linux /boot/vmlinuz…
bore no relation to the other UUIDs because _this_ linux… line is
copied directly from grub.cfg files scattered elsewhere on the disks.

I can illustrate this with the following. Here's the first menuentry
(top level) in my wheezy partition, freshly generated, but then doctored
by inserting "doctored" in the title and zeroing the first two digits
of the UUID throughout this entry. (This grub.cfg is never actioned
because the MBR doesn't point to it.)


### BEGIN /etc/grub.d/10_linux ###
menuentry 'doctored Debian GNU/Linux, with Linux 3.2.0-4-686-pae' --class 
debian --class gnu-linux --class gnu --class os {
load_video
insmod gzio
insmod part_msdos
insmod ext2
set root='(hd0,msdos2)'
search --no-floppy --fs-uuid --set=root 
0097eef1-c934-4b8f-a76b-4b084d6cf6f0
echo'Loading Linux 3.2.0-4-686-pae ...'
linux   /boot/vmlinuz-3.2.0-4-686-pae 
root=UUID=0097eef1-c934-4b8f-a76b-4b084d6cf6f0 ro  quiet
echo'Loading initial ramdisk ...'
initrd  /boot/initrd.img-3.2.0-4-686-pae
}


Now here are the first two menuentries in section 30 of my jessie
partition (which the MBR points to), generated after the above.
The first menuentry is the top-level one, the second is the
duplicate inside the submenu.

### BEGIN /etc/grub.d/30_os-prober ###
menuentry 'Debian GNU/Linux (7.11) (on /dev/sda2)' --class gnu-linux --class 
gnu --class os $menuentry_id_option 
'osprober-gnulinux-simple-dd97eef1-c934-4b8f-a76b-4b084d6cf6f0' {
insmod part_msdos
insmod ext2
set root='hd0,msdos2'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos2 
--hint-efi=hd0,msdos2 --hint-baremetal=ahci0,msdos2 --hint='hd0,msdos2'  
dd97eef1-c934-4b8f-a76b-4b084d6cf6f0
else
  search --no-floppy --fs-uuid --set=root 
dd97eef1-c934-4b8f-a76b-4b084d6cf6f0
fi
linux /boot/vmlinuz-3.2.0-4-686-pae 
root=UUID=0097eef1-c934-4b8f-a76b-4b084d6cf6f0 ro quiet
initrd /boot/initrd.img-3.2.0-4-686-pae
}
submenu 'Advanced options for Debian GNU/Linux (7.11) (on /dev/sda2)' 
$menuentry_id_option 
'osprober-gnulinux-advanced-dd97eef1-c934-4b8f-a76b-4b084d6cf6f0' {
menuentry 'doctored Debian GNU/Linux, with Linux 3.2.0-4-686-pae (on 
/dev/sda2)' --class gnu-linux --class gnu --class os $menuentry_id_option 
'osprober-gnulinux-/boot/vmlinuz-3.2.0-4-686-pae--dd97eef1-c934-4b8f-a76b-4b084d6cf6f0'
 {
insmod part_msdos
insmod ext2
set root='hd0,msdos2'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root 
--hint-bios=hd0,msdos2 

Re: drive names and UUIDs, was Re: Intresting dd fsck grub uuid fstab action

[Fungi4All]

 Original Message 
Subject: drive names and UUIDs, was Re: Intresting dd fsck grub uuid fstab 
action
From: joel.r...@gmail.com
To: Fungi4All <fungil...@protonmail.com>

Fungi4All-san,

I'll try explaining what we don't know whether you understand or not.

I understand everything you have written below.

So when you dd a parition/volume, you copy the
UUID for that partition/volume, too.

I asked specifically for this and I never got a straight answer. Is it possible 
though to
copy something and NOT get the same uuid but the partition to retain its 
original uuid?
Because this is what happened. When I made a list of the uuids after dd the 
source and
targets were not all the same. Some had transferred some did not.

And when you dd the whole device, you copy all the UUIDs on every
partition/volume on the device.

I use gparted for partition work. I made a new experiment to see in action what 
may happen
with a usb drive being coppied. The stick had two partitions, ext4, and the 
targets were two
partitions on hd0 (the only hard drive). In this case the targets got the same 
uuids. I tried
over and over again to change the stick's uuid's and gparted will not allow me.

In order to have both the duplicate and the original connected to the
computer at the same time, you have to figure out a way it can tell
them apart.

As far as I can tell when a duplicate exists it just doesn't get mounted. With 
gparted you
see it fine, it is there with a duplicate.

...
If you read through this and understand it, and can tell us what you did
in a way that we can tell you understand this, we can continue to try to
help.

Why don't just skip all this that we are in perfect agreement with and go to 
the juicy part.
After all uuids are unique and fstab are all correct, updating-grub would mix 
match uuids in writing
its grub.cfg
Two uuids on the same entry! Over and over again till I edited it out to 
the correct ones and it all worked.
Why does everyone choses to skip on this issue and keeps explaining me over and 
over
what I have well understood by now?

Grub when installed it has a way of picking up if there is a bootable system in 
each partition and makes a
table of entries/menu-items
Menu 1 Debian 8
Menu 2 Debian 9
Menu 3 Zlumbudu

Here I have one entry Menu2 and within the same instruction two different uuids 
exist, one is the same as
menu 1 the next is the same of menu2  you chose 2 and 1 starts. Over and 
over I tried to update it, it
showed as a new file, the same error. I correct it manually everything works 
great!

--
Joel Rees

delusions of being a novelist:
http://joel-rees-economics.blogspot.com/2017/01/soc500-00-00-toc.html

drive names and UUIDs, was Re: Intresting dd fsck grub uuid fstab action

[Joel Rees]

Fungi4All-san,

I'll try explaining what we don't know whether you understand or not.
First, about

   /dev/sda
   /dev/sdb
   ...

When you turn the machine on, these "names" do not exist. Well, at
least, the computer does not know which physical device is /dev/sda
and which is /dev/sdb, etc.

When the BIOS is finding hard disks and other disk-like persistent
storage devices right after you turn the computer on, it remembers them
in the order it finds them. That means that, if one device finishes
powering up before another, it is likely to have a lower device name.

But even that is a probability, not a guarantee, because BIOS is
generally not looking at each device at the exact moment it powers up.

Because of this, /dev/sda may be your boot drive one time, and may be
your backup drive another time. Or, if you have multiple boot drives, it
may be your MSWIndows boot drive one time and your debian boot
drive the next, etc.

You want to think it should be more simple, but it's not. And it's not your
fault, and it's not Debian's fault. (Who's fault? Microsoft, Intel, Apple,
Maxtor, Seagate, Commodore, Atari, Radio Shack, IBM, DEC,
Honeywell, ..., pretty much all the companies involved.)

Why can't the drive itself just say it's /dev/sda? Well, what happens
when you go to the store and need a /dev/sdc, but all the drives in stock
are named /dev/sdb?

Labels of various sorts were tried, but when labels tended to be too
simply done, like "accounting" instead of something like "ACTG20170601".

So UUIDs were invented as a new sort of label that would theoretically
never be duplicated. They are separate from the labels that the are
called labels in /etc/fstab and gparted's listing, etc.

This explanation is too simple, but close enough to what's happening
to explain what we thing has happened to your drives.

UUIDs or other kinds of labels, where are they stored?

In the storage area of the drive itself, along with the MBR, the partition
information, the file system information, and the program and data files.

That means that, when you use dd to duplicate your storage device,
even the UUID is duplicated.

Now, it turns out that it's convenient to label or name partitions/volumes
within the device, and UUIDs are now generally assigned to each
partition/volume. These are different UUIDs, and they are also stored
on the disk itself. So when you dd a parition/volume, you copy the
UUID for that partition/volume, too.

And when you dd the whole device, you copy all the UUIDs on every
partition/volume on the device.

In order to have both the duplicate and the original connected to the
computer at the same time, you have to figure out a way it can tell
them apart.

The easiest way is to first change the UUID for the new device, and
then the UUIDs for each partition/volume, as well. and then edit
/etc/fstab on the new device to point to the changed UUIDs. You can do
this with an install CD or a live OS on a USB, etc.

You can't do it easily by booting either the new or old device, even if
you boot a different OS on either the new or old device.

It can be done,by giving the necessary volumes and the device itself
labels (that are not UUIDs) and changing /etc/fstab on the device that
will be booted to use labels instead of UUIDs.

Anything you changed before you changed either the UUIDs or labels,
edited /etc/fstab to use the new ones, and rebooted, you really don't
know which drive you did it to.

Running fsck before you take care of that could be really dangerous.

If you read through this and understand it, and can tell us what you did
in a way that we can tell you understand this, we can continue to try to
help.

--
Joel Rees

delusions of being a novelist:
http://joel-rees-economics.blogspot.com/2017/01/soc500-00-00-toc.html

Re: Intresting dd fsck grub uuid fstab action

[Fungi4All]

From: deb...@lionunicorn.co.uk

I'm not going to wade through all these imaginings again; sorry.
Just some points.

Your use of the term imaginings emphasizes clearly how you do not
reas but "assume" that this all mighty perfect system can't be at fault
and it all lies in my mistakes, without ever referring on what should
have been done and didn't.

>> How would I know which fstab you edited? […]
>> IOW how did you decide which disk the kernel had decided to call sda?
>> How did you tell which was the original and which was the identical copy?
>
> Because the source was on hd0 (physical unit) to target hd1,
> I suspect grub does not worry much about sda/sdb label
> but according to bios knows the master and slave.

What does grub have to do with it. By the time you edited fstab,
the linux kernel was running, sda and sdb are indeterminate, and
choices have been made between duplicate UUIDs.

You are focusing on grub out of context. You ask one clarification,
you get it and you keep IMAGINING things because you did not read.
And you waste my time repeating everything again and again.
If you read previous posts I have stated clearly three times at least,
that I did not edit fstab from "the live" kernel of the systems mentioned
but by "another" system. You want me to be specific? I used a debian
system installed on a usb-stick. to edit the fstab on both debian
installations. Does it matter? Even if dos5.0 would give access to
an ext4 disk I could edit fstab and grub.cfg from there. Why bother
with a live kernel on a system that wouldn't boot.

Whatever system started and booted the system its live identification
of partitions and UUIDs were very accurate at any given time. Those
transferred to fstab and a reboot was an attempt for them to boot correctly.

Assuming you want to avoid the issues involved with cloning an
active root filesystem, close down your A system and boot up a
system on a stick.
Copy the partitions from A to B.
Check for errors. /var/log/kern/log is worth inspecting.
Use tune2fs -U random /dev/sdbN to assign new UUIDs to the
copied filesystems.
If you've copied a swap partition, mkswap /dev/sdbN will create
a new UUID by default.
You can now reboot into your A system as per usual.
Assuming you want to make the copied partitions available when
the A system is running, edit A's fstab to add mountpoints for
your new partitions, then mount -a.

I have described the process in extemely boring detail, you both
fail to read and try to force situations that never existed.
I DID NOT TRY TO COPY A LIVE SYSTEM, it was DEAD! This was
very clear to anyone who bothers and can read basic english.
I did not expect a system that is functional and running well, it is then
shutdown then a "different" system is used to copy it and then
edit the fstab of the COPY to match the uuids on the system.

> Also please explain the logic of the developer of dd in "copying and 
> transfering"
> a UUID that is supposedly meant to be unique. Why not always asign a new
> one every time? Because it is meant for backup work and not for copying?

dd doesn't know what a filesystem is, or what a UUID is. It just
copies what you tell it to (or converts it in specific ways).

Does it duplicate uuids or doesn't it? Does it do it consistently or does
it sometimes it leaves the uuid as it was and sometimes it duplicated one?
WHY Because not all changed, or were duplicated, it was a mismatch.
I understand swap because it wasn't copied, but the other 4 3 had new ids
and one had a duplicate. I say I understand that dd made all duplicates and
fsck only assigned new uuids to partitions that were realigned, covering
the 1MB gap that was left by gparted.

The steps you described as doing in the order given were effectively
like trying to make a poached egg out of an egg you'd already scrambled.

You are trying to draw pictures and I have no clue what your intentions are.
Are you offended because you wrote the code for dd and grub? I have been
doing this long enough to recognize an irregular behavior. If you are trying
to figure out what the problem is don't refer to me anymore because I
said over and over again I fixed the unreasonably complex mess that was
created especially by having to edit grub.cfg as automatically it kept 
configuring
itsled WRONG Both installations were perfect, 10 unique UUIDs and
responding fstabs. Grub would update and mix two systems. All I changed was
the second like of the entry to the correct ID and both systems worked
flawlessly (eventually).

David.

Over and out!

Re: Intresting dd fsck grub uuid fstab action

[David Wright]

I'm not going to wade through all these imaginings again; sorry.
Just some points.

On Mon 29 May 2017 at 15:56:52 (-0400), Fungi4All wrote:
>> How would I know which fstab you edited? […]
>> IOW how did you decide which disk the kernel had decided to call sda?
>> How did you tell which was the original and which was the identical copy?
> 
> Because the source was on hd0 (physical unit) to target hd1,
> I suspect grub does not worry much about sda/sdb label
> but according to bios knows the master and slave.

What does grub have to do with it. By the time you edited fstab,
the linux kernel was running, sda and sdb are indeterminate, and
choices have been made between duplicate UUIDs.

> Well, that was my initial question. I could understand something
> going bad with the copy but not the source. The only logic is that
> the fsck asked to replace the uuids and I said "y" and it did.

You see, you've decided on what to blame and make up anything to
justify it.

> OK, since you blame every single bit of it in what I did

I'm trying to explain what may have happened under the circumstances
that you have presented us with. You are the one trying to apportion
blame, and you've decided on fsck and grub.

> why don't you
> tell me what are the correct steps for doing exactly what I wanted to
> do, copy an installation on an other hard disk and having it boot by
> the same original grub that handled all other systems.
> Because I am clueless at this point on what I exactly I did wrong and
> why such a simple procedure becomes so enormous source of problems.

Assuming you want to avoid the issues involved with cloning an
active root filesystem, close down your A system and boot up a
 system on a stick.
Copy the partitions from A to B.
Check for errors. /var/log/kern/log is worth inspecting.
Use tune2fs -U random /dev/sdbN to assign new UUIDs to the
 copied filesystems.
If you've copied a swap partition, mkswap /dev/sdbN will create
 a new UUID by default.
You can now reboot into your A system as per usual.
Assuming you want to make the copied partitions available when
 the A system is running, edit A's fstab to add mountpoints for
 your new partitions, then mount -a.

> Also please explain the logic of the developer of dd in "copying and 
> transfering"
> a UUID that is supposedly meant to be unique. Why not always asign a new
> one every time? Because it is meant for backup work and not for copying?

dd doesn't know what a filesystem is, or what a UUID is. It just
copies what you tell it to (or converts it in specific ways).

A dollar bill has a unique serial number. If you make a perfect copy,
it is no longer unique. The copyist doesn't change the number.
Neither does dd.

> I am telling you over and over again, I had no clue and never in my
> experience had I realise that dd "sometimes" does this. Not consistent!
> I discovered this in practice and "fixed it", by using gparted and making
> new uuids for anything duplicate (which I believe only happened in 2
> out of 5 partitions). That is when I fixed the fstabs and then booted
> sda5 and asked it update its grub. Then the problem became one of
> grub alone. I don't know how else can I explain it. If I say I do this
> in step 4 you tell me I shouldn't because there was a problem in step2
> I tell you what I did to fix it to get to 4 you tell me what I did in 2
> caused 4.

The steps you described as doing in the order given were effectively
like trying to make a poached egg out of an egg you'd already scrambled.

> It shouldn't take 5hours of guessing to copy a 5gb system from one
> disk to the next and expect to be able to run both in the next hour
> or so. This is rediculusly complex and inconsistent. Why so much
> redundancy? And this Uuid thing has got me thinking in ways that
> I didn't before. Why within a single system (not a WAN) does uniqueness
> has to be in the order of trillions and trillions?

You obviously don't understand the purpose of using UUIDs.
They solve the problem of creating a label for something (a
partition or a filesystem in your case) which is unique¹, but
without needing reference to anything else; no looking anything
up or checking anywhere.

Imagine you wanted to open an account with a bank, and they said
"Fine, give us the account number." You could run
$ dbus-uuidgen
7df4b93e13b8374cd491b440592cf5d8
$ 
and give them that. Nobody else will have this number for
their bank account, anywhere. If several banks merge, their
customers' account numbers will all remain distinct.

¹to all intents and purposes.

Cheers,
David.

Re: Intresting dd fsck grub uuid fstab action

[Fungi4All]

From: deb...@lionunicorn.co.uk

On Fri 26 May 2017 at 17:52:33 (-0400), Fungi4All wrote:
> From: deb...@lionunicorn.co.uk
>> On Thu 25 May 2017 at 16:41:37 (-0400), Fungi4All wrote:
>> > Le 25/05/2017 à 05:11, Fungi4All a écrit :
>> > > I experimented in switching a clone of my sid installation to an 
>> > > experimental, that was the plan.
>> > > Since I was doing other things I thought I'll let the cloning take place 
>> > > unattended.
>> > > Let's say sda5/6/7/8/9 were to be cloned to sdb5-9 (5 / 6 var 7 sw 8 tmp 
>> > > 9 home) all b partitions were slightly larger.
>> > > I used dd bs=1M for each one and though all I had to do is log in sda5 
>> > > edit the sdb5 fstab and then update-grub.
>>
>> … and I get the impression that that's exactly what you did:
>> 1 dd
>> 2a "log in sda5" (is that mount, or boot?)
>>
> Boot
>>
>> 2b edit an fstab file (to what purpose, at this point?)
>>
> Why would I edit the fstab in the original installation that I copied from?

How would I know which fstab you edited? You have this situation:

fsUUIDs Disk A Disk B
↓
MBR points to any fsUUID ABCD unknown MBR
some other partitions some other partitions
ABCD sda5 identical with sdb5
EF23 sda6 identical with sdb6
4567 sda7 identical with sdb7
89AB sda8 identical with sdb8
CDEF sda9 identical with sdb9

or do you? Why not:

fsUUIDs Disk A Disk B
↓
MBR points to any fsUUID ABCD unknown MBR
some other partitions some other partitions
ABCD sdb5 identical with sda5
EF23 sdb6 identical with sda6
4567 sdb7 identical with sda7
89AB sdb8 identical with sda8
CDEF sdb9 identical with sda9

IOW how did you decide which disk the kernel had decided to call sda?
How did you tell which was the original and which was the identical copy?

Because the source was on hd0 (physical unit) to target hd1,
I suspect grub does not worry much about sda/sdb label
but according to bios knows the master and slave. Gparted
will not be confused where is what and just based on contents and
size it is clear which is which, even when it decides to recognize a
usb-stick as sda. It does care about uuids
What is strange is that my error of not paying attention for
uniqueness was not consistent. Not all 5 partitions had the same uuids

> From sda5 I edited the new copy on sdb5 but sda5 did not boot my first
> time around.

I don't know what "my first time around" means. I've struggled to
reconstruct what you did from a list of numbered questions/comments
in your OP (not a strict list of actions) and some responses to
comments in your follow-up.

Let's say I used a live debian to copy 1 partition to another
(let's eliminate the confusion of distributed installations).
Let's say sda3 (uuid=abcd) to sdb3 () and I go read
the uuid for sdb3 and write it to the fstab on sdb3.
Then I intent to boot on sda3 and update-grub to include a
boot entry for sdb3. When I attempt to do so the fsck finds
errors, confusion, boots sda3 in an emergency mode and
does not see /var /tmp /home /swp
So now on this "first time around" I am trying to understand
why anything changed and sda3 (a functional installation)
does not boot when the fstab on it matches the uuids which
I never changed. So I double check and I am wrong, they
have changed. Was it because fsck while booting assigned
new uuids to stop the confusion.
Rounds 2 and 3 eventually results on booting the original.
Then grub is updated (maybe again). Then I try to boot sdb3
and it boots on sda3 but with the rest of partitions (var tmp
home swp) from sdb. By now I have installed a pseudo file
indicators on each desktop so I can tell which is which, and
by looking at what partitions on file manager seem as mountable
but not yet mounted (partitions from other installations).

I asked "to what purpose do you edit fstab" because at this
stage I see no sense in it. The original disk has an MBR that
allegedly boots partition 5. Presumably it then mounts the
partitions of interest, 6-9 using it's fstab.

Well, that was my initial question. I could understand something
going bad with the copy but not the source. The only logic is that
the fsck asked to replace the uuids and I said "y" and it did. Not
necessaraly understanding fully what the problem was with
the partitions. I remember having to do with alignment and covering
up free space between two partitions. Possibly this create new uuids?
Sometimes when you create a subsequent partition with gparted
there is a 1Mb between that can't be covered.

The "other" disk has an unknown MBR but is otherwise the same.
Were you to disconnect the original disk's data cable, you
could boot a rescue/live stick and install the correct MBR/grub
on the "other" disk and be able to boot *its" partition 5.
Its fstab would remain the same. All the UUIDs would still tie
up. Everything would be hunky dory so long as the disks never
"met" each other.

The disks coexist for some time now. The sdb MBR is not used.
Grub on sda 's Mbr is responsible for booting all systems installed.
It does so by 

Re: Intresting dd fsck grub uuid fstab action

[David Wright]

On Fri 26 May 2017 at 17:52:33 (-0400), Fungi4All wrote:
> From: deb...@lionunicorn.co.uk
>> On Thu 25 May 2017 at 16:41:37 (-0400), Fungi4All wrote:
>> > Le 25/05/2017 à 05:11, Fungi4All a écrit :
>> > > I experimented in switching a clone of my sid installation to an 
>> > > experimental, that was the plan.
>> > > Since I was doing other things I thought I'll let the cloning take place 
>> > > unattended.
>> > > Let's say sda5/6/7/8/9 were to be cloned to sdb5-9 (5 / 6 var 7 sw 8 tmp 
>> > > 9 home) all b partitions were slightly larger.
>> > > I used dd bs=1M for each one and though all I had to do is log in sda5 
>> > > edit the sdb5 fstab and then update-grub.
>> 
>> … and I get the impression that that's exactly what you did:
>> 1 dd
>> 2a "log in sda5" (is that mount, or boot?)
>> 
> Boot
>> 
>> 2b edit an fstab file (to what purpose, at this point?)
>> 
> Why would I edit the fstab in the original installation that I copied from?

How would I know which fstab you edited? You have this situation:

fsUUIDs Disk A  Disk B
 ↓
MBR points to any fsUUID ABCD   unknown MBR
some other partitions   some other partitions
ABCDsda5 identical with sdb5
EF23sda6 identical with sdb6
4567sda7 identical with sdb7
89ABsda8 identical with sdb8
CDEFsda9 identical with sdb9

or do you? Why not:

fsUUIDs Disk A  Disk B
 ↓
MBR points to any fsUUID ABCD   unknown MBR
some other partitions   some other partitions
ABCDsdb5 identical with sda5
EF23sdb6 identical with sda6
4567sdb7 identical with sda7
89ABsdb8 identical with sda8
CDEFsdb9 identical with sda9

IOW how did you decide which disk the kernel had decided to call sda?
How did you tell which was the original and which was the identical copy?

> From sda5 I edited the new copy on sdb5 but sda5 did not boot my first
> time around.

I don't know what "my first time around" means. I've struggled to
reconstruct what you did from a list of numbered questions/comments
in your OP (not a strict list of actions) and some responses to
comments in your follow-up.

I asked "to what purpose do you edit fstab" because at this
stage I see no sense in it. The original disk has an MBR that
allegedly boots partition 5. Presumably it then mounts the
partitions of interest, 6-9 using it's fstab.

The "other" disk has an unknown MBR but is otherwise the same.
Were you to disconnect the original disk's data cable, you
could boot a rescue/live stick and install the correct MBR/grub
on the "other" disk and be able to boot *its" partition 5.
Its fstab would remain the same. All the UUIDs would still tie
up. Everything would be hunky dory so long as the disks never
"met" each other.

> The fsck messages I could only think were pertaining to
> something wrong in the new partitions but even IT did not make sense.

I have no evidence to suggest that problems could be caused by
the trailing garbage in the copied partitions (which you said
were bigger), but it would spur me into finding out.

>> 
>> 3 run update-grub
>> 
> Yes, thinking that at least this would fix the original booting up.
> Either source or target wouldn't matter.

But I think this would screw up the (which?) grub.cfg royally.
And, unless you're actually a forensic scientist prepared to
carry out experiments and examine the code (because there's
some seriously valuable deductions to be made), picking over
the messed up file is probably a waste of time.

>> 
>> Assuming that your copies are good, and that the apparent
>> "alterations" to sda… arise from misunderstanding or misinterpreting
>> UUIDs (or that they actually post-date your running update-grub), then
>> your problems arise from one or both of:
>> 
> I had no intention to alter anything in sda, why would I. It was just a
> source to copy from an already functional system. The only change
> would have been an additional entry to grub.cfg in sda5 for booting to
> sdb5.

Computers don't take notice of intentions, only actions.

>> 
>> Booting into sda5 with both devices sda and sdb connected,
>> Running update-grub with both devices sda and sdb connected.
>> 
> That only happened after I had already copied the "new" uuids on sdb5
> to the fstab there.

That's no help. You've already booted a system with duplicate UUIDs
using a version of grub that finding things by UUID (choosing between
original and copied partitions at random?, who knows?) and a kernel
that guesses at random which disk to label /dev/sda and which to
label /dev/sdb.

> Simple copy-paste of 5 uuids from /dev/disk/by-uuid

I don't understand this. You seem to be 

Re: Intresting dd fsck grub uuid fstab action

[Fungi4All]

 Original Message 
Subject: Re: Intresting dd fsck grub uuid fstab action
UTC Time: May 26, 2017 5:35 AM
From: deb...@lionunicorn.co.uk

On Thu 25 May 2017 at 16:41:37 (-0400), Fungi4All wrote:
> Le 25/05/2017 à 05:11, Fungi4All a écrit :
> > I experimented in switching a clone of my sid installation to an 
> > experimental, that was the plan.
> > Since I was doing other things I thought I'll let the cloning take place 
> > unattended.
> > Let's say sda5/6/7/8/9 were to be cloned to sdb5-9 (5 / 6 var 7 sw 8 tmp 9 
> > home) all b partitions were slightly larger.
> > I used dd bs=1M for each one and though all I had to do is log in sda5 edit 
> > the sdb5 fstab and then update-grub.

… and I get the impression that that's exactly what you did:
1 dd
2a "log in sda5" (is that mount, or boot?)

Boot

2b edit an fstab file (to what purpose, at this point?)

Why would I edit the fstab in the original installation that I copied from?
From sda5 I edited the new copy on sdb5 but sda5 did not boot my first
time around. The fsck messages I could only think were pertaining to
something wrong in the new partitions but even IT did not make sense.

3 run update-grub

Yes, thinking that at least this would fix the original booting up.
Either source or target wouldn't matter.

Assuming that your copies are good, and that the apparent
"alterations" to sda… arise from misunderstanding or misinterpreting
UUIDs (or that they actually post-date your running update-grub), then
your problems arise from one or both of:

I had no intention to alter anything in sda, why would I. It was just a
source to copy from an already functional system. The only change
would have been an additional entry to grub.cfg in sda5 for booting to
sdb5.

Booting into sda5 with both devices sda and sdb connected,
Running update-grub with both devices sda and sdb connected.

That only happened after I had already copied the "new" uuids on sdb5
to the fstab there. Simple copy-paste of 5 uuids from /dev/disk/by-uuid

> > 1 - Does dd leave uuid targets as they are, does it create new ones, or 
> > does it copy uuid from source to target?

I assume that's your reporting of step 1.

???

> > 2 - When I rebooted sda5 was not booting up properly. Eventually I checked
> its fstab and corrected it but at this poing the system run an fsck and I got
> some errors and hit y,y,y,y, for fixing. I don't know if the errors came from
> uuids existing in the system twice.

Something about partition inconsistencies but the print was too small for me to 
read
in detail, never mind remembering.

Is that "log in"? What did you correct in fstab? What was wrong?

The only thing I did to fstab up to this point was at the target installation, 
not the original.
It took a few reboots trying to figure out why the source was so affected. At 
this point
I think I got the target to boot but what it was doing as the entry of Debian 
on sdb5 it
would boot sda5 while using sdb6,7,8,9. The reason I guess is because sda5 and 
sdb5
still had the same uuid. At some point when I used gparted to check all 10 
partitions and
create new uuids and labels, and redid both fstab to the new uuids I went back 
to sda5
and reupdated grub. Then the entry for sdb5 was still booting up from sda5!
That is when I went into grub.cfg and discovered the two uuids used for that 
entry had
2 different uuids, for the same entry. The first was sdb5 the next was sda5's 
uuid.

Is sda the same disk as it was when you ran dd? There's no reason why
it should be if the kernel races to label them sda and sdb. How did
you check?

Yes, dd run from a stick sdc "dd if=sda5 of=sdb5 bs=1M" then 6,8,9
7 was swap created with gparted, why copy it?

Why does a system suddenly run an fsck?

It run on recovery so I can run update grub before a full boot-up.
Before the login root prompt it reported the disk problems that never
had before with the same disks. But I'm pretty sure it checks filesystems
in every boot because I remember it says hit ctrl-C to skip filechecks.

> > 3 - Updating grub seemed to have made a bigger mess as now I could
> boot up but the partitions were mixed up between sda and sdb parts. It
> would boot-up on root / of sda and have a home in sdb9 

Yes. I would expect grub to be completely confused about what to
write in …wherever/grub/grub.cfg and possibly have no idea which
MBR it related to, with two disks present.

With the same 2 disks grub had been updated before, at least 2 times
recently. I don't understand you reasoning, but again: After all uuid's
were unique and both fstabs have been edited and double-checked.
update-grub runs and on the next reboot it is asked on menu entry
Debian on sdb5 to boot and it boots sda5 because on its cfg file it has
2 different uuids for the same entry.
Any explanations?

> The first thing I di

Re: Intresting dd fsck grub uuid fstab action

[David Wright]

On Thu 25 May 2017 at 16:41:37 (-0400), Fungi4All wrote:
> Le 25/05/2017 à 05:11, Fungi4All a écrit :
> > I experimented in switching a clone of my sid installation to an 
> > experimental, that was the plan.
> > Since I was doing other things I thought I'll let the cloning take place 
> > unattended.
> > Let's say sda5/6/7/8/9 were to be cloned to sdb5-9 (5 / 6 var 7 sw 8 tmp 9 
> > home) all b partitions were slightly larger.
> > I used dd bs=1M for each one and though all I had to do is log in sda5 edit 
> > the sdb5 fstab and then update-grub.

… and I get the impression that that's exactly what you did:
  1  dd
  2a "log in sda5" (is that mount, or boot?)
  2b edit an fstab file (to what purpose, at this point?)
  3  run update-grub

Assuming that your copies are good, and that the apparent
"alterations" to sda… arise from misunderstanding or misinterpreting
UUIDs (or that they actually post-date your running update-grub), then
your problems arise from one or both of:

Booting into sda5 with both devices sda and sdb connected,
Running update-grub with both devices sda and sdb connected.

> > 1 - Does dd leave uuid targets as they are, does it create new ones, or 
> > does it copy uuid from source to target?

I assume that's your reporting of step 1.

> > 2 - When I rebooted sda5 was not booting up properly. Eventually I checked
> its fstab and corrected it but at this poing the system run an fsck and I got
> some errors and hit y,y,y,y, for fixing. I don't know if the errors came from
> uuids existing in the system twice.

Is that "log in"? What did you correct in fstab? What was wrong?

Is sda the same disk as it was when you ran dd? There's no reason why
it should be if the kernel races to label them sda and sdb. How did
you check?

Why does a system suddenly run an fsck?

> > 3 - Updating grub seemed to have made a bigger mess as now I could
> boot up but the partitions were mixed up between sda and sdb parts. It
> would boot-up on root / of sda and have a home in sdb9 

Yes. I would expect grub to be completely confused about what to
write in …wherever/grub/grub.cfg and possibly have no idea which
MBR it related to, with two disks present.

> The first thing I did after dd was done, was to make a txt table of all
> the sda* sdb* in order with uuids and rewrote the fstab on sdb, I thought
> the sda would be as it was. But wasn't. It wasn't all uuids that transfered
> but seem enough for me not to notice they were the same and for sda5
> to boot up and discover the discrepancy. I speculate that fsck then instead
> of giving sdb a new uuid it altered sda. By the time the check was done
> and I run grub-upd some of the uuid in grub belonged to sda partitions and
> some on sdb.
> By copying from sda to sdb I never thought the original would be affected,
> only the target needed fixing. That threw me off. But on the new grub.cfg
> both entried would boot from the sda but with remaining partitions being
> a mismatch.

This makes me wonder whether you've taken on board Pascal's comments
re different types of UUIDs.

And when did fsck start handing out UUIDs to filesystems, let alone devices.

> > 4 - Eventually to stay safe I booted from live usb and edited all the
> partitions, gave them separate labels to avoid any confusion (as "Deb
> on Sda6 var..."_) and switched to all new uuids to clear up the mess.
> Re-edited fstab on bot sda and sdb (hd0 and hd1 dos5) and rebooted
> sda and updated-grub again.

And we're left guessing what the contents are.
Do remind me, what is msdos3 here?

> > 5 - Here is the unexplained part: My sda5 system booted up fine and normal, 
> > when I would try sdb5 the second line would say debian clean on sda5!!!
> > So I went into grub.cfg to see what is going on. On the menu entry of the 
> > second installation the first line of each entry would have the proper sdb5 
> > uuid, the following line - which I believe is the actual command - had the 
> > uuid from sda5. So, am I to assume that grub when it updates only replaces 
> > the first line of each entry expecting the next uuid will be the same, even 
> > if it is not?

No idea. Again, we don't know what you've written where, when you say
"switched to all new uuids to clear up the mess". And what's the
"second installation"? What are you calling an entry, and a menu entry?

> I know if I boot
> up something on sda12 and install or reconfigure grub in there then 12 will
> take control of booting all other bootable partitions in the system.

Well, that depends on whether you install it or just configure it.

When you generate a new grub.cfg file, grub collects bits and pieces
from the grub.cfg files it finds on the various partitions it guesses
are linux systems. This new grub.cfg will normally be written in
whatever filesystem contains /boot/grub/. However, that's not
necessarily the grub.cfg that was read and acted upon when you just
booted, nor necessarily the grub.cfg that will be active when you next
boot. You can 

Re: Intresting dd fsck grub uuid fstab action

[Fungi4All]

 Original Message 
Subject: Re: Intresting dd fsck grub uuid fstab action
UTC Time: May 25, 2017 8:57 AM
From: pas...@plouf.fr.eu.org

Your lines are too long.

I didn't know that was a problem, I've learned to write and only hit return on 
-->
a paragraph.

Le 25/05/2017 à 05:11, Fungi4All a écrit :
> I experimented in switching a clone of my sid installation to an 
> experimental, that was the plan.
> Since I was doing other things I thought I'll let the cloning take place 
> unattended.
> Let's say sda5/6/7/8/9 were to be cloned to sdb5-9 (5 / 6 var 7 sw 8 tmp 9 
> home) all b partitions were slightly larger.
> I used dd bs=1M for each one and though all I had to do is log in sda5 edit 
> the sdb5 fstab and then update-grub.
>
> 1 - Does dd leave uuid targets as they are, does it create new ones, or does 
> it copy uuid from source to target?

dd does not care about UUID or any other metadata. It just copies every
bit of data. Filesystem UUIDs and labels are part of the data, so are
cloned verbatim if you clone the partition. Partition UUIDs and labels
(mostly used on GPT partition tables) are part of the whole disk data,
so if you clone the whole disk, you also clone the partition UUIDs and
labels.

I didn't know this, I thought uuids are only created during partition creation
and are not part of the partition's contents. I've used dd for backup several
times but in temporaty mediums, not within the system. I rarely have two
drives in one box. I didn't copy the whole disk, just specific partitions of the
one disk into similar ones on the 2nd.

> 2 - When I rebooted sda5 was not booting up properly. Eventually I checked
its fstab and corrected it but at this poing the system run an fsck and I got
some errors and hit y,y,y,y, for fixing. I don't know if the errors came from
uuids existing in the system twice.

Did you clone the partition while it was mounted and used ? Then you
cloned a filesystem in a dirty state, no wonder why fsck found errors.

No, I'd never do that, but I did once trying to clone a live copy of a system on
top of a mininmal debian installation. That tought me not to copy dev and
media while they are alive, becuase that may incoprporate all the mounted
disks :) No I used a different system on a different partition to copy a parked
debian sid.

> 3 - Updating grub seemed to have made a bigger mess as now I could
boot up but the partitions were mixed up between sda and sdb parts. It
would boot-up on root / of sda and have a home in sdb9 

If the cloned disk is intended to be used in the same machine as the
original one, you should change all identifiers, UUIDs and labels,
otherwise GRUB and libblkid will be confused with duplicates. One "U" in
UUID stands for "unique".

I know, it is in trillions or more a chance of creating two of the same in a 
life
time. The first thing I did after dd was done, was to make a txt table of all
the sda* sdb* in order with uuids and rewrote the fstab on sdb, I thought
the sda would be as it was. But wasn't. It wasn't all uuids that transfered
but seem enough for me not to notice they were the same and for sda5
to boot up and discover the discrepancy. I speculate that fsck then instead
of giving sdb a new uuid it altered sda. By the time the check was done
and I run grub-upd some of the uuid in grub belonged to sda partitions and
some on sdb.
By copying from sda to sdb I never thought the original would be affected,
only the target needed fixing. That threw me off. But on the new grub.cfg
both entried would boot from the sda but with remaining partitions being
a mismatch.

> 4 - Eventually to stay safe I booted from live usb and edited all the
partitions, gave them separate labels to avoid any confusion (as "Deb
on Sda6 var..."_) and switched to all new uuids to clear up the mess.
Re-edited fstab on bot sda and sdb (hd0 and hd1 dos5) and rebooted
sda and updated-grub again.

By default update-grub uses only filesystem UUIDs, not labels. So even
if you change labels, if multiple filesystem have the same UUID, any of
them may be selected.

You missed the point.
Here are some lines as an example

if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos3 
--hint-efi=hd0,msdos3 --hint-baremetal=ahci0,msdos3 --hint='hd0,msdos3' UUID on 
sda3 XXX
else
search --no-floppy --fs-uuid --set=root UUID on sda3 YYY
fi
insmod png

On a correct grub both of those uuids are the same. I ended up with two 
different ones without editing grub.cfg
I discovered this after I had made sure that all UUIDs are unique and both 
fstab files were correct. I updated grub
and it would still only boot sda on the sdb entry. The entry label itself would 
say Debian on sdb5 and it would boot
on sda5. That made me think of looking at grub. I replaced the mismatched uuids 
manually and it booted like A1

> 5 - Here is the unexplained part:

Re: Intresting dd fsck grub uuid fstab action

[Pascal Hambourg]

Your lines are too long.

Le 25/05/2017 à 05:11, Fungi4All a écrit :

I experimented in switching a clone of my sid installation to an experimental, 
that was the plan.
Since I was doing other things I thought I'll let the cloning take place 
unattended.
Let's say sda5/6/7/8/9 were to be cloned to sdb5-9 (5 / 6 var 7 sw 8 tmp 9 
home) all b partitions were slightly larger.
I used dd bs=1M for each one and though all I had to do is log in sda5 edit the 
sdb5 fstab and then update-grub.

1 - Does dd leave uuid targets as they are, does it create new ones, or does it 
copy uuid from source to target?


dd does not care about UUID or any other metadata. It just copies every 
bit of data. Filesystem UUIDs and labels are part of the data, so are 
cloned verbatim if you clone the partition. Partition UUIDs and labels 
(mostly used on GPT partition tables) are part of the whole disk data, 
so if you clone the whole disk, you also clone the partition UUIDs and 
labels.



2 - When I rebooted sda5 was not booting up properly. Eventually I checked its 
fstab and corrected it but at this poing the system run an fsck and I got some 
errors and hit y,y,y,y, for fixing. I don't know if the errors came from uuids 
existing in the system twice.


Did you clone the partition while it was mounted and used ? Then you 
cloned a filesystem in a dirty state, no wonder why fsck found errors.



3 - Updating grub seemed to have made a bigger mess as now I could boot up but 
the partitions were mixed up between sda and sdb parts. It would boot-up on 
root / of sda and have a home in sdb9 


If the cloned disk is intended to be used in the same machine as the 
original one, you should change all identifiers, UUIDs and labels, 
otherwise GRUB and libblkid will be confused with duplicates. One "U" in 
UUID stands for "unique".



4 - Eventually to stay safe I booted from live usb and edited all the partitions, gave 
them separate labels to avoid any confusion (as "Deb on Sda6 var..."_) and 
switched to all new uuids to clear up the mess. Re-edited fstab on bot sda and sdb (hd0 
and hd1 dos5) and rebooted sda and updated-grub again.


By default update-grub uses only filesystem UUIDs, not labels. So even 
if you change labels, if multiple filesystem have the same UUID, any of 
them may be selected.



5 - Here is the unexplained part: My sda5 system booted up fine and normal, 
when I would try sdb5 the second line would say debian clean on sda5!!!

So I went into grub.cfg to see what is going on. On the menu entry of the 
second installation the first line of each entry would have the proper sdb5 
uuid, the following line - which I believe is the actual command - had the uuid 
from sda5. So, am I to assume that grub when it updates only replaces the first 
line of each entry expecting the next uuid will be the same, even if it is not?


When creating menu entries for another Linux system, update-grub copies 
the contents from grub.cfg of the other Linux system, if it exists. So 
check that grub.cfg on sdb has beed updated properly.

Intresting dd fsck grub uuid fstab action

[Fungi4All]

I experimented in switching a clone of my sid installation to an experimental, 
that was the plan.
Since I was doing other things I thought I'll let the cloning take place 
unattended.
Let's say sda5/6/7/8/9 were to be cloned to sdb5-9 (5 / 6 var 7 sw 8 tmp 9 
home) all b partitions were slightly larger.
I used dd bs=1M for each one and though all I had to do is log in sda5 edit the 
sdb5 fstab and then update-grub.

1 - Does dd leave uuid targets as they are, does it create new ones, or does it 
copy uuid from source to target?
2 - When I rebooted sda5 was not booting up properly. Eventually I checked its 
fstab and corrected it but at this poing the system run an fsck and I got some 
errors and hit y,y,y,y, for fixing. I don't know if the errors came from uuids 
existing in the system twice.
3 - Updating grub seemed to have made a bigger mess as now I could boot up but 
the partitions were mixed up between sda and sdb parts. It would boot-up on 
root / of sda and have a home in sdb9 
4 - Eventually to stay safe I booted from live usb and edited all the 
partitions, gave them separate labels to avoid any confusion (as "Deb on Sda6 
var..."_) and switched to all new uuids to clear up the mess. Re-edited fstab 
on bot sda and sdb (hd0 and hd1 dos5) and rebooted sda and updated-grub again.
5 - Here is the unexplained part: My sda5 system booted up fine and normal, 
when I would try sdb5 the second line would say debian clean on sda5!!!

So I went into grub.cfg to see what is going on. On the menu entry of the 
second installation the first line of each entry would have the proper sdb5 
uuid, the following line - which I believe is the actual command - had the uuid 
from sda5. So, am I to assume that grub when it updates only replaces the first 
line of each entry expecting the next uuid will be the same, even if it is not?

So I got it all fixed, not much to see in the experimental part for my interest 
except some kernels 4.10 and 4.11, but the puzzle is not solved. Where exactly 
did I go wrong? I'm just proud of myself for fixing it, but could there be a 
bug somewhere? For a minute or two I thought maybe grub was falsely detecting 
some kind of raid action between the two drives and mirror partitions, but I 
have no raid (intentionally).

Feel free to replicate the mess and I learned to hate gparted for taking so 
damn long from each operation to the next.

(AK)