Re: [ubuntu-uk] Script to dd from an SD card only partitioned area

[Colin Law]

On 20 September 2016 at 08:55, Neil Greenwood
 wrote:
>
>
> On 19 September 2016 21:07:13 BST, Colin Law  wrote:
>>On 19 September 2016 at 20:13, Neil Greenwood
>> wrote:
>>> Sorry for the top post, I'm on my phone.
>>>
>>> I think partimage does what you want already. Clonezilla gives a
>>(very
>>> slightly) friendlier front end, but I've not used either for several
>>> years...
>>
>>I believe partimage does not handle ext4 which would be a problem (at
>>least that is what [1] says)
>>
>
> You are right. It looks like it may work with ext4, but it's not supported. 
> fsarchiver is an alternative that does support ext4 and is written by the 
> partimage developer.

I think fsarchiver will only archive one partition at a time rather
than the complete device.

>
>>I had thought about clonezilla and will have another look at it. The
>>last time I used it (which was some time ago) it seemed overly
>>complex, but perhaps I just need to put a bit more effort in to see
>>how to use it from a script.
>>
>
> Clonezilla definitely supports ext4, using partclone which may be more easily 
> scripted...

I can report that clonezilla does exactly what I need.  Having worked
my way through the rather tortuous UI it is, in fact, very easy to
script.  To clone the SD card I just need to unmount any mounted
partitions on the device (sdb in my case):
for n in /dev/sdb* ; do umount $n ; done

Then mount the folder I want to save the image in as /home/partimag
sudo mount --bind $FOLDERTOCLONETO /home/partimag
and run the command
sudo /usr/sbin/ocs-sr -q2 -c -j2 -z1p -i 4096 -fsck-src-part -p choose
savedisk  sdb

and to restore to a card, unmount its partitions and mount
/home/partimag as above and
sudo /usr/sbin/ocs-sr -g auto -e1 auto -e2 -c -r -icds -j2 -p true
restoredisk  sdb

This seems to work perfectly, and has the added bonus that the image
is much smaller that dd (even when gzipped).  The Raspbian Lite image
which contains about 1.3GiB of files in a 2GB partition (plus the
other bits and pieces) is squashed down to just over 0.5GiB when
cloned, but is 1.2GiB when dumped with dd and then gzipped.

Thanks all for the suggestions

Colin

-- 
ubuntu-uk@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk
https://wiki.ubuntu.com/UKTeam/

Re: [ubuntu-uk] Script to dd from an SD card only partitioned area

[Neil Greenwood]

On 19 September 2016 21:07:13 BST, Colin Law  wrote:
>On 19 September 2016 at 20:13, Neil Greenwood
> wrote:
>> Sorry for the top post, I'm on my phone.
>>
>> I think partimage does what you want already. Clonezilla gives a
>(very
>> slightly) friendlier front end, but I've not used either for several
>> years...
>
>I believe partimage does not handle ext4 which would be a problem (at
>least that is what [1] says)
>

You are right. It looks like it may work with ext4, but it's not supported. 
fsarchiver is an alternative that does support ext4 and is written by the 
partimage developer. 

>I had thought about clonezilla and will have another look at it. The
>last time I used it (which was some time ago) it seemed overly
>complex, but perhaps I just need to put a bit more effort in to see
>how to use it from a script.
>

Clonezilla definitely supports ext4, using partclone which may be more easily 
scripted... 

>Thanks
>
>Colin
>
>
>[1] https://www.partimage.org/Main_Page
>

Neil 
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
ubuntu-uk@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk
https://wiki.ubuntu.com/UKTeam/

Re: [ubuntu-uk] Script to dd from an SD card only partitioned area

[Colin Law]

On 19 September 2016 at 20:13, Neil Greenwood
 wrote:
> Sorry for the top post, I'm on my phone.
>
> I think partimage does what you want already. Clonezilla gives a (very
> slightly) friendlier front end, but I've not used either for several
> years...

I believe partimage does not handle ext4 which would be a problem (at
least that is what [1] says)

I had thought about clonezilla and will have another look at it. The
last time I used it (which was some time ago) it seemed overly
complex, but perhaps I just need to put a bit more effort in to see
how to use it from a script.

Thanks

Colin


[1] https://www.partimage.org/Main_Page

>
>
> Neil
>
> On 19 September 2016 17:49:44 BST, Colin Law  wrote:
>>
>> On 19 September 2016 at 17:18, Robert McWilliam  wrote:
>>>
>>>  On Mon, 19 Sep 2016, at 14:21, Colin Law wrote:

  I do a fair amount of work with SD cards and use dd to create an image
  for backup or for burning onto other cards. If I burn an image from an
  8GB card onto a 16GB card then I get a card which is only half used.
  If I then make an image from that one then I get a 16GB image (of
  which only 8GB or less is partitioned) which is larger than it needs
  to be and also if I burn that onto another 8GB card then it fails as
  the card is not large enough (or at least it says it has failed, the
  card will in fact be ok).
>>>
>>>
>>>  You can copy a single
>>> partition by pointing dd at the partition rather
>>>  than the device, e.g. sda1 rather than sda. I expect that would achieve
>>>  the same thing as  giving dd offset and size that you can get from fdisk
>>>  (but less likely to get those wrong).
>>>
>>>  Neither approach will give you an image that you can (reliably) put back
>>>  onto a card with (just) dd. It won't include the partition table.
>>
>>
>> Is not the partition table in the space before the first partition? So
>> in the example I posted where I had
>>
>> Device Boot  Start End Sectors Size Id Type
>> /dev/sdb1 8192  137215  129024  63M  c W95 FAT32 (LBA)
>> /dev/sdb2   137216 4233215 4096000   2G 83 Linux
>>
>> is not the partition table in sectors 0 to 8191? So if I copy sectors
>> 0 to 2333215 that should include the partition table and all the
>> partitions.  Is that not correct?
>>
>>>  It
>>>  will work if the destination card is partitioned the same as the source
>>>  and you write to the same offset, or if you've got a partition the same
>>>  size and you update the offset to hit that, but otherwise you'd need to
>>>  update the partition table (and other partitions) to make an
>>>  appropriately sized gap for it and then write to that.
>>>
>>>  I think it's better to look at what you're trying to do, and see if dd
>>>  is the right tool. I can understand wanting to use dd for archiving or
>>>  backing up cards since it'll also catch things that have been deleted or
>>>  lost to filesystem corruption that you can then (try to) recover once
>>>  you've noticed that something is missing. I'm less convinced it's a good
>>>  idea going the other way; it causes the problems you're seeing when
>>>  sizes aren't the same and it means you're writing more to the cards than
>>>  you need to. I
>>> think you'd be better to mount the image file and copy
>>>  the files across to the card.
>>
>>
>> To do it that way I believe I would have to write a script to pick up
>> the partition info from the original card, mount and copy the files in
>> each partition, and save the partition info with the files. Then to
>> restore it I would need a script to re-partition the new card and copy
>> the files across to each partition.
>>
>> Colin
>
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
> --
> ubuntu-uk@lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk
> https://wiki.ubuntu.com/UKTeam/
>

-- 
ubuntu-uk@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk
https://wiki.ubuntu.com/UKTeam/

Re: [ubuntu-uk] Script to dd from an SD card only partitioned area

[Colin Law]

On 19 September 2016 at 18:37, Robert McWilliam  wrote:
> On Mon, 19 Sep 2016, at 17:49, Colin Law wrote:
>> On 19 September 2016 at 17:18, Robert McWilliam  wrote:
>> Is not the partition table in the space before the first partition? So
>> in the example I posted where I had
>>
>> Device Boot  Start End Sectors Size Id Type
>> /dev/sdb1 8192  137215  129024  63M  c W95 FAT32 (LBA)
>> /dev/sdb2   137216 4233215 4096000   2G 83 Linux
>>
>> is not the partition table in sectors 0 to 8191? So if I copy sectors
>> 0 to 2333215 that should include the partition table and all the
>> partitions.  Is that not correct?
>
> Put a 4 in front of that end sector number, lose one of the 3's, and
> yes, I think you'd get partition table and those two partitions. That
> kind of transcription error is why I don't think you should play with
> offsets directly. That looks like a human transcription error so you
> wouldn't have exactly that type of error in a script, but parsing text
> to get a number is really fragile (what appears in the "Boot" column
> there when it has something? Will your parsing be thrown off that? What
> else might change?).

Indeed, that is precisely why I hoped there would either be better
ways of determining the partion information or a better way of doing
the whole exercise.

> ...
> For a whole card image you can make loopback devices for the partitions
> in it:
> $sudo losetup -Pf --show disk_image.raw
> then mount the partitions
> $sudo mount /dev/loop0p1 /mnt/point
> then copy the files to a card (or anywhere else) with normal file
> management tools, unmount the image:
> $sudo umount /mnt/point
> and detach the loopback device
> $sudo losetup -d /dev/loop0

That is interesting, thanks.

>
> For the single partition image you don't need the loopback device and
> can mount it directly with:
> $sudo mount -o loop partition_image.raw /mnt/point
>
> As for partitioning the destination card: I'd keep them all partitioned
> with one partition the full size of the card. If you want a clean card
> to restore an image too: rm -rf /where/its/mounted (be very, very,
> careful if putting that in a script with something to determine the
> mount point...).

I don't have full control of the partitions. The example I showed is
is a bootable raspbian lite card. Another is a NOOBS card running
raspbian, another is a SheevaPlug running Ubuntu 9.04 and so on. So if
I want an automatic script it will have to clean and re-partition the
new card appropriately.

Regards

Colin

-- 
ubuntu-uk@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk
https://wiki.ubuntu.com/UKTeam/

Re: [ubuntu-uk] Script to dd from an SD card only partitioned area

[Neil Greenwood]

Sorry for the top post, I'm on my phone. 

I think partimage does what you want already. Clonezilla gives a (very 
slightly) friendlier front end, but I've not used either for several years... 


Neil 

On 19 September 2016 17:49:44 BST, Colin Law  wrote:
>On 19 September 2016 at 17:18, Robert McWilliam 
>wrote:
>> On Mon, 19 Sep 2016, at 14:21, Colin Law wrote:
>>> I do a fair amount of work with SD cards and use dd to create an
>image
>>> for backup or for burning onto other cards. If I burn an image from
>an
>>> 8GB card onto a 16GB card then I get a card which is only half used.
>>> If I then make an image from that one then I get a 16GB image (of
>>> which only 8GB or less is partitioned) which is larger than it needs
>>> to be and also if I burn that onto another 8GB card then it fails as
>>> the card is not large enough (or at least it says it has failed, the
>>> card will in fact be ok).
>>
>> You can copy a single partition by pointing dd at the partition
>rather
>> than the device, e.g. sda1 rather than sda. I expect that would
>achieve
>> the same thing as  giving dd offset and size that you can get from
>fdisk
>> (but less likely to get those wrong).
>>
>> Neither approach will give you an image that you can (reliably) put
>back
>> onto a card with (just) dd. It won't include the partition table.
>
>Is not the partition table in the space before the first partition? So
>in the example I posted where I had
>
>Device Boot  Start End Sectors Size Id Type
>/dev/sdb1 8192  137215  129024  63M  c W95 FAT32 (LBA)
>/dev/sdb2   137216 4233215 4096000   2G 83 Linux
>
>is not the partition table in sectors 0 to 8191? So if I copy sectors
>0 to 2333215 that should include the partition table and all the
>partitions.  Is that not correct?
>
>> It
>> will work if the destination card is partitioned the same as the
>source
>> and you write to the same offset, or if you've got a partition the
>same
>> size and you update the offset to hit that, but otherwise you'd need
>to
>> update the partition table (and other partitions) to make an
>> appropriately sized gap for it and then write to that.
>>
>> I think it's better to look at what you're trying to do, and see if
>dd
>> is the right tool. I can understand wanting to use dd for archiving
>or
>> backing up cards since it'll also catch things that have been deleted
>or
>> lost to filesystem corruption that you can then (try to) recover once
>> you've noticed that something is missing. I'm less convinced it's a
>good
>> idea going the other way; it causes the problems you're seeing when
>> sizes aren't the same and it means you're writing more to the cards
>than
>> you need to. I think you'd be better to mount the image file and copy
>> the files across to the card.
>
>To do it that way I believe I would have to write a script to pick up
>the partition info from the original card, mount and copy the files in
>each partition, and save the partition info with the files. Then to
>restore it I would need a script to re-partition the new card and copy
>the files across to each partition.
>
>Colin
>
>-- 
>ubuntu-uk@lists.ubuntu.com
>https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk
>https://wiki.ubuntu.com/UKTeam/

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.-- 
ubuntu-uk@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk
https://wiki.ubuntu.com/UKTeam/

Re: [ubuntu-uk] Script to dd from an SD card only partitioned area

[Robert McWilliam]

On Mon, 19 Sep 2016, at 17:49, Colin Law wrote:
> On 19 September 2016 at 17:18, Robert McWilliam  wrote:
> Is not the partition table in the space before the first partition? So
> in the example I posted where I had
> 
> Device Boot  Start End Sectors Size Id Type
> /dev/sdb1 8192  137215  129024  63M  c W95 FAT32 (LBA)
> /dev/sdb2   137216 4233215 4096000   2G 83 Linux
> 
> is not the partition table in sectors 0 to 8191? So if I copy sectors
> 0 to 2333215 that should include the partition table and all the
> partitions.  Is that not correct?

Put a 4 in front of that end sector number, lose one of the 3's, and 
yes, I think you'd get partition table and those two partitions. That
kind of transcription error is why I don't think you should play with
offsets directly. That looks like a human transcription error so you
wouldn't have exactly that type of error in a script, but parsing text
to get a number is really fragile (what appears in the "Boot" column
there when it has something? Will your parsing be thrown off that? What
else might change?). 

I had thought you were going to grab just the partition you wanted. If
you are grabbing everything up to the end of the last partition in the
partition table then yeah that'll work and can be written back to any
card big enough to take it and be usable (with some unusable space at
the end if the card is bigger than the partitions). 

> To do it that way I believe I would have to write a script to pick up
> the partition info from the original card, mount and copy the files in
> each partition, and save the partition info with the files. Then to
> restore it I would need a script to re-partition the new card and copy
> the files across to each partition.

If you have an image of the whole card then the partition table is in
it, if you have an image of just the partition then you don't need the
partition table. 

For a whole card image you can make loopback devices for the partitions
in it:
$sudo losetup -Pf --show disk_image.raw
then mount the partitions
$sudo mount /dev/loop0p1 /mnt/point
then copy the files to a card (or anywhere else) with normal file
management tools, unmount the image:
$sudo umount /mnt/point
and detach the loopback device
$sudo losetup -d /dev/loop0
 

For the single partition image you don't need the loopback device and
can mount it directly with:
$sudo mount -o loop partition_image.raw /mnt/point

As for partitioning the destination card: I'd keep them all partitioned
with one partition the full size of the card. If you want a clean card
to restore an image too: rm -rf /where/its/mounted (be very, very,
careful if putting that in a script with something to determine the
mount point...). 

Robert

-- 
ubuntu-uk@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk
https://wiki.ubuntu.com/UKTeam/

Re: [ubuntu-uk] Script to dd from an SD card only partitioned area

[Colin Law]

On 19 September 2016 at 17:18, Robert McWilliam  wrote:
> On Mon, 19 Sep 2016, at 14:21, Colin Law wrote:
>> I do a fair amount of work with SD cards and use dd to create an image
>> for backup or for burning onto other cards. If I burn an image from an
>> 8GB card onto a 16GB card then I get a card which is only half used.
>> If I then make an image from that one then I get a 16GB image (of
>> which only 8GB or less is partitioned) which is larger than it needs
>> to be and also if I burn that onto another 8GB card then it fails as
>> the card is not large enough (or at least it says it has failed, the
>> card will in fact be ok).
>
> You can copy a single partition by pointing dd at the partition rather
> than the device, e.g. sda1 rather than sda. I expect that would achieve
> the same thing as  giving dd offset and size that you can get from fdisk
> (but less likely to get those wrong).
>
> Neither approach will give you an image that you can (reliably) put back
> onto a card with (just) dd. It won't include the partition table.

Is not the partition table in the space before the first partition? So
in the example I posted where I had

Device Boot  Start End Sectors Size Id Type
/dev/sdb1 8192  137215  129024  63M  c W95 FAT32 (LBA)
/dev/sdb2   137216 4233215 4096000   2G 83 Linux

is not the partition table in sectors 0 to 8191? So if I copy sectors
0 to 2333215 that should include the partition table and all the
partitions.  Is that not correct?

> It
> will work if the destination card is partitioned the same as the source
> and you write to the same offset, or if you've got a partition the same
> size and you update the offset to hit that, but otherwise you'd need to
> update the partition table (and other partitions) to make an
> appropriately sized gap for it and then write to that.
>
> I think it's better to look at what you're trying to do, and see if dd
> is the right tool. I can understand wanting to use dd for archiving or
> backing up cards since it'll also catch things that have been deleted or
> lost to filesystem corruption that you can then (try to) recover once
> you've noticed that something is missing. I'm less convinced it's a good
> idea going the other way; it causes the problems you're seeing when
> sizes aren't the same and it means you're writing more to the cards than
> you need to. I think you'd be better to mount the image file and copy
> the files across to the card.

To do it that way I believe I would have to write a script to pick up
the partition info from the original card, mount and copy the files in
each partition, and save the partition info with the files. Then to
restore it I would need a script to re-partition the new card and copy
the files across to each partition.

Colin

-- 
ubuntu-uk@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk
https://wiki.ubuntu.com/UKTeam/

Re: [ubuntu-uk] Script to dd from an SD card only partitioned area

[Robert McWilliam]

On Mon, 19 Sep 2016, at 14:21, Colin Law wrote:
> I do a fair amount of work with SD cards and use dd to create an image
> for backup or for burning onto other cards. If I burn an image from an
> 8GB card onto a 16GB card then I get a card which is only half used.
> If I then make an image from that one then I get a 16GB image (of
> which only 8GB or less is partitioned) which is larger than it needs
> to be and also if I burn that onto another 8GB card then it fails as
> the card is not large enough (or at least it says it has failed, the
> card will in fact be ok).

You can copy a single partition by pointing dd at the partition rather
than the device, e.g. sda1 rather than sda. I expect that would achieve
the same thing as  giving dd offset and size that you can get from fdisk
(but less likely to get those wrong). 

Neither approach will give you an image that you can (reliably) put back
onto a card with (just) dd. It won't include the partition table. It
will work if the destination card is partitioned the same as the source
and you write to the same offset, or if you've got a partition the same
size and you update the offset to hit that, but otherwise you'd need to
update the partition table (and other partitions) to make an
appropriately sized gap for it and then write to that.

I think it's better to look at what you're trying to do, and see if dd
is the right tool. I can understand wanting to use dd for archiving or
backing up cards since it'll also catch things that have been deleted or
lost to filesystem corruption that you can then (try to) recover once
you've noticed that something is missing. I'm less convinced it's a good
idea going the other way; it causes the problems you're seeing when
sizes aren't the same and it means you're writing more to the cards than
you need to. I think you'd be better to mount the image file and copy
the files across to the card. 

Robert
--
Robert McWilliam  r...@allmail.netwww.ormiret.com

Apparently, three out of four people make up 75% of the population.

-- 
ubuntu-uk@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk
https://wiki.ubuntu.com/UKTeam/