Re: [Hampshire] Grouping Files for backup

2019-05-29 Thread Simon Reap via Hampshire 

To use multi-volume, do something like:

    cd to directory with files
    tar cvf /dir/out1.tar -M -L 52428800 *
(where /dir/ is a place to hold the 50GB tar files, and 52428800 is 
50*1024*1024, the number of 1k blocks in 50GB)


It will write the first 50GB to /dir/out1.tar, then prompt with:
  Prepare volume #2 for '/dir/out1.tar' and hit return:
Type in
    n /dir/out2.tar
(that's 'n space /dir' etc) and press return. The next 50GB goes to 
/dir/out2.tar and it prompts with

    Prepare volume #3 for '/dir/out2.tar' and hit return:
Type in
    n /dir/out3.tar
and press return. Continue for 37 more files!

(When I started writing this, I didn't realise how much effort it was!).



Another solution could be:

    tar cvf - * | split -b 52428800 -d - /dir/out

This will write /dir/out00, /dir/out01, /dir/out02 and so on to 
/dir/out40. The tar writes to stdout and pipes to split. The '-b' splits 
the file into that number of bytes per file, and the '-d' uses numeric 
suffices on the filenames (otherwise it would be aa, ab, ac etc to b0). 
The '-' reads from stdin, and the '/dir/out' is the prefix for the 
output filenames.




This splits the files at a given number of bytes, so all parts need to 
be connected together to extract data. The benefit of the 'tar -M' 
version is that each part is a valid tar file (though some files may be 
split across two tar files).


Simon

On 29/05/2019 16:12, James Courtier-Dutton via Hampshire wrote:

Hi,
 Tar already has such a feature. A long time ago, it would have needed 
to spread data across backup tapes.

look at the man page for tag for something like --multi-volume.

Kind Regards

James


On Wed, 29 May 2019 at 12:12, Rob Malpass via Hampshire 
mailto:hampshire@mailman.lug.org.uk>> 
wrote:


Hi all

I’ve never been a bash script expert (to say the least!) but I
need a clever script (I assume that’s the easiest way but please
say if not) that can turn 120 files (amounting to 2Tb of data)
into 40 files of <=50Gb for an archiving project I’m about to
embark on.   Can anyone help / give me some pointers?

I can guarantee none of the files are over 50Gb in size. So it’s a
question of something like:

Create an empty archive (call it n.tar.gz)

Repeat

Find next (uncompressed) file

Compress it to tmp.tar.gz

If size of tmp.tar.gz < (50Gb – current size of n.tar.gz) then

    Append tmp.tar.gz to n.tar.gz

Else

    Close n.tar.gz

Alert me so I can write n.tar.gz to MD

    Create next empty archive ((n+1).tar.gz)

Add tmp.tar.gz to (n+1).tar.gz

End If

Until all files have been compressed

The end product should be 40 files called n.tar.gz (where 0-- 
Please post to: Hampshire@mailman.lug.org.uk


Web Interface: https://mailman.lug.org.uk/mailman/listinfo/hampshire
LUG URL: http://www.hantslug.org.uk
--


 
	Virus-free. www.avg.com 
 



<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>



-- 
Please post to: Hampshire@mailman.lug.org.uk
Web Interface: https://mailman.lug.org.uk/mailman/listinfo/hampshire
LUG URL: http://www.hantslug.org.uk
--

Re: [Hampshire] Grouping Files for backup

2019-05-29 Thread James Courtier-Dutton via Hampshire 
Hi,
 Tar already has such a feature. A long time ago, it would have needed to
spread data across backup tapes.
look at the man page for tag for something like --multi-volume.

Kind Regards

James


On Wed, 29 May 2019 at 12:12, Rob Malpass via Hampshire <
hampshire@mailman.lug.org.uk> wrote:

> Hi all
>
>
>
> I’ve never been a bash script expert (to say the least!) but I need a
> clever script (I assume that’s the easiest way but please say if not) that
> can turn 120 files (amounting to 2Tb of data) into 40 files of <=50Gb for
> an archiving project I’m about to embark on.   Can anyone help / give me
> some pointers?
>
>
>
> I can guarantee none of the files are over 50Gb in size.   So it’s a
> question of something like:
>
> Create an empty archive (call it n.tar.gz)
>
> Repeat
>
> Find next (uncompressed) file
>
> Compress it to tmp.tar.gz
>
> If size of tmp.tar.gz < (50Gb – current size of n.tar.gz) then
>
> Append tmp.tar.gz to n.tar.gz
>
> Else
>
> Close n.tar.gz
>
> Alert me so I can write n.tar.gz to MD
>
> Create next empty archive ((n+1).tar.gz)
>
> Add tmp.tar.gz to (n+1).tar.gz
>
> End If
>
> Until all files have been compressed
>
>
>
> The end product should be 40 files called n.tar.gz (where 0
>
>
> Any (constructive!) ideas very welcome.
>
>
>
> Cheers
>
>
>
>
> --
> Please post to: Hampshire@mailman.lug.org.uk
> Web Interface: https://mailman.lug.org.uk/mailman/listinfo/hampshire
> LUG URL: http://www.hantslug.org.uk
> --
-- 
Please post to: Hampshire@mailman.lug.org.uk
Web Interface: https://mailman.lug.org.uk/mailman/listinfo/hampshire
LUG URL: http://www.hantslug.org.uk
--

[Hampshire] Grouping Files for backup

2019-05-29 Thread Rob Malpass via Hampshire 
Hi all

 

I've never been a bash script expert (to say the least!) but I need a clever
script (I assume that's the easiest way but please say if not) that can turn
120 files (amounting to 2Tb of data) into 40 files of <=50Gb for an
archiving project I'm about to embark on.   Can anyone help / give me some
pointers?

 

I can guarantee none of the files are over 50Gb in size.   So it's a
question of something like:

Create an empty archive (call it n.tar.gz)

Repeat

Find next (uncompressed) file

Compress it to tmp.tar.gz

If size of tmp.tar.gz < (50Gb - current size of n.tar.gz) then

Append tmp.tar.gz to n.tar.gz

Else

Close n.tar.gz

Alert me so I can write n.tar.gz to MD

Create next empty archive ((n+1).tar.gz)

Add tmp.tar.gz to (n+1).tar.gz

End If

Until all files have been compressed

 

The end product should be 40 files called n.tar.gz (where 0-- 
Please post to: Hampshire@mailman.lug.org.uk
Web Interface: https://mailman.lug.org.uk/mailman/listinfo/hampshire
LUG URL: http://www.hantslug.org.uk
--