Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Peter Humphrey]

On Friday 09 Sep 2016 23:25:10 Andrew Lowe wrote:

>   I've just spent 7 hours in a bottleshop in an entertainment area,
> putting up with idiots swearing all night. Now I come home and catch up
> on what's happening on this list and what do I get? More drop kicks
> swearing their heads off. It's just an indication of the lack of your
> grasp of the English language when you start carrying on like this.
> 
> ... There is a time and a place for everything, a public technical email
> list is not one of them for swearing.

A man after my own heart.

When a new couple took over my local pub last year, they split the village* 
beer drinkers down the middle by enforcing a no-swearing policy. Now all 
those men who can't say anything without an f*** every other word go to the 
other main pub, leaving the one I use to those who can. Trade has soared, 
and local people have been appearing who were never seen in a pub before. 
Winners all round.

* Tideswell, Derbyshire, in case you're interested.

-- 
Rgds
Peter

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Andrew Lowe]

On 01/09/16 17:01, Alan McKinnon wrote:

On 01/09/2016 09:18, gevisz wrote:

2016-09-01 9:13 GMT+03:00 Alan McKinnon :

On 01/09/2016 08:04, gevisz wrote:


[snip]


[snip]
...
...
[snip]


That is the most stupid dumbass argument I've heard in weeks.
It doesn't even deserve a response.

Who the fuck is promoting this shit?




	I've just spent 7 hours in a bottleshop in an entertainment area, 
putting up with idiots swearing all night. Now I come home and catch up 
on what's happening on this list and what do I get? More drop kicks 
swearing their heads off. It's just an indication of the lack of your 
grasp of the English language when you start carrying on like this.


	Don't get me wrong, I can swear with the best of them, I spent my late 
teens working as a barman in a pub opposite one of the hardest gaols you 
will find anywhere. There is a time and a place for everything, a public 
technical email list is not one of them for swearing.


	How about you just pull your head in until you learn some "big boy" 
words and can contribute in a civil manner?


Just my 5c worth,
Andrew

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Volker Armin Hemmann]

Am 08.09.2016 um 00:47 schrieb Alan McKinnon:
> On 08/09/2016 00:12, Volker Armin Hemmann wrote:
>> Am 07.09.2016 um 08:18 schrieb Alan McKinnon:
>>> On 07/09/2016 01:57, Volker Armin Hemmann wrote:
 Am 01.09.2016 um 11:01 schrieb Alan McKinnon:
> On 01/09/2016 09:18, gevisz wrote:
>> 2016-09-01 9:13 GMT+03:00 Alan McKinnon :
>>> On 01/09/2016 08:04, gevisz wrote:
> [snip]
>
>>> it will take about 5 seconds to partition it.
>>> And a few more to mkfs it.
>> Just to partition - may be, but I very much doubt
>> that it will take seconds to create a full-fledged
>> ext4 file system on these 5TB via USB2 connention.
> Do it. Tell me how long it tool.
>
> Discussing it without doing it and offering someone else's opinion
> is a
> 100% worthless activity
>
>> Even more: my aquiantance from the Window world
>> that recomended me this disc scared me that it may
>> take days...
> Mickey Mouse told me it takes microseconds. So what?
>
> Do it. Tell me how long it took.
>
 Is it still advisable to partition a big hard drive
 into smaller logical ones and why?
>>> The only reason to partition a drive is to get 2 or more
>>> smaller ones that differ somehow (size, inode ratio, mount
>>> options, etc)
>>>
>>> Go with no partition table by all means, but if you one day find
>>> you
>>> need one, you will have to copy all your data off, repartition,
>>> and copy
>>> your data back. If you are certain that will not happen (eg you
>>> will
>>> rather buy a second drive) then by all means dispense with
>>> partitions.
>>>
>>> They are after all nothing more than a Microsoft invention from
>>> the 80s
>>> so people could install UCSD Pascal next to MS-DOS
>> I definitely will not need more than one mount point for this
>> hard drive
>> but I do remember some arguments that partitioning a large hard
>> drive
>> into smaller logical ones gives me more safety in case a file system
>> suddenly will get corrupted because in this case I will loose my
>> data
>> only on one of the logical partitions and not on the whole drive.
>>
>> Is this argument still valid nowadays?
> That is the most stupid dumbass argument I've heard in weeks.
> It doesn't even deserve a response.
>
> Who the fuck is promoting this shit?
>
>
 people who had to deal with corrupted filesystems in the past?


>>> The way to deal with the problem of fs corruption is to have reliable
>>> tested backups.
>>>
>>> The wrong way to deal with the problem of fs corruption is to get into
>>> cargo-cult manoeuvrers thinking that lots of little bits making a whole
>>> is going to solve the problem.
>>>
>>> Especially when the part of the disk statistically most at risk is the
>>> valuable data itself. OS code can be rebuilt easily, without backups
>>> data can't.
>>>
>>
>> the bigger the drive, the greater the chance of fs corruption. Just by
>> statistics. Better one minor partition is lost than everything.
>
> What are the statistical chances of that one minor partition being the
> one that gets corrupted? Statistically the odds are very small.
>
> Think about it, if the minor partition is say 5% of the disk and if
> all other things are exactly equal, the odds are 1 in 20.
>
> Apart from inherent defects in the drive itself, the sectors that are
> more prone to failing are those that are read the most and to a larger
> extent those that are written the most.
>
> What is read the most? OS and Data
> What is written the most? Data
> What has by far the greatest likelihood of suffering fs corruption? Data

and that is why spreading data over several partitions is not a bad idea.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[waltdnes]

On Thu, Sep 08, 2016 at 12:12:07AM +0200, Volker Armin Hemmann wrote

> You can disagree as much as you like, but with the size of drives and
> the current error rate of consumer hard drives it is not a question of
> 'if' but just a matter of 'when'.

  It's not just the drive; it's the entire PC.  My main desktop at home
has had a few panics recently.  It's several years old, and I'll be
getting rid of it, because I can't really trust it.  I've switched to my
"hot backup", and am currently setting up a new machine as the new "hot
backup".  After doing the initial Gentoo install, I copied over the
config files, with appropriate changes.  I copied /var/lib/portage/world
and launched "emerge --changed-use --deep --update @world".  A few
minutes ago, emerge was on package 228 of 337.

  I have 3 USB backup drives and I use them all.  This does not include
the monthly copying over of /home/waltdnes and /home/misc from the main
desktop to the "hot backup".

  BTW, this is probably the first email sent out from this machine to
the Gentoo list.

-- 
Walter Dnes 
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Alan McKinnon]

On 08/09/2016 00:12, Volker Armin Hemmann wrote:

Am 07.09.2016 um 08:18 schrieb Alan McKinnon:

On 07/09/2016 01:57, Volker Armin Hemmann wrote:

Am 01.09.2016 um 11:01 schrieb Alan McKinnon:

On 01/09/2016 09:18, gevisz wrote:

2016-09-01 9:13 GMT+03:00 Alan McKinnon :

On 01/09/2016 08:04, gevisz wrote:

[snip]


it will take about 5 seconds to partition it.
And a few more to mkfs it.

Just to partition - may be, but I very much doubt
that it will take seconds to create a full-fledged
ext4 file system on these 5TB via USB2 connention.

Do it. Tell me how long it tool.

Discussing it without doing it and offering someone else's opinion is a
100% worthless activity


Even more: my aquiantance from the Window world
that recomended me this disc scared me that it may
take days...

Mickey Mouse told me it takes microseconds. So what?

Do it. Tell me how long it took.


Is it still advisable to partition a big hard drive
into smaller logical ones and why?

The only reason to partition a drive is to get 2 or more
smaller ones that differ somehow (size, inode ratio, mount options, etc)

Go with no partition table by all means, but if you one day find you
need one, you will have to copy all your data off, repartition, and copy
your data back. If you are certain that will not happen (eg you will
rather buy a second drive) then by all means dispense with partitions.

They are after all nothing more than a Microsoft invention from the 80s
so people could install UCSD Pascal next to MS-DOS

I definitely will not need more than one mount point for this hard drive
but I do remember some arguments that partitioning a large hard drive
into smaller logical ones gives me more safety in case a file system
suddenly will get corrupted because in this case I will loose my data
only on one of the logical partitions and not on the whole drive.

Is this argument still valid nowadays?

That is the most stupid dumbass argument I've heard in weeks.
It doesn't even deserve a response.

Who the fuck is promoting this shit?



people who had to deal with corrupted filesystems in the past?



The way to deal with the problem of fs corruption is to have reliable
tested backups.

The wrong way to deal with the problem of fs corruption is to get into
cargo-cult manoeuvrers thinking that lots of little bits making a whole
is going to solve the problem.

Especially when the part of the disk statistically most at risk is the
valuable data itself. OS code can be rebuilt easily, without backups
data can't.



the bigger the drive, the greater the chance of fs corruption. Just by
statistics. Better one minor partition is lost than everything.


What are the statistical chances of that one minor partition being the 
one that gets corrupted? Statistically the odds are very small.


Think about it, if the minor partition is say 5% of the disk and if all 
other things are exactly equal, the odds are 1 in 20.


Apart from inherent defects in the drive itself, the sectors that are 
more prone to failing are those that are read the most and to a larger 
extent those that are written the most.


What is read the most? OS and Data
What is written the most? Data
What has by far the greatest likelihood of suffering fs corruption? Data





You can disagree as much as you like, but with the size of drives and
the current error rate of consumer hard drives it is not a question of
'if' but just a matter of 'when'.



I don't disagree with you. I'm disagreeing with cargo cult mentality 
that dividing a disk up into lots of smaller partitions somehow 
magically confers significant safety margins of some magical kind. Go 
read the OPs opening statement again, he's quoting a friend from 20 
years ago and the statement consists entirely of woo-woo magic 
hand-wavey statements, the kind of shit I have to deal with every day 
from twits with just enough IQ to read executive white papers.


Yes, drives fail. Yes, consumer drives are crap. With 3TB now being 
common place and prices plunging, we have 20G or so for OS and 2980GB 
full of data. That 20G is so small and immaterial in terms of risk we 
can just disregard it and assume the only thing that can be damaged is 
2980G of data.


Solution: back up the whole damn lot properly and forget what we did 20 
years ago. That was farting in a breeze, nowadays it's farting in a 
hurricane.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Volker Armin Hemmann]

Am 07.09.2016 um 08:18 schrieb Alan McKinnon:
> On 07/09/2016 01:57, Volker Armin Hemmann wrote:
>> Am 01.09.2016 um 11:01 schrieb Alan McKinnon:
>>> On 01/09/2016 09:18, gevisz wrote:
 2016-09-01 9:13 GMT+03:00 Alan McKinnon :
> On 01/09/2016 08:04, gevisz wrote:
>>> [snip]
>>>
> it will take about 5 seconds to partition it.
> And a few more to mkfs it.
 Just to partition - may be, but I very much doubt
 that it will take seconds to create a full-fledged
 ext4 file system on these 5TB via USB2 connention.
>>> Do it. Tell me how long it tool.
>>>
>>> Discussing it without doing it and offering someone else's opinion is a
>>> 100% worthless activity
>>>
 Even more: my aquiantance from the Window world
 that recomended me this disc scared me that it may
 take days...
>>> Mickey Mouse told me it takes microseconds. So what?
>>>
>>> Do it. Tell me how long it took.
>>>
>> Is it still advisable to partition a big hard drive
>> into smaller logical ones and why?
> The only reason to partition a drive is to get 2 or more
> smaller ones that differ somehow (size, inode ratio, mount options, etc)
>
> Go with no partition table by all means, but if you one day find you
> need one, you will have to copy all your data off, repartition, and copy
> your data back. If you are certain that will not happen (eg you will
> rather buy a second drive) then by all means dispense with partitions.
>
> They are after all nothing more than a Microsoft invention from the 80s
> so people could install UCSD Pascal next to MS-DOS
 I definitely will not need more than one mount point for this hard drive
 but I do remember some arguments that partitioning a large hard drive
 into smaller logical ones gives me more safety in case a file system
 suddenly will get corrupted because in this case I will loose my data
 only on one of the logical partitions and not on the whole drive.

 Is this argument still valid nowadays?
>>> That is the most stupid dumbass argument I've heard in weeks.
>>> It doesn't even deserve a response.
>>>
>>> Who the fuck is promoting this shit?
>>>
>>>
>> people who had to deal with corrupted filesystems in the past?
>>
>>
> The way to deal with the problem of fs corruption is to have reliable
> tested backups.
>
> The wrong way to deal with the problem of fs corruption is to get into
> cargo-cult manoeuvrers thinking that lots of little bits making a whole
> is going to solve the problem.
>
> Especially when the part of the disk statistically most at risk is the
> valuable data itself. OS code can be rebuilt easily, without backups
> data can't.
>

the bigger the drive, the greater the chance of fs corruption. Just by
statistics. Better one minor partition is lost than everything.

You can disagree as much as you like, but with the size of drives and
the current error rate of consumer hard drives it is not a question of
'if' but just a matter of 'when'.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Alan McKinnon]

On 07/09/2016 01:57, Volker Armin Hemmann wrote:
> Am 01.09.2016 um 11:01 schrieb Alan McKinnon:
>> On 01/09/2016 09:18, gevisz wrote:
>>> 2016-09-01 9:13 GMT+03:00 Alan McKinnon :
 On 01/09/2016 08:04, gevisz wrote:
>> [snip]
>>
 it will take about 5 seconds to partition it.
 And a few more to mkfs it.
>>> Just to partition - may be, but I very much doubt
>>> that it will take seconds to create a full-fledged
>>> ext4 file system on these 5TB via USB2 connention.
>>
>> Do it. Tell me how long it tool.
>>
>> Discussing it without doing it and offering someone else's opinion is a
>> 100% worthless activity
>>
>>> Even more: my aquiantance from the Window world
>>> that recomended me this disc scared me that it may
>>> take days...
>> Mickey Mouse told me it takes microseconds. So what?
>>
>> Do it. Tell me how long it took.
>>
> Is it still advisable to partition a big hard drive
> into smaller logical ones and why?
 The only reason to partition a drive is to get 2 or more
 smaller ones that differ somehow (size, inode ratio, mount options, etc)

 Go with no partition table by all means, but if you one day find you
 need one, you will have to copy all your data off, repartition, and copy
 your data back. If you are certain that will not happen (eg you will
 rather buy a second drive) then by all means dispense with partitions.

 They are after all nothing more than a Microsoft invention from the 80s
 so people could install UCSD Pascal next to MS-DOS
>>> I definitely will not need more than one mount point for this hard drive
>>> but I do remember some arguments that partitioning a large hard drive
>>> into smaller logical ones gives me more safety in case a file system
>>> suddenly will get corrupted because in this case I will loose my data
>>> only on one of the logical partitions and not on the whole drive.
>>>
>>> Is this argument still valid nowadays?
>> That is the most stupid dumbass argument I've heard in weeks.
>> It doesn't even deserve a response.
>>
>> Who the fuck is promoting this shit?
>>
>>
> people who had to deal with corrupted filesystems in the past?
> 
> 

The way to deal with the problem of fs corruption is to have reliable
tested backups.

The wrong way to deal with the problem of fs corruption is to get into
cargo-cult manoeuvrers thinking that lots of little bits making a whole
is going to solve the problem.

Especially when the part of the disk statistically most at risk is the
valuable data itself. OS code can be rebuilt easily, without backups
data can't.

-- 
Alan McKinnon
alan.mckin...@gmail.com

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Volker Armin Hemmann]

Am 01.09.2016 um 11:01 schrieb Alan McKinnon:
> On 01/09/2016 09:18, gevisz wrote:
>> 2016-09-01 9:13 GMT+03:00 Alan McKinnon :
>>> On 01/09/2016 08:04, gevisz wrote:
> [snip]
>
>>> it will take about 5 seconds to partition it.
>>> And a few more to mkfs it.
>> Just to partition - may be, but I very much doubt
>> that it will take seconds to create a full-fledged
>> ext4 file system on these 5TB via USB2 connention.
>
> Do it. Tell me how long it tool.
>
> Discussing it without doing it and offering someone else's opinion is a
> 100% worthless activity
>
>> Even more: my aquiantance from the Window world
>> that recomended me this disc scared me that it may
>> take days...
> Mickey Mouse told me it takes microseconds. So what?
>
> Do it. Tell me how long it took.
>
 Is it still advisable to partition a big hard drive
 into smaller logical ones and why?
>>> The only reason to partition a drive is to get 2 or more
>>> smaller ones that differ somehow (size, inode ratio, mount options, etc)
>>>
>>> Go with no partition table by all means, but if you one day find you
>>> need one, you will have to copy all your data off, repartition, and copy
>>> your data back. If you are certain that will not happen (eg you will
>>> rather buy a second drive) then by all means dispense with partitions.
>>>
>>> They are after all nothing more than a Microsoft invention from the 80s
>>> so people could install UCSD Pascal next to MS-DOS
>> I definitely will not need more than one mount point for this hard drive
>> but I do remember some arguments that partitioning a large hard drive
>> into smaller logical ones gives me more safety in case a file system
>> suddenly will get corrupted because in this case I will loose my data
>> only on one of the logical partitions and not on the whole drive.
>>
>> Is this argument still valid nowadays?
> That is the most stupid dumbass argument I've heard in weeks.
> It doesn't even deserve a response.
>
> Who the fuck is promoting this shit?
>
>
people who had to deal with corrupted filesystems in the past?

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Mick]

On Sunday 04 Sep 2016 17:48:14 Stroller wrote:
> > On 3 Sep 2016, at 17:50, Mick  wrote:

> > Yes, flash drives (unlike spinning drivers) are completely digital.  In
> > addition, wear levelling algorithms invariably kick in and bits and bytes
> > are sprayed all over the pages/modules of the memory chips.  So you could
> > say they are fragmented by design.
> 
> That would seem to dismiss the problem, "oh, they're fragmented by design,
> thus it's unimportant".

I'm far from an expert on NOR/NAND flash drives and therefore I didn't mean to 
sound dismissive.  I was merely highlighting the fact that the memory 
controller on these cards interferes with whatever our OS is trying to write 
on them, as the card's chip controller implements various wear levelling 
algorithms.


> My understanding is that defragmenting a flash device (although I think,
> personally, I would only do this by deleting all the files on the drive,
> and copying them back) can make for faster access.
> 
> • http://www.lagom.nl/misc/flash_fragmentation.html
> •
> http://www.wizcode.com/articles/comments/flash_memory_fragmentation_myths_a
> nd_facts/
> 
> Stroller.

Some of these tests assume that flushing the OS cache *also* flushes the cache 
on the flash drive.  This is not so, especially on more modern flush drives.  
I've been watching the behaviour of a Verbatim 32G USB stick I use more or 
less daily and I am convinced that running sync following a copy operation on 
my PC, in no way means the cache on the flash controller is also flushed.

What these tests prove is that when the card is full it takes longer to write 
content on it, because blocks will have to be erased before they can be 
written on.  The cluster size is quite important for this performance, as is 
the size of the file(s) being copied.

What I am saying is that the write operation performance is determined by the 
cluster size, the file size, the flash drive's cache size and most importantly 
by the flash drive controller's wear levelling algorithms.  There is no 
guarantee that data will be written contiguously, although they will be 
written in one-block-at-a-time.  The blocks themselves almost certainly will 
not be contiguous on a used drive.  Formatting it with unsuitable logical 
block sizes for its physical block size will almost certainly incur a write 
penalty (always depending on the size of the file being written).

This is what I meant when I said USB flash drives are fragmented by design.
-- 
Regards,
Mick

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Stroller]

> On 3 Sep 2016, at 17:50, Mick  wrote:
>> 
>> I understood that fragmentation can also occur on flash-based disks.
>> 
>> Although the effect of it is not so noticeable, I understood that it still
>> has one.
> 
> Yes, flash drives (unlike spinning drivers) are completely digital.  In 
> addition, wear levelling algorithms invariably kick in and bits and bytes are 
> sprayed all over the pages/modules of the memory chips.  So you could say 
> they 
> are fragmented by design.

That would seem to dismiss the problem, "oh, they're fragmented by design, thus 
it's unimportant".

My understanding is that defragmenting a flash device (although I think, 
personally, I would only do this by deleting all the files on the drive, and 
copying them back) can make for faster access.

• http://www.lagom.nl/misc/flash_fragmentation.html
• 
http://www.wizcode.com/articles/comments/flash_memory_fragmentation_myths_and_facts/

Stroller.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Mick]

On Saturday 03 Sep 2016 16:39:03 Stroller wrote:
> > On 2 Sep 2016, at 23:03, Mick  wrote:
> > 
> > … a potentially more effective defrag method irrespective of fs
> > (we're talking about spinning disks where this issue applies) i
> 
> I understood that fragmentation can also occur on flash-based disks.
> 
> Although the effect of it is not so noticeable, I understood that it still
> has one.
> 
> Stroller.

Yes, flash drives (unlike spinning drivers) are completely digital.  In 
addition, wear levelling algorithms invariably kick in and bits and bytes are 
sprayed all over the pages/modules of the memory chips.  So you could say they 
are fragmented by design.

-- 
Regards,
Mick

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Stroller]

> On 2 Sep 2016, at 23:03, Mick  wrote:
> 
> … a potentially more effective defrag method irrespective of fs 
> (we're talking about spinning disks where this issue applies) i

I understood that fragmentation can also occur on flash-based disks.

Although the effect of it is not so noticeable, I understood that it still has 
one.

Stroller.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

On Fri, 2 Sep 2016 07:48:17 -0400, Rich Freeman wrote:

> > I use a script to handle my snapshots, so snapshotting multiple
> > subvolumes is less of an issue, but an option to snapshot a subvolume
> > and all its children, or even the whole filesystem, would be nice.
> >  
> 
> Note that what I want is a snapshot that crosses subvolume boundaries,
> so that it is atomic.  Not a program that just iterates creating
> individual snapshots that don't all happen at the exact same time.

Yes, I get that. It would be a nice feature.


-- 
Neil Bothwick

If the bank returns your cheque marked "Insufficient Funds," call them
 and ask if they mean you or them. :-)


pgpo9ekX8SOvU.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Rich Freeman]

On Fri, Sep 2, 2016 at 6:59 AM, Neil Bothwick  wrote:
> On Thu, 1 Sep 2016 13:54:40 -0400, Rich Freeman wrote:
>
>> > Bind mounts? I thought you would use btrfs subvolumes!
>> >
>>
>> Often the bind mounts point to btrfs subvolumes.
>>
>> Yeah, I guess I could directly mount all those subvolumes, but I find
>> symlinks or bind mounts easier.  The other factor is that if I have
>> unnecessary subvolumes then I'm having to manage snapshots across more
>> of them and my snapshots are less atomic, since snapshots don't cross
>> subvolume boundaries (which is something which ought to be
>> configurable).
>
> I use a script to handle my snapshots, so snapshotting multiple
> subvolumes is less of an issue, but an option to snapshot a subvolume and
> all its children, or even the whole filesystem, would be nice.
>

Note that what I want is a snapshot that crosses subvolume boundaries,
so that it is atomic.  Not a program that just iterates creating
individual snapshots that don't all happen at the exact same time.

I'd have to look a little more closely at how the filesystem roots
work to see if that is actually possible.  I don't know if the root
node actually covers all the subvolumes it contains, and how exactly
subvolumes are bound to their containing directories.

I guess if the structure of the tree doesn't allow a single snapshot
at the data structure level another option would be for the filesystem
to create a write barrier / lock of some kind while the snapshots are
being created, so they end up being consistent anyway.  This approach
could work even across different filesystems.

-- 
Rich

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

On Thu, 1 Sep 2016 13:54:40 -0400, Rich Freeman wrote:

> > Bind mounts? I thought you would use btrfs subvolumes!
> >  
> 
> Often the bind mounts point to btrfs subvolumes.
> 
> Yeah, I guess I could directly mount all those subvolumes, but I find
> symlinks or bind mounts easier.  The other factor is that if I have
> unnecessary subvolumes then I'm having to manage snapshots across more
> of them and my snapshots are less atomic, since snapshots don't cross
> subvolume boundaries (which is something which ought to be
> configurable).

I use a script to handle my snapshots, so snapshotting multiple
subvolumes is less of an issue, but an option to snapshot a subvolume and
all its children, or even the whole filesystem, would be nice.


-- 
Neil Bothwick

Remember, it takes 47 muscles to frown
And only 4 to pull the trigger of a sniper rifle


pgpcS56fycjMK.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-02 7:23 GMT+03:00 gevisz :
> 2016-09-01 11:55 GMT+03:00 Frank Steinmetzger :
>> On Thu, Sep 01, 2016 at 11:44:19AM +0300, gevisz wrote:
>>
>>> > Some people do a full systems check (i.e. badblocks) before entrusting a
>>> > drive with anything important.
>>>
>>> It is a good advice! I have already thought of this but I am sorry to
>>> acknowledge that, since the "old good times" of MS DOS 6.22, I never did
>>> this in Linux. :(
>>> […]
>>> So, can you, please, advice me about the program or utility that can do
>>> badblocks check for me?
>>
>> Badblocks is part of e2fsprogs. But since you’re using USB2, this will
>> really take a while. At best I get 39 MB/s out of it. Another way is a
>> S.M.A.R.T. test, methinks `smartctl -t full` is the command for that. But I
>> don’t know what exactly is being tested there. But it runs fully internal of
>> the disk, so no USB2-bottleneck. Others may chime in if I tell fairy tales.
>
> So far, the hard drive passed two (small) smart tests started by commands:
> # smartctl -c -t short -d sat /dev/sdc
> and
> # smartctl -t conveyance -d sat /dev/sdc



> However, after running
> # smartctl -t long -d sat /dev/sdc
> I have no indication that it has been passed:
> # smartctl -t long -d sat /dev/sdc
> smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.4.6-gentoo] (local build)
> Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
> Sending command: "Execute SMART Extended self-test routine immediately
> in off-line mode".
> Drive command "Execute SMART Extended self-test routine immediately in
> off-line mode" successful.
> Testing has begun.
> Please wait 571 minutes for test to complete.
> Test will complete after Fri Sep  2 04:02:18 2016
>
> Use smartctl -X to abort test.
>
> Fri Sep 2 6:10
> # smartctl -l selftest -d sat /dev/sdc
> smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.4.6-gentoo] (local build)
> Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org
>
> Read Device Identity failed: scsi error device will be ready soon
>
> A mandatory SMART command failed: exiting. To continue, add one or
> more '-T permissive' options.

Well, may be, it has not been finished yet.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 11:55 GMT+03:00 Frank Steinmetzger :
> On Thu, Sep 01, 2016 at 11:44:19AM +0300, gevisz wrote:
>
>> > Some people do a full systems check (i.e. badblocks) before entrusting a
>> > drive with anything important.
>>
>> It is a good advice! I have already thought of this but I am sorry to
>> acknowledge that, since the "old good times" of MS DOS 6.22, I never did
>> this in Linux. :(
>> […]
>> So, can you, please, advice me about the program or utility that can do
>> badblocks check for me?
>
> Badblocks is part of e2fsprogs. But since you’re using USB2, this will
> really take a while. At best I get 39 MB/s out of it. Another way is a
> S.M.A.R.T. test, methinks `smartctl -t full` is the command for that. But I
> don’t know what exactly is being tested there. But it runs fully internal of
> the disk, so no USB2-bottleneck. Others may chime in if I tell fairy tales.

So far, the hard drive passed two (small) smart tests started by commands:
# smartctl -c -t short -d sat /dev/sdc
and
# smartctl -t conveyance -d sat /dev/sdc
as is indicated by
# smartctl -l selftest -d sat /dev/sdc
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.4.6-gentoo] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_DescriptionStatus  Remaining
LifeTime(hours)  LBA_of_first_error
# 1  Conveyance offline  Completed without error   00% 0 -
# 2  Short offline   Completed without error   00% 0 -

However, after running
# smartctl -t long -d sat /dev/sdc
I have no indication that it has been passed:
# smartctl -t long -d sat /dev/sdc
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.4.6-gentoo] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately
in off-line mode".
Drive command "Execute SMART Extended self-test routine immediately in
off-line mode" successful.
Testing has begun.
Please wait 571 minutes for test to complete.
Test will complete after Fri Sep  2 04:02:18 2016

Use smartctl -X to abort test.

Fri Sep 2 6:10
# smartctl -l selftest -d sat /dev/sdc
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.4.6-gentoo] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

Read Device Identity failed: scsi error device will be ready soon

A mandatory SMART command failed: exiting. To continue, add one or
more '-T permissive' options.

# smartctl -a -d sat /dev/sdc
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.4.6-gentoo] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: WDC WD50EZRZ-00GZ5B1
Serial Number:   
LU WWN Device Id: 
Firmware Version: 80.00A80
User Capacity:5,000,981,078,016 bytes [5.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate:5700 rpm
Device is:Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:Fri Sep  2 06:12:50 2016 EEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80)Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status:  (   0)The previous self-test
routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (57180) seconds.
Offline data collection
capabilities:  (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities:(0x0003)Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability:(0x01)Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time:  (   2) minutes.
Extended self-test routine
recommended polling time:  ( 571) minutes.
Conveyance self-test routine
recommended polling time:  (   5) minutes.
SCT capabilities:(0x3035)SCT Status supported.
 

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Alan McKinnon]

On 01/09/2016 10:49, gevisz wrote:

2016-09-01 10:30 GMT+03:00 Matthias Hanft :

gevisz wrote:


But what are disadvantages of not partitioning a big
hard drive into smaller logical ones?


If your filesystem becomes corrupt (and you are unable to
repair it), *all* of your data is lost (instead of just
one partition). That's the only disadvantage I can think
of.


That is exactly what I am afraid of!

So, the 20-years old rule of thumb is still valid. :(


No, it is not valid, and it is not true.

Data corruption on-disk does not by and large (unless you are very 
unlucky) corrupt file systems. It corrupts files.


Secondly, by and large, most people have all the files they really care 
about on one partition, called DATA or similar. Everything else except 
your data can usually be reconstructed, especially the OS itself. You 
probably store all that data in one volume simply because it makes 
logical sense to do so. Data is read and written far more than anything 
else on your disk so if you are unlucky enough to suffer volume 
corruption it's likely to be on a) the biggest volume and b) the busiest 
volume. In both cases it is your data, meaning your data is what is 
exposed to risk and everything else not so much.


Yes, this is a real factor you mention. It is detectable and 
measureable. It's also minute and statistically irrelevant if you 
haven't dealt with environmental factors that cause data damage (dodgy 
ram, cables, psus, over-temps, brownouts). If those things happen, and 
they WILL happen, you are 10-20 times at least more likely to lose your 
data than anything else, no matter how you partitioned the disk.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

On Thu, 1 Sep 2016 22:04:22 +0300, gevisz wrote:

> > LVM is neither encrypted nor compressed. The filesystems on it are no
> > different to the filesystems on physical partitions, and subject to
> > the same risks. An LVM logical volume is just a block device that is
> > treated the same as a physical partition on a non-LVM setup.  
> 
> Thank you for the explanation, I have also just refreshed my memory
> about LVM before replying to you but still can not see any reason why
> I may need LVM on an external hard drive...

You gave on in the post that I replied to suggesting LVM in the first
place - that's why I suggested it.

You were worrying about the difficulty of altering a partition layout
once it is committed to disk and filled with data. LVM removes that
problem, because volumes and filesystems can be resized, added and
deleted at will.

However, at no point did I state that you "need" it, only that it may be
useful. The location of the drive is less relevant that its capacity when
considering this.


-- 
Neil Bothwick

A clean desk is a sign of a cluttered desk drawer.


pgp4KApoZXOVf.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 22:12 GMT+03:00 Rich Freeman :
> On Thu, Sep 1, 2016 at 2:58 PM, gevisz  wrote:
>> 2016-09-01 14:55 GMT+03:00 Rich Freeman :
>>
>>> 2. Set it up as an LVM partition.  Unless you're using filesystems
>>> like zfs/btrfs that have their own way of doing volume management,
>>> this just makes things less painful down the road.
>>>
>>> 3. I'd probably just set it up as one big logical volume, unless you
>>> know you don't need all the space and you think you might use it for
>>> something else later.  You can change your mind on this with ext4+lvm
>>> either way, but better to start out whichever way seems best.
>>
>> I had to refresh my memory about LVM before replying to you
>> but still can not see why I may need LVM on an external
>> hard drive...
>
> It just gives you more options in the future,

Yes, thank you.

> it is easy to move LVM volumes to other drives, re-partition them later,
> and so on.

I still suspect that this extra level of complexity can complicate recovery
of the data, if anything happens to the disk under LVM management (except
for stealing the hard drive, of course :).

>  I agree it is probably overkill on a removable device, but it doesn't hurt.
> This is a 5TB drive after all.  But, I don't think it is super-critical 
> either.
>
>>
>>> It will take you all of 30 seconds to format this, unless you're
>>> running badblocks (which almost nobody does, because...
>>
>> it takes too much time?
>>
>> I currently running a smart test on it, and it promised to take
>> 10 hours to complete...
>
> That's basically it.  If it didn't take time people would of course
> run it first.  I think a SMART test would be about as good and likely
> a lot faster.  However, the drive should be managing bad blocks on its
> own (granted, many drives seem to get that wrong in my experience,
> which is part of why I run btrfs, but I probably wouldn't use
> btrfs/zfs for a drive you're moving all over the place since who knows
> what kind of kernel you'll have when you use it and heaven help you if
> you ever need to read it on Windows).

It is not a question of using the disk with Windows, but I too often see
some reports about problems in using btrfs on this list to try using it
myself...

>>> You seem to be concerned about losing data.  You should be.  This is a
>>> physical storage device.  You WILL lose everything stored on it at
>>> some point in time.
>>
>> Last time, I have managed to restore all the data from my 2.5" hard
>> drive that suddenly died about 7 years ago and hope to do it again
>> if any. :)
>
> Well, if the data is redundant then you're fine (it is essentially
> already backed up).

No, that data was not backed up.

But I am not guaranteed to be so lucky again, of course. :(

That is why I decided to finally start to back up my data. :)

>  But, you should check those backups from time to time.
>
> You should never rely on the ability to recover data from a hard
> drive.  For starters, if you just lose the thing (portable things can
> sometimes grow legs; you're talking about 5 libraries of congress in a
> bag that could get stolen) or it is catastrophically destroyed that
> isn't going to work.  Short of that there is a fair chance you can get
> a lot of data off the drive, and it is fairly likely if you're using
> some kind of expensive recovery service, but you can't promise that
> the specific file you care about most will get recovered.
>
> Backups are annoying.

Yes. :)

>  I don't do them as well as ideally I should

Who does? :)

Well, probably, one who just lost a lot of data because of not doing backup. :)

> (way too much data to get it all offsite), but I make a conscious
> decision about what does/doesn't get backed up and how.  I
> occasionally restore my encrypted cloud backups to confirm they
> contain what I expect them to.  I actually get the log summary emailed
> daily to make sure it is running (if I had more hosts I could use some
> kind of monitoring for that...).  I've never needed to use the online
> cloud backups, but they're there for a reason and they cover anything
> I actually care about (documents and such).  I also backup all my
> cloud services (evernote, google drive, etc) to local storage
> occassionally; that doesn't require further backup since it is the
> backup.  You just need two copies of everything, with one copy
> preferably being inaccessible from the other and not at the same
> physical site.

Well, thank you for your advices.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 15:51 GMT+03:00 Michael Mol :
>
> On Thursday, September 01, 2016 12:09:09 PM gevisz wrote:
>> 2016-09-01 11:54 GMT+03:00 Neil Bothwick :
>> > On Thu, 1 Sep 2016 11:49:43 +0300, gevisz wrote:
>> >> > If your filesystem becomes corrupt (and you are unable to
>> >> > repair it), *all* of your data is lost (instead of just
>> >> > one partition). That's the only disadvantage I can think
>> >> > of.
>> >>
>> >> That is exactly what I am afraid of!
>> >>
>> >> So, the 20-years old rule of thumb is still valid. :(
>> >>
>> >> > I don't like partitions either (after some years, I
>> >> > always found that sizes don't match my requirements any
>> >> > more),
>> >>
>> >> And this is exactly the reason why I do not want to partition
>> >> my new hard drive! :)
>> >
>> > Have you considered LVM? You get the benefits of separate filesystems
>> > without the limitations of inflexible partitioning.
>>
>> I am afraid of LVM because of the same reason as described below:
>>
>> returning to the "old good times" of MS DOS 6.22, I do remember that working
>> then on 40MB (yes, megabytes) hard drive I used some program that
>> compressed all the data before saving them on that hard drive.
>> Unfortunately, one day, because of the corruption, I lost all the data on
>> that hard drive. Since then, I am very much afraid of compressed or
>> encrypted hard drives.
>
> LVM doesn't *need* to do any of that. It will only do as much as you tell it
> to do. If you only want to use it as a way of reshaping relatively simple
> partitions, you can use it for that.
>
> Honestly, I tend not to create separate partitions for separate mount points
> these days. At least, not on personal systems. For servers, it's can be
> beneficial to have /var separate from /, or /var/log separate from /var, or
> /var/spool, or /var/lib/mysql, or what have you. But the biggest driver for
> that, IME, is if one of those fills up, it can't take down the rest of the
> host.
>
> In your case, I'd suggest using a single / filesystem. If it works, it works.
> If it doesn't, you'll know in the future where you need to be more flexible;
> there's no single panacea.

Thank you for the reply. And I even agree with you to the point that
on a Linux desktop it may be enough to have just 3 different partitions:
one - for /, second - for swap (yes, one can do without it nowadays),
and third - for /home. But you probably missed the point that it goes
about an external drive dedicated to backups only.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Rich Freeman]

On Thu, Sep 1, 2016 at 2:58 PM, gevisz  wrote:
> 2016-09-01 14:55 GMT+03:00 Rich Freeman :
>
>> 2. Set it up as an LVM partition.  Unless you're using filesystems
>> like zfs/btrfs that have their own way of doing volume management,
>> this just makes things less painful down the road.
>>
>> 3. I'd probably just set it up as one big logical volume, unless you
>> know you don't need all the space and you think you might use it for
>> something else later.  You can change your mind on this with ext4+lvm
>> either way, but better to start out whichever way seems best.
>
> I had to refresh my memory about LVM before replying to you
> but still can not see why I may need LVM on an external
> hard drive...

It just gives you more options in the future, it is easy to move LVM
volumes to other drives, re-partition them later, and so on.  I agree
it is probably overkill on a removable device, but it doesn't hurt.
This is a 5TB drive after all.  But, I don't think it is
super-critical either.

>
>> It will take you all of 30 seconds to format this, unless you're
>> running badblocks (which almost nobody does, because...
>
> it takes too much time?
>
> I currently running a smart test on it, and it promised to take
> 10 hours to complete...

That's basically it.  If it didn't take time people would of course
run it first.  I think a SMART test would be about as good and likely
a lot faster.  However, the drive should be managing bad blocks on its
own (granted, many drives seem to get that wrong in my experience,
which is part of why I run btrfs, but I probably wouldn't use
btrfs/zfs for a drive you're moving all over the place since who knows
what kind of kernel you'll have when you use it and heaven help you if
you ever need to read it on Windows).

>
>> You seem to be concerned about losing data.  You should be.  This is a
>> physical storage device.  You WILL lose everything stored on it at
>> some point in time.
>
> Last time, I have managed to restore all the data from my 2.5" hard
> drive that suddenly died about 7 years ago and hope to do it again
> if any. :)

Well, if the data is redundant then you're fine (it is essentially
already backed up).  But, you should check those backups from time to
time.

You should never rely on the ability to recover data from a hard
drive.  For starters, if you just lose the thing (portable things can
sometimes grow legs; you're talking about 5 libraries of congress in a
bag that could get stolen) or it is catastrophically destroyed that
isn't going to work.  Short of that there is a fair chance you can get
a lot of data off the drive, and it is fairly likely if you're using
some kind of expensive recovery service, but you can't promise that
the specific file you care about most will get recovered.

Backups are annoying.  I don't do them as well as ideally I should
(way too much data to get it all offsite), but I make a conscious
decision about what does/doesn't get backed up and how.  I
occasionally restore my encrypted cloud backups to confirm they
contain what I expect them to.  I actually get the log summary emailed
daily to make sure it is running (if I had more hosts I could use some
kind of monitoring for that...).  I've never needed to use the online
cloud backups, but they're there for a reason and they cover anything
I actually care about (documents and such).  I also backup all my
cloud services (evernote, google drive, etc) to local storage
occassionally; that doesn't require further backup since it is the
backup.  You just need two copies of everything, with one copy
preferably being inaccessible from the other and not at the same
physical site.

-- 
Rich

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 15:21 GMT+03:00 Neil Bothwick :
> On Thu, 1 Sep 2016 12:09:09 +0300, gevisz wrote:
>
>> > Have you considered LVM? You get the benefits of separate filesystems
>> > without the limitations of inflexible partitioning.
>>
>> I am afraid of LVM because of the same reason as described below:
>>
>> returning to the "old good times" of MS DOS 6.22, I do remember that
>> working then on 40MB (yes, megabytes) hard drive I used some program
>> that compressed all the data before saving them on that hard drive.
>> Unfortunately, one day, because of the corruption, I lost all the data
>> on that hard drive. Since then, I am very much afraid of compressed or
>> encrypted hard drives.
>
> LVM is neither encrypted nor compressed. The filesystems on it are no
> different to the filesystems on physical partitions, and subject to the
> same risks. An LVM logical volume is just a block device that is treated
> the same as a physical partition on a non-LVM setup.

Thank you for the explanation, I have also just refreshed my memory
about LVM before replying to you but still can not see any reason why
I may need LVM on an external hard drive...

> Sp far, you have come up with reasons, good or otherwise, for not taking
> each of the available choices. You need to decide what you really need
> and what is important to you. Only then can you decide on the best
> arrangement for your needs.
>
> Neil Bothwick
>
> Evolution stops when stupidity is no longer fatal!

:)

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 14:55 GMT+03:00 Rich Freeman :
> On Thu, Sep 1, 2016 at 2:04 AM, gevisz  wrote:
>>
>> Is it still advisable to partition a big hard drive
>> into smaller logical ones and why?
>>
>
> Assuming this is only used on Linux machines (you mentioned moving
> files around), here is what I would do:
>
> 1. Definitely create a partition table.  Yes, I know some like to
> stick filesystems on raw drives, but you're basically going to fight
> all the automation in existence if you do this.

I will do it with gparted and guess that it will create a partition table
for me anyway.

> 2. Set it up as an LVM partition.  Unless you're using filesystems
> like zfs/btrfs that have their own way of doing volume management,
> this just makes things less painful down the road.
>
> 3. I'd probably just set it up as one big logical volume, unless you
> know you don't need all the space and you think you might use it for
> something else later.  You can change your mind on this with ext4+lvm
> either way, but better to start out whichever way seems best.

I had to refresh my memory about LVM before replying to you
but still can not see why I may need LVM on an external
hard drive...

> It will take you all of 30 seconds to format this, unless you're
> running badblocks (which almost nobody does, because...

it takes too much time?

I currently running a smart test on it, and it promised to take
10 hours to complete...

> You seem to be concerned about losing data.  You should be.  This is a
> physical storage device.  You WILL lose everything stored on it at
> some point in time.

Last time, I have managed to restore all the data from my 2.5" hard
drive that suddenly died about 7 years ago and hope to do it again
if any. :)

>  You mitigate this by one or more of:
> 1.  Not storing anything you mind losing on the drive, and then not
> complaining when you lose it.
> 2.  Keeping backups, preferably at a different physical location,
> using a periodically tested recovery methodology.
> 3.  Availability solutions like RAID (not the same as a backup, but it
> will mean less downtime WHEN you WILL have a drive failure).  Some
> filesystems like zfs/btrfs have specific ways of achieving this (and
> are generally more resistant to unreliable storage devices, which all
> storage devices are).
>
> I've actually had LVM eat my data once due to some kind of really rare
> bug (found one discussion of similar issues on some forum somewhere).

Aha!

> That isn't a good reason not to use LVM.  Wanting to plug the drive
> into a bunch of Windows machines would be a good reason not to use
> LVM, or ext4 for that matter.
>
> Most of the historic reasons for not having large volumes had to do
> with addressing limits, whether it be drive geometry limits,
> filesystem limits, etc.  Modern partition tables like GPT and
> filesystems can handle volumes MUCH larger than 5TB.
>
> Most modern journaling filesystems should also tend to avoid failure
> modes like losing the entire filesystem during a power failure (when
> correctly used, heaven help you if you follow a random friend's advice
> with mount options, like not using at least ordered data or disabling
> barriers).  But, bugs can exist, which is a big reason to have backups
> and not just trust your filesystem unless you don't care much about
> the data.

Thank you for replying.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Rich Freeman]

On Thu, Sep 1, 2016 at 1:42 PM, Neil Bothwick  wrote:
> On Thu, 1 Sep 2016 09:27:39 -0400, Rich Freeman wrote:
>
>> > Honestly, I tend not to create separate partitions for separate mount
>> > points these days. At least, not on personal systems. For servers,
>> > it's can be beneficial to have /var separate from /, or /var/log
>> > separate from /var, or /var/spool, or /var/lib/mysql, or what have
>> > you. But the biggest driver for that, IME, is if one of those fills
>> > up, it can't take down the rest of the host.
>
>> The other big use case these days would be SSDs.  I tend to have one
>> SSD filesystem for root, and one SSD filesystem for everything else.
>> That means a lot of bind mounts, but it all works.
>
> Bind mounts? I thought you would use btrfs subvolumes!
>

Often the bind mounts point to btrfs subvolumes.

Yeah, I guess I could directly mount all those subvolumes, but I find
symlinks or bind mounts easier.  The other factor is that if I have
unnecessary subvolumes then I'm having to manage snapshots across more
of them and my snapshots are less atomic, since snapshots don't cross
subvolume boundaries (which is something which ought to be
configurable).


-- 
Rich

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

On Thu, 1 Sep 2016 09:27:39 -0400, Rich Freeman wrote:

> > Honestly, I tend not to create separate partitions for separate mount
> > points these days. At least, not on personal systems. For servers,
> > it's can be beneficial to have /var separate from /, or /var/log
> > separate from /var, or /var/spool, or /var/lib/mysql, or what have
> > you. But the biggest driver for that, IME, is if one of those fills
> > up, it can't take down the rest of the host.

> The other big use case these days would be SSDs.  I tend to have one
> SSD filesystem for root, and one SSD filesystem for everything else.
> That means a lot of bind mounts, but it all works.

Bind mounts? I thought you would use btrfs subvolumes!


-- 
Neil Bothwick

Make it idiot proof and someone will make a better idiot.


pgpWky1T94bMH.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Rich Freeman]

On Thu, Sep 1, 2016 at 8:51 AM, Michael Mol  wrote:
>
> Honestly, I tend not to create separate partitions for separate mount points
> these days. At least, not on personal systems. For servers, it's can be
> beneficial to have /var separate from /, or /var/log separate from /var, or
> /var/spool, or /var/lib/mysql, or what have you. But the biggest driver for
> that, IME, is if one of those fills up, it can't take down the rest of the
> host.
>

The other big use case these days would be SSDs.  I tend to have one
SSD filesystem for root, and one SSD filesystem for everything else.
That means a lot of bind mounts, but it all works.  I'm not about to
get into separate filesystems for random directories in var that tend
to get big.

-- 
Rich

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Michael Mol]

On Thursday, September 01, 2016 12:09:09 PM gevisz wrote:
> 2016-09-01 11:54 GMT+03:00 Neil Bothwick :
> > On Thu, 1 Sep 2016 11:49:43 +0300, gevisz wrote:
> >> > If your filesystem becomes corrupt (and you are unable to
> >> > repair it), *all* of your data is lost (instead of just
> >> > one partition). That's the only disadvantage I can think
> >> > of.
> >> 
> >> That is exactly what I am afraid of!
> >> 
> >> So, the 20-years old rule of thumb is still valid. :(
> >> 
> >> > I don't like partitions either (after some years, I
> >> > always found that sizes don't match my requirements any
> >> > more),
> >> 
> >> And this is exactly the reason why I do not want to partition
> >> my new hard drive! :)
> > 
> > Have you considered LVM? You get the benefits of separate filesystems
> > without the limitations of inflexible partitioning.
> 
> I am afraid of LVM because of the same reason as described below:
> 
> returning to the "old good times" of MS DOS 6.22, I do remember that working
> then on 40MB (yes, megabytes) hard drive I used some program that
> compressed all the data before saving them on that hard drive.
> Unfortunately, one day, because of the corruption, I lost all the data on
> that hard drive. Since then, I am very much afraid of compressed or
> encrypted hard drives.

LVM doesn't *need* to do any of that. It will only do as much as you tell it 
to do. If you only want to use it as a way of reshaping relatively simple 
partitions, you can use it for that.

Honestly, I tend not to create separate partitions for separate mount points 
these days. At least, not on personal systems. For servers, it's can be 
beneficial to have /var separate from /, or /var/log separate from /var, or 
/var/spool, or /var/lib/mysql, or what have you. But the biggest driver for 
that, IME, is if one of those fills up, it can't take down the rest of the 
host.

In your case, I'd suggest using a single / filesystem. If it works, it works. 
If it doesn't, you'll know in the future where you need to be more flexible; 
there's no single panacea.

-- 
:wq

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Dale]

gevisz wrote:
> 2016-09-01 9:13 GMT+03:00 Alan McKinnon :
>> On 01/09/2016 08:04, gevisz wrote:
>>> I have bought an external 5TB Western Digital hard drive
>>> that I am going to use mainly for backing up some files
>>> in my home directory and carrying a very big files, for
>>> example a virtual machine image file, from one computer
>>> to another. This hard drive is preformatted with NTFS.
>>> Now, I am going to format it with ext4 which probably
>>> will take a lot of time taking into account that it is
>>> going to be done via USB connection. So, before formatting
>>> this hard drive I would like to know if it is still
>>> advisable to partition big hard drives into smaller
>>> logical ones.
>> it will take about 5 seconds to partition it.
>> And a few more to mkfs it.
> Just to partition - may be, but I very much doubt
> that it will take seconds to create a full-fledged
> ext4 file system on these 5TB via USB2 connention.
>
> Even more: my aquiantance from the Window world
> that recomended me this disc scared me that it may
> take days...
>

Something to think on.  You have a 5TB drive.  You format the whole
thing and let's say it takes 30 seconds.  Or, you break it into two
2.5TB partitions and then format those, which take 20 or 25 seconds
each.  That adds up to 40 to 50 seconds format time.  Isn't it faster to
format one large partition instead of two?  After all, you have to type
the command in to format it too which also takes a few seconds, assuming
you up arrow and just edit the partition letter.  No matter whether you
break the drive up into parts or not, you are still formatting 5TBs
worth of drive.  The only way you can save time is to not format the
whole thing. 

Things break.  They always have and always will.  Sure you can prepare
for that lose but if not careful, you could lose it while you are
second, third, forth etc etc etc guessing yourself and what tool you are
going to use.  I suspect that every file system out there has caused a
person to lose data before.  I'm sure that every brand and even model of
hard drive out there has caused someone to lose data before.  When it
gets as complex has a hard drive and the tools used on them, it has to
break at some point.  The best bet, duplicate your files just in case
something in the above list goes bad.  The only advice I think would be
good on this, don't use the same brand and model drive for both main and
backup.  One could even say not to use the same file system, that way if
one goes bad due to bad coding in the kernel, likely the other shouldn't
be affected, but even that can't be a for sure thing. 

I hope you aren't to worried about making a backup now that you can
think on how everything fails eventually.  ;-)

Dale

:-)  :-) 

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

On Thu, 1 Sep 2016 12:09:09 +0300, gevisz wrote:

> > Have you considered LVM? You get the benefits of separate filesystems
> > without the limitations of inflexible partitioning.  
> 
> I am afraid of LVM because of the same reason as described below:
> 
> returning to the "old good times" of MS DOS 6.22, I do remember that
> working then on 40MB (yes, megabytes) hard drive I used some program
> that compressed all the data before saving them on that hard drive.
> Unfortunately, one day, because of the corruption, I lost all the data
> on that hard drive. Since then, I am very much afraid of compressed or
> encrypted hard drives.

LVM is neither encrypted nor compressed. The filesystems on it are no
different to the filesystems on physical partitions, and subject to the
same risks. An LVM logical volume is just a block device that is treated
the same as a physical partition on a non-LVM setup.

Sp far, you have come up with reasons, good or otherwise, for not taking
each of the available choices. You need to decide what you really need
and what is important to you. Only then can you decide on the best
arrangement for your needs.


-- 
Neil Bothwick

Evolution stops when stupidity is no longer fatal!


pgpCgd0ppLsNs.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Rich Freeman]

On Thu, Sep 1, 2016 at 2:04 AM, gevisz  wrote:
>
> Is it still advisable to partition a big hard drive
> into smaller logical ones and why?
>

Assuming this is only used on Linux machines (you mentioned moving
files around), here is what I would do:

1. Definitely create a partition table.  Yes, I know some like to
stick filesystems on raw drives, but you're basically going to fight
all the automation in existence if you do this.
2. Set it up as an LVM partition.  Unless you're using filesystems
like zfs/btrfs that have their own way of doing volume management,
this just makes things less painful down the road.
3. I'd probably just set it up as one big logical volume, unless you
know you don't need all the space and you think you might use it for
something else later.  You can change your mind on this with ext4+lvm
either way, but better to start out whichever way seems best.

It will take you all of 30 seconds to format this, unless you're
running badblocks (which almost nobody does, because...).

You seem to be concerned about losing data.  You should be.  This is a
physical storage device.  You WILL lose everything stored on it at
some point in time.  You mitigate this by one or more of:
1.  Not storing anything you mind losing on the drive, and then not
complaining when you lose it.
2.  Keeping backups, preferably at a different physical location,
using a periodically tested recovery methodology.
3.  Availability solutions like RAID (not the same as a backup, but it
will mean less downtime WHEN you WILL have a drive failure).  Some
filesystems like zfs/btrfs have specific ways of achieving this (and
are generally more resistant to unreliable storage devices, which all
storage devices are).

I've actually had LVM eat my data once due to some kind of really rare
bug (found one discussion of similar issues on some forum somewhere).
That isn't a good reason not to use LVM.  Wanting to plug the drive
into a bunch of Windows machines would be a good reason not to use
LVM, or ext4 for that matter.

Most of the historic reasons for not having large volumes had to do
with addressing limits, whether it be drive geometry limits,
filesystem limits, etc.  Modern partition tables like GPT and
filesystems can handle volumes MUCH larger than 5TB.

Most modern journaling filesystems should also tend to avoid failure
modes like losing the entire filesystem during a power failure (when
correctly used, heaven help you if you follow a random friend's advice
with mount options, like not using at least ordered data or disabling
barriers).  But, bugs can exist, which is a big reason to have backups
and not just trust your filesystem unless you don't care much about
the data.

-- 
Rich

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 12:01 GMT+03:00 Alan McKinnon :
> On 01/09/2016 09:18, gevisz wrote:
>> 2016-09-01 9:13 GMT+03:00 Alan McKinnon :
>>> On 01/09/2016 08:04, gevisz wrote:
>
> [snip]
>
 Is it still advisable to partition a big hard drive
 into smaller logical ones and why?
>>>
>>> The only reason to partition a drive is to get 2 or more
>>> smaller ones that differ somehow (size, inode ratio, mount options, etc)
>>>
>>> Go with no partition table by all means, but if you one day find you
>>> need one, you will have to copy all your data off, repartition, and copy
>>> your data back. If you are certain that will not happen (eg you will
>>> rather buy a second drive) then by all means dispense with partitions.
>>>
>>> They are after all nothing more than a Microsoft invention from the 80s
>>> so people could install UCSD Pascal next to MS-DOS
>>
>> I definitely will not need more than one mount point for this hard drive
>> but I do remember some arguments that partitioning a large hard drive
>> into smaller logical ones gives me more safety in case a file system
>> suddenly will get corrupted because in this case I will loose my data
>> only on one of the logical partitions and not on the whole drive.
>>
>> Is this argument still valid nowadays?
>
> That is the most stupid dumbass argument I've heard in weeks.
> It doesn't even deserve a response.
>
> Who the fuck is promoting this shit?

Even somebody in this thread (in addition to me and independent of me)
made the same arguments.

But I do not state anything, I am just asking.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Alan McKinnon]

On 01/09/2016 10:44, gevisz wrote:
> 2016-09-01 10:23 GMT+03:00 Frank Steinmetzger :
>> On Thu, Sep 01, 2016 at 08:13:23AM +0200, Alan McKinnon wrote:
>>
>>> it will take about 5 seconds to partition it.
>>> And a few more to mkfs it.
>>>
>>> Are you sure you aren't thinking of mkfs with ext2 (which did take hours
>>> for a drive that size?
>>
>> Some people do a full systems check (i.e. badblocks) before entrusting a
>> drive with anything important.
> 
> It is a good advice! I have already thought of this but I am sorry to
> acknowledge
> that, since the "old good times" of MS DOS 6.22, I never did this in Linux. :(
> 
> And except for one 2.5" disk failure on my old laptop about 7 years ago,
> I had no problem with this so far. :)
> 
> All other my hard disks work for about 10 years without any intervention
> from my side and even without any backups so far. That's why I started
> to think about it now. :)
> 
> So, can you, please, advice me about the program or utility that can do
> badblocks check for me?
> 
 Is it still advisable to partition a big hard drive
 into smaller logical ones and why?
>>>
>>> The only reason to partition a drive is to get 2 or more
>>> smaller ones that differ somehow (size, inode ratio, mount options, etc)
>>
>> If you want to do backups, then of course the file system is important, so
>> it retains permissions and stuff. Your ext4 choice is the right one in that
>> case. However, I partitioned by backupdrive into two partitions, so the one
>> with the sensitive data can be encrypted. The big partition that holds media
>> files has not got that treatment.
> 
> It is, again, a good advice but, again, returning to the "old good times"
> of MS DOS 6.22, I do remember that working then on 40MB (yes, megabytes)
> hard drive I used some program that compressed all the data before saving
> them on that hard drive. Unfortunately, one day, because of the corruption,
> I lost all the data on that hard drive. Since then, I am very much afraid of
> compressed or encrypted hard drives.
> 
>>> Go with no partition table by all means, but if you one day find you
>>> need one, you will have to copy all your data off, repartition, and copy
>>> your data back.
>>
>> When I do the mentioned partitioning scheme, I put the biggest partition at
>> the beginning of the drive and the smaller one(s) at the back. That way,
>> should I ever actually need to resize a partition, I only have to export the
>> smaller partition for the process (or none at all, if it’s just a backup
>> itself and I have another backup on another drive).
>> Of course there’s LVM these days, but up until recently, I used NTFS for the
>> media partition so I could also read it in $DUMB_OS, which doesn’t know LVM.
>> Only a short while back, I also switched to ext4 for that, so I can retain
>> file names with : and ? in them. But I still refrained from using LVM,
>> though.
> 
> I am afraid of LVM because of the same reason I described above.

You are allowing your fears of 20 years ago to determine your present
day attitudes. That's silly.

There's a misconception that a drive is somehow a pristine storage
device with pigeon holes where data goes and nothing below the fs level
can have any effect. Nothing could be further from the truth.

The sheer amount of encoding that goes into a drive is almost beyond
belief. It is NOT a digital device, it is analog - exactly like tape,
just many times more complex. It doesn't store bits, it stores
fluctuating regions of local magnetism and all of that gets decoded by
analog circuitry to represent digital bits. About 50% of your drive's
capacity (measured by what the heads see) is devoted to marking where
the tracks go, where the sectors start and end, checksumming and may
other firmware safeguards.

All that stuff can go wrong.

Yes, encryption on-drive can go wrong. So can SSL traffic, gpg keys,
encrypted mail, vpns and all your traffic over the internet (if you're
on a corporate network I can almost assure you it's all encrypted inside
an IPSec tunnel).

That stuff doesn't break. Neither does your disk encryption.

LVM can break, but it's really hard. All a PV is, is a block device with
a 2k signature at the start then some metadata. All a VG is, is a bunch
of PVs and some metadata to list them. An LV is really nothing more than
an indirection lookup table: The fs knows which inode and sectors the
file uses, and the LV has a mapping table to track which real disk
sectors that is.

The kernel does this all the time with every smidgen of RAM you have
(it's how a vmm works). It's the same technology in essence.

So yeah, stuff can break. But that same stuff is used in many other
highly critical areas where you likely don't know of it, and that
doesn't change that it's there.

I know of know recent reports where disk encryption or volume management
broke data solely due to a code bug in stable production versions. Devs
are not that stupid :-)

So honestly, your 

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Alan McKinnon]

On 01/09/2016 10:59, gevisz wrote:
> 2016-09-01 11:03 GMT+03:00 Neil Bothwick :
>> On Thu, 1 Sep 2016 10:18:29 +0300, gevisz wrote:
>>
 it will take about 5 seconds to partition it.
 And a few more to mkfs it.
>>>
>>> Just to partition - may be, but I very much doubt
>>> that it will take seconds to create a full-fledged
>>> ext4 file system on these 5TB via USB2 connention.
>>
>> Even if that were the case, does it really matter?
> 
> You are right: it does not matter for the main question.
> However, it is an additional reason to think twice and
> ask a knowledgeable people before starting. :)
> 
>> You said you wanted to use this drive for backups,
>> surely doing it right is more important than doing
>> it quickly. It's not like you have to hold its hand
>> while mkfs is running.
> 
> But I would have to keep my fingers crossed so that
> there would not be a sudden blackout during the formatting
> hard disk anyway. :)

Even if it does, so what?

Unplug, start over. It's a mkfs, it lays down where the inodes are. You
can do it over and over and over and over  safely


-- 
Alan McKinnon
alan.mckin...@gmail.com

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 12:04 GMT+03:00 Neil Bothwick :
> On Thu, 1 Sep 2016 11:59:55 +0300, gevisz wrote:
>
>> > You said you wanted to use this drive for backups,
>> > surely doing it right is more important than doing
>> > it quickly. It's not like you have to hold its hand
>> > while mkfs is running.
>>
>> But I would have to keep my fingers crossed so that
>> there would not be a sudden blackout during the formatting
>> hard disk anyway. :)
>
> Nah, just start again. It's only after the power goes out for the third
> time that you decide the Universe hates you and give up!

:)

> Neil Bothwick
>
> Q: What's the second worst sound you can hear a sysadmin make?
> A: Uh-oh
> Q: And the worst sound?
> A: Oops

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 11:54 GMT+03:00 Neil Bothwick :
> On Thu, 1 Sep 2016 11:49:43 +0300, gevisz wrote:
>
>> > If your filesystem becomes corrupt (and you are unable to
>> > repair it), *all* of your data is lost (instead of just
>> > one partition). That's the only disadvantage I can think
>> > of.
>>
>> That is exactly what I am afraid of!
>>
>> So, the 20-years old rule of thumb is still valid. :(
>>
>> > I don't like partitions either (after some years, I
>> > always found that sizes don't match my requirements any
>> > more),
>>
>> And this is exactly the reason why I do not want to partition
>> my new hard drive! :)
>
> Have you considered LVM? You get the benefits of separate filesystems
> without the limitations of inflexible partitioning.

I am afraid of LVM because of the same reason as described below:

returning to the "old good times" of MS DOS 6.22, I do remember that working
then on 40MB (yes, megabytes) hard drive I used some program that compressed
all the data before saving them on that hard drive. Unfortunately, one day,
because of the corruption, I lost all the data on that hard drive. Since then,
I am very much afraid of compressed or encrypted hard drives.

> Neil Bothwick
>
> For a list of all the ways technology has failed to improve the
> quality of life, please press three.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 11:55 GMT+03:00 Frank Steinmetzger :
> On Thu, Sep 01, 2016 at 11:44:19AM +0300, gevisz wrote:
>
>> > Some people do a full systems check (i.e. badblocks) before entrusting a
>> > drive with anything important.
>>
>> It is a good advice! I have already thought of this but I am sorry to
>> acknowledge that, since the "old good times" of MS DOS 6.22, I never did
>> this in Linux. :(
>> […]
>> So, can you, please, advice me about the program or utility that can do
>> badblocks check for me?
>
> Badblocks is part of e2fsprogs. But since you’re using USB2, this will
> really take a while. At best I get 39 MB/s out of it. Another way is a
> S.M.A.R.T. test, methinks `smartctl -t full` is the command for that. But I
> don’t know what exactly is being tested there. But it runs fully internal of
> the disk, so no USB2-bottleneck. Others may chime in if I tell fairy tales.

Thank you for your advices. I will try both.

> Gruß | Greetings | Qapla'
> I cna ytpe 300 wrods pre mniuet!!!

:)

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

On Thu, 1 Sep 2016 11:59:55 +0300, gevisz wrote:

> > You said you wanted to use this drive for backups,
> > surely doing it right is more important than doing
> > it quickly. It's not like you have to hold its hand
> > while mkfs is running.  
> 
> But I would have to keep my fingers crossed so that
> there would not be a sudden blackout during the formatting
> hard disk anyway. :)

Nah, just start again. It's only after the power goes out for the third
time that you decide the Universe hates you and give up!


-- 
Neil Bothwick

Q: What's the second worst sound you can hear a sysadmin make?
A: Uh-oh
Q: And the worst sound?
A: Oops


pgp0jGVSn93tr.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Alan McKinnon]

On 01/09/2016 09:18, gevisz wrote:
> 2016-09-01 9:13 GMT+03:00 Alan McKinnon :
>> On 01/09/2016 08:04, gevisz wrote:

[snip]

>> it will take about 5 seconds to partition it.
>> And a few more to mkfs it.
> 
> Just to partition - may be, but I very much doubt
> that it will take seconds to create a full-fledged
> ext4 file system on these 5TB via USB2 connention.


Do it. Tell me how long it tool.

Discussing it without doing it and offering someone else's opinion is a
100% worthless activity

> 
> Even more: my aquiantance from the Window world
> that recomended me this disc scared me that it may
> take days...

Mickey Mouse told me it takes microseconds. So what?

Do it. Tell me how long it took.

>>> Is it still advisable to partition a big hard drive
>>> into smaller logical ones and why?
>>
>> The only reason to partition a drive is to get 2 or more
>> smaller ones that differ somehow (size, inode ratio, mount options, etc)
>>
>> Go with no partition table by all means, but if you one day find you
>> need one, you will have to copy all your data off, repartition, and copy
>> your data back. If you are certain that will not happen (eg you will
>> rather buy a second drive) then by all means dispense with partitions.
>>
>> They are after all nothing more than a Microsoft invention from the 80s
>> so people could install UCSD Pascal next to MS-DOS
> 
> I definitely will not need more than one mount point for this hard drive
> but I do remember some arguments that partitioning a large hard drive
> into smaller logical ones gives me more safety in case a file system
> suddenly will get corrupted because in this case I will loose my data
> only on one of the logical partitions and not on the whole drive.
> 
> Is this argument still valid nowadays?

That is the most stupid dumbass argument I've heard in weeks.
It doesn't even deserve a response.

Who the fuck is promoting this shit?


-- 
Alan McKinnon
alan.mckin...@gmail.com

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 11:03 GMT+03:00 Neil Bothwick :
> On Thu, 1 Sep 2016 10:18:29 +0300, gevisz wrote:
>
>> > it will take about 5 seconds to partition it.
>> > And a few more to mkfs it.
>>
>> Just to partition - may be, but I very much doubt
>> that it will take seconds to create a full-fledged
>> ext4 file system on these 5TB via USB2 connention.
>
> Even if that were the case, does it really matter?

You are right: it does not matter for the main question.
However, it is an additional reason to think twice and
ask a knowledgeable people before starting. :)

> You said you wanted to use this drive for backups,
> surely doing it right is more important than doing
> it quickly. It's not like you have to hold its hand
> while mkfs is running.

But I would have to keep my fingers crossed so that
there would not be a sudden blackout during the formatting
hard disk anyway. :)

> Neil Bothwick
>
> Bagpipe for free: Stuff cat under arm. Pull legs, chew tail.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Frank Steinmetzger]

On Thu, Sep 01, 2016 at 11:44:19AM +0300, gevisz wrote:

> > Some people do a full systems check (i.e. badblocks) before entrusting a
> > drive with anything important.
> 
> It is a good advice! I have already thought of this but I am sorry to
> acknowledge that, since the "old good times" of MS DOS 6.22, I never did
> this in Linux. :(
> […]
> So, can you, please, advice me about the program or utility that can do
> badblocks check for me?

Badblocks is part of e2fsprogs. But since you’re using USB2, this will
really take a while. At best I get 39 MB/s out of it. Another way is a
S.M.A.R.T. test, methinks `smartctl -t full` is the command for that. But I
don’t know what exactly is being tested there. But it runs fully internal of
the disk, so no USB2-bottleneck. Others may chime in if I tell fairy tales.

-- 
Gruß | Greetings | Qapla'
I cna ytpe 300 wrods pre mniuet!!!

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

On Thu, 1 Sep 2016 11:49:43 +0300, gevisz wrote:

> > If your filesystem becomes corrupt (and you are unable to
> > repair it), *all* of your data is lost (instead of just
> > one partition). That's the only disadvantage I can think
> > of.  
> 
> That is exactly what I am afraid of!
> 
> So, the 20-years old rule of thumb is still valid. :(
> 
> > I don't like partitions either (after some years, I
> > always found that sizes don't match my requirements any
> > more),  
> 
> And this is exactly the reason why I do not want to partition
> my new hard drive! :)

Have you considered LVM? You get the benefits of separate filesystems
without the limitations of inflexible partitioning.


-- 
Neil Bothwick

For a list of all the ways technology has failed to improve the
quality of life, please press three.


pgpmVQ8DlMG5Z.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 10:30 GMT+03:00 Matthias Hanft :
> gevisz wrote:
>>
>> But what are disadvantages of not partitioning a big
>> hard drive into smaller logical ones?
>
> If your filesystem becomes corrupt (and you are unable to
> repair it), *all* of your data is lost (instead of just
> one partition). That's the only disadvantage I can think
> of.

That is exactly what I am afraid of!

So, the 20-years old rule of thumb is still valid. :(

> I don't like partitions either (after some years, I
> always found that sizes don't match my requirements any
> more),

And this is exactly the reason why I do not want to partition
my new hard drive! :)

> and therefore, on my new server, I didn't create
> any other partitions than "boot":
>
> home01 ~ # df -h -T
> Filesystem Type  Size  Used Avail Use% Mounted on
> /dev/sda4  xfs17T   14T  2.9T  83% /
> devtmpfs   devtmpfs   10M 0   10M   0% /dev
> tmpfs  tmpfs 3.2G  644K  3.2G   1% /run
> shmtmpfs  16G  512K   16G   1% /dev/shm
> /dev/sda2  ext2  124M   46M   72M  39% /boot
> ACDFusefuse.ACDFuse  100T  284M  100T   1% /mnt/acd
> /dev/sdb1  ext3  2.7T  707G  1.9T  28% /mnt/toshiba
> home01 ~ #
>
> Backup of important and/or secret files goes to an external
> USB hard drive (sdb1) which is formatted with ext3 for
> maximum compatibility (every other Linux can read this
> without kernel hacks); backup of not-so-secret files goes
> to Amazon Cloud Drive (acd) or some other "cloud"; unimportant
> files (videos which I have on DVD or BD anyway, or can re-buy)
> just aren't backed up.
>
> -Matt
>
>

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 10:23 GMT+03:00 Frank Steinmetzger :
> On Thu, Sep 01, 2016 at 08:13:23AM +0200, Alan McKinnon wrote:
>
>> it will take about 5 seconds to partition it.
>> And a few more to mkfs it.
>>
>> Are you sure you aren't thinking of mkfs with ext2 (which did take hours
>> for a drive that size?
>
> Some people do a full systems check (i.e. badblocks) before entrusting a
> drive with anything important.

It is a good advice! I have already thought of this but I am sorry to
acknowledge
that, since the "old good times" of MS DOS 6.22, I never did this in Linux. :(

And except for one 2.5" disk failure on my old laptop about 7 years ago,
I had no problem with this so far. :)

All other my hard disks work for about 10 years without any intervention
from my side and even without any backups so far. That's why I started
to think about it now. :)

So, can you, please, advice me about the program or utility that can do
badblocks check for me?

>> > Is it still advisable to partition a big hard drive
>> > into smaller logical ones and why?
>>
>> The only reason to partition a drive is to get 2 or more
>> smaller ones that differ somehow (size, inode ratio, mount options, etc)
>
> If you want to do backups, then of course the file system is important, so
> it retains permissions and stuff. Your ext4 choice is the right one in that
> case. However, I partitioned by backupdrive into two partitions, so the one
> with the sensitive data can be encrypted. The big partition that holds media
> files has not got that treatment.

It is, again, a good advice but, again, returning to the "old good times"
of MS DOS 6.22, I do remember that working then on 40MB (yes, megabytes)
hard drive I used some program that compressed all the data before saving
them on that hard drive. Unfortunately, one day, because of the corruption,
I lost all the data on that hard drive. Since then, I am very much afraid of
compressed or encrypted hard drives.

>> Go with no partition table by all means, but if you one day find you
>> need one, you will have to copy all your data off, repartition, and copy
>> your data back.
>
> When I do the mentioned partitioning scheme, I put the biggest partition at
> the beginning of the drive and the smaller one(s) at the back. That way,
> should I ever actually need to resize a partition, I only have to export the
> smaller partition for the process (or none at all, if it’s just a backup
> itself and I have another backup on another drive).
> Of course there’s LVM these days, but up until recently, I used NTFS for the
> media partition so I could also read it in $DUMB_OS, which doesn’t know LVM.
> Only a short while back, I also switched to ext4 for that, so I can retain
> file names with : and ? in them. But I still refrained from using LVM,
> though.

I am afraid of LVM because of the same reason I described above.

> Gruß | Greetings | Qapla’
> ’ve been using vi for 15 years, because I don’t know with which command
> to close it.

:)

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Neil Bothwick]

On Thu, 1 Sep 2016 10:18:29 +0300, gevisz wrote:

> > it will take about 5 seconds to partition it.
> > And a few more to mkfs it.  
> 
> Just to partition - may be, but I very much doubt
> that it will take seconds to create a full-fledged
> ext4 file system on these 5TB via USB2 connention.

Even if that were the case, does it really matter? You said you wanted to
use this drive for backups, surely doing it right is more important than
doing it quickly. It's not like you have to hold its hand while mkfs is
running.


-- 
Neil Bothwick

Bagpipe for free: Stuff cat under arm. Pull legs, chew tail.


pgp8QcADrX2iP.pgp
Description: OpenPGP digital signature

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Matthias Hanft]

gevisz wrote:
> 
> But what are disadvantages of not partitioning a big
> hard drive into smaller logical ones?

If your filesystem becomes corrupt (and you are unable to
repair it), *all* of your data is lost (instead of just
one partition). That's the only disadvantage I can think
of. I don't like partitions either (after some years, I
always found that sizes don't match my requirements any
more), and therefore, on my new server, I didn't create
any other partitions than "boot":

home01 ~ # df -h -T
Filesystem Type  Size  Used Avail Use% Mounted on
/dev/sda4  xfs17T   14T  2.9T  83% /
devtmpfs   devtmpfs   10M 0   10M   0% /dev
tmpfs  tmpfs 3.2G  644K  3.2G   1% /run
shmtmpfs  16G  512K   16G   1% /dev/shm
/dev/sda2  ext2  124M   46M   72M  39% /boot
ACDFusefuse.ACDFuse  100T  284M  100T   1% /mnt/acd
/dev/sdb1  ext3  2.7T  707G  1.9T  28% /mnt/toshiba
home01 ~ #

Backup of important and/or secret files goes to an external
USB hard drive (sdb1) which is formatted with ext3 for
maximum compatibility (every other Linux can read this
without kernel hacks); backup of not-so-secret files goes
to Amazon Cloud Drive (acd) or some other "cloud"; unimportant
files (videos which I have on DVD or BD anyway, or can re-buy)
just aren't backed up.

-Matt

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Frank Steinmetzger]

On Thu, Sep 01, 2016 at 08:13:23AM +0200, Alan McKinnon wrote:

> it will take about 5 seconds to partition it.
> And a few more to mkfs it.
>
> Are you sure you aren't thinking of mkfs with ext2 (which did take hours
> for a drive that size?

Some people do a full systems check (i.e. badblocks) before entrusting a
drive with anything important.

> > Is it still advisable to partition a big hard drive
> > into smaller logical ones and why?
> 
> The only reason to partition a drive is to get 2 or more
> smaller ones that differ somehow (size, inode ratio, mount options, etc)

If you want to do backups, then of course the file system is important, so
it retains permissions and stuff. Your ext4 choice is the right one in that
case. However, I partitioned by backupdrive into two partitions, so the one
with the sensitive data can be encrypted. The big partition that holds media
files has not got that treatment.

> Go with no partition table by all means, but if you one day find you
> need one, you will have to copy all your data off, repartition, and copy
> your data back.

When I do the mentioned partitioning sceme, I put the biggest partition at
the beginning of the drive and the smaller one(s) at the back. That way,
should I ever actually need to resize a partition, I only have to export the
smaller partition for the process (or none at all, if it’s just a backup
itself and I have another backup on another drive).
Of course there’s LVM these days, but up until recently, I used NTFS for the
media partition so I could also read it in $DUMB_OS, which doesn’t know LVM.
Only a short while back, I also switched to ext4 for that, so I can retain
file names with : and ? in them. But I still refrained from using LVM,
though.

-- 
Gruß | Greetings | Qapla’
’ve been using vi for 15 years, because I don’t know with which command
to close it.

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

2016-09-01 9:13 GMT+03:00 Alan McKinnon :
> On 01/09/2016 08:04, gevisz wrote:
>> I have bought an external 5TB Western Digital hard drive
>> that I am going to use mainly for backing up some files
>> in my home directory and carrying a very big files, for
>> example a virtual machine image file, from one computer
>> to another. This hard drive is preformatted with NTFS.
>> Now, I am going to format it with ext4 which probably
>> will take a lot of time taking into account that it is
>> going to be done via USB connection. So, before formatting
>> this hard drive I would like to know if it is still
>> advisable to partition big hard drives into smaller
>> logical ones.
>
> it will take about 5 seconds to partition it.
> And a few more to mkfs it.

Just to partition - may be, but I very much doubt
that it will take seconds to create a full-fledged
ext4 file system on these 5TB via USB2 connention.

Even more: my aquiantance from the Window world
that recomended me this disc scared me that it may
take days...

> Are you sure you aren't thinking of mkfs with ext2
> (which did take hours for a drive that size?
>
>>
>> For about 20 last years, following an advice of my older
>> colleague, I always partitioned all my hard drives into
>> the smaller logical ones and do very well know all
>> disadvantages of doing so. :)
>
> So you are following 20 year-old advice for hardware relevant to 20
> years ago and not taking tech advances into account ? :-)

Yes. But, please, take into account that after these 20 years
I decided to reconsider the old "rule of thumb." :)

>> But what are disadvantages of not partitioning a big
>> hard drive into smaller logical ones?
>
> You only get 1 mount point
> Some ancient software might whinge and complain about not having a
> partition table present.
> The drive vendor no longer has a place to put their magic sekrit
> phone-home data collection stuff. Oh wait, that's a benefit and belongs
> below
>
>>
>> Is it still advisable to partition a big hard drive
>> into smaller logical ones and why?
>
> The only reason to partition a drive is to get 2 or more
> smaller ones that differ somehow (size, inode ratio, mount options, etc)
>
> Go with no partition table by all means, but if you one day find you
> need one, you will have to copy all your data off, repartition, and copy
> your data back. If you are certain that will not happen (eg you will
> rather buy a second drive) then by all means dispense with partitions.
>
> They are after all nothing more than a Microsoft invention from the 80s
> so people could install UCSD Pascal next to MS-DOS

I definitely will not need more than one mount point for this hard drive
but I do remember some arguments that partitioning a large hard drive
into smaller logical ones gives me more safety in case a file system
suddenly will get corrupted because in this case I will loose my data
only on one of the logical partitions and not on the whole drive.

Is this argument still valid nowadays?

Re: [gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[Alan McKinnon]

On 01/09/2016 08:04, gevisz wrote:
> I have bought an external 5TB Western Digital hard drive
> that I am going to use mainly for backing up some files
> in my home directory and carrying a very big files, for
> example a virtual machine image file, from one computer
> to another. This hard drive is preformatted with NTFS.
> Now, I am going to format it with ext4 which probably
> will take a lot of time taking into account that it is
> going to be done via USB connection. So, before formatting
> this hard drive I would like to know if it is still
> advisable to partition big hard drives into smaller
> logical ones.

it will take about 5 seconds to partition it.
And a few more to mkfs it.

Are you sure you aren't thinking of mkfs with ext2 (which did take hours
for a drive that size?

> 
> For about 20 last years, following an advice of my older
> colleague, I always partitioned all my hard drives into
> the smaller logical ones and do very well know all
> disadvantages of doing so. :)

So you are following 20 year-old advice for hardware relevant to 20
years ago and not taking tech advances into account ? :-)

> 
> But what are disadvantages of not partitioning a big
> hard drive into smaller logical ones?

You only get 1 mount point
Some ancient software might whinge and complain about not having a
partition table present.
The drive vendor no longer has a place to put their magic sekrit
phone-home data collection stuff. Oh wait, that's a benefit and belongs
below

> 
> Is it still advisable to partition a big hard drive
> into smaller logical ones and why?

The only reason to partition a drive is to get 2 or more
smaller ones that differ somehow (size, inode ratio, mount options, etc)

Go with no partition table by all means, but if you one day find you
need one, you will have to copy all your data off, repartition, and copy
your data back. If you are certain that will not happen (eg you will
rather buy a second drive) then by all means dispense with partitions.

They are after all nothing more than a Microsoft invention from the 80s
so people could install UCSD Pascal next to MS-DOS


-- 
Alan McKinnon
alan.mckin...@gmail.com

[gentoo-user] [OT] Is it still advisable to partition a big hard drive?

[gevisz]

I have bought an external 5TB Western Digital hard drive
that I am going to use mainly for backing up some files
in my home directory and carrying a very big files, for
example a virtual machine image file, from one computer
to another. This hard drive is preformatted with NTFS.
Now, I am going to format it with ext4 which probably
will take a lot of time taking into account that it is
going to be done via USB connection. So, before formatting
this hard drive I would like to know if it is still
advisable to partition big hard drives into smaller
logical ones.

For about 20 last years, following an advice of my older
colleague, I always partitioned all my hard drives into
the smaller logical ones and do very well know all
disadvantages of doing so. :)

But what are disadvantages of not partitioning a big
hard drive into smaller logical ones?

Is it still advisable to partition a big hard drive
into smaller logical ones and why?