Re: 7.1 sparc64 softraid0 1.5TB/2TB partition limit of RAID 5 + c

Michael Truog Fri, 23 Sep 2022 01:36:36 -0700

On 9/18/22 03:42, Klemens Nanni wrote:

On Fri, Sep 16, 2022 at 05:59:20PM -0700, Michael Truog wrote:

Hi,


I was attempting to have a RAID 5 softraid0 setup on a sparc64 machine (boot
log output below) but ran into problems when attempting to create a single
partition with the size 5.5TB (RAID 5 with 4 x 2TB hard drives).  I found an
interesting problem when using disklabel on the softraid0 hard drive device,
when attempting to make this 5.5TB partition.  The partition "a" would only
be allowed as 1.5TB and any partition >= "d" would only be allowed as 2TB,
however the limit occurred silently after disklabel had exited.  When inside
disklabel, I could allocate a single "a" partition to be 5.5TB successfully
and was able to write the partition successfully.  However, when the
disklabel process exited, either with the q command or a kill signal 9, the
partition would be shrunk to the limit described above.  If the disklabel
process was suspended (ctrl-Z), this wouldn't happen and newfs would see the
5.5TB partition, though usage of the partition wouldn't work.  The partition
would have inaccessible blocks that fsck showed extreme anger at, when it
saw it at boot time.

It would really help to showcase your issue with commands/output.

This issue is not related to softraid(4), it is most probably an old
sparc(64) quirk:

1. create big dummy disk for a single filesystem:

        $ ldomctl create-vdisk -s 10T sparse-10T.img

2. pass it to guest domain in order to have a "real" 10T sized sd(4):

        # dmesg | grep ^sd2
        sd2 at scsibus3 targ 0 lun 0: <SUN, Virtual Disk, 1.1>
        sd2: 10485760MB, 512 bytes/sector, 21474836480 sectors

I attempted doing this. The ldomctl manpage provided a good path forgetting that working. However, at least for my Sun SPARC EnterpriseT5220 (T2) I needed to explicitly declare the "primary" domain becauseotherwise the configuration would not be used after an ILOM resetoccurred. I had a working ldom.conf configuration, but once I attempted(after getting the ldom.conf working) to do "ldomctl delete openbsd""ldomctl download openbsd" ILOM "reset /SYS" the machine failed toboot. Then the machine failed to allow an ILOM "stop /SYS" to occur. So, I then removed both power plugs, inserted them again, updated thefirmware without retaining the configuration and all was well again.


My configuration was:
domain "primary" {
    vcpu 56
    memory 31G
}
domain "test" {
    vcpu 8
    memory 32G
    vdisk "/home/sparse-10T.img"
    vnet
}

I didn't initialize the vnet before the delete/download and it generatedWARNINGs on the console, so that may have led to the problem. I wasusing the 7.2 snapshot from 2022-09-19 for everything described in thisemail reply.

I did attempt to replicate the kernel panic from 7.1 with the 7.2snapshot using softraid0 5 and c. The limit on sparc64 partition sizesdoesn't appear to be very clearly defined, with the limit changing basedon the situation. In your situation below it has allowed partition a tobe resized to 2 TB when using the ldomctl vdisk. When I used softraid0with level c, partition a was allowed to be 1.6 TB. When I usedsoftraid0 with level 5, partition a was allowed to be 1.5 TB. With the7.2 snapshot it appears that this limit applies to the first partition,not necessarily partition a. With 7.1, I always saw it as an issue withpartition a. If partition d is created after a shrinks to 1.5 TB, withsoftraid0 level 5, partition d is allowed to be 2 TB. However, if anattempt is made to create a partition e after that, it shrinks to matchthe same partition as d (to become an invalid partition). I have outputthat helps to show this.

I was unable to replicate the 7.1 kernel panic when using the 7.2snapshot, so there isn't a clear failure related to the partitionresizing itself when disklabel exits. With the 7.2 snapshot, I am ableto suspend the disklabel process (after the write, but before the quit),do newfs, kill -9 on the disklabel process and mount/write to the newfilesystem without any kernel panic. The new filesystem will still failwhen fsck is attempted, but the first write of a file is able to succeed(tried the disklabel/suspend/newfs/kill/write-file sequence with bothsoftraid0 level 5 and c).

        # echo '/ 1M-* 100%' | disklabel -wAT/dev/stdin sd2
        # disklabel -h sd2
        # /dev/rsd2c:
        type: SCSI
        disk: SCSI disk
        label: Virtual Disk
        duid: c4befc09bf56efed
        flags: vendor
        bytes/sector: 512
        sectors/track: 255
        tracks/cylinder: 511
        sectors/cylinder: 130305
        cylinders: 164804
        total sectors: 21474836480 # total bytes: 10.0T
        boundstart: 0
        boundend: 21474836480

        16 partitions:
        #                size           offset  fstype [fsize bsize   cpg]
          a:             2.0T                0  4.2BSD   8192 65536     1
          c:            10.0T                0  unused
        disklabel: warning, partition a: size % cylinder-size != 0
Can you try reproducing your disklabel/suspend/newfs/kill/write-file
dance without mpi(4) and softraid(4) in between?

Does it also crash if you create partitions that are smaller than 2.0T?

I only saw the kernel panic when attempting to make the truncatedpartition work, specifically with 7.1 using softraid0 with level 5 (4x2TB disks). I never had problems with smaller partition sizes.


I have the output from the snapshot softraid0 use below:

# disklabel -E sd0