Large disk drives
Was there any resolution as to how large disk drives would be handled if their interface did not support the "capacity" request that would tell how large they were? Or as an alternative, is there any way to avoid buying USB-SCSI interfaces that do not support the large-capacity request? Unfortunately, such devices work OK with Windows (since Windows trusts what the partition table says), you can't just say to the salesperson "It has to work on drives over 3 TB." Dale -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
On Mon, 2014-11-03 at 16:06 -0500, Dale R. Worley wrote: > Was there any resolution as to how large disk drives would be handled > if their interface did not support the "capacity" request that would > tell how large they were? Realistically no ... unless someone comes up with a reliable heuristic to give us the size. > Or as an alternative, is there any way to avoid buying USB-SCSI > interfaces that do not support the large-capacity request? > Unfortunately, such devices work OK with Windows (since Windows trusts > what the partition table says), you can't just say to the salesperson > "It has to work on drives over 3 TB." This is a stopgap: your 3TB drive can be guessed as the 16 bit capacity plus 2TB, but the same won't happen for a 5TB device. Believing the partition table gives us a chicken and egg problem because something still has to get the partition table on to the device. I don't think "don't buy something that doesn't work" is a hugely unreasonable response to this. James > Dale > -- > To unsubscribe from this list: send the line "unsubscribe linux-scsi" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
On Tue, 4 Nov 2014, James Bottomley wrote: > On Mon, 2014-11-03 at 16:06 -0500, Dale R. Worley wrote: > > Was there any resolution as to how large disk drives would be handled > > if their interface did not support the "capacity" request that would > > tell how large they were? > > Realistically no ... unless someone comes up with a reliable heuristic > to give us the size. I posted a patch to allow the user to override the reported capacity: http://marc.info/?l=linux-scsi&m=140993840113445&w=2 Nobody responded to it. > > Unfortunately, such devices work OK with Windows (since Windows trusts > > what the partition table says), you can't just say to the salesperson > > "It has to work on drives over 3 TB." > > This is a stopgap: your 3TB drive can be guessed as the 16 bit capacity > plus 2TB, but the same won't happen for a 5TB device. Believing the > partition table gives us a chicken and egg problem because something > still has to get the partition table on to the device. > > I don't think "don't buy something that doesn't work" is a hugely > unreasonable response to this. The problem is knowing beforehand whether it will work. Once you buy the device and can test it, returning it is annoying and time-consuming at best. Alan Stern -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
Replying to two messages at once: > Date: Tue, 4 Nov 2014 11:14:39 -0500 (EST) > From: Alan Stern > cc: "Dale R. Worley" , , > > > On Tue, 4 Nov 2014, James Bottomley wrote: > > > On Mon, 2014-11-03 at 16:06 -0500, Dale R. Worley wrote: > > > Was there any resolution as to how large disk drives would be handled > > > if their interface did not support the "capacity" request that would > > > tell how large they were? > > > > Realistically no ... unless someone comes up with a reliable heuristic > > to give us the size. I can understand why the linux-scsi project would not want to take up what is really a hack to work around a hardware deficiency. > I posted a patch to allow the user to override the reported capacity: > > http://marc.info/?l=linux-scsi&m=140993840113445&w=2 > > Nobody responded to it. > > > > Unfortunately, such devices work OK with Windows (since Windows trusts > > > what the partition table says), you can't just say to the salesperson > > > "It has to work on drives over 3 TB." > > > > This is a stopgap: your 3TB drive can be guessed as the 16 bit capacity > > plus 2TB, but the same won't happen for a 5TB device. Believing the > > partition table gives us a chicken and egg problem because something > > still has to get the partition table on to the device. > > > > I don't think "don't buy something that doesn't work" is a hugely > > unreasonable response to this. > > The problem is knowing beforehand whether it will work. Once you buy > the device and can test it, returning it is annoying and time-consuming > at best. Or as I would phrase it, How do you turn "Don't buy something that doesn't work!" into an algorithm? That is: I'm standing in MicroCenter, holding a box in my hand that contains a USB-to-SCSI adapter. How do I determine whether or not it supports large disks? The problem being that it *works with Windows*, so I can't just ask the friendly salesperson, Does this work with 3 terabyte disks? In my case, the Diablotek device works, while the Kingwin EZ-Connect does not, despite being labeled in essentially the same way. (Neither says that Linux is supported.) I admit that my problem may be my deficiency in dealing with the retail situation, as I'm a much better software engineer than shopper. Dale -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
On Tue, 2014-11-04 at 11:14 -0500, Alan Stern wrote: > On Tue, 4 Nov 2014, James Bottomley wrote: > > > On Mon, 2014-11-03 at 16:06 -0500, Dale R. Worley wrote: > > > Was there any resolution as to how large disk drives would be handled > > > if their interface did not support the "capacity" request that would > > > tell how large they were? > > > > Realistically no ... unless someone comes up with a reliable heuristic > > to give us the size. > > I posted a patch to allow the user to override the reported capacity: > > http://marc.info/?l=linux-scsi&m=140993840113445&w=2 > > Nobody responded to it. Sorry, meant to. In principle I'm OK with this as the lever for the hack (largely because it means we don't need to do anything) but will the distributions support it? > > > Unfortunately, such devices work OK with Windows (since Windows trusts > > > what the partition table says), you can't just say to the salesperson > > > "It has to work on drives over 3 TB." > > > > This is a stopgap: your 3TB drive can be guessed as the 16 bit capacity > > plus 2TB, but the same won't happen for a 5TB device. Believing the > > partition table gives us a chicken and egg problem because something > > still has to get the partition table on to the device. > > > > I don't think "don't buy something that doesn't work" is a hugely > > unreasonable response to this. > > The problem is knowing beforehand whether it will work. Once you buy > the device and can test it, returning it is annoying and time-consuming > at best. OK, but I still don't understand how windows gets the partition table on there in the first place ... that must surely be some sort of guess the disk size hack. James > Alan Stern > > -- > To unsubscribe from this list: send the line "unsubscribe linux-scsi" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
On Wed, 5 Nov 2014, James Bottomley wrote: > On Tue, 2014-11-04 at 11:14 -0500, Alan Stern wrote: > > On Tue, 4 Nov 2014, James Bottomley wrote: > > > > > On Mon, 2014-11-03 at 16:06 -0500, Dale R. Worley wrote: > > > > Was there any resolution as to how large disk drives would be handled > > > > if their interface did not support the "capacity" request that would > > > > tell how large they were? > > > > > > Realistically no ... unless someone comes up with a reliable heuristic > > > to give us the size. > > > > I posted a patch to allow the user to override the reported capacity: > > > > http://marc.info/?l=linux-scsi&m=140993840113445&w=2 > > > > Nobody responded to it. > > Sorry, meant to. In principle I'm OK with this as the lever for the > hack (largely because it means we don't need to do anything) but will > the distributions support it? While I can't speak for the distributions, it's reasonable to assume that if something becomes part of the upstream kernel then they will include it. > OK, but I still don't understand how windows gets the partition table on > there in the first place ... that must surely be some sort of guess the > disk size hack. It's simpler than that: The drive is attached directly to the computer (i.e., via SATA rather than USB) when the partition table is created. With no USB-SATA bridge chip to mess things up, there's no problem determining the correct capacity. Alan Stern -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
On 11/05/2014 06:34 PM, Alan Stern wrote: <> > > It's simpler than that: The drive is attached directly to the computer > (i.e., via SATA rather than USB) when the partition table is created. > With no USB-SATA bridge chip to mess things up, there's no problem > determining the correct capacity. > Right! I think it should be very simple to, just as we send a READ_CAPACITY16 / READ_CAPACITY32 / READ_CAPACITY64 or is it a GET_CODE_PAGE ? whatever we send today we can also have READ_CAPACITY_PART() which will send a READ of sector 0 the raw PC_COMMAND way and run it through some partition analyzer. We only need to support gpt or msdos partitions I think in this case. Then if READ_CAPACITY16 returns all (s) and READ_CAPACITY32 is not supported and/or blacklisted, then yes attempt a READ_CAPACITY_PART() which should be sam-2 compatible. It should not take longer than a weekend afternoon over cup of Machha. If any one sends a bad device my way, I'll do it. But anyone should be able to code something so simple. > Alan Stern > Cheers Boaz -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
On Wed, Nov 05, 2014 at 05:07:48PM +0100, James Bottomley wrote: > > OK, but I still don't understand how windows gets the partition table on > there in the first place ... that must surely be some sort of guess the > disk size hack. 99% of the time the partition table was provided by the drive manufacturer; I would assume they have a way of doing that as a part of their manufacturing processes that don't require an OS individually running the equivalent of fdisk or gdisk on said drive. - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
On Wed, Nov 05, 2014 at 11:34:11AM -0500, Alan Stern wrote: > > Sorry, meant to. In principle I'm OK with this as the lever for the > > hack (largely because it means we don't need to do anything) but will > > the distributions support it? > > While I can't speak for the distributions, it's reasonable to assume > that if something becomes part of the upstream kernel then they will > include it. Btw, we do have precedence for looking at partition tables from SCSI code with scsi_partsize(), so the layering violation of looking at EFI labels for disks sizes wouldn't be something entirely new even if we did it in kernel space. -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
> From: Alan Stern > I posted a patch to allow the user to override the reported capacity: > > http://marc.info/?l=linux-scsi&m=140993840113445&w=2 I see the patch, and I feel confident I could install it if I needed to. What command do I execute to "write to the capacity_override attribute", before I do "blockdev --rereadpt /dev/sdX"? Dale -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
On Wed, 2014-11-05 at 11:30 -0800, Christoph Hellwig wrote: > On Wed, Nov 05, 2014 at 11:34:11AM -0500, Alan Stern wrote: > > > Sorry, meant to. In principle I'm OK with this as the lever for the > > > hack (largely because it means we don't need to do anything) but will > > > the distributions support it? > > > > While I can't speak for the distributions, it's reasonable to assume > > that if something becomes part of the upstream kernel then they will > > include it. > > Btw, we do have precedence for looking at partition tables from SCSI > code with scsi_partsize(), so the layering violation of looking at > EFI labels for disks sizes wouldn't be something entirely new even > if we did it in kernel space. We really don't want to make the decision within the kernel of whether we believe the partition size or the disk capacity. For these disk problems we need it to be the former, but if we choose that always, we'll get weird results on mispartitioned devices. The usual rule is no policy in the kernel and which to choose is policy, so just export the knob (as Alan's patch does) and then let userspace decide. James -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
On 11/06/2014 12:30 PM, James Bottomley wrote: > On Wed, 2014-11-05 at 11:30 -0800, Christoph Hellwig wrote: >> On Wed, Nov 05, 2014 at 11:34:11AM -0500, Alan Stern wrote: Sorry, meant to. In principle I'm OK with this as the lever for the hack (largely because it means we don't need to do anything) but will the distributions support it? >>> >>> While I can't speak for the distributions, it's reasonable to assume >>> that if something becomes part of the upstream kernel then they will >>> include it. >> >> Btw, we do have precedence for looking at partition tables from SCSI >> code with scsi_partsize(), so the layering violation of looking at >> EFI labels for disks sizes wouldn't be something entirely new even >> if we did it in kernel space. > > We really don't want to make the decision within the kernel of whether > we believe the partition size or the disk capacity. For these disk > problems we need it to be the former, but if we choose that always, > we'll get weird results on mispartitioned devices. > > The usual rule is no policy in the kernel and which to choose is policy, > so just export the knob (as Alan's patch does) and then let userspace > decide. > I do not think it is a matter of policy. >From what I understand the situation is that read_capacity_10() which is 32bit, Max 2T byte disks. fails with 0x and read_capacity_16() (64bit) is not supported because of wrong scsi-version or because it is blacklisted. (or device returns not-supported) Now If sd_read_capacity() succeed then off-course we choose it. What I'm saying is if read-capacity fails, then should we attempt a new read_capacity_part()? And sure a user-mode knob if he wants to make policy, like you said, is fine by me. But just the simple case of read-capacity failure should we then? > James Thanks Boaz -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
On Thu, 2014-11-06 at 16:33 +0200, Boaz Harrosh wrote: > On 11/06/2014 12:30 PM, James Bottomley wrote: > > On Wed, 2014-11-05 at 11:30 -0800, Christoph Hellwig wrote: > >> On Wed, Nov 05, 2014 at 11:34:11AM -0500, Alan Stern wrote: > Sorry, meant to. In principle I'm OK with this as the lever for the > hack (largely because it means we don't need to do anything) but will > the distributions support it? > >>> > >>> While I can't speak for the distributions, it's reasonable to assume > >>> that if something becomes part of the upstream kernel then they will > >>> include it. > >> > >> Btw, we do have precedence for looking at partition tables from SCSI > >> code with scsi_partsize(), so the layering violation of looking at > >> EFI labels for disks sizes wouldn't be something entirely new even > >> if we did it in kernel space. > > > > We really don't want to make the decision within the kernel of whether > > we believe the partition size or the disk capacity. For these disk > > problems we need it to be the former, but if we choose that always, > > we'll get weird results on mispartitioned devices. > > > > The usual rule is no policy in the kernel and which to choose is policy, > > so just export the knob (as Alan's patch does) and then let userspace > > decide. > > > > I do not think it is a matter of policy. > > From what I understand the situation is that read_capacity_10() which is > 32bit, Max 2T byte disks. fails with 0x and read_capacity_16() (64bit) > is not supported because of wrong scsi-version or because it is blacklisted. > (or device returns not-supported) Actually, no, RC(10) returns the lowest 32 bits of the actual result. Which is out of spec, but hey, this is a USB bridge ... > Now If sd_read_capacity() succeed then off-course we choose it. Um, the problem is we can't tell ... sd_read_capacity() returns a number, it just may be wrong by a factor of 2TB. > What I'm saying > is if read-capacity fails, then should we attempt a new read_capacity_part()? We can't tell we have a failure. We can tell if the partition capacity and the read capacity differ that one of them must be wrong ... > And sure a user-mode knob if he wants to make policy, like you said, is fine > by me. > > But just the simple case of read-capacity failure should we then? We don't have a failure. This is the problem. Determining that a problem exists James -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
On Thu, 6 Nov 2014, Boaz Harrosh wrote: > On 11/06/2014 12:30 PM, James Bottomley wrote: > > We really don't want to make the decision within the kernel of whether > > we believe the partition size or the disk capacity. For these disk > > problems we need it to be the former, but if we choose that always, > > we'll get weird results on mispartitioned devices. > > > > The usual rule is no policy in the kernel and which to choose is policy, > > so just export the knob (as Alan's patch does) and then let userspace > > decide. Another alternative would be to export a policy setting for whether or not to take the capacity from the partition table. That would be more convenient for users, because then they wouldn't have to figure out the number of blocks in the drive. On the other hand, it wouldn't work so well if there is no partition table and the user wants to create one. > I do not think it is a matter of policy. > > From what I understand the situation is that read_capacity_10() which is > 32bit, Max 2T byte disks. fails with 0x and read_capacity_16() (64bit) > is not supported because of wrong scsi-version or because it is blacklisted. > (or device returns not-supported) No, that is not the problem. read_capacity_10() succeeds but gives an incorrect value. read_capacity_16() fails. > Now If sd_read_capacity() succeed then off-course we choose it. What I'm > saying > is if read-capacity fails, then should we attempt a new read_capacity_part()? sd_read_capacity() succeeds, but the value it gets is wrong. > And sure a user-mode knob if he wants to make policy, like you said, is fine > by me. > > But just the simple case of read-capacity failure should we then? That's a separate question. As far as I know, the case you are describing has not come up. Alan Stern -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
On 11/06/2014 05:54 PM, James Bottomley wrote: > > We don't have a failure. This is the problem. Determining that a > problem exists > OK Sorry. I assumed the bridge is smart enough to do nothing, ie READ_CAPACITY_10 is passed as is via sata to the device that actually supports READ_CAPACITY_16, as I understand then the actual good drive is not suppose to send size-modulue-2G in response to READ_CAPACITY_10. Should it ? Then the bridge just sends that back to me, now if I send READ_CAPACITY_16 the bridge will return NOT-SUPPORTED because it is unexpected. But what are you saying that the bridge was smart enough to do READ_CAPACITY_16 get a 64bit value from the drive and then return the lower 32bit to me ? Really ? I would not imagine in the life of me someone so dumb. And surly it is against any spec. Are you sure ? I think you are wrong I think the guy reported that he can only see 2T out of his 3T drive which means the bridge returned 0x, exactly 2T > James > > But hey, I guess stupidity is limitless, this one here pushing real far. Cheers Boaz -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
On 11/06/2014 05:53 PM, Alan Stern wrote: >> But just the simple case of read-capacity failure should we then? > > That's a separate question. As far as I know, the case you are > describing has not come up. > BTW: what we should do is when the partition parser at the block layer see that the partition capacity as written in the partition-table is bigger then the capacity reported for the device we can put a fat message at dmesg with both sizes and user can decide. And/or add an extra field at block attributes for part_capacity. It is info that is known and parsed for today. > Alan Stern Thanks Boaz -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
On Thu, 6 Nov 2014, Boaz Harrosh wrote: > On 11/06/2014 05:54 PM, James Bottomley wrote: > > > > We don't have a failure. This is the problem. Determining that a > > problem exists > > > > OK Sorry. I assumed the bridge is smart enough to do nothing, > > ie READ_CAPACITY_10 is passed as is via sata to the device that > actually supports READ_CAPACITY_16, as I understand then the > actual good drive is not suppose to send size-modulue-2G in > response to READ_CAPACITY_10. Should it ? The bridge does not send SCSI commands to the drive; it sends ATA commands. When the bridge receives a READ CAPACITY(10) command from the host computer, if the drive's actual capacity is >= 2^32 blocks then the bridge is supposed to send back a result buffer containing 0x. Not the actual capacity modulo 2^32. > Then the bridge just sends that back to me, now if I send > READ_CAPACITY_16 the bridge will return NOT-SUPPORTED because > it is unexpected. Yes. > But what are you saying that the bridge was smart enough to > do READ_CAPACITY_16 get a 64bit value from the drive and then > return the lower 32bit to me ? Really ? I would not imagine > in the life of me someone so dumb. And surly it is against > any spec. The bridge does not send READ CAPACITY(16) to the drive. It sends an ATA command -- and it probably sends the same command for READ CAPACITY(10). > Are you sure ? I think you are wrong I think the guy reported > that he can only see 2T out of his 3T drive which means the > bridge returned 0x, exactly 2T We are sure. Dale reported that the size he got was less than 2 TB, and it was equal to the actual capacity modulo 2^32. You can check for yourself; here's the original email: http://marc.info/?l=linux-scsi&m=140908235510961&w=2 Alan Stern -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Large disk drives
From: Boaz Harrosh > On 11/06/2014 05:53 PM, Alan Stern wrote: > >> But just the simple case of read-capacity failure should we then? > > > > That's a separate question. As far as I know, the case you are > > describing has not come up. > > > > BTW: what we should do is when the partition parser at the block layer > see that the partition capacity as written in the partition-table is > bigger then the capacity reported for the device we can put a fat > message at dmesg with both sizes and user can decide. One possibility is to try to read the last sector of the actual partition. If it succeeds there are 2 possibilities: 1) The disk is as bit as the partition table indicates. 2) The high bit(s) of the sector number have been masked. (or the disk locks up) In many cases the latter can be verified by reading the other sector. But if the media is new/erased then all the sectors will be 0xff and you'd have to do a write to ensure there was no sector aliasing. David
Re: Large disk drives
On Thu, 2014-11-06 at 17:08 +, David Laight wrote: > From: Boaz Harrosh > > On 11/06/2014 05:53 PM, Alan Stern wrote: > > >> But just the simple case of read-capacity failure should we then? > > > > > > That's a separate question. As far as I know, the case you are > > > describing has not come up. > > > > > > > BTW: what we should do is when the partition parser at the block layer > > see that the partition capacity as written in the partition-table is > > bigger then the capacity reported for the device we can put a fat > > message at dmesg with both sizes and user can decide. > > One possibility is to try to read the last sector of the actual > partition. If it succeeds there are 2 possibilities: These are USB devices (bridges) not well behaved SCSI devices. The last time we tried to poke USB devices beyond their defined capacity (the USB capacity off by one problem) the firmware on some crashed and we eventually gave up. That's why, unless we can find simple, functional heuristics for the kernel, it's safer to punt to userspace. James > 1) The disk is as bit as the partition table indicates. > 2) The high bit(s) of the sector number have been masked. > (or the disk locks up) > > In many cases the latter can be verified by reading the other sector. > But if the media is new/erased then all the sectors will be 0xff > and you'd have to do a write to ensure there was no sector aliasing. > > David > > NrybXǧv^){.n+{"{ayʇڙ,jfhzwj:+vwjmzZ+ݢj"! -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
On Wed, 5 Nov 2014, Dale R. Worley wrote: > > From: Alan Stern > > > I posted a patch to allow the user to override the reported capacity: > > > > http://marc.info/?l=linux-scsi&m=140993840113445&w=2 > > I see the patch, and I feel confident I could install it if I needed > to. What command do I execute to "write to the capacity_override > attribute", before I do "blockdev --rereadpt /dev/sdX"? As root: echo N >/sys/class/scsi_disk/D:D:D:D/capacity_override where N is the number of blocks and D:D:D:D is the device name corresponding to the sdX disk. Or if you prefer, an equivalent path would be /sys/block/sdX/device/scsi_disk/D:D:D:D/capacity_override and here you wouldn't have to guess at the value of D:D:D:D because it would be the only entry in the /sys/block/sdX/device/scsi_disk directory. Alan Stern -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
> From: "Theodore Ts'o" > > On Wed, Nov 05, 2014 at 05:07:48PM +0100, James Bottomley wrote: > > > > OK, but I still don't understand how windows gets the partition table on > > there in the first place ... that must surely be some sort of guess the > > disk size hack. > > 99% of the time the partition table was provided by the drive > manufacturer; I would assume they have a way of doing that as a part > of their manufacturing processes that don't require an OS individually > running the equivalent of fdisk or gdisk on said drive. I would expect it as part of the manufacturer's testing process. After all, the disks are going to be plugged into a machine whose whole purpose is to run the disk through its paces, and the machine knows in advance exactly how big the disk is. Dale -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
> From: James Bottomley > > We really don't want to make the decision within the kernel of whether > we believe the partition size or the disk capacity. For these disk > problems we need it to be the former, but if we choose that always, > we'll get weird results on mispartitioned devices. > > The usual rule is no policy in the kernel and which to choose is policy, > so just export the knob (as Alan's patch does) and then let userspace > decide. I can see some significant advantages to setting up the hack in user-space, above the architectural ones: - We can get the user to confirm how big we think the disk is -- because its size is printed on the label of the disk. And if the partition table is unreadable in some way, the user can manually enter the size. - We can warn the user that the data path to his disk is deficient in not supporting which ever SCSI request, so that he knows to upgrade his hardware. There is one thing that seems like it might be a problem: We have to ensure that the SCSI driver can read the partition tables (in the standard locations) even if it doesn't know how big the disk is. Which leads me to wonder what happens if one reads /dev/sdX until one hits end-of file. People have written that we don't want to read the disk from locations beyond end-of-data because some disks react badly to out-of-range reads. But if that is so in general, there would be problems simply copying /dev/sdX. (Indeed, if all disks gave a proper error for out-of-range reads, a bisection search would find the size of the disk easily enough.) Dale -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
On Thu, 6 Nov 2014, Dale R. Worley wrote: > There is one thing that seems like it might be a problem: We have to > ensure that the SCSI driver can read the partition tables (in the > standard locations) even if it doesn't know how big the disk is. A DOS partition table is stored in the first 512 bytes of the disk. A GPT partition table occupies the first 33 or so blocks. (It also has a copy occupying the last few blocks, but if you don't know how large the disk is, that's not much help.) > Which leads me to wonder what happens if one reads /dev/sdX until one > hits end-of file. People have written that we don't want to read the > disk from locations beyond end-of-data because some disks react badly > to out-of-range reads. But if that is so in general, there would be > problems simply copying /dev/sdX. (Indeed, if all disks gave a proper > error for out-of-range reads, a bisection search would find the size > of the disk easily enough.) Most drives will work fine if you try to read beyond the end. You'll just get an appropriate error return. But some devices (typically, cheap consumer-grade USB devices) go wacky. Often enough, they crash or hang. Some of them may get going again in response to a reset; others have to be powered off to recover. Alan Stern -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
If READ_CAPACITY_10 returns something that looks valid but might be off by a multiple of 2TB, and READ_CAPACITY_16 fails, what do we really want to do when we read the partition table? If the partition table indicates that everything fits in the capacity returned by READ_CAPACITY_10, great, it isn't a large disk drive and we can do all 10-byte commands. If the partition table indicates that the disk drive really is large and the USB bridge has problems, I'd say we'd better not touch the drive any further (with possible exceptions). If the USB bridge fails READ_CAPACITY_16 but the USB bridge pretends to succeed with READ_16 and WRITE_16, well maybe it wraps around at 2TB and writes the wrong sector on the drive. If READ_CAPACITY_16 fails, I wouldn't trust the USB bridge with any 16-byte command. If the partition table indicates that the drive is larger, the user had better be told to find a different USB bridge or use a real SATA or SCSI controller. A possible exception is that if a USB bridge is well tested and confirmed to handle READ_16 and WRITE_16 correctly while failing READ_CAPACITY_16, an antiquirk could be configured to say we rely on the bridge. By the way, I've seen some USB bridges that lie about whether they performed various SAT commands (ATA passthrough), but told the truth about performing an ATA IDENTIFY DEVICE through SAT. So we could attempt ATA passthrough with an IDENTIFY DEVICE command, and if it happens to return a sane looking result, we could comparre it to the bridge's own translation of READ_CAPACITY_10 and READ_CAPACITY_16. If the passthrough yielded a sane result and the capacity doesn't match, we know not to trust the bridge with 16-byte commands (except if tested, as mentioned). In such a case, we don't even have to look at the partition table. Of course if the ATA passthrough fails or if the result is garbage then abandon this test and maybe look at the partition table. By the way, it's not just USB bridges. I have a particularly obnoxious ExpressCard with an eSATA connection. Where the SATA drive operates on 512-byte LBAs (whether real or emulated doesn't matter), the eSATA bridge translates groups of 8 of them into emulated 4096-byte LBAs. This should make it possible for a 3TB drive to be used when the PC's drivers don't know how to use ATA commands on sector numbers bigger than 32 bits, as long as the PC's drivers can handle 4096-byte sectors, right? Well, no. This eSATA bridge's firmware wraps around at 2TB. The vendor went to all this effort just to screw up and maximize the amount of damage that occurs by hiding their screw up. I don't know if eBay's warranty covers this kind of defect, but it wouldn't be worth paying for postage to return this POS. I should have bought the Brooklyn Bridge instead. To compound the oddity, the same ExpressCard contains a working USB 3.0 bridge. (Sorry I can't construct headers to get this message threaded correctly in the mailing list.) -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Large disk drives
From: Norman Diamond ... > By the way, I've seen some USB bridges that lie about whether they > performed various SAT commands (ATA passthrough), but told the truth > about performing an ATA IDENTIFY DEVICE through SAT. So we could attempt > ATA passthrough with an IDENTIFY DEVICE command, and if it happens to > return a sane looking result, we could comparre it to the bridge's own > translation of READ_CAPACITY_10 and READ_CAPACITY_16. If the passthrough > yielded a sane result and the capacity doesn't match, we know not to > trust the bridge with 16-byte commands (except if tested, as mentioned). > In such a case, we don't even have to look at the partition table. Of > course if the ATA passthrough fails or if the result is garbage then > abandon this test and maybe look at the partition table. ... Don't count on devices answering ATA IDENTIFY correctly either. I had some CF cards from one of the main 'labels' that reported an incorrectly layed out identify response, and 2 others (almost identical) that locked solid when the request was issued. We tried to send them back, but they were returned with a nice shiny new MBR table. David -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large disk drives
From: David Laight > > From: Norman Diamond > ... >> By the way, I've seen some USB bridges that lie about whether they >> performed various SAT commands (ATA passthrough), but told the truth >> about performing an ATA IDENTIFY DEVICE through SAT. So we could attempt >> ATA passthrough with an IDENTIFY DEVICE command, and if it happens to >> return a sane looking result, we could comparre it to the bridge's own >> translation of READ_CAPACITY_10 and READ_CAPACITY_16. If the passthrough >> yielded a sane result and the capacity doesn't match, we know not to >> trust the bridge with 16-byte commands (except if tested, as mentioned). >> In such a case, we don't even have to look at the partition table. Of >> course if the ATA passthrough fails or if the result is garbage then >> abandon this test and maybe look at the partition table. > ... > > Don't count on devices answering ATA IDENTIFY correctly either. Of course. That's why you quoted me saying this: >> Of >> course if the ATA passthrough fails or if the result is garbage then >> abandon this test and maybe look at the partition table. -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html