Re: New SCSI Driver for Mac OS X

[Charles Davis]

On 4/28/10 11:08 PM, Ken Thomases wrote:
> On Apr 28, 2010, at 8:28 PM, Charles Davis wrote:
> 
>> On 4/28/10 5:48 PM, Ken Thomases wrote:
>>> On Apr 28, 2010, at 6:18 PM, Charles Davis wrote:
>>>
 Besides, to use it, we'd have to call DADiskMount() directly instead of
 going through diskutil.
>>>
>>> Huh?  Who goes through diskutil?
>> NTDLL, when it gets a FSCTL_DISMOUNT_VOLUME.
> 
> Huh.  Weird.  I did not know that.
> 
> I can't speak for Alexandre, but I suspect he would be fine using the 
> appropriate Mac-native framework, whether that would be DiskArbitration or 
> CoreServices (FSUnmountVolumeSync()), over invoking a Mac-specific command 
> via system().  (So long as it's a C API, of course. ;)
Don't know about that. The way it's implemented (and it's implemented by
a function in dlls/ntdll/directory.c called 'DIR_unmount_volume), it
invokes a command on both Mac OS X and other UNIXes/Linux. I think
whoever implemented it used diskutil(8) on Mac OS X because it was the
path of least resistance (given that it was already system(3)'ing
umount(8) on other platforms).
> 
> 
>>> But, anyway, there's no barrier to registering such an approval callback.
>> Like I said, it will work when ejection and unmount requests go through
>> DA, but not for raw umount(2) or ioctl(2) with DKIOCEJECT.
> 
> Is this something you've tested?  I have not, but I would not be at all 
> surprised if DiskArbitration had a kernel component.
> 
> For example, if I mount a disk image via the Finder and then use umount(8) to 
> unmount it, I get the relevant NSWorkspaceDidUnmountNotification.  
> NSWorkspace is built on DiskArbitration's callbacks but umount(8) is not.  It 
> just uses the unmount(2) system call.
I just used a similar test case with a real disk, and I still got the
DidUnmount note. But...
> 
> (The above test was the quickest to hand.  I didn't take the time to write a 
> direct DiskArbitration test case, with or without an unmount approval 
> callback.)
When I used DiskArbitration directly, and unmounted the disk with
umount(8), it did not ask me if it was OK to unmount the disk. When I
went through DA, however, I did get asked. I tried to test the same with
DKIOCEJECT instead of unmount(2), but I couldn't open the whole media
device file (got EBUSY).

It's probably true that AppKit has indeed registered callbacks with DA.
I looked at its dependencies with otool, and DiskArbitration was one of
them. But since it delivered the note when umount(8) was used and DA
didn't, AppKit must therefore be using some other mechanism to discover
this. (I thought it might be kqueues, but I looked at its imports with
nm and couldn't find any references to kqueue(2) or kevent(2).)

Chip

Re: New SCSI Driver for Mac OS X

[Ken Thomases]

On Apr 28, 2010, at 8:28 PM, Charles Davis wrote:

> On 4/28/10 5:48 PM, Ken Thomases wrote:
>> On Apr 28, 2010, at 6:18 PM, Charles Davis wrote:
>> 
>>> Besides, to use it, we'd have to call DADiskMount() directly instead of
>>> going through diskutil.
>> 
>> Huh?  Who goes through diskutil?
> NTDLL, when it gets a FSCTL_DISMOUNT_VOLUME.

Huh.  Weird.  I did not know that.

I can't speak for Alexandre, but I suspect he would be fine using the 
appropriate Mac-native framework, whether that would be DiskArbitration or 
CoreServices (FSUnmountVolumeSync()), over invoking a Mac-specific command via 
system().  (So long as it's a C API, of course. ;)


>> But, anyway, there's no barrier to registering such an approval callback.
> Like I said, it will work when ejection and unmount requests go through
> DA, but not for raw umount(2) or ioctl(2) with DKIOCEJECT.

Is this something you've tested?  I have not, but I would not be at all 
surprised if DiskArbitration had a kernel component.

For example, if I mount a disk image via the Finder and then use umount(8) to 
unmount it, I get the relevant NSWorkspaceDidUnmountNotification.  NSWorkspace 
is built on DiskArbitration's callbacks but umount(8) is not.  It just uses the 
unmount(2) system call.

(The above test was the quickest to hand.  I didn't take the time to write a 
direct DiskArbitration test case, with or without an unmount approval callback.)

-Ken

Re: New SCSI Driver for Mac OS X

[Charles Davis]

On 4/28/10 5:48 PM, Ken Thomases wrote:
> On Apr 28, 2010, at 6:18 PM, Charles Davis wrote:
> 
>> Besides, to use it, we'd have to call DADiskMount() directly instead of
>> going through diskutil.
> 
> Huh?  Who goes through diskutil?
NTDLL, when it gets a FSCTL_DISMOUNT_VOLUME.
>  Who would have to call DADiskMount()?
The same.
>  And why do you think DADiskMount() is necessary to register approval 
> callbacks?  That's not my understanding.  One gets the DADiskRefs for the 
> mounted disks via the disk-appeared callback.
Sorry, I should have taken a closer look. I forgot about the approval
callbacks.
> 
>> Which means we'd have to add some code to
>> support DA to NTDLL. I don't know what AJ thinks of that.
> 
> There's already DiskArbitration support in mountmgr.sys, added by Alexandre.  
> :)
I knew that.
> 
> At one point he tried using an unmount approval callback to have Wine close 
> all of its handles for files on the device, but DA wouldn't even ask the 
> callbacks if there were files open.
Right, because it will refuse unless you tell it force-unmount.
>  I believe that has changed in Snow Leopard and we could revive that 
> approach.  That way, unmounting volumes would just work seamlessly without 
> the need for the user to invoke eject.exe.so.
Cool. Maybe we should look into that.
> 
> But, anyway, there's no barrier to registering such an approval callback.
Like I said, it will work when ejection and unmount requests go through
DA, but not for raw umount(2) or ioctl(2) with DKIOCEJECT.
> 
> 
>>> Also, I don't know enough about the driver model to know if multiple user 
>>> clients can compete or conflict for locking the media.  For example, if two 
>>> processes lock the media and only one unlocks it, is it locked or unlocked? 
>>>  DiskArbitration avoids these pitfalls.
>> It's unlocked. It's like the old handles. 10 calls to HLock() would be
>> undone by a single HUnlock(). I need to fix that. Luckily, the kernel
>> gave me the process that initiated the (un)lock, so I can keep track of
>> that. (Then I can also force the disk unlocked when it terminates.)
> 
> OK, although I don't know if you have an easy way to actually be told when a 
> process terminates,
I don't, but I have *a* way:

The driver registers a kernel event socket () and
kernel control socket (). A daemon connects to both
sockets. The daemon listens for events on the event socket. Every time
the driver is told to lock the drive, it sends an event to the daemon.
When the daemon gets this event, it adds the process it got to a list of
processes for which it waits to die. When a process dies, the daemon
sends a control message to the driver, and the driver unlocks the drive.
Finally, every time the driver is told to unlock the drive, it sends
another event to the daemon, which removes the process from the list.
> or to link the lock to the process so it's automatically removed on process 
> termination.
Keep an OSSet of proc_t's that have locked the drive. Then when I get a
note from the daemon, I unlock the drive.
> 
> -Ken
> 

Re: New SCSI Driver for Mac OS X

[Ken Thomases]

On Apr 28, 2010, at 6:18 PM, Charles Davis wrote:

> Besides, to use it, we'd have to call DADiskMount() directly instead of
> going through diskutil.

Huh?  Who goes through diskutil?  Who would have to call DADiskMount()?  And 
why do you think DADiskMount() is necessary to register approval callbacks?  
That's not my understanding.  One gets the DADiskRefs for the mounted disks via 
the disk-appeared callback.

> Which means we'd have to add some code to
> support DA to NTDLL. I don't know what AJ thinks of that.

There's already DiskArbitration support in mountmgr.sys, added by Alexandre.  :)

At one point he tried using an unmount approval callback to have Wine close all 
of its handles for files on the device, but DA wouldn't even ask the callbacks 
if there were files open.  I believe that has changed in Snow Leopard and we 
could revive that approach.  That way, unmounting volumes would just work 
seamlessly without the need for the user to invoke eject.exe.so.

But, anyway, there's no barrier to registering such an approval callback.


>> Among other things, if Wine uses your driver to lock the media and then 
>> crashes, is there any way for the user to recover and unlock the media?
> Just send the device file a DKIOCALLOW, and then everything will be back
> to normal.

OK, but that's not exactly something you can just tell a user to do.

>> Also, I don't know enough about the driver model to know if multiple user 
>> clients can compete or conflict for locking the media.  For example, if two 
>> processes lock the media and only one unlocks it, is it locked or unlocked?  
>> DiskArbitration avoids these pitfalls.
> It's unlocked. It's like the old handles. 10 calls to HLock() would be
> undone by a single HUnlock(). I need to fix that. Luckily, the kernel
> gave me the process that initiated the (un)lock, so I can keep track of
> that. (Then I can also force the disk unlocked when it terminates.)

OK, although I don't know if you have an easy way to actually be told when a 
process terminates, or to link the lock to the process so it's automatically 
removed on process termination.

-Ken

Re: New SCSI Driver for Mac OS X

[Charles Davis]

On 4/28/10 5:02 PM, Ken Thomases wrote:
> Hi Charles,
> 
> Are the DKIOCPREVENT and DKIOCALLOW ioctls necessary?
They were the reason I first wrote the driver. I added DKIOCSCSICOMMAND
later.
>  DiskArbitration already has a facility for clients to refuse 
> mount/unmount/eject requests.
> 
> http://developer.apple.com/mac/library/documentation/Darwin/Reference/DiscArbitrationFramework/DiskArbitration_h/
That only works on requests that go through DiskArbitration. I'm pretty
sure that won't stop a raw DKIOCEJECT (which by the way is what Wine
uses). How can it, when DKIOCEJECT goes straight to the kernel?

Besides, to use it, we'd have to call DADiskMount() directly instead of
going through diskutil. Which means we'd have to add some code to
support DA to NTDLL. I don't know what AJ thinks of that.
> 
> Among other things, if Wine uses your driver to lock the media and then 
> crashes, is there any way for the user to recover and unlock the media?
Just send the device file a DKIOCALLOW, and then everything will be back
to normal.
>  Also, I don't know enough about the driver model to know if multiple user 
> clients can compete or conflict for locking the media.  For example, if two 
> processes lock the media and only one unlocks it, is it locked or unlocked?  
> DiskArbitration avoids these pitfalls.
It's unlocked. It's like the old handles. 10 calls to HLock() would be
undone by a single HUnlock(). I need to fix that. Luckily, the kernel
gave me the process that initiated the (un)lock, so I can keep track of
that. (Then I can also force the disk unlocked when it terminates.)
> 
> 
> Also, at 
> ,
>  there's a mismatch between "path+6" and "pathlen-5", no?  That's an argument 
> for using strlcat, instead, isn't it? 
You're right. It's fixed now. (Personally, I'm surprised it didn't panic.)

Chip

Re: New SCSI Driver for Mac OS X

[Ken Thomases]

Hi Charles,

Are the DKIOCPREVENT and DKIOCALLOW ioctls necessary?  DiskArbitration already 
has a facility for clients to refuse mount/unmount/eject requests.

http://developer.apple.com/mac/library/documentation/Darwin/Reference/DiscArbitrationFramework/DiskArbitration_h/

Among other things, if Wine uses your driver to lock the media and then 
crashes, is there any way for the user to recover and unlock the media?  Also, 
I don't know enough about the driver model to know if multiple user clients can 
compete or conflict for locking the media.  For example, if two processes lock 
the media and only one unlocks it, is it locked or unlocked?  DiskArbitration 
avoids these pitfalls.


Also, at 
,
 there's a mismatch between "path+6" and "pathlen-5", no?  That's an argument 
for using strlcat, instead, isn't it? 

(By no means does the above mean I've done a thorough code review.  It's just 
something I noticed.)

Regards,
Ken

Re: New SCSI Driver for Mac OS X

[Charles Davis]

On 4/28/10 9:35 AM, Charles Davis wrote:
> I don't have a public repository yet. My plan is to create a SourceForge
> project for this (like AJ suggested).
And here it is:

 http://sourceforge.net/projects/xnu-extras/

Chip

Re: New SCSI Driver for Mac OS X

[Charles Davis]

On 4/28/10 2:14 PM, Stefan Dösinger wrote:
> 
> Am 28.04.2010 um 17:35 schrieb Charles Davis:
> 
>> On 4/28/10 4:03 AM, Stefan Dösinger wrote:
>>>
>>> Am 27.04.2010 um 22:10 schrieb Charles Davis:
>>>
 tar jxf ExtMediaBSDClient.tar.bz2
 cd ExtMediaBSDClient
>>> Do you have the sourcecode somewhere with instructions on how to compile it?
>> Sure. I've attached the source here. To compile it, you need Xcode
>> installed.
> Looks interesting. I prefer to have the sourcecode before I load something 
> into my kernel :-)
That's understandable.
> 
>> Once it's loaded, all SCSI disks that are inserted afterward will accept 
>> the IOCTL.
> I assume that means I have to remove my cdrom from my drive and reinsert it 
> after loading the driver, correct? Or does it mean removing the drive(as in 
> unplugging and replugging a USB drive for example?)
If it's a removable disk drive, you don't have to physically unplug it.
:) Reinserting the CD-ROM should work.

Chip

Re: New SCSI Driver for Mac OS X

[Stefan Dösinger]

Am 28.04.2010 um 17:35 schrieb Charles Davis:

> On 4/28/10 4:03 AM, Stefan Dösinger wrote:
>> 
>> Am 27.04.2010 um 22:10 schrieb Charles Davis:
>> 
>>> tar jxf ExtMediaBSDClient.tar.bz2
>>> cd ExtMediaBSDClient
>> Do you have the sourcecode somewhere with instructions on how to compile it?
> Sure. I've attached the source here. To compile it, you need Xcode
> installed.
Looks interesting. I prefer to have the sourcecode before I load something into 
my kernel :-)

> Once it's loaded, all SCSI disks that are inserted afterward will accept 
> the IOCTL.
I assume that means I have to remove my cdrom from my drive and reinsert it 
after loading the driver, correct? Or does it mean removing the drive(as in 
unplugging and replugging a USB drive for example?)

Re: New SCSI Driver for Mac OS X

[Stefan Dösinger]

Am 27.04.2010 um 22:10 schrieb Charles Davis:

> tar jxf ExtMediaBSDClient.tar.bz2
> cd ExtMediaBSDClient
Do you have the sourcecode somewhere with instructions on how to compile it?

Re: New SCSI Driver for Mac OS X

[Charles Davis]

On 4/27/10 8:54 PM, C.W. Betts wrote:
> Excuse me, but why is this needed again? What IOCTL is missing in Mac OS X's 
> native implementation?
Do I have to explain everything?!

All right. Here goes.

Way back in October of last year, I started a discussion on this list
about SCSI on Mac OS X:

http://www.winehq.org/pipermail/wine-devel/2009-October/079257.html

Then AJ objected to the first approach, saying that we really can't
require the disk to be unmounted. This leaves option 2.

This driver is part of an attempt to implement option 2.

In summary:
- There's no IOCTL for sending SCSI commands.
- There's an API for sending them, but it's unrealistic for Wine.
Therefore, we want to use an IOCTL, instead.
- To implement an IOCTL, we need a driver.
- This driver implements an IOCTL for sending SCSI commands.

And don't suggest the SCSITask user client. I already did (as you can
see in the linked post).

Chip

Re: New SCSI Driver for Mac OS X

[C.W. Betts]

Excuse me, but why is this needed again? What IOCTL is missing in Mac OS X's 
native implementation?
On Apr 27, 2010, at 2:10 PM, Charles Davis wrote:

> Hi,
> 
> A while ago on this list, I was talking about implementing SCSI support
> on Mac OS X in Wine, and we all agreed the best way to do that would be
> to have a custom driver that provides an IOCTL interface.
> 
> I've attached the driver (which is now finished) here. I've tested it on
> my system, and it works with this small test program that's also
> attached. But I want to make sure it's totally stable before I start
> working on the Wine side. So if you've got a Mac, and you're feeling
> brave, feel free to install my driver and run my test program.
> 
> Extract the archive, then cd to the directory inside:
> 
> tar jxf ExtMediaBSDClient.tar.bz2
> cd ExtMediaBSDClient
> 
> Install the extension to /Library/Extensions:
> 
> cp -r ExtMediaBSDClient.kext /Library/Extensions
> 
> If you're feeling really brave, you can install it to
> /System/Library/Extensions (and then it will be loaded on boot):
> 
> cp -r ExtMediaBSDClient.kext /System/Library/Extensions
> 
> The driver bundle has to be owned by root and belong to the wheel group,
> so make sure it does before trying to load it:
> 
> sudo chown -R root:wheel /path/to/ExtMediaBSDClient.kext
> 
> You can load the driver with the kextload command:
> 
> kextload /path/to/ExtMediaBSDClient.kext
> 
> You can verify that the driver loaded by running:
> 
> kextstat | grep "ExtMediaBSDClient"
> 
> Once it's loaded, all SCSI disks that are inserted afterward will accept
> the IOCTL. All disks that were inserted before won't however. To make
> sure a disk is using my driver, run the ioreg tool (it's part of Xcode)
> and look for "ExtMediaBSDClient":
> 
> ioreg | grep "ExtMediaBSDClient"
> 
> To test it, run the program 'ExtMediaTest' that I included. Pass it the
> filename of a device file for the disk (it should be one of the raw
> device files, e.g. /dev/rdisk1 instead of /dev/disk1) on which you want
> to test this:
> 
> ./ExtMediaTest /path/to/device/file
> 
> If it works, the program will print some of the INQUIRY data for that
> disk to the Terminal. If it doesn't work, it will print an error to the
> Terminal (and the driver will have logged something in the kernel log;
> see the Console app), or worse, the kernel will panic. (If it says
> "Inappropriate ioctl for device", that probably means it's not using my
> driver. Try ejecting the disk and reinserting it. And if it says the
> device is busy, try the program on one of the slices, e.g. /dev/rdisk1s1
> instead of /dev/rdisk1.)
> 
> If anything goes wrong (error messages, kernel panics, etc.), I want to
> know about it. If you get an error, paste the whole error message. If
> it's from ExtMediaTest, show me the lines in the kernel log that came
> from the driver (they have "ExtMediaBSDClient" or some such in them). If
> the kernel panics, attach the panic log. Also, tell me which version of
> Mac OS X you're using (so I know which KernelDebugKit to use).
> 
> As soon as I verify that the driver works well, I'm going to start
> submitting patches to implement support for it in Wine.
> 
> Chip
>