Re: Request for cooperation with all burn backends
[EMAIL PROTECTED] wrote: Hi, Joerg Schilling: Cooperative locking is needed in a way that allows and is based not on device nodes but on real hardware targets. Bill Davidsen: I'm less sure about cooperative locking, any method which fails if any one program behaves badly is not scalable. It seems to be a hard constraint in the burn community that the locking mechanism has to allow free access to the drive if the burn program deems its actions harmless. Full responsibility has as precondition a sufficient knowledge of the situation, nevertheless. So the burn programs need at least some means to announce their disturbance-sensitive activities on a drive and to detect such announcements before starting any activity which assumes exclusive usage of the drive. Bill: A perfect solution also must address locking of partitions vs. locking the entire device. Partitions with burner programs ? How do partitions apply to us ? Unless you believe that we can get device locking in for just one device and ignore the rest, a proposal would have to address locking the device for all devices, disk, tape, burner, flash, etc. I doubt that a partial solution would find much support, while a solution over all device might. SysV message queue msgget, msgsnd, ... Indeed a candidate. Installed on my Linux. Then there is mq_open, mq_send, ... Labeled "POSIX". Not installed on my Linux. I will consider to use this. It has not the advantage of TCP/IP to detect the demise of a lock holder by the end of the connection. Have a nice day :) Thomas -- bill davidsen <[EMAIL PROTECTED]> CTO TMR Associates, Inc Doing interesting things with small computers since 1979 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
Hi, Joerg Schilling: >> Cooperative locking is needed in a way that allows and is based not on >> device nodes but on real hardware targets. Bill Davidsen: > I'm less sure about cooperative locking, any method which fails if any one > program behaves badly is not scalable. It seems to be a hard constraint in the burn community that the locking mechanism has to allow free access to the drive if the burn program deems its actions harmless. Full responsibility has as precondition a sufficient knowledge of the situation, nevertheless. So the burn programs need at least some means to announce their disturbance-sensitive activities on a drive and to detect such announcements before starting any activity which assumes exclusive usage of the drive. Bill: > A perfect solution also must address locking of > partitions vs. locking the entire device. Partitions with burner programs ? How do partitions apply to us ? > I have a little paper somewhere If possible i would be interested to read. As said, i do not strive for a perfect solution but for one that gives some hope and chance for mutual good-will programming. Main concerns currently: - An unambigous mapping from drive addresses as known to the burn programs to the physical drives which we actually want to address. I consider to put the full load of this to a server process. - A minimal footprint in the code of the burn programs so that everybody is willing to include the necessary software and to participate in the cooperation. This client code shall only provide a unique process id, transfer the known (reversely ambigous) address of the drive to the server and recieve the reply wether the (unambigous) drive could be exclusively reserved or not. > I don't know that any solution which depends on every program > cooperating will be practical, and in fact automounters seem to ignore > the rest of the world. The good willing programs and developers are the ones whom i want to address here. If we ever get a practical locking convention implemented on the most important operating systems then the next step might be a coordinated move of burn developers towards automounter developers. It would spare both sides lots of anger if we would not have our software fighting. > I'm leaning toward Joerg's thought that locks on inodes referencing > physical devices should work at the device level to avoid aliasing issues. This is the other hard constraint that emerged from the discussion so far: device file locking is insufficient. Even device driver locking is not what we need. We need to cooperate on the drive. One drive - one lock. Not so clear are: - How much is it worth to us ? How much effort will each developer accept in order to participate ? - What operating systems need to be covered by a first suite of locking servers and how complicated will it be to implement unique drive identification on each ? My ideas for Linux are not 100% fool proof but would work for the configurations known to me: /dev/hdX device driver or (emulated) SCSI devices. > SysV message queue msgget, msgsnd, ... Indeed a candidate. Installed on my Linux. Then there is mq_open, mq_send, ... Labeled "POSIX". Not installed on my Linux. I will consider to use this. It has not the advantage of TCP/IP to detect the demise of a lock holder by the end of the connection. Have a nice day :) Thomas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
[EMAIL PROTECTED] wrote: Hi, Andy: int grab_sg (int blkfd) me: Seems to work well for me and my two drives sr0 and sr1. Joerg: Forget about this "method". It is known not to work reliably on Linux and similar moethods will not work at all on other OS. This is a kind of emergency patch for a particular problem with (some ?) Linux kernels 2.4 . I am very thankful to Andy that he addresses old kernel 2.4 at all. His proposal will allow me to detect growisofs runs on a drive and to stay off. The problem on Linux is device aliasing that results in many independent device nodes. Yep. O_EXCL fails to provide the needed functionality under many circumstances. Above function will allow growisofs and libburn to meet at the same Linux /dev/sgN and there locking should work. (Same works fine between cdrecord and libburn.) Cooperative locking is needed in a way that allows and is based not on device nodes but on real hardware targets. I agree to this statement in general. (Above sg grabbing is not a widely usable solution.) My ideas are about a central dispatcher service which arbitrates locking requests. It could encapsulate nearly all OS dependency if we manage to find an OS independend communications method and make it understand all our permissible address formats (permissible on the particlar OS). I don't know that any solution which depends on every program cooperating will be practical, and in fact automounters seem to ignore the rest of the world. But if you did want to do a control program, I might suggest looking at the SysV message queue features, which are portable and provide serialized N-to-M communication, removal of unread messages on terminations, etc. I'm leaning toward Joerg's thought that locks on inodes referencing physical devices should work at the device level to avoid aliasing issues. -- bill davidsen <[EMAIL PROTECTED]> CTO TMR Associates, Inc Doing interesting things with small computers since 1979 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
Joerg Schilling wrote:
[EMAIL PROTECTED] wrote:
Hi,
int grab_sg (int blkfd)
Seems to work well for me and my two drives sr0 and sr1.
I threw out two of my three functions and made
try_to_lock_linux_sg() using grab_sg() instead.
The call in builtin_dd() has changed a bit:
if (i == 3 && fd >= 0)
try_to_lock_linux_sg(ioctl_device, fd);
void try_to_lock_linux_sg(char *ioctl_device, int ioctl_dev_fd)
{
int fd_sg = -1;
if ( ( ( strncmp (ioctl_device, "/dev/sr", 7) == 0
&& isdigit(ioctl_device[7]) ) ||
( strncmp (ioctl_device, "/dev/scd", 8) == 0
&& isdigit(ioctl_device[8]) )
) )
fd_sg = grab_sg(ioctl_dev_fd);
if(fd_sg == -2)
fprintf (stderr,":-( unable to O_EXCL sg equivalent of %s: "
"Other burn program active on drive ?\n",
ioctl_device),
exit (FATAL_START(errno));
}
Forget about this "method". It is known not to work reliably on Linux
and similar moethods will not work at all on other OS.
The problem on Linux is device aliasing that results in many independent
device nodes.
I don't believe Solaris handles locking well if there is a hard link
created from a device inode. I saw this when some init script created a
hard rather than symbolic link to a device, admiditedly a few years ago.
BTW: any attempt to completely lock the device would prevent correct usability
even if there war no device aliasing.
Cooperative locking is needed in a way that allows and is based not on device
nodes but on real hardware targets.
Agree. Some form of locking on the actual device is desirable. I'm less
sure about cooperative locking, any method which fails if any one
program behaves badly is not scalable. Worse yet, bad behaviour by one
often shows as a failure in another, increasing time to solution.
A perfect solution also must address locking of partitions vs. locking
the entire device. I have a little paper somewhere around on just this
issue, and I referenced Plauger's "Law of Least Astonishment" therein. I
see the locking issues in Linux as partially political, but there are
certainly enouigh technical issues to go around.
--
bill davidsen <[EMAIL PROTECTED]>
CTO TMR Associates, Inc
Doing interesting things with small computers since 1979
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
Hi, Andy: > > > int grab_sg (int blkfd) me: > > Seems to work well for me and my two drives sr0 and sr1. Joerg: > Forget about this "method". It is known not to work reliably on Linux > and similar moethods will not work at all on other OS. This is a kind of emergency patch for a particular problem with (some ?) Linux kernels 2.4 . I am very thankful to Andy that he addresses old kernel 2.4 at all. His proposal will allow me to detect growisofs runs on a drive and to stay off. > The problem on Linux is device aliasing that results in many independent > device nodes. Yep. O_EXCL fails to provide the needed functionality under many circumstances. Above function will allow growisofs and libburn to meet at the same Linux /dev/sgN and there locking should work. (Same works fine between cdrecord and libburn.) > Cooperative locking is needed in a way that allows and is based > not on device nodes but on real hardware targets. I agree to this statement in general. (Above sg grabbing is not a widely usable solution.) My ideas are about a central dispatcher service which arbitrates locking requests. It could encapsulate nearly all OS dependency if we manage to find an OS independend communications method and make it understand all our permissible address formats (permissible on the particlar OS). If it can use a reliable OS provided locking mechanism: Very good. It should do. If not: it has to provide some own functionality for locking. On Linux i would implement it via scsi address parameters resp. major,minor device number pairs. Possibly configurable to teach it about hardware coincidences of distinct device drivers. Andy: > > > Have you seen resmgrd? me: > > I found this overview of 2006-09-29: > > http://forgeftp.novell.com//resmgr/web/README.html Joerg: > Last time, I did look at this software, it was full of conceptional bugs I agree that it does not qualify as solution to our problem. It offers locking, but in a way that i can top by my own experiments or by an old NFS-wide locking tool from early 1990s which marks lock files with an expiration date. (It is a bit fat, though.) > > Next i will try to find out wether HAL would be of more > > help. > > HAL is known to be a non-cooperative program that interrupts > CD/DVD writing. If its concept allows the type of locking we need, then one could try to negociate a less disturbing operation with the HAL developers. In our general usage scenario, HAL would only be one implementation of a lock mechanism, if ever. I still plan to evaluate wether it would roughly provide the needed gestures. But i do not plan to depend on it. > Sun is just working on a new vold implementation > for better GNOME support. Let us wait until this has been finished Got an URL for me to learn about its suitability for locking ? On Solaris our arbiter could be a frontend to vold if that provides the needed functionality. That would bring us in sync with other vold locks. In this case the aribter's only service would be to provide the standardized communcations protocol for the burn programs and a gateway to vold. me: > > I had a significant increase in DVD misburns as long as > > libburn opened and inquired the drive for its bus scan. Joerg: > Then you did something wrong. That's well possible. I will try to find out, eventually. But given my current lack of drive- and transport-level knowledge this may last a while. I am still suckling on the previously existing libburn code in that aspect. So i better try to stay off drives used by programs which know better. Andy: > > [Once again] given that we're talking about Linux, Joerg: > If you are talking about Linux only, we should stop this discussion. We got two levels of abstractions in this topic: 1) The particular 2.4 problem between growisofs and libburn. 2) The general problem that we are quite blind towards the activities of concurrent burn processes. Number 1) is on a good way thanks to Andy. (I'm just waiting for seeing how he integrates grab_sg() into growisofs.) Number 2) is still collecting divergent opinions and might need some time until a proposal emerges which fulfills the minimal needs without imposing too much load on the developers. > It would rather make sense to implement a > demonstrator on a OS that has a better design and then tell the Linux kernel > developers what to do. I am still in favor of doing our own thing in user space. One would eventually use OS facilities of sufficient quality for that, but also to have a generic implementation concept which is "very portable". The clients shall not see much diversity between OSes. They will need to get a unique process id and to perform the necessary communications with the arbiter. Communication transport candidates are plain files and TCP/IP for now. The convention about the file addresses resp. TCP/IP port might become system dependent. We should not dismiss the idea just because we do not have a good solution yet.
Re: Request for cooperation with all burn backends
[EMAIL PROTECTED] wrote: > "Very portable" almost alwas == "equally crippled on all platforms". > I'm so tired of 'very portable' software. This is not true and there are many software packages that demonstrate how portable software may work nicely. > > Have you seen resmgrd? Well, it didn't seem to catch up, but anyway... > > Why didn't it catch up? If you want my opinion, I think that all > > attempts to achieve the goal *purely* in user-land are doomed. 2.6 > > O_EXCL on block device appears to be sufficient for intended purpose and > > I personally would rather prefer it back-ported to 2.4 than some > > user-land facility. > > I had it explained to me in great detail how it was perfectly correct > to cripple the RedHat version of CDParanoia such that only one could > ever be running at a time-- because its autoscanning would always > block on advisory locks if any other program was using any other > CDROM/DVDROM device on the machine. Why? Because it was somehow > easier than fixing Nautilus's automount/autoplay tendencies to turn > every burn into a coaster. I am not sure whether I understand you correctly, but Redhat also did cripple cdrecord instead of fixing the newer programs that also like to access CD/DVD writers. > I like the idea of having a convention-- but I would argue against it > locking down devices against all access. A CDROM device is perfectly > capable of answering, eg, ' are you a cdrom?' whil e burning. I > realize that deciding what access is 'safe' is underspecified right > now. Correct! This is why I dislike the way of any attempts of locking I did see for now on Linux. I like to make cdrecord -scanbus work always as expected but be able to tell the user that a specific drive is currently in use. > > Note that it doesn't have to be /dev/sg. > > /dev/sg is dead. Long live SG_IO. This way of wording looks to me as if you did not understand the problem. SG_IO is used in /dev/sg for a ling time and the real problem with SG_IO is that _other_ related ioctls do not work correctly for all non-/dev/sg devices. > I will not be obeying O_EXCL in cdparanoia, at least in its current > form. However, I also want to make cdparanoia safe in the context of > cdrom devices ripping. Off topic: Monty, do you like to start working on cdparanoia again? >From my impression, you stopped doing so around y2000. This is why I did take the paranoia code 4.5 years ago, made it portable and a library that is now used by cdda2wav. Since then, I did fix several minor bugs in the code and you should have a look at the current libparanoia, to see what I did. Please also note that the basic CD reading stuff in cdparanoia is still similar to what cdda2wav did in 1997. Cdda2wav did evolve a lot since then and this is why cdda2wav may currently read a lot more CDs than cdparania currently does. This in special is true for "unCDs". BTW: I am planning to add C2-pointer support to libparanoia next year. partially back...: I am planning to change all users of libscg to scan the bus by default and in case that there is only a single CD/DVD device in the system, to use this device. This unfortunately is impossible on Linux since device aliasing was added and since not all SG_IO related ioctls return correct and complete information in all cases anymore Jörg -- EMail:[EMAIL PROTECTED] (home) Jörg Schilling D-13353 Berlin [EMAIL PROTECTED](uni) [EMAIL PROTECTED] (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
Andy Polyakov <[EMAIL PROTECTED]> wrote: > [Once again] given that we're talking about Linux, 2.6 O_EXCL serializes > only O_EXCL opens. So the program that only wishes to query unit can do > so, while program that wants to pull the data or modify page settings > should rather adhere to O_EXCL to serialize the access. If you are talking about Linux only, we should stop this discussion. Linux has many problems with it's kernel design that make it nearly impossible to implement a working method. It would rather make sense to implement a demonstrator on a OS that has a better design and then tell the Linux kernel developers what to do. Jörg -- EMail:[EMAIL PROTECTED] (home) Jörg Schilling D-13353 Berlin [EMAIL PROTECTED](uni) [EMAIL PROTECTED] (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
[EMAIL PROTECTED] wrote: > I had a significant increase in DVD misburns as long as > libburn opened and inquired the drive for its bus scan. Then you did something wrong. Jörg -- EMail:[EMAIL PROTECTED] (home) Jörg Schilling D-13353 Berlin [EMAIL PROTECTED](uni) [EMAIL PROTECTED] (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
[EMAIL PROTECTED] wrote: > Andy Polyakov wrote: > > Have you seen resmgrd? > > I found this overview of 2006-09-29: > http://forgeftp.novell.com//resmgr/web/README.html > which differs a bit from the impression i got from > the SuSE 9.0 man pages. > > One could execute program > resmgr lock /dev/xyz > and then open the device by normal means, possibly even > with O_EXCL. Last time, I did look at this software, it was full of conceptional bugs that tend to make the whole a much biger security risk than a clean solution in libscg or cdrecord. The program dishonored: - Problems from pattern matching - Problems from device aliasing - General attack scenarios on the basic idea. It looked to me as a program that has been written by a person that just discovered the possibility of sending open file descriptors via a socket connection although this method exists since 1979. I contacted the Author in hope to discuss the security problems in his concept but he was not interested in a discussion. Let me only comment on the non-obvious problems. > The sysadmin has to set up an appropriate resmgrd > configuration before this. Due to the pattern matching, this carries a high risk. > Further ideas: > > Next i will try to find out wether HAL would be of more > help. HAL is known to be a non-cooperative program that interrupts CD/DVD writing. Sun is just working on a new vold implementation for better GNOME support. Let us wait until this has been finished Jörg -- EMail:[EMAIL PROTECTED] (home) Jörg Schilling D-13353 Berlin [EMAIL PROTECTED](uni) [EMAIL PROTECTED] (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
[EMAIL PROTECTED] wrote:
> Hi,
>
> > int grab_sg (int blkfd)
>
> Seems to work well for me and my two drives sr0 and sr1.
>
> I threw out two of my three functions and made
> try_to_lock_linux_sg() using grab_sg() instead.
>
> The call in builtin_dd() has changed a bit:
>
> if (i == 3 && fd >= 0)
> try_to_lock_linux_sg(ioctl_device, fd);
>
> void try_to_lock_linux_sg(char *ioctl_device, int ioctl_dev_fd)
> {
> int fd_sg = -1;
>
> if ( ( ( strncmp (ioctl_device, "/dev/sr", 7) == 0
> && isdigit(ioctl_device[7]) ) ||
>( strncmp (ioctl_device, "/dev/scd", 8) == 0
> && isdigit(ioctl_device[8]) )
>) )
> fd_sg = grab_sg(ioctl_dev_fd);
> if(fd_sg == -2)
> fprintf (stderr,":-( unable to O_EXCL sg equivalent of %s: "
> "Other burn program active on drive ?\n",
> ioctl_device),
> exit (FATAL_START(errno));
> }
>
Forget about this "method". It is known not to work reliably on Linux
and similar moethods will not work at all on other OS.
The problem on Linux is device aliasing that results in many independent
device nodes.
BTW: any attempt to completely lock the device would prevent correct usability
even if there war no device aliasing.
Cooperative locking is needed in a way that allows and is based not on device
nodes but on real hardware targets.
Jörg
--
EMail:[EMAIL PROTECTED] (home) Jörg Schilling D-13353 Berlin
[EMAIL PROTECTED](uni)
[EMAIL PROTECTED] (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
Hi,
> int grab_sg (int blkfd)
Seems to work well for me and my two drives sr0 and sr1.
I threw out two of my three functions and made
try_to_lock_linux_sg() using grab_sg() instead.
The call in builtin_dd() has changed a bit:
if (i == 3 && fd >= 0)
try_to_lock_linux_sg(ioctl_device, fd);
I made a small change in grab_sg() in order to distinguish
a failed locking attempt from failure to get to that attempt.
#include
int grab_sg (int blkfd)
{
...
if (sgfd>=0)
{
...
}
else if (errno == EBUSY)
sgfd = -2;
...
}
void try_to_lock_linux_sg(char *ioctl_device, int ioctl_dev_fd)
{
int fd_sg = -1;
if ( ( ( strncmp (ioctl_device, "/dev/sr", 7) == 0
&& isdigit(ioctl_device[7]) ) ||
( strncmp (ioctl_device, "/dev/scd", 8) == 0
&& isdigit(ioctl_device[8]) )
) )
fd_sg = grab_sg(ioctl_dev_fd);
if(fd_sg == -2)
fprintf (stderr,":-( unable to O_EXCL sg equivalent of %s: "
"Other burn program active on drive ?\n",
ioctl_device),
exit (FATAL_START(errno));
}
I will now use this variation of growisofs 7.0 for my daily
backups. It will last a few days until i get opportunity
to test it on kernel 2.6 and /dev/hdX.
If my code is as intended then this should not even call
grab_sg(), and of course never abort.
Give me a note as soon as you decided for a final implementation
and i will test as good as i can.
Have a nice day :)
Thomas
PS: About -Z /dev/sr1=imagefile :
My ext3 is ok as source for image files.
My reiserfs is not. ":-( write failed: Invalid argument".
Mangling #ifdef O_DIRECT in growisofs.c enabled burning from
that reiserfs too.
Only good i watch other people's problems on this list. :))
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
Is it really a problem between recording programs?
It is especially a problem between growisofs and libburn
on my 2.4 system.
- growisofs burns of DVD+RW experience data damage in about
25 % of the cases after libburn did a bus scan on the burning drive.
- growisofs burns of DVD-RW stall libburn bus scan as soon as the
active drive is enumerated. Affected growisofs burns have about
50 % probability to be damaged.
...
I can offer a patch of 100 to 150 lines in growisofs.c
to achieve locking of /dev/srN or /dev/scdN via the
corresponding /dev/sgM.
Since i implemented that on my system i am free of trouble
between growisofs and libburn.
I published the test version (loudly declaring itself as
inofficial hack) on the [EMAIL PROTECTED] mailing list
in the hope of some test results ... then i wanted to approach
you.
No echo. Everybody is on 2.6 already.
See diff -puN at
http://scdbackup.sourceforge.net/dvd+rw-tools-7.0.tsA60930.diff.txt
If "sg scanning" is so intrusive, why are you using at all? Isn't there
a way to list sg binding, say though ... /proc? And look, there is! It's
even possible to see what kind each device is and limit search to type
5, a.k.a. CD-ROM, devices... In either case below is function which will
appear in next release. Test it. A.
int grab_sg (int blkfd)
{ struct { unsigned int dev_id,host_unique_id; } idlunblk,idlunsg;
FILE *fp;
int host_no,channel,lun,id,type,i,sgfd=-1;
char str[128];
struct stat sb;
if (ioctl (blkfd,SCSI_IOCTL_GET_IDLUN,&idlunblk) < 0) return -1;
if ((fp=fopen ("/proc/scsi/sg/devices","r")) == NULL) return -1;
for (i=0;i>=0;i++)
{ if (fgets (str,sizeof(str),fp) == NULL) break;
if (sscanf (str,"%d\t%d\t%d\t%d\t%d",
&host_no,&channel,&id,&lun,&type) != 5)
continue;
if (idlunblk.dev_id == ((id & 0xff) +
((lun & 0xff) << 8) +
((channel & 0xff) << 16) +
((host_no & 0xff) << 24)))
{ sprintf (str,"/dev/sg%d",i);
if (stat (str,&sb) < 0) break;
errno = ENOENT;
if (minor(sb.st_rdev) != i) break;
sgfd = open (str,O_RDWR|O_NONBLOCK|O_EXCL);
if (sgfd>=0)
{ if (ioctl (sgfd,SCSI_IOCTL_GET_IDLUN,&idlunsg) < 0 ||
idlunblk.dev_id != idlunsg.dev_id ||
idlunblk.host_unique_id != idlunsg.host_unique_id)
close (sgfd), sgfd = -1, errno = ENOENT;
}
break;
}
}
fclose (fp);
return sgfd;
}
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
On 10/11/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: I will not be obeying O_EXCL in cdparanoia, at least in its current form. However, I also want to make cdparanoia safe in the context of cdrom devices ripping. "burning" not "ripping". So sorry. Monty -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
Have you seen resmgrd? Keep in mind that I brought it as example of something that didn't quite catch up. And I did this to sort of emphasize the point that I don't believe in pure user-land solutions:-) I mean I kind of wanted to say "look, it kind of does the things you propose, yet, it did *not* really make it." And consequently "time is better spent adhering to block device even under 2.4, back-porting O_EXCL for block device[!], etc..." For us programmers it would be nice to know the resmgr protocol and thus to get rid of forking an external program. (A study of resmgr client code should help.) resmgr comes with libresmgr.so.1 with documented interface providing all the functionality exposed through command line directly to application. I far as I understand... I believe this scheme is extensible enough for other operating systems. As mentioned, the reality is rather that it's either "very portable" or "with explicit [and disjount] support for every system," because most of other system do provide means for serializing access and not using them is not an option. Cheers:-) A. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
On 10/10/06, Andy Polyakov <[EMAIL PROTECTED]> wrote: As for locking, or rather serializing access to [relevant] devices. "Very portable" customarily means support for different operating systems. But the trouble is that the other systems, other than Linux that is, might and already do have own ways to serialize the access. It might be impossible and/or simply inappropriate not to use these mechanisms. Doesn't this kind of doom all "very portable" attempts as simply unachievable? "Very portable" almost alwas == "equally crippled on all platforms". I'm so tired of 'very portable' software. Secondly. Why do you address back-end developers? Is it really a problem between recording programs? Isn't auto-mounting/-playing facilities interfering with ongoing recording *bigger* problem? [...] Have you seen resmgrd? Well, it didn't seem to catch up, but anyway... Why didn't it catch up? If you want my opinion, I think that all attempts to achieve the goal *purely* in user-land are doomed. 2.6 O_EXCL on block device appears to be sufficient for intended purpose and I personally would rather prefer it back-ported to 2.4 than some user-land facility. I had it explained to me in great detail how it was perfectly correct to cripple the RedHat version of CDParanoia such that only one could ever be running at a time-- because its autoscanning would always block on advisory locks if any other program was using any other CDROM/DVDROM device on the machine. Why? Because it was somehow easier than fixing Nautilus's automount/autoplay tendencies to turn every burn into a coaster. I like the idea of having a convention-- but I would argue against it locking down devices against all access. A CDROM device is perfectly capable of answering, eg, ' are you a cdrom?' whil e burning. I realize that deciding what access is 'safe' is underspecified right now. Note that it doesn't have to be /dev/sg. /dev/sg is dead. Long live SG_IO. To summarize. My vote goes for block device addressing, back-porting of O_EXCL to 2.4 and convincing auto-mounting/-playing developers to stick to it. I will not be obeying O_EXCL in cdparanoia, at least in its current form. However, I also want to make cdparanoia safe in the context of cdrom devices ripping. Monty -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
... But the trouble is that the other systems, other than Linux that is, might and already do have own ways to serialize the access. It might be impossible and/or simply inappropriate not to use these mechanisms. Doesn't this kind of doom all "very portable" attempts as simply unachievable? "Very portable" almost alwas == "equally crippled on all platforms". I'm so tired of 'very portable' software. That's kind of what I wanted to say. I mean it's either "very portable" or "with explicit support for every system." But as it seems we're talking about one particular OS, Linux in this case, after all, we should abstain for using "very portable" term in context of this discussion. At least I will:-) I like the idea of having a convention-- but I would argue against it locking down devices against all access. [Once again] given that we're talking about Linux, 2.6 O_EXCL serializes only O_EXCL opens. So the program that only wishes to query unit can do so, while program that wants to pull the data or modify page settings should rather adhere to O_EXCL to serialize the access. A CDROM device is perfectly capable of answering, eg, ' are you a cdrom?' whil e burning. As far as I interpret Thomas' experience he has problems with growisofs performing recording on /dev/cdrom being disrupted by scan attempt on /dev/sg. This apparently turns to be far from simple "are you a cdrom?" question... I realize that deciding what access is 'safe' is underspecified right now. ... because from personal experience I can tell that I've never had problems running dvd+rw-medianfo during ongoing growisofs recording. dvd+rw-medianfo does issue "are you a cdrom?" and a bunch of other queries. In other words unit *is* indeed capable to answer the limited amount of questions and the fact that Thomas reports problem indicates rather a kernel issue. To summarize. My vote goes for block device addressing, back-porting of O_EXCL to 2.4 and convincing auto-mounting/-playing developers to stick to it. I will not be obeying O_EXCL in cdparanoia, at least in its current form. However, I also want to make cdparanoia safe in the context of cdrom devices [ripping] burning. I don't quite understand why not adhere to O_EXCL when it actually comes to action. I mean you can perform initial open without O_EXCL so that you can pull TOC, offer options to user, etc, but just before you're about to modify page parameters and start pulling CDDA, you can as well claim O_EXCL... But once again, I don't see applications such as cdrskin or cdparanoia as biggest problem, because user still *controls* them, but [s]he does not control auto-mounting/-playing facilities. Cheers. A. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
Hi, [EMAIL PROTECTED] wrote: > "Very portable" almost alwas == "equally crippled on all platforms". > I'm so tired of 'very portable' software. A general locking facility would have to be system dependent in respect to device identification and naming. All other aspects, especially the generic locking aspects can be done in a portable way. TCP/IP and file system access are portable concepts. > I like the idea of having a convention-- but I would argue against it > locking down devices against all access. A CDROM device is perfectly > capable of answering, eg, ' are you a cdrom?' whil e burning. I > realize that deciding what access is 'safe' is underspecified right > now. A big problem is that it is unclear which ioctls and what SG_IO payload is safe and what possibly disturbs ongoing burns. I had a significant increase in DVD misburns as long as libburn opened and inquired the drive for its bus scan. Libburn's bus scan is an original design feature. Its suppression was a matter of heart to me and lead to my current involvement. Nevertheless the main usage model for libburn with interactive programs would include such a bus scan with each program run. Even multiple scans per run would make sense. So i need to make it safe against other libburn instances and want to make it mutually safe with the other burn programs. > > Note that it doesn't have to be /dev/sg. > > /dev/sg is dead. Long live SG_IO. What about SG_IO via /dev/sg ? :)) > I will not be obeying O_EXCL in cdparanoia, at least in its current > form. However, I also want to make cdparanoia safe in the context of > cdrom devices burning The charms of O_EXCL do not outweight its disadvantages, indeed. (But for now it works for me. That outweights much.) The general solution still seems to be a systemwide central locking service which manages tickets about devices without touching them. The burn programs may decide on their own which of their activities need such a lock ticket or what they deem to be absolutely safe towards interference with others. It is clear that all participants of the locking service need a common naming system (OS specific) and that the service must not be prone to stale locks or race conditions. Open questions: What is the cheapest and most widely available connection mechanism which allows the server to hand out lock tickets and to learn about the demise of a client ? How do our needs intersect with services provided by HAL, resmgr et al ? (For resmgr the answer is rather negative. by now.) Have a nice day :) Thomas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
Hi,
Andy Polyakov wrote:
> Have you seen resmgrd?
I found this overview of 2006-09-29:
http://forgeftp.novell.com//resmgr/web/README.html
which differs a bit from the impression i got from
the SuSE 9.0 man pages.
One could execute program
resmgr lock /dev/xyz
and then open the device by normal means, possibly even
with O_EXCL.
Obstacles:
I can see no guarantee that a lock automatically vanishes
when the obtaining process ends, or that a "stale" lock
is recognized with sufficient probability.
Like O_EXCL this seems to lock device file inodes and not
devices. Thus /dev/sr0 and /dev/sg0 might be not coordinated.
The sysadmin has to set up an appropriate resmgrd
configuration before this.
For us programmers it would be nice to know the resmgr
protocol and thus to get rid of forking an external
program. (A study of resmgr client code should help.)
Conclusion:
For our purposes resmgr is inferior to a working O_EXCL
lock. Except its possible acceptance with auto-mounters.
The current resmgr usage model seems to rather invite
the user to lock the device before growisofs, cdrecord,
wodim or cdrskin get started.
With a shell trap it should be possible to get this
end-user-safe. At least all experiments could be done on
shell wrapper level without changeing the code of the
participiating burn programs.
I now consider to offer this locking on the level of my
backup tool. Maybe it helps against automounters.
But with libburn i am quite unhappy about the outlook
on resmgrd. Neither can i assume to have it set up
properly on the majority of systems, nor does it really
ensure that i do not suffer from outdated locks.
A server with a persistent connection to the client
would be an appropriate implementation of our locking
demands. This server could revoke the lock as soon as
the connection to the client ends.
resmgrd is not designed to fulfill this task, i fear.
Further ideas:
Next i will try to find out wether HAL would be of more
help.
I consider to combine the naming scheme of my own lock
file experiment with a simple server rather than with
persistent files on disk.
The naming scheme hopefully coordinates devices and
not device file inodes. At least with SCSI- and ATAPI-
burners on Linux:
/dev/srN, dev/scdN, /dev/sgN get mapped to
"scsi_${host}_${channel}_${id}_${lun}"
other device file paths get mapped to
"rdev_${major}_${minor}
with the device numbers obtained from struct stat.st_rdev .
I believe this scheme is extensible enough for other
operating systems.
Have a nice day :)
Thomas
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
Hi,
> > Is it possible we define a common locking mechanism for drives
> > which does not depend on hardly documented Linux O_EXCL ?
> > Something simple and very portable would be needed.
>
> As for locking, or rather serializing access to [relevant] devices.
> "Very portable" customarily means support for different operating
> systems.
I tested growisofs with an advisory locking system based
on lock file naming conventions, unique owner ids and
synchronization by waiting and verifying survival of owner
ids.
In principle it works - if there weren't all the exit()
in growisofs.c which do not play well with the need to remove
a lock file when work is done.
> Doesn't this kind of doom all "very portable" attempts as
> simply unachievable?
It turned out to be quite obtrusive on growisofs code,
indeed. It gave up that idea for now. Especially since
Joerg seems to be not convinced of it either.
> Secondly. Why do you address back-end developers?
Because i joined that club as junior member by accident.
> Is it really a problem between recording programs?
It is especially a problem between growisofs and libburn
on my 2.4 system.
- growisofs burns of DVD+RW experience data damage in about
25 % of the cases after libburn did a bus scan on the burning drive.
- growisofs burns of DVD-RW stall libburn bus scan as soon as the
active drive is enumerated. Affected growisofs burns have about
50 % probability to be damaged.
> Isn't auto-mounting/-playing facilities
> interfering with ongoing recording *bigger* problem? So if you want to
> make the noble attempt and spend some time convincing developers time is
> better spent talking to that developers. IMHO that is:-)
Well, after i proved my Web 2.0 skills by forging a lock
alliance of cdrtools, dvd+rw-tools, cdrkit and libburn
i will probably approach the auto-community for joining
our convention. :))
> I think that all
> attempts to achieve the goal *purely* in user-land are doomed. 2.6
> O_EXCL on block device appears to be sufficient for intended purpose and
> I personally would rather prefer it back-ported to 2.4 than some
> user-land facility.
O_EXCL locking is promised for 2.4 in the Linux sg HOWTO :
http://www.tldp.org/HOWTO/SCSI-Generic-HOWTO/open.html
I can offer a patch of 100 to 150 lines in growisofs.c
to achieve locking of /dev/srN or /dev/scdN via the
corresponding /dev/sgM.
Since i implemented that on my system i am free of trouble
between growisofs and libburn.
I published the test version (loudly declaring itself as
inofficial hack) on the [EMAIL PROTECTED] mailing list
in the hope of some test results ... then i wanted to approach
you.
No echo. Everybody is on 2.6 already.
See diff -puN at
http://scdbackup.sourceforge.net/dvd+rw-tools-7.0.tsA60930.diff.txt
The complete package for testing
http://scdbackup.sourceforge.net/dvd+rw-tools-7.0.tsA60930.tar.gz
My own tests on kernel 2.4.21 showed no problems.
> Note that it doesn't have to be /dev/sg. Even though different, both 2.4
> and 2.6 provide interfaces for passing SCSI commands from user-land
> through /dev/cdrom. Well, I have to admit I've never tested
> CDROM_SEND_PACKET interface for non-data recordings... Did libburn
> developers do?
libburn is using /dev/sgN with N out of {0..31} and /dev/hdX with
X out of {'a'..'z'} . Theoretically they could be mixed but in
practice /dev/sgN is for 2.4 with ide-scsi and /dev/hdX is for 2.6.
Both device families are used via the same SG_IO ioctl calls.
I did not thoroughly explore the transport and drive
command level yet. Not to speak of the more artful matters of
disc, session, and track semantics. (Most need for work is
currently in the usability aspects of the library.)
> To summarize. My vote goes for block device addressing,
We both use /dev/hdX on Linux 2.6 and locking seems to work fine.
(Allthough i believe O_EXCL locking should happen earlier
in growisofs.)
> back-porting of O_EXCL to 2.4
Please consider above locking proposal via /dev/sgM.
It should keep away trouble with libburn and hopefully with
cdrecord and wodim. libburn and cdrecord do respect their mutual
locks on kernel 2.4. I tested that several times by accident.
> and convincing auto-mounting/-playing developers to stick
> to it.
For now i would be glad to see Linux 2.4 mutual exclusion working
as it does on my own system now. The eventual abort of growisofs
comes a bit late but elsewise it seems to be ok.
> Note that there
> are left-overs from experiment with resmsg in dvd+rw-tools code [look
> for dev_open], so in case of alternative outcome,
I will have a look at resmgrd (and its relation to HAL
and D-Bus). It could be a model example of a locking service
and one could try to make an implementation on base of it.
Such a proposal could be acceptable to automounters, too.
Have a nice day :)
Thomas
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
Thomas, this is a request towards all developers of burn backends. Is it possible we define a common locking mechanism for drives which does not depend on hardly documented Linux O_EXCL ? Something simple and very portable would be needed. I don't quite understand couple of things. As for locking, or rather serializing access to [relevant] devices. "Very portable" customarily means support for different operating systems. But the trouble is that the other systems, other than Linux that is, might and already do have own ways to serialize the access. It might be impossible and/or simply inappropriate not to use these mechanisms. Doesn't this kind of doom all "very portable" attempts as simply unachievable? Secondly. Why do you address back-end developers? Is it really a problem between recording programs? Isn't auto-mounting/-playing facilities interfering with ongoing recording *bigger* problem? So if you want to make the noble attempt and spend some time convincing developers time is better spent talking to that developers. IMHO that is:-) Advisory locking would be sufficient, i think. So shall we assume that we're actually talking about Linux and Linux only? At least the rest of *this* message implies Linux alone. Please keep it in mind... Have you seen resmgrd? Well, it didn't seem to catch up, but anyway... Why didn't it catch up? If you want my opinion, I think that all attempts to achieve the goal *purely* in user-land are doomed. 2.6 O_EXCL on block device appears to be sufficient for intended purpose and I personally would rather prefer it back-ported to 2.4 than some user-land facility. My motivation currently is about growisofs and /dev/srN : I had the ito open(2) O_EXCL those /dev/srN and /dev/scdM which have the same SCSI address as the /dev/sgK used by libburn. Note that it doesn't have to be /dev/sg. Even though different, both 2.4 and 2.6 provide interfaces for passing SCSI commands from user-land through /dev/cdrom. Well, I have to admit I've never tested CDROM_SEND_PACKET interface for non-data recordings... Did libburn developers do? I definitely would (if I were to develop non-data cd recording program:-). Addressing by block device is really more natural, because you share same view as auto-mounting/-playing and hot-plugging facilities and even initial OS installation procedure, and consequently even user view. Not to mention that that's how it works in most OSes anyway. To summarize. My vote goes for block device addressing, back-porting of O_EXCL to 2.4 and convincing auto-mounting/-playing developers to stick to it. My vote naturally doesn't make it just happen, but unfortunately I don't have time [or energy] to suggest alternative. Note that there are left-overs from experiment with resmsg in dvd+rw-tools code [look for dev_open], so in case of alternative outcome, I'd prefer to see something similar to that, i.e. link to an .so and if available modify behavior at run time. Cheers. A. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
Hi,
> > which does not depend on hardly documented Linux O_EXCL ?
> > Something simple and very portable would be needed.
>
> - Linux has a silly problem caused by device aliasing:
Yeah. But *that* i could circumvent.
The drivers know the SCSI quatruple (Host,Channel,Id,Lun)
and by that i can determine the matching sr for my sg.
Now with that info in hand i would only need a central
instance which tells me wether some other burn process
works on a thing with the same info.
That's what O_EXCL fails to do on sr with my oldish kernel.
I want a new more reliable central instance for that
in user space. A convention of burn backends so they
don't spoil each others burn.
Of course that thing will not save us from automounters
as long as they do not join our convention. But how
should they - as long as there is no convention ?
> - Record Locking only works on files.
I would anyway prefer to have no system locking mechanism
involved but only vanilla file operations.
link(2) and rename(2) can make a file appear quite atomicly.
The only race condition would be the asynchronizity between
lookup of not-yet-existent lock file and creation of the own
lock file.
For that i got a solution from the mid 1980s.
A lock is obtained this way:
- lookup the lock file (have lost locking contest if it exists)
- create lock file with on id stamp as content
- wait three seconds
- lookup lock file and check wether own id is still there
(if not there any more: have lost locking contest)
The 3 seconds were sufficient for NFS on a 10 Mbit LAN
with 50 workstations of 25 MHz each. We could wait less
than a second for that.
Lock files have names of form:
/tmp/burner_backend_lock_${info_code}
Each burner device leads to a unique $info_code.
> If Linux would make [...]
Linux won't make. We need to make.
Linux is there and will be there.
The alternative is not Solaris but Vista, i fear.
HAL could probably do. Its server would be ideal.
But we should strive for something much more simple.
My porposal is:
- Above file synchronization protocol which will be
very portable.
- A info_code convention which for now covers:
- SCSI on Linux (and probably on most other systems)
${Host}_${Channel}_${Id}_${Lun}
All four numbers as plain decimals with no leading 0s.
- ATAPI on Linux (and on other systems ?)
${stat.st_rdev}
As hexadecimal number but without trailing "0x".
To my knowledge there is only one (Major,Minor) device
number pair per ATAPI burner in Linux.
- transports of your choice on systems of your choice
(currently i only have to deal with above two)
Thus if cdrecord wants to access burner "0,1,0" it looks up
${Host}_${Channel}_${Id}_${Lun} (how is channel+id related
to cdrecord target, btw ?) and attempts to lock via
/tmp/burner_backend_lock_0_0_1_0
growisofs using /dev/scd1 would obtain the same quatruple
by SCSI_IOCTL_GET_IDLUN and compute the same file address.
The same for cdrskin using /dev/sg1 .
With ATAPI, growisofs and cdrskin use both "/dev/hdc"
and would come by a simple stat(2) to a file address like
/tmp/burner_backend_lock_300
cdrecord would have to find a way to come from "ATA:1,0,0"
to "/dev/hdc" resp. the device number.
If you know a characteristic info of a ATAPI burner that
is easier to obtain for cdrecord: make a proposal.
> What you mentioned here is highly Linux specific and does nto help
> to find a OS independent way.
The info conventions for the file names can become OS specific,
i confess. Let's see how many OSes have no Host,Channel,Id,Lun
for SCSI and no unique device number for ATAPI burners.
Have a nice day :)
Thomas
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Request for cooperation with all burn backends
[EMAIL PROTECTED] wrote: > this is a request towards all developers of burn backends. > > Is it possible we define a common locking mechanism for drives > which does not depend on hardly documented Linux O_EXCL ? This way of locking is known not to work. I believe it has been introduced by Redhat without and discussion and all Linux vendors who introduce it get into similar problems. > Something simple and very portable would be needed. > Advisory locking would be sufficient, i think. It is unfortunately not for two reasons: - Linux has a silly problem caused by device aliasing: You may open the same device under various different device drivers that avoid knowing from each other in oder to cause the biggest impact on clean aplications like cdrecord. - Record Locking only works on files. > It should cover the problem of one SCSI address having several > "official" device files. Thus i would propose a lock file which > contains characteristic unique info about the burner. That would > also help with device aliases that are no symbolic links. If Linux would make the internal device instance information available to user space programs, it would be possible to properly join all device aliases even on Linux using code inside libscg. As long as Linux does not offer enough internal information for all kind of SCSI targets to user space programs (mainly caused by the unwillingness of Jens Axboe) in order to impact cdrtools, this is impossible. > The mechanism on each OS would be allowed to be OS-specific > since it is supposed to be used only within one system at a time. > It would not have to be totally safe against race conditions. > It should suffice to write the own process id into the locking file, > wait a due time, and check wether the own id survived. If all applications use the same SCSI transport library and all OS do not have device aliasing, this is possible. [Linux specific part removed] What you mentioned here is highly Linux specific and does nto help to find a OS independent way. Unfortunately, it is impossible to have a fact based discussion with the Linux kernel people on this topic, so starting such a discussion in the vicinity of Linux does not look like a good idea. Note that cdrecors currently has no problems on Solaris because it waits for 10 seconds before it's operation starts. For this reason, the minimal gracetime in cdrecord is 3 seconds because this is needed for vold to notice that there is nothing of interest for vold in the drive (a blank medium). As Solaris is currently trying to support gnome by modifying vold in a way that gives better interfaces to gnome and as people on Solaris do care about cdrecord, let us wait to the end of this year and let us solve the problem on the future implementations on Solaris first. Jörg -- EMail:[EMAIL PROTECTED] (home) Jörg Schilling D-13353 Berlin [EMAIL PROTECTED](uni) [EMAIL PROTECTED] (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
