Creating lock file for compilers that don't support -c -o

[Albert Chin]

If a compiler doesn't support -c -o, libtool creates a lock file by
hard linking the libtool binary with the lock file. This is a problem
if the libtool binary is on a different file system than the lock
file. Why don't we use a symbolic link?

-- 
albert chin ([EMAIL PROTECTED])


___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Bob Friesenhahn]

On Mon, 25 Aug 2003, Albert Chin wrote:

> If a compiler doesn't support -c -o, libtool creates a lock file by
> hard linking the libtool binary with the lock file. This is a problem
> if the libtool binary is on a different file system than the lock
> file. Why don't we use a symbolic link?

Creating a hard link is an atomic operation whereas a symbolic link is
not so it can't be used as a locking mechanism.

Maybe there is some other hard-link which can be used instead?

Bob
==
Bob Friesenhahn
[EMAIL PROTECTED]
http://www.simplesystems.org/users/bfriesen



___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Paul Jarc]

Bob Friesenhahn <[EMAIL PROTECTED]> wrote:
> Creating a hard link is an atomic operation whereas a symbolic link is
> not

How so?


paul


___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Bob Friesenhahn]

On Mon, 25 Aug 2003, Paul Jarc wrote:

> Bob Friesenhahn <[EMAIL PROTECTED]> wrote:
> > Creating a hard link is an atomic operation whereas a symbolic link is
> > not
>
> How so?

It has to do with the complexity of the operation, particularly if a
network is involved.  Creating a hard link does not create a new file
(allocate a new inode), it simply updates a directory table to
reference an existing file.  This requires a single network request.
Most importantly, it is documented to be atomic.  Creating a symbolic
link requires testing for an existing file, and then (if the file does
not exist) creating a new file, and a directory entry to reference it.
This requires multiple network transactions with an opportunity for
race-conditions.

NAME
 link - make a hard file link

LIBRARY
 Standard C Library (libc, -lc)

SYNOPSIS
 #include 

 int
 link(const char *name1, const char *name2);

DESCRIPTION
 The link() function call atomically creates the specified directory entry
 (hard link) name2 with the attributes of the underlying object pointed at
 by name1. ...



NAME
 symlink - make symbolic link to a file

LIBRARY
 Standard C Library (libc, -lc)

SYNOPSIS
 #include 

 int
 symlink(const char *name1, const char *name2);

DESCRIPTION
 A symbolic link name2 is created to name1 (name2 is the name of
the file
 created, name1 is the string used in creating the symbolic link).
Either
 name may be an arbitrary path name; the files need not be on the
same
 file system.

==
Bob Friesenhahn
[EMAIL PROTECTED]
http://www.simplesystems.org/users/bfriesen



___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Paul Jarc]

Bob Friesenhahn <[EMAIL PROTECTED]> wrote:
> Creating a symbolic link requires testing for an existing file, and
> then (if the file does not exist) creating a new file, and a
> directory entry to reference it.  This requires multiple network
> transactions with an opportunity for race-conditions.

open() with O_CREAT|O_EXCL also creates a new file, yet that does not
subject it to race conditions.  symlink() has equivalent semantics to
O_CREAT|O_EXCL.  It may be that some network filesystems fail to
preserve the atomicity; I wouldn't know.  But at least for local
filesystems, I don't see any problems with symlinks.


paul


___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Robert Collins]

On Tue, 2003-08-26 at 03:58, Bob Friesenhahn wrote:
> On Mon, 25 Aug 2003, Albert Chin wrote:
> Maybe there is some other hard-link which can be used instead?

Renames, and directory create/delete are atomic AFAIK.

The arch/tla code uses file system semantics for locking - some study of
the code would likely give ideas.

Rob

-- 
GPG key available at: .



___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Bob Friesenhahn]

On Tue, 26 Aug 2003, Robert Collins wrote:

> On Tue, 2003-08-26 at 03:58, Bob Friesenhahn wrote:
> > On Mon, 25 Aug 2003, Albert Chin wrote:
> > Maybe there is some other hard-link which can be used instead?
>
> Renames, and directory create/delete are atomic AFAIK.
>
> The arch/tla code uses file system semantics for locking - some study of
> the code would likely give ideas.

The problem is ultimately how to use something in the filesystem where
object files are written rather than the filesystem where the source
files happen to be located.  The alternative is to figure out how to
use files/directories in the system's temporary file directory to do
the locking.

Bob
==
Bob Friesenhahn
[EMAIL PROTECTED]
http://www.simplesystems.org/users/bfriesen




___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Albert Chin]

On Mon, Aug 25, 2003 at 02:06:06PM -0500, Bob Friesenhahn wrote:
> On Mon, 25 Aug 2003, Paul Jarc wrote:
> 
> > Bob Friesenhahn <[EMAIL PROTECTED]> wrote:
> > > Creating a hard link is an atomic operation whereas a symbolic link is
> > > not
> >
> > How so?
> 
> It has to do with the complexity of the operation, particularly if a
> network is involved.  Creating a hard link does not create a new file
> (allocate a new inode), it simply updates a directory table to
> reference an existing file.  This requires a single network request.
> Most importantly, it is documented to be atomic.  Creating a symbolic
> link requires testing for an existing file, and then (if the file does
> not exist) creating a new file, and a directory entry to reference it.
> This requires multiple network transactions with an opportunity for
> race-conditions.
>
> NAME
>  link - make a hard file link
> 
> LIBRARY
>  Standard C Library (libc, -lc)
> 
> SYNOPSIS
>  #include 
> 
>  int
>  link(const char *name1, const char *name2);
> 
> DESCRIPTION
>  The link() function call atomically creates the specified directory entry
>  (hard link) name2 with the attributes of the underlying object pointed at
>  by name1. ...
>
> ...
>

I'd rather see a link to the POSIX standard defining link as atomic.

-- 
albert chin ([EMAIL PROTECTED])


___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Albert Chin]

On Mon, Aug 25, 2003 at 12:58:07PM -0500, Bob Friesenhahn wrote:
> On Mon, 25 Aug 2003, Albert Chin wrote:
> 
> > If a compiler doesn't support -c -o, libtool creates a lock file by
> > hard linking the libtool binary with the lock file. This is a problem
> > if the libtool binary is on a different file system than the lock
> > file. Why don't we use a symbolic link?
> 
> Creating a hard link is an atomic operation whereas a symbolic link is
> not so it can't be used as a locking mechanism.
> 
> Maybe there is some other hard-link which can be used instead?

Why not use $srcfile and "$srcfile.lock" as the lock file? So, rather
than:
  until $run ln "$0" "$lockfile" 2>/dev/null; do
we would have:
  lockfile="$srcfile.lock"
  ...
  until $run ln "$srcfile" "$lockfile" 2>/dev/null; do

-- 
albert chin ([EMAIL PROTECTED])


___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Robert Collins]

On Tue, 2003-08-26 at 12:58, Albert Chin wrote:


> Why not use $srcfile and "$srcfile.lock" as the lock file? So, rather
> than:

They may be on different fs's.

Rob

-- 
GPG key available at: .



___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Sascha Schumann]

> I'd rather see a link to the POSIX standard defining link as atomic.

IEEE Std 1003.1-2003

http://www.opengroup.org/onlinepubs/007904975/functions/link.html

"The link() function shall atomically create a new link for
the existing file and the link count of the file shall be
incremented by one."

- Sascha


___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Albert Chin]

On Tue, Aug 26, 2003 at 01:49:10PM +1000, Robert Collins wrote:
> On Tue, 2003-08-26 at 12:58, Albert Chin wrote:
> 
> > Why not use $srcfile and "$srcfile.lock" as the lock file? So, rather
> > than:
> 
> They may be on different fs's.

How is that possible? If we use $srcfile.lock as the lock file it is
guaranteed to be on the same filesystem as $srcfile.

-- 
albert chin ([EMAIL PROTECTED])


___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Earnie Boyd]

Paul Jarc wrote:
Bob Friesenhahn <[EMAIL PROTECTED]> wrote:

Creating a symbolic link requires testing for an existing file, and
then (if the file does not exist) creating a new file, and a
directory entry to reference it.  This requires multiple network
transactions with an opportunity for race-conditions.


open() with O_CREAT|O_EXCL also creates a new file, yet that does not
subject it to race conditions.  symlink() has equivalent semantics to
O_CREAT|O_EXCL.  It may be that some network filesystems fail to
preserve the atomicity; I wouldn't know.  But at least for local
filesystems, I don't see any problems with symlinks.
Problem with symlinks, and hardlinks for that matter, is portability. 
Not all systems support them.

Earnie.



___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Albert Chin]

On Tue, Aug 26, 2003 at 07:13:34AM -0400, Earnie Boyd wrote:
> Paul Jarc wrote:
> >Bob Friesenhahn <[EMAIL PROTECTED]> wrote:
> >
> >>Creating a symbolic link requires testing for an existing file, and
> >>then (if the file does not exist) creating a new file, and a
> >>directory entry to reference it.  This requires multiple network
> >>transactions with an opportunity for race-conditions.
> >
> >
> >open() with O_CREAT|O_EXCL also creates a new file, yet that does not
> >subject it to race conditions.  symlink() has equivalent semantics to
> >O_CREAT|O_EXCL.  It may be that some network filesystems fail to
> >preserve the atomicity; I wouldn't know.  But at least for local
> >filesystems, I don't see any problems with symlinks.
> >
> 
> Problem with symlinks, and hardlinks for that matter, is portability. 
> Not all systems support them.

We only need to worry about it on systems without -c -o support.

-- 
albert chin ([EMAIL PROTECTED])


___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Bob Friesenhahn]

On Tue, 26 Aug 2003, Albert Chin wrote:

> On Tue, Aug 26, 2003 at 01:49:10PM +1000, Robert Collins wrote:
> > On Tue, 2003-08-26 at 12:58, Albert Chin wrote:
> >
> > > Why not use $srcfile and "$srcfile.lock" as the lock file? So, rather
> > > than:
> >
> > They may be on different fs's.
>
> How is that possible? If we use $srcfile.lock as the lock file it is
> guaranteed to be on the same filesystem as $srcfile.

The source files may be located on a read-only filesystem.  They may
also be shared (via NFS or SMB) by simultaneous builds on a number of
other hosts.  We can't/shouldn't update the source file directory or
the directory where libtool resides.  The only place we are allowed to
write is in the object directory, or the temporary file directory.

Bob
==
Bob Friesenhahn
[EMAIL PROTECTED]
http://www.simplesystems.org/users/bfriesen



___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Bob Friesenhahn]

On Tue, 26 Aug 2003, Robert Collins wrote:

> On Tue, 2003-08-26 at 12:58, Albert Chin wrote:
>
>
> > Why not use $srcfile and "$srcfile.lock" as the lock file? So, rather
> > than:
>
> They may be on different fs's.

Right.  It is possible for libtool, the source files, and the object
files, to reside on three different file systems.

Bob
==
Bob Friesenhahn
[EMAIL PROTECTED]
http://www.simplesystems.org/users/bfriesen



___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Albert Chin]

On Tue, Aug 26, 2003 at 10:56:47AM -0500, Bob Friesenhahn wrote:
> On Tue, 26 Aug 2003, Robert Collins wrote:
> 
> > On Tue, 2003-08-26 at 12:58, Albert Chin wrote:
> >
> >
> > > Why not use $srcfile and "$srcfile.lock" as the lock file? So, rather
> > > than:
> >
> > They may be on different fs's.
> 
> Right.  It is possible for libtool, the source files, and the object
> files, to reside on three different file systems.

So it looks like we have no alternative but to create a small program
at runtime that creates a file with O_CREAT|O_EXCL?

-- 
albert chin ([EMAIL PROTECTED])


___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool

Re: Creating lock file for compilers that don't support -c -o

[Bob Friesenhahn]

On Tue, 26 Aug 2003, Albert Chin wrote:

> On Tue, Aug 26, 2003 at 10:56:47AM -0500, Bob Friesenhahn wrote:
> > On Tue, 26 Aug 2003, Robert Collins wrote:
> >
> > > On Tue, 2003-08-26 at 12:58, Albert Chin wrote:
> > >
> > >
> > > > Why not use $srcfile and "$srcfile.lock" as the lock file? So, rather
> > > > than:
> > >
> > > They may be on different fs's.
> >
> > Right.  It is possible for libtool, the source files, and the object
> > files, to reside on three different file systems.
>
> So it looks like we have no alternative but to create a small program
> at runtime that creates a file with O_CREAT|O_EXCL?

This is not any better than creating a file in the object directory
when libtool is first run which can be used for link-based locking.
The user (i.e.  Makefile) would be responsible for removing the file
(if the user chooses to).  If the small program is created and removed
every time libtool is run, the performance would be terrible.

Bob
==
Bob Friesenhahn
[EMAIL PROTECTED]
http://www.simplesystems.org/users/bfriesen



___
Libtool mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/libtool