Re: [Devel] locking on SunOS
A little late, but applied. Do test and let me know. On Dec 09, 2009, at 06:16, Ian Donaldson wrote: > >Thanks. > > > Would you kindly send a unified diff (preferably against CVS)? > > I've applied my patch with adjustments for differences in the CVS > version, however I'm unable to test it; autoconf falls over > with this: > -- > configure.ac:27: error: possibly undefined macro: AM_INIT_AUTOMAKE > If this token and others are legitimate, please use m4_pattern_allow. > See the Autoconf documentation. > configure.ac:29: error: possibly undefined macro: AM_MAINTAINER_MODE > configure.ac:34: error: possibly undefined macro: AC_PROG_LIBTOOL > configure.ac:324: error: possibly undefined macro: AM_CONDITIONAL > -- > both on Fedora FC11 (autoconf-2.63.2) and on Solaris 9 (with autoconf-2.65) > > I've spent a bit of time trying to figure this out to no avail; > I'm not an expert in any way with autoconf. > > Lots of similar complaints on google, with nothing obvious. > > I'm attaching the patch anyway; can somebody else see if its ok perhaps > with an older autoconf. > > Ian D > ___ > Devel mailing list > Devel@mbuni.org > http://lists.mbuni.org/mailman/listinfo/devel ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel
Re: [Devel] locking on SunOS
Thanks. Would you kindly send a unified diff (preferably against CVS)? On Dec 08, 2009, at 08:47, Ian Donaldson wrote: > This topic has gone quiet and I've not seen any of the patches > appear in CVS; its been over a year! > > I've just applied the patches and some tweaks to the 1.4.0 base > and they seem to work ok. I also tweaked some errno > handling in the patch and fixed a bug whereby a failed fstat > would return no result to a lock attempt. > > The #ifdef SunOS probably should be replaced with #ifdef HAS_BROKEN_FLOCK > in most cases, as there are probably other SVR4 systems out > there that could benefit from this multi-threaded locking support. > > I enclose my patch relative to the 1.4.0 base (with more context > than the original patch as patching from that gave some errors). > > I also enclose a fix for Solaris and any other SVR4 based system > to support logfile rotation using SIGHUP... the original > code would die on the 2nd SIGHUP due to the signal handler > not being reinstated. Using sigaction instead on such systems. > (#ifdef SA_RESTART) > > I also tweaked configure for Solaris 9 x86, as it only supported > recognition of sparc with gcc. > > Ian D > ___ > Devel mailing list > Devel@mbuni.org > http://lists.mbuni.org/mailman/listinfo/devel ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel
Re: [Devel] locking on SunOS
Unfortunately, the same true for the version of MBuni I have. I applied the patch (with a few changes) to my version (roughly a 1.4) and tested with that. The patch against the CVS version is attached. This isn't actually tested (with CVS version) though. If I get the time, I will try get into the office and try it out on a Sun box - if not - I'll get around to it monday. Even if there might still be issues on Sun, it doesn't change anything on other systems though. Note, that I haven't used unlock_and_close everywhere, only the places where I think it is necessary. Using it everywhere might actually be destructive to use it everywhere since, fdopen is sometimes called on an already open (and locked) file descriptor and then closed again, where the original file descriptor is still open and should not yet be unlocked. Regards, Christian Attached is a patch against latest cvs. On Fri, Sep 19, 2008 at 5:40 AM, Jason Pollock < [EMAIL PROTECTED]> wrote: > P. A. Bagyenda wrote: > >> Jason, >> >> Thanks. It would be nice to see a full patch against latest CVS for this. >> > > As I've said, my version of Mbuni is drastically different from CVS. :) > I'll see what I can do. > > > On Sep 19, 2008, at 02:16, Jason Pollock wrote: >> >> P. A. Bagyenda wrote: >>> Thanks for the review. If you would, kindly share a non-destructive patch for further discussion. >>> >>> Here are the changes that I applied based on the comments. >>> >>> >>> Index: mms_util.c >>> === >>> RCS file: /home/cvs/MBUNI/mmlib/mms_util.c,v >>> retrieving revision 1.4 >>> diff -a -u -r1.4 mms_util.c >>> --- mms_util.c10 Sep 2008 04:04:09 -1.4 >>> +++ mms_util.c18 Sep 2008 23:15:09 - >>> @@ -22,9 +22,9 @@ >>> #include >>> #include >>> #include >>> -#include >>> #endif >>> >>> +#include >>> #include >>> #include >>> #include >>> @@ -929,9 +929,8 @@ >>>key.dev = buf.st_dev; >>> >>>release_file_lock(fd, &key); >>> -#else >>> -return fclose(fp); >>> #endif >>> +return fclose(fp); >>> } >>> >>> /* Compare a file_lock(lhs) to the file_key(rhs) >>> ___ >>> Devel mailing list >>> Devel@mbuni.org >>> http://lists.mbuni.org/mailman/listinfo/devel >>> >> >> >> > ___ > Devel mailing list > Devel@mbuni.org > http://lists.mbuni.org/mailman/listinfo/devel > sunlock_cvs_patch.diff Description: Binary data ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel
Re: [Devel] locking on SunOS
P. A. Bagyenda wrote: Jason, Thanks. It would be nice to see a full patch against latest CVS for this. As I've said, my version of Mbuni is drastically different from CVS. :) I'll see what I can do. On Sep 19, 2008, at 02:16, Jason Pollock wrote: P. A. Bagyenda wrote: Thanks for the review. If you would, kindly share a non-destructive patch for further discussion. Here are the changes that I applied based on the comments. Index: mms_util.c === RCS file: /home/cvs/MBUNI/mmlib/mms_util.c,v retrieving revision 1.4 diff -a -u -r1.4 mms_util.c --- mms_util.c10 Sep 2008 04:04:09 -1.4 +++ mms_util.c18 Sep 2008 23:15:09 - @@ -22,9 +22,9 @@ #include #include #include -#include #endif +#include #include #include #include @@ -929,9 +929,8 @@ key.dev = buf.st_dev; release_file_lock(fd, &key); -#else -return fclose(fp); #endif +return fclose(fp); } /* Compare a file_lock(lhs) to the file_key(rhs) ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel
Re: [Devel] locking on SunOS
Jason, Thanks. It would be nice to see a full patch against latest CVS for this. On Sep 19, 2008, at 02:16, Jason Pollock wrote: P. A. Bagyenda wrote: Thanks for the review. If you would, kindly share a non-destructive patch for further discussion. Here are the changes that I applied based on the comments. Index: mms_util.c === RCS file: /home/cvs/MBUNI/mmlib/mms_util.c,v retrieving revision 1.4 diff -a -u -r1.4 mms_util.c --- mms_util.c 10 Sep 2008 04:04:09 - 1.4 +++ mms_util.c 18 Sep 2008 23:15:09 - @@ -22,9 +22,9 @@ #include #include #include -#include #endif +#include #include #include #include @@ -929,9 +929,8 @@ key.dev = buf.st_dev; release_file_lock(fd, &key); -#else -return fclose(fp); #endif +return fclose(fp); } /* Compare a file_lock(lhs) to the file_key(rhs) ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel
Re: [Devel] locking on SunOS
P. A. Bagyenda wrote: Thanks for the review. If you would, kindly share a non-destructive patch for further discussion. Here are the changes that I applied based on the comments. Index: mms_util.c === RCS file: /home/cvs/MBUNI/mmlib/mms_util.c,v retrieving revision 1.4 diff -a -u -r1.4 mms_util.c --- mms_util.c 10 Sep 2008 04:04:09 - 1.4 +++ mms_util.c 18 Sep 2008 23:15:09 - @@ -22,9 +22,9 @@ #include #include #include -#include #endif +#include #include #include #include @@ -929,9 +929,8 @@ key.dev = buf.st_dev; release_file_lock(fd, &key); -#else -return fclose(fp); #endif +return fclose(fp); } /* Compare a file_lock(lhs) to the file_key(rhs) ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel
Re: [Devel] locking on SunOS
Christian Theil Have wrote: I've been testing the patch and it seems to work out allright :-) A few notes; #include must be outside #ifdef SunOS - other systems needs it too in unlock_and_fclose(FILE *fp) fclose is only called in #else, I suppose that should have been like for unlock_and_close or we'll run out of filehandles ;-) Hah, thanks for finding that one. :) ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel
Re: [Devel] locking on SunOS
Thanks for the review.
If you would, kindly share a non-destructive patch for further
discussion.
P.
On Sep 18, 2008, at 17:05, Christian Theil Have wrote:
I've been testing the patch and it seems to work out allright :-)
A few notes;
#include
must be outside #ifdef SunOS - other systems needs it too
in unlock_and_fclose(FILE *fp)
fclose is only called in #else, I suppose that should have been like
for unlock_and_close or we'll run out of filehandles ;-)
Best regards,
Christian
On Wed, Sep 10, 2008 at 10:05 PM, Jason Pollock <[EMAIL PROTECTED]
> wrote:
Christian Theil Have wrote:
Hi
I think it looks good :-) I was working on some similar code based
on a dict, which I was still testing out, but your patch is cleaner.
I look forward to trying it out, but I wont have the opportunity to
test it until Monday though.
About changing close and fclose everywhere in the codebase, you
could get around this without having to change the rest the of
codebase by using a few macros , eg.
#ifdef SunOS
#define close(fd) unlock_and_close(fd)
#endif
int unlock_and_close(fd) {
#ifdef close(fd)
#undef close(fd)
printf("unlock_and_close do the stuff..\n");
return close(fd);
#define close(fd) unlock_and_close(fd)
#endif
}
and similarly for fclose...
Christian.
That would work. I wasn't sure if the Mbuni team would want to go
that way, or the Kannel way:
#define close(x) you_should_not_call_close_directly(x)
I personally decided against the #define because I wanted to be able
to check that all instances of close/fclose had been modified in the
source I was working on. I did look at trying to figure out how to
wrap the libc call, but I thought that might be a bit too
destructive to my schedule. ;)
I'm still new to the gwlib code, I didn't realise it had a dict!
Too bad it uses octstr's as keys.
Let me know if you run into any problems. If I find any, I'll let
the list know.
Jason
___
Devel mailing list
Devel@mbuni.org
http://lists.mbuni.org/mailman/listinfo/devel
___
Devel mailing list
Devel@mbuni.org
http://lists.mbuni.org/mailman/listinfo/devel
Re: [Devel] locking on SunOS
I've been testing the patch and it seems to work out allright :-)
A few notes;
#include
must be outside #ifdef SunOS - other systems needs it too
in unlock_and_fclose(FILE *fp)
fclose is only called in #else, I suppose that should have been like for
unlock_and_close or we'll run out of filehandles ;-)
Best regards,
Christian
On Wed, Sep 10, 2008 at 10:05 PM, Jason Pollock <
[EMAIL PROTECTED]> wrote:
>
>
> Christian Theil Have wrote:
>
>> Hi
>>
>> I think it looks good :-) I was working on some similar code based on a
>> dict, which I was still testing out, but your patch is cleaner.
>> I look forward to trying it out, but I wont have the opportunity to test
>> it until Monday though.
>>
>> About changing close and fclose everywhere in the codebase, you could get
>> around this without having to change the rest the of codebase by using a few
>> macros , eg.
>>
>> #ifdef SunOS
>> #define close(fd) unlock_and_close(fd)
>> #endif
>>
>>
>>
>> int unlock_and_close(fd) {
>> #ifdef close(fd)
>> #undef close(fd)
>>printf("unlock_and_close do the stuff..\n");
>>return close(fd);
>> #define close(fd) unlock_and_close(fd)
>> #endif
>> }
>>
>> and similarly for fclose...
>>
>> Christian.
>>
>
> That would work. I wasn't sure if the Mbuni team would want to go that
> way, or the Kannel way:
>
> #define close(x) you_should_not_call_close_directly(x)
>
> I personally decided against the #define because I wanted to be able to
> check that all instances of close/fclose had been modified in the source I
> was working on. I did look at trying to figure out how to wrap the libc
> call, but I thought that might be a bit too destructive to my schedule. ;)
>
> I'm still new to the gwlib code, I didn't realise it had a dict! Too bad
> it uses octstr's as keys.
>
> Let me know if you run into any problems. If I find any, I'll let the list
> know.
>
> Jason
>
___
Devel mailing list
Devel@mbuni.org
http://lists.mbuni.org/mailman/listinfo/devel
Re: [Devel] locking on SunOS
Christian Theil Have wrote:
Hi
I think it looks good :-) I was working on some similar code based on
a dict, which I was still testing out, but your patch is cleaner.
I look forward to trying it out, but I wont have the opportunity to
test it until Monday though.
About changing close and fclose everywhere in the codebase, you could
get around this without having to change the rest the of codebase by
using a few macros , eg.
#ifdef SunOS
#define close(fd) unlock_and_close(fd)
#endif
int unlock_and_close(fd) {
#ifdef close(fd)
#undef close(fd)
printf("unlock_and_close do the stuff..\n");
return close(fd);
#define close(fd) unlock_and_close(fd)
#endif
}
and similarly for fclose...
Christian.
That would work. I wasn't sure if the Mbuni team would want to go that
way, or the Kannel way:
#define close(x) you_should_not_call_close_directly(x)
I personally decided against the #define because I wanted to be able to
check that all instances of close/fclose had been modified in the source
I was working on. I did look at trying to figure out how to wrap the
libc call, but I thought that might be a bit too destructive to my
schedule. ;)
I'm still new to the gwlib code, I didn't realise it had a dict! Too
bad it uses octstr's as keys.
Let me know if you run into any problems. If I find any, I'll let the
list know.
Jason
___
Devel mailing list
Devel@mbuni.org
http://lists.mbuni.org/mailman/listinfo/devel
Re: [Devel] locking on SunOS
Hi
I think it looks good :-) I was working on some similar code based on
a dict, which I was still testing out, but your patch is cleaner.
I look forward to trying it out, but I wont have the opportunity to
test it until Monday though.
About changing close and fclose everywhere in the codebase, you could
get around this without having to change the rest the of codebase by
using a few macros , eg.
#ifdef SunOS
#define close(fd) unlock_and_close(fd)
#endif
int unlock_and_close(fd) {
#ifdef close(fd)
#undef close(fd)
printf("unlock_and_close do the stuff..\n");
return close(fd);
#define close(fd) unlock_and_close(fd)
#endif
}
and similarly for fclose...
Christian.
On Sep 10, 2008, at 6:35 AM, P. A. Bagyenda wrote:
Thank you Jason,
Comments welcome from all interested parties, before we apply patch.
Paul.
On Sep 10, 2008, at 07:13, Jason Pollock wrote:
PostgreSQL isn't really an option for me here at this point.
I've modified the lock functions to (hopefully) allow linux's flock
semantics.
The code now maintains a list of inode/dev pairs. When a lock
request arrives for an existing inode/dev, it
is forced to wait on a pthread_cond_t.
The change does require that close and fclose be changed to two
functions in the library to maintain the list. Otherwise
the list will get out of sync with the locks. This may not be
viable for the main code base.
I've attached my changes, I would appreciate any comments. Just a
warning - I haven't tested it extensively, just enough to see that
it stops the queue runner from starting a message a second time.
P. A. Bagyenda wrote:
We welcome a solution when one is found! The chickens among us
would, if using Solaris, rather use the PostgreSQL-based queue
storage module instead :)
That brings a number of advantages:
- Queue processing is faster for larger queues, since unlike the
file-based storage which scans the entire queue directory tree on
each run, this one only picks up messages due for processing
- Message archival is built in.
On Sep 04, 2008, at 00:02, Jason Pollock wrote:
Jason Pollock wrote:
I have noticed the same problem. With fcntl locks, the lock
isn't associated with a file descriptor, it's associated with
the file, so if someone opens the file, checks the lock and
closes the file in the same process, the lock is released.
Since it's a required lock, perhaps Solaris mandatory locking
would be a better way to deal with the problem than a dict?
http://sunsite.uakom.sk/sunworldonline/swol-04-1998/swol-04-insidesolaris.html
Closer reading seems to indicate that that won't work either.
Nuts.
___
Devel mailing list
Devel@mbuni.org
http://lists.mbuni.org/mailman/listinfo/devel
___
Devel mailing list
Devel@mbuni.org
http://lists.mbuni.org/mailman/listinfo/devel
___
Devel mailing list
Devel@mbuni.org
http://lists.mbuni.org/mailman/listinfo/devel
___
Devel mailing list
Devel@mbuni.org
http://lists.mbuni.org/mailman/listinfo/devel
Re: [Devel] locking on SunOS
Thank you Jason, Comments welcome from all interested parties, before we apply patch. Paul. On Sep 10, 2008, at 07:13, Jason Pollock wrote: PostgreSQL isn't really an option for me here at this point. I've modified the lock functions to (hopefully) allow linux's flock semantics. The code now maintains a list of inode/dev pairs. When a lock request arrives for an existing inode/dev, it is forced to wait on a pthread_cond_t. The change does require that close and fclose be changed to two functions in the library to maintain the list. Otherwise the list will get out of sync with the locks. This may not be viable for the main code base. I've attached my changes, I would appreciate any comments. Just a warning - I haven't tested it extensively, just enough to see that it stops the queue runner from starting a message a second time. P. A. Bagyenda wrote: We welcome a solution when one is found! The chickens among us would, if using Solaris, rather use the PostgreSQL-based queue storage module instead :) That brings a number of advantages: - Queue processing is faster for larger queues, since unlike the file-based storage which scans the entire queue directory tree on each run, this one only picks up messages due for processing - Message archival is built in. On Sep 04, 2008, at 00:02, Jason Pollock wrote: Jason Pollock wrote: I have noticed the same problem. With fcntl locks, the lock isn't associated with a file descriptor, it's associated with the file, so if someone opens the file, checks the lock and closes the file in the same process, the lock is released. Since it's a required lock, perhaps Solaris mandatory locking would be a better way to deal with the problem than a dict? http://sunsite.uakom.sk/sunworldonline/swol-04-1998/swol-04-insidesolaris.html Closer reading seems to indicate that that won't work either. Nuts. ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel
Re: [Devel] locking on SunOS
PostgreSQL isn't really an option for me here at this point. I've modified the lock functions to (hopefully) allow linux's flock semantics. The code now maintains a list of inode/dev pairs. When a lock request arrives for an existing inode/dev, it is forced to wait on a pthread_cond_t. The change does require that close and fclose be changed to two functions in the library to maintain the list. Otherwise the list will get out of sync with the locks. This may not be viable for the main code base. I've attached my changes, I would appreciate any comments. Just a warning - I haven't tested it extensively, just enough to see that it stops the queue runner from starting a message a second time. P. A. Bagyenda wrote: We welcome a solution when one is found! The chickens among us would, if using Solaris, rather use the PostgreSQL-based queue storage module instead :) That brings a number of advantages: - Queue processing is faster for larger queues, since unlike the file-based storage which scans the entire queue directory tree on each run, this one only picks up messages due for processing - Message archival is built in. On Sep 04, 2008, at 00:02, Jason Pollock wrote: Jason Pollock wrote: I have noticed the same problem. With fcntl locks, the lock isn't associated with a file descriptor, it's associated with the file, so if someone opens the file, checks the lock and closes the file in the same process, the lock is released. Since it's a required lock, perhaps Solaris mandatory locking would be a better way to deal with the problem than a dict? http://sunsite.uakom.sk/sunworldonline/swol-04-1998/swol-04-insidesolaris.html Closer reading seems to indicate that that won't work either. Nuts. ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel code_changes.tgz Description: GNU Zip compressed data ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel
Re: [Devel] locking on SunOS
We welcome a solution when one is found! The chickens among us would, if using Solaris, rather use the PostgreSQL-based queue storage module instead :) That brings a number of advantages: - Queue processing is faster for larger queues, since unlike the file-based storage which scans the entire queue directory tree on each run, this one only picks up messages due for processing - Message archival is built in. On Sep 04, 2008, at 00:02, Jason Pollock wrote: Jason Pollock wrote: I have noticed the same problem. With fcntl locks, the lock isn't associated with a file descriptor, it's associated with the file, so if someone opens the file, checks the lock and closes the file in the same process, the lock is released. Since it's a required lock, perhaps Solaris mandatory locking would be a better way to deal with the problem than a dict? http://sunsite.uakom.sk/sunworldonline/swol-04-1998/swol-04-insidesolaris.html Closer reading seems to indicate that that won't work either. Nuts. ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel
Re: [Devel] locking on SunOS
Jason Pollock wrote: I have noticed the same problem. With fcntl locks, the lock isn't associated with a file descriptor, it's associated with the file, so if someone opens the file, checks the lock and closes the file in the same process, the lock is released. Since it's a required lock, perhaps Solaris mandatory locking would be a better way to deal with the problem than a dict? http://sunsite.uakom.sk/sunworldonline/swol-04-1998/swol-04-insidesolaris.html Closer reading seems to indicate that that won't work either. Nuts. ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel
Re: [Devel] locking on SunOS
I have noticed the same problem. With fcntl locks, the lock isn't associated with a file descriptor, it's associated with the file, so if someone opens the file, checks the lock and closes the file in the same process, the lock is released. Since it's a required lock, perhaps Solaris mandatory locking would be a better way to deal with the problem than a dict? http://sunsite.uakom.sk/sunworldonline/swol-04-1998/swol-04-insidesolaris.html Christian Theil Have wrote: Hi The file locking code for SunOS in CVS is not working properly. It will lock the files for other processes, but when accessed from the same process (the mms queue process) the lock will be ignored. This can cause some problems, since the messages are produced to multiple deliver deliver threads, which each attempts to deliver the message :-( I am contemplating writing a fix for this, but I am not sure how to best go around this: I have tried implementing it using share reservations, which seems to work for different threads of the same process, but since the queue runs in the same process _and_ thread, share reservations doesn't really work out either. I think a dict or similar could easily be used to keep track of the open files in the queue, but not so sure if this is the best solution. It might be satisfactory though. What do you think? Best regards, Christian Theil Have ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel ___ Devel mailing list Devel@mbuni.org http://lists.mbuni.org/mailman/listinfo/devel
