Re: apache:session and mod perl
On Fri, 2004-12-31 at 09:51 -0800, Chris Ochs wrote: > A deadlock happens when you have an updater that acquires a write > lock, but it's waiting for the first updater that is also in a SELECT > FOR UPDATE but has not yet acquired a write lock. There is no such thing as a write lock for the Postgres subclass. It grabs a lock when you materialize the session and doesn't give it up or try to get another one until the session object is DESTROYed. > The first updater > can't finish because it can't acquire a write lock that the second > updater has already obtained, and the second updater can't update > until the first one finishes, which it can't. Only one can get a lock on a particular session at once, which is by design. There is no deadlock though, since only a single lock is involved here. > A workable hack would probably be to have Apache::Session::Lock::File > always set Transaction to 1 if the backend store was postgres. You *really* should not need Apache::Session::Lock::File with Postgres if you are not doing anything crazy with Postgres isolation levels and transactions. I regret bringing it up now. You need to back up and figure out what's going wrong with your Postgres stuff. Are you doing commits on this same database handle that would cause the lock to be released too early? Going back to an earlier mail: > If it is a scoping issue it's probably in one of the Apache::Session > modules I think someone would have found that by now. > The session is kept in a > global request object I use for all variables that need to be passed > around. In the cleanup handler I untie the session and undefined the > request object, and log it all with log4perl. That should work, but no other request will be able to read the same session until your cleanup handler has run. > I just figured that File locking wasn't supported or tested > with the Postgresql backend store It's not. > > I put some loggin into apache::session::lock::file, and then switched > > between using apache::session::file and apache::session::flex with > > postgres and Lock set to File. Can you try it with just Postgres? - Perrin -- Report problems: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: apache:session and mod perl
I know what's happening now, it dawned on me after I wrote this last night. A deadlock happens when you have an updater that acquires a write lock, but it's waiting for the first updater that is also in a SELECT FOR UPDATE but has not yet acquired a write lock. The first updater can't finish because it can't acquire a write lock that the second updater has already obtained, and the second updater can't update until the first one finishes, which it can't. Which also explains why Apache::Session::Flex doesn't list File as one of the supported Lock options. A workable hack would probably be to have Apache::Session::Lock::File always set Transaction to 1 if the backend store was postgres. Chris On Thu, 30 Dec 2004 23:41:16 -0800, Chris Ochs <[EMAIL PROTECTED]> wrote: > > > I tried setting Lock to File instead of Null, but there is some sort > > > of contention issue because after the first request all other requests > > > hang like they are waiting for a lock to be release. > > > > This usually means you have a scoping bug in your code. If the session > > object never goes out of scope, it will not release the lock. > > I put some loggin into apache::session::lock::file, and then switched > between using apache::session::file and apache::session::flex with > postgres and Lock set to File. > With postgres it calls acquire_read lock a bunch of times in a row > with an acquire_write_lock thrown in and then something blocks. Using > apache::session::file it alternates correctly and doesn't do that. > There has to be some logic in apache::session that is different than > apache::session::file. > > Here is a log when using apache::session::file. I put the pid in front. > > 96836 acquire_read_lock > 96836 acquire_write_lock > 96836 release_all_lock > 96836 DESTROY > 96836 release_all_lock > 96822 acquire_read_lock > 96822 acquire_write_lock > 96822 release_all_lock > 96822 DESTROY > 96822 release_all_lock > 96836 acquire_read_lock > 96836 acquire_write_lock > 96836 release_all_lock > 96836 DESTROY > 96836 release_all_lock > > Here is a log using flex with postgres and Lock set to File. > 96824 acquire_read_lock > 96834 acquire_read_lock > 96826 acquire_read_lock > 96824 acquire_write_lock > 96822 acquire_read_lock > > I'm not sure exactly what pages were being called in the log snippets > above, but this pattern is always the same. > > Chris > -- Report problems: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: apache:session and mod perl
> > I tried setting Lock to File instead of Null, but there is some sort > > of contention issue because after the first request all other requests > > hang like they are waiting for a lock to be release. > > This usually means you have a scoping bug in your code. If the session > object never goes out of scope, it will not release the lock. I put some loggin into apache::session::lock::file, and then switched between using apache::session::file and apache::session::flex with postgres and Lock set to File. With postgres it calls acquire_read lock a bunch of times in a row with an acquire_write_lock thrown in and then something blocks. Using apache::session::file it alternates correctly and doesn't do that. There has to be some logic in apache::session that is different than apache::session::file. Here is a log when using apache::session::file. I put the pid in front. 96836 acquire_read_lock 96836 acquire_write_lock 96836 release_all_lock 96836 DESTROY 96836 release_all_lock 96822 acquire_read_lock 96822 acquire_write_lock 96822 release_all_lock 96822 DESTROY 96822 release_all_lock 96836 acquire_read_lock 96836 acquire_write_lock 96836 release_all_lock 96836 DESTROY 96836 release_all_lock Here is a log using flex with postgres and Lock set to File. 96824 acquire_read_lock 96834 acquire_read_lock 96826 acquire_read_lock 96824 acquire_write_lock 96822 acquire_read_lock I'm not sure exactly what pages were being called in the log snippets above, but this pattern is always the same. Chris -- Report problems: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: apache:session and mod perl
> > > I tried setting Lock to File instead of Null, but there is some sort > > of contention issue because after the first request all other requests > > hang like they are waiting for a lock to be release. > > This usually means you have a scoping bug in your code. If the session > object never goes out of scope, it will not release the lock. > If it is a scoping issue it's probably in one of the Apache::Session modules, or maybe I'm not supplying all the necessary parameters when using File locking with Postgresql. Literally all I did is change Lock to File instead of Null and that's it. The session is kept in a global request object I use for all variables that need to be passed around. In the cleanup handler I untie the session and undefined the request object, and log it all with log4perl. If there were scoping issues with the global request object, it would show up all over the place. I just figured that File locking wasn't supported or tested with the Postgresql backend store, so I didn't spend any more time on it. > Yes, you should not need anything more than the Postgres locking. > > > I use the default isolation in postgresql. > > Then it really should be shielding you from this problem. Make sure your > scoping is correct (you could throw a warn into the A::S DESTROY() sub to > make sure it is getting called) and make sure you are issuing a commit > somewhere in there to save your changes. SELECT FOR UPDATE doesn't block reads. I put in the File backend store with the Transaction argument and it puts an exclusive lock on reads, which makes everything work as I wanted it to. Now I will go back and look closer at Apache::Session and see if the File locking is supported for the Postgresql store. Chris -- Report problems: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: apache:session and mod perl
Chris Ochs said: > I have found three things that need to be worked around like this in > order to use this approach. One is frames like you said, the other is > not pulling in stylesheets via an href, and the other is making sure > images are not loaded through mod perl. If I do all of that I don't > have any problems. I still don't like it, but it might be our only > option. It's not your only option, but you really don't want to be pulling in sessions for things like CSS and images, or serving them through mod_perl. > I tried setting Lock to File instead of Null, but there is some sort > of contention issue because after the first request all other requests > hang like they are waiting for a lock to be release. This usually means you have a scoping bug in your code. If the session object never goes out of scope, it will not release the lock. > Also, I found out through further testing that Apache::Session doesn't > do exclusive locking on read, only write. Apache::Session::Store::Postgres uses an exclusive lock via SELECT FOR UPDATE, and it grabs the lock when you load the session. Other subclasses behave differently. I don't believe the "Transaction" option has any effect on the Postgres subclass. > And that also > means that there isn't any need for additional locking with postgresql > because using SELECT FOR UPDATE basically does the same thing as the > file locking. Yes, you should not need anything more than the Postgres locking. > I use the default isolation in postgresql. Then it really should be shielding you from this problem. Make sure your scoping is correct (you could throw a warn into the A::S DESTROY() sub to make sure it is getting called) and make sure you are issuing a commit somewhere in there to save your changes. - Perrin -- Report problems: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: apache:session and mod perl
> So you have a lost update problem. There is probably a way to structure > things to avoid this (maybe not using sessions for it), but some form of > mutually exclusive locking would fix it, at the expense of making your > site slower, since each frame will have to wait for its turn. You might > try to make your code avoid loading sessions in frames where they are not > absolutely needed. I have found three things that need to be worked around like this in order to use this approach. One is frames like you said, the other is not pulling in stylesheets via an href, and the other is making sure images are not loaded through mod perl. If I do all of that I don't have any problems. I still don't like it, but it might be our only option. > The thing that makes the difference here is really the locking module. > Take a look at the code inside Apache::Session::Postgres. It uses the > Null locking module because Postgres is supposed to handle the locking. > You could have it use the locking from the file-based or MySQL-based > sessions if you wanted to. Apache::Session::Flex makes this easy to > specify. I tried setting Lock to File instead of Null, but there is some sort of contention issue because after the first request all other requests hang like they are waiting for a lock to be release. I did preload Apache::Session:Lock::File. Also, I found out through further testing that Apache::Session doesn't do exclusive locking on read, only write. Which is what I would expect actually after thinking about it for a minute. And that also means that there isn't any need for additional locking with postgresql because using SELECT FOR UPDATE basically does the same thing as the file locking. > You're allowing dirty reads? That's not the normal isolation level for > Postgres. That is likely the source of your problems. Can you change > that to the default level, or is there something else that will break? > No, my mistake I was using the wrong terminology. I use the default isolation in postgresql. -- Report problems: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: apache:session and mod perl
Chris Ochs said: > The issue I am having is that the postgresql store doesn't do any > locking It should. It uses "SELECT FOR UPDATE" which grabs an exclusive lock on the row in question until a commit is issued. A::S doesn't issue the commit, so you have to do that yourself. Have you been messing with isolation levels in a way that would defeat the SELECT FOR UPDATE read lock? > What is happening is when a page that is loaded loads other pages > (such as a frameset), sometimes the request for the first page doesn't > finish before the other requests have already read the session data. This can be a problem with multiple browser windows, fast reloads, etc. Even without frames, you always have to expect concurrent access. > It goes like this. > > - Request 1 fetches session > - Request 2 fetches session > - Request 1 changes some session information > - Request 1 writes out session info > - Request 2 writes out session info > > In our case a flag stating whether a user is logged in or not is > stored in the session. If Request 1 logs out a user, Request 2 flags > the user as logged in again when it writes out the session data. So you have a lost update problem. There is probably a way to structure things to avoid this (maybe not using sessions for it), but some form of mutually exclusive locking would fix it, at the expense of making your site slower, since each frame will have to wait for its turn. You might try to make your code avoid loading sessions in frames where they are not absolutely needed. > Solutions I have thought of so far include.. > > 1. Switch back to file based sessions, not what I want. The thing that makes the difference here is really the locking module. Take a look at the code inside Apache::Session::Postgres. It uses the Null locking module because Postgres is supposed to handle the locking. You could have it use the locking from the file-based or MySQL-based sessions if you wanted to. Apache::Session::Flex makes this easy to specify. > 2. Keep a separate pool of cached connections just for the session > database, and have those connections use a transaction isolation level > that guarantees no dirty reads. You're allowing dirty reads? That's not the normal isolation level for Postgres. That is likely the source of your problems. Can you change that to the default level, or is there something else that will break? - Perrin -- Report problems: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: apache:session and mod perl
On Thu, 30 Dec 2004 10:00:32 +0200, Octavian Rasnita <[EMAIL PROTECTED]> wrote: > If this happens only in pages with frames, try creating a page without > frames. Umm no, I want a solution not a work around. This application is used by a lot of people on a lot of different sites with the file backend store. I'll go back to that if I have to. -- Report problems: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: apache:session and mod perl
If this happens only in pages with frames, try creating a page without frames. Teddy - Original Message - From: "Chris Ochs" <[EMAIL PROTECTED]> To: Sent: Thursday, December 30, 2004 9:42 AM Subject: apache:session and mod perl This question could go to one of several different lists, but I thought this might be a good place to start. I have an application based on mod perl and the Template Toolkit. For session tracking I am using Apache::Session with the postgresql backend store. The issue I am having is that the postgresql store doesn't do any locking, which is causing some requests to read bad data, but I really want to keep the database as the backend store because it will scale a lot better if we need to have multiple servers access the same session data. What is happening is when a page that is loaded loads other pages (such as a frameset), sometimes the request for the first page doesn't -- Report problems: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html List etiquette: http://perl.apache.org/maillist/email-etiquette.html
