[sqlalchemy] Re: Old event listeners firing on the wrong session?

2015-02-09 Thread Jonathan Vanasco 
I just wanted to suggest another approach that may work, at least as a 
fallback.

When I upload files to Amazon S3 I track the activity in a logging table 
(via sqlalchemy) that looks a bit like this:

  id | filename | timestamp_upload_start | upload_status (bool) | 
timestamp_deleted

Before uploading I create and flush an object with: id(sequence), filename, 
timestamp_start

If it uploads, I set upload_success = True , and flush
If it fails, I set upload_status = False, and flush.

If i can catch the failure, I'll delete the file and update 
`timestamp_deleted`.  If i can't catch it, a taskrunner often checks for 
failed uploads that didn't delete and performs a cleanup.

-- 
You received this message because you are subscribed to the Google Groups 
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[sqlalchemy] Re: Old event listeners firing on the wrong session?

2015-02-09 Thread Russ 
Problem solved.

Michael - you nailed it when you guessed:

if you have set an event listener on a single session that is definitely 
 the 
 actual Session object and not the thread local or the scoped_session or 
 anything


sa.event.listen was being passed the scoped_session instance, not an actual 
instance.  This scoped_session usage is all through the code, per what 
seems to be an old way of doing things... that being that the object 
returned by scoped_session() object had all of the Session objects as a 
convenience, so they were pretty much interchangeable.  Clearly not with 
the event system!

Calling the object to get the actual session prevents any unexpected 
rollback handlers from firing shots from the Session graveyard.

Thanks for the help!

Russ

PS: had to dig through some old docs to check my sanity on the 
scoped_session-as-a-Session confusion.  It was clearly a legit way to do it 
back in 0.6:
http://docs.sqlalchemy.org/en/rel_0_6/orm/session.html?highlight=remove#creating-a-thread-local-context

... but new docs have no such comments/recommendations.  I think the new 
method and docs are clearer.

I'm thinking I should scrub all code for this type of usage and fix it up 
so there's no blurry line between Sessions and 
sessionmakers/scoped_sessions.  Do you recommend that?


On Monday, February 9, 2015 at 9:47:23 PM UTC-5, Russ wrote:

 Thanks for the idea, Jonathan!  I was actually discussing such a fallback 
 watchdog on #postgresql earlier today.  Now that I've been having event 
 troubles and it is highlighting atomicity issues with the db/filesystem 
 split, I'm definitely going to implement such a safety net.

 I think the problem is slightly different than yours, though.  In my case 
 the file have been written in mid-transaction, and I want to delete it 
 if/when the transaction rolls back.  If my rollback detector is failing (my 
 bug, I'm sure) I have no db flag to check.  The watchdog needs to check the 
 filesystem for dangling files (files with no db reference).  This will 
 likely be two phase check (with a big gap) to avoid race conditions, but 
 should be easy enough since file dirs+names are done by database id.  If 
 the id is 123456789 it stores to `/files/123/456/789`. The dir splitting 
 should help minimize requirements on filesystem awareness... glad I did 
 that.

 My bigger problem right now is that the late-firing zombie rollback 
 handler is actually sniping files it shouldn't be!  Rollbacks in 
 subsequent/unrelated sessions are sniping files from sessions that ran to 
 commit without issue.  Watchdog or not, I need to fix that!!

 I really wish databases handled hybrid db/filesystem storage better.  Any 
 errors in file deletion code are deadly.  SQL Server has that nice 
 FILESTREAM object for precisely this situation, but it doesn't seem that 
 others do yet. 



-- 
You received this message because you are subscribed to the Google Groups 
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[sqlalchemy] Re: Old event listeners firing on the wrong session?

2015-02-09 Thread Russ 
Thanks for the idea, Jonathan!  I was actually discussing such a fallback 
watchdog on #postgresql earlier today.  Now that I've been having event 
troubles and it is highlighting atomicity issues with the db/filesystem 
split, I'm definitely going to implement such a safety net.

I think the problem is slightly different than yours, though.  In my case 
the file have been written in mid-transaction, and I want to delete it 
if/when the transaction rolls back.  If my rollback detector is failing (my 
bug, I'm sure) I have no db flag to check.  The watchdog needs to check the 
filesystem for dangling files (files with no db reference).  This will 
likely be two phase check (with a big gap) to avoid race conditions, but 
should be easy enough since file dirs+names are done by database id.  If 
the id is 123456789 it stores to `/files/123/456/789`. The dir splitting 
should help minimize requirements on filesystem awareness... glad I did 
that.

My bigger problem right now is that the late-firing zombie rollback handler 
is actually sniping files it shouldn't be!  Rollbacks in 
subsequent/unrelated sessions are sniping files from sessions that ran to 
commit without issue.  Watchdog or not, I need to fix that!!

I really wish databases handled hybrid db/filesystem storage better.  Any 
errors in file deletion code are deadly.  SQL Server has that nice 
FILESTREAM object for precisely this situation, but it doesn't seem that 
others do yet. 

-- 
You received this message because you are subscribed to the Google Groups 
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.