Re: What's the best way to optimize database connections in Pyramid?
This worked for me fine for another non-Pyramid app:
class MongoModel(object):
__dbconn = None
def __init__(self, properties=None):
self._id = None
if properties:
for k,v in properties.iteritems():
setattr(self, k, v if k != '_id' else
pymongo.objectid.ObjectId(v))
def collection(self):
if MongoModel.__dbconn is None:
MongoModel.__dbconn =
pymongo.Connection(host=conf["mongo.sets"].split(" "))[conf["mongo.db"]]
return MongoModel.__dbconn[self._collection]
...
This class also exposes the collection methods which internally call
those methods on self.collection() ... Then each model class would
inherit from MongoModel (and specify _collection argument in its
__init__ method).
However this is not threadsafe (for some other reasons I used prefork
process-only flup), so I suppose you'd need to make a threadlocal class
argument which contains __dbconn.
Since MongoDB is not transactional, I don't think you'll need middleware
to hook into request-response flow for the purposes of handling
autocommits or rollbacks, but ymmv.
.oO V Oo.
On 03/16/2011 06:38 PM, Stephen Lacy wrote:
Since mod_wsgi uses long-lived processes, I've decided to keep the
connection open for the life of the process (and auto-reconnect if
there's a failure).
I'm actually a little surprised that people here are connecting &
disconnecting with each request, as it feels unnecessary. Is there
some disadvantage of long-lived connections? Of course, it likely
varies per database, but FWIW, I'm using MongoDB as well.
Steve
--
You received this message because you are subscribed to the Google Groups
"pylons-devel" group.
To post to this group, send email to pylons-devel@googlegroups.com.
To unsubscribe from this group, send email to
pylons-devel+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/pylons-devel?hl=en.
Re: What's the best way to optimize database connections in Pyramid?
The normal strategy for databases is to use a connection pool. Conceptually the code gets a new connection each request but the pool may return an existing connection instead. Depending on the database this can be a big win; some databases connect so quickly that it hardly makes a difference. -- You received this message because you are subscribed to the Google Groups "pylons-devel" group. To post to this group, send email to pylons-devel@googlegroups.com. To unsubscribe from this group, send email to pylons-devel+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/pylons-devel?hl=en.
Re: What's the best way to optimize database connections in Pyramid?
Since mod_wsgi uses long-lived processes, I've decided to keep the connection open for the life of the process (and auto-reconnect if there's a failure). I'm actually a little surprised that people here are connecting & disconnecting with each request, as it feels unnecessary. Is there some disadvantage of long-lived connections? Of course, it likely varies per database, but FWIW, I'm using MongoDB as well. Steve On Tue, Mar 15, 2011 at 4:53 PM, oO wrote: > I've looked at the request example, but I've decided to put the > MongoDB connection inside the > resource instead. It's actually in the resource_factory object, and my > resource objects are not > recreated on every request, only when the resource tree actually > changes. This is pretty similar to > the way Pyramid implements the ZODB layer. > > The view code (or god forbid the template code ;-) ) still has access > to the connection from either > request.context.db or request.root.db > > It has the advantage that each resource object could actually be > fetching data from completely different > data-store. and calling the db directly from the view code is just a > development backdoor convenience. > > This, of course works better with traversal, but even in a > url_dispatch system you still have at least one resource object. > > oO > > > On Mar 14, 2:47 pm, Seth wrote: > > Thank you everyone for your insightful comments. I would really love to > see > > code examples (if available) of each of these different solutions you all > > are describing. I feel like I'm wading into the deep end of the pool > here. > > > > For the record (and for those wondering), I'm using mod_wsgi (daemon > mode) > > for the hosting so "true" static routes aren't being handled by Pyramid, > but > > cycling a new dbconn for every request is still not "the right way to do > it" > > in my book. > > > > Seth > > -- > You received this message because you are subscribed to the Google Groups > "pylons-devel" group. > To post to this group, send email to pylons-devel@googlegroups.com. > To unsubscribe from this group, send email to > pylons-devel+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/pylons-devel?hl=en. > > -- You received this message because you are subscribed to the Google Groups "pylons-devel" group. To post to this group, send email to pylons-devel@googlegroups.com. To unsubscribe from this group, send email to pylons-devel+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/pylons-devel?hl=en.
Re: What's the best way to optimize database connections in Pyramid?
I've looked at the request example, but I've decided to put the MongoDB connection inside the resource instead. It's actually in the resource_factory object, and my resource objects are not recreated on every request, only when the resource tree actually changes. This is pretty similar to the way Pyramid implements the ZODB layer. The view code (or god forbid the template code ;-) ) still has access to the connection from either request.context.db or request.root.db It has the advantage that each resource object could actually be fetching data from completely different data-store. and calling the db directly from the view code is just a development backdoor convenience. This, of course works better with traversal, but even in a url_dispatch system you still have at least one resource object. oO On Mar 14, 2:47 pm, Seth wrote: > Thank you everyone for your insightful comments. I would really love to see > code examples (if available) of each of these different solutions you all > are describing. I feel like I'm wading into the deep end of the pool here. > > For the record (and for those wondering), I'm using mod_wsgi (daemon mode) > for the hosting so "true" static routes aren't being handled by Pyramid, but > cycling a new dbconn for every request is still not "the right way to do it" > in my book. > > Seth -- You received this message because you are subscribed to the Google Groups "pylons-devel" group. To post to this group, send email to pylons-devel@googlegroups.com. To unsubscribe from this group, send email to pylons-devel+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/pylons-devel?hl=en.
Re: What's the best way to optimize database connections in Pyramid?
Thank you everyone for your insightful comments. I would really love to see code examples (if available) of each of these different solutions you all are describing. I feel like I'm wading into the deep end of the pool here. For the record (and for those wondering), I'm using mod_wsgi (daemon mode) for the hosting so "true" static routes aren't being handled by Pyramid, but cycling a new dbconn for every request is still not "the right way to do it" in my book. Seth -- You received this message because you are subscribed to the Google Groups "pylons-devel" group. To post to this group, send email to pylons-devel@googlegroups.com. To unsubscribe from this group, send email to pylons-devel+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/pylons-devel?hl=en.
Re: What's the best way to optimize database connections in Pyramid?
Perhaps I am missing something here but why would you serve static files from your Pyramid app? Why don't you serve the static files by the webserver directly? As for MongoDB, how about a wrapper around the pymongo classes/objects and use lazy connectors, ie. do the actual connection only if one does not exist in the thread, whenever a pymongo method is called? Vlad On 03/13/2011 09:54 AM, Seth wrote: I was advised a while back by someone to throw my MongoDB connection call into a pyramid.events.NewRequest subscriber. However, this has proven to be a bad idea because the NewRequest subscriber gets called even if the request is a static_route request, and since the event.request object in the NewRequest subscriber doesn't have a matched_route attribute, there's no bullet-proof way to filter those out. How are other people connecting to a db with high-traffic sites using Pyramid and avoiding the static_route re-connection nonsense? Perhaps we could request to get a matched_route object on the NewRequest event object? Thanks, Seth -- You received this message because you are subscribed to the Google Groups "pylons-devel" group. To post to this group, send email to pylons-devel@googlegroups.com. To unsubscribe from this group, send email to pylons-devel+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/pylons-devel?hl=en.
Re: What's the best way to optimize database connections in Pyramid?
On 13/03/2011 14:51, Chris Rossi wrote: If it were me I would attach a connection factory instead of an open connection. The factory would open a connection and stash it on the request the first time it is invoked and then just return the stashed connection on subsequent calls. This preserves the convenience of your pattern but allows you to only cause a connection to be opened for requests that need it. I used the component-architecture-hidden-behind-functions approach that has worked so well for Pyramid when I did mortar_rdb: http://packages.python.org/mortar_rdb/api.html#mortar_rdb.registerSession http://packages.python.org/mortar_rdb/api.html#mortar_rdb.getSession Obviously, that won't help with MongoDB, but the thread-local and zope.component utility approach might work for Seth too. cheers, Chris PS: I'm at PyCon and sprinting on Pyramid, please come grab me if you want to play aroun with this stuff :-) -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk -- You received this message because you are subscribed to the Google Groups "pylons-devel" group. To post to this group, send email to pylons-devel@googlegroups.com. To unsubscribe from this group, send email to pylons-devel+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/pylons-devel?hl=en.
Re: What's the best way to optimize database connections in Pyramid?
On Mar 13, 4:54 am, Seth wrote: > I was advised a while back by someone to throw my MongoDB connection call > into a pyramid.events.NewRequest subscriber. However, this has proven to be > a bad idea because the NewRequest subscriber gets called even if the request > is a static_route request, and since the event.request object in the > NewRequest subscriber doesn't have a matched_route attribute, there's no > bullet-proof way to filter those out. > > How are other people connecting to a db with high-traffic sites using > Pyramid and avoiding the static_route re-connection nonsense? Perhaps we > could request to get a matched_route object on the NewRequest event object? > If it were me I would attach a connection factory instead of an open connection. The factory would open a connection and stash it on the request the first time it is invoked and then just return the stashed connection on subsequent calls. This preserves the convenience of your pattern but allows you to only cause a connection to be opened for requests that need it. Chris -- You received this message because you are subscribed to the Google Groups "pylons-devel" group. To post to this group, send email to pylons-devel@googlegroups.com. To unsubscribe from this group, send email to pylons-devel+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/pylons-devel?hl=en.
Re: What's the best way to optimize database connections in Pyramid?
In this case it only means "replace the function with its return value as a kind of cache" -- You received this message because you are subscribed to the Google Groups "pylons-devel" group. To post to this group, send email to pylons-devel@googlegroups.com. To unsubscribe from this group, send email to pylons-devel+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/pylons-devel?hl=en.
Re: What's the best way to optimize database connections in Pyramid?
Damn(or hurrah), another new concept to learn - wikipedia: Reification (computer science), making a data model for a previously abstract concept I simply can't wait till the day when a day gets some more hours to it... fanx a lot! jj On Mon, Mar 14, 2011 at 1:02 AM, Michael Merickel wrote: > I've been a fan of tacking the connection onto the request object using > @reify, thus if your request never calls "request.db" it doesn't have any db > overhead. > Thanks to reify, it reuses the same connection throughout that request. If > you want to ensure that the connection is closed at the end of the request, > you can also add a finished_callback to handle that. > class MyRequest(Request): > @reify > def db(self): > # do stuff to get connection either from middleware in previous > email > # or actually create a connection > conn = > # optional: add callback to cleanup after request is complete > def _cleanup(request): > conn.close() > self.add_finished_callback(_cleanup) > return conn > The cleanup and things is handled by repoze.tm2 if using something like > SQLAlchemy with the ZopeTransactionExtension, but otherwise you'll probably > want to handle it yourself. > > Michael > > > On Sun, Mar 13, 2011 at 8:54 AM, Fernando Correa Neto > wrote: >> >> Hi >> >> On Sun, Mar 13, 2011 at 5:54 AM, Seth wrote: >> [snip] >> > How are other people connecting to a db with high-traffic sites using >> > Pyramid and avoiding the static_route re-connection nonsense? Perhaps we >> > could request to get a matched_route object on the NewRequest event >> > object? >> >> I usually try to stick to a connection that is always on environ. >> You'd have a wsgi middleware that always try to get the connection >> from the threadinglocal and expose that under some environ key, say, >> 'conn.mongo'. >> If the middleware fails to get that from the thread, you'd create >> another connection and then re-expose it under the same key. >> >> Regards, >> Fernando >> >> -- >> You received this message because you are subscribed to the Google Groups >> "pylons-devel" group. >> To post to this group, send email to pylons-devel@googlegroups.com. >> To unsubscribe from this group, send email to >> pylons-devel+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/pylons-devel?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "pylons-devel" group. > To post to this group, send email to pylons-devel@googlegroups.com. > To unsubscribe from this group, send email to > pylons-devel+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/pylons-devel?hl=en. > -- med venlig hilsen / with kind regards Jørgen G. Jørgensen. --- -- You received this message because you are subscribed to the Google Groups "pylons-devel" group. To post to this group, send email to pylons-devel@googlegroups.com. To unsubscribe from this group, send email to pylons-devel+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/pylons-devel?hl=en.
Re: What's the best way to optimize database connections in Pyramid?
When failing to get it from the thread is that always because the connection is closed or could it also be because some other thread had locked the previously opened connection? On Sun, Mar 13, 2011 at 9:54 PM, Fernando Correa Neto wrote: > Hi > > On Sun, Mar 13, 2011 at 5:54 AM, Seth wrote: > [snip] >> How are other people connecting to a db with high-traffic sites using >> Pyramid and avoiding the static_route re-connection nonsense? Perhaps we >> could request to get a matched_route object on the NewRequest event object? > > I usually try to stick to a connection that is always on environ. > You'd have a wsgi middleware that always try to get the connection > from the threadinglocal and expose that under some environ key, say, > 'conn.mongo'. > If the middleware fails to get that from the thread, you'd create > another connection and then re-expose it under the same key. > > Regards, > Fernando > > -- > You received this message because you are subscribed to the Google Groups > "pylons-devel" group. > To post to this group, send email to pylons-devel@googlegroups.com. > To unsubscribe from this group, send email to > pylons-devel+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/pylons-devel?hl=en. > > -- med venlig hilsen / with kind regards Jørgen G. Jørgensen. --- -- You received this message because you are subscribed to the Google Groups "pylons-devel" group. To post to this group, send email to pylons-devel@googlegroups.com. To unsubscribe from this group, send email to pylons-devel+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/pylons-devel?hl=en.
Re: What's the best way to optimize database connections in Pyramid?
I've been a fan of tacking the connection onto the request object using @reify, thus if your request never calls "request.db" it doesn't have any db overhead. Thanks to reify, it reuses the same connection throughout that request. If you want to ensure that the connection is closed at the end of the request, you can also add a finished_callback to handle that. class MyRequest(Request): @reify def db(self): # do stuff to get connection either from middleware in previous email # or actually create a connection conn = # optional: add callback to cleanup after request is complete def _cleanup(request): conn.close() self.add_finished_callback(_cleanup) return conn The cleanup and things is handled by repoze.tm2 if using something like SQLAlchemy with the ZopeTransactionExtension, but otherwise you'll probably want to handle it yourself. Michael On Sun, Mar 13, 2011 at 8:54 AM, Fernando Correa Neto wrote: > Hi > > On Sun, Mar 13, 2011 at 5:54 AM, Seth wrote: > [snip] > > How are other people connecting to a db with high-traffic sites using > > Pyramid and avoiding the static_route re-connection nonsense? Perhaps we > > could request to get a matched_route object on the NewRequest event > object? > > I usually try to stick to a connection that is always on environ. > You'd have a wsgi middleware that always try to get the connection > from the threadinglocal and expose that under some environ key, say, > 'conn.mongo'. > If the middleware fails to get that from the thread, you'd create > another connection and then re-expose it under the same key. > > Regards, > Fernando > > -- > You received this message because you are subscribed to the Google Groups > "pylons-devel" group. > To post to this group, send email to pylons-devel@googlegroups.com. > To unsubscribe from this group, send email to > pylons-devel+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/pylons-devel?hl=en. > > -- You received this message because you are subscribed to the Google Groups "pylons-devel" group. To post to this group, send email to pylons-devel@googlegroups.com. To unsubscribe from this group, send email to pylons-devel+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/pylons-devel?hl=en.
Re: What's the best way to optimize database connections in Pyramid?
Hi On Sun, Mar 13, 2011 at 5:54 AM, Seth wrote: [snip] > How are other people connecting to a db with high-traffic sites using > Pyramid and avoiding the static_route re-connection nonsense? Perhaps we > could request to get a matched_route object on the NewRequest event object? I usually try to stick to a connection that is always on environ. You'd have a wsgi middleware that always try to get the connection from the threadinglocal and expose that under some environ key, say, 'conn.mongo'. If the middleware fails to get that from the thread, you'd create another connection and then re-expose it under the same key. Regards, Fernando -- You received this message because you are subscribed to the Google Groups "pylons-devel" group. To post to this group, send email to pylons-devel@googlegroups.com. To unsubscribe from this group, send email to pylons-devel+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/pylons-devel?hl=en.
