[sqlalchemy] Re: DynamicMetaData question
On Mar 13, 2007, at 4:26 PM, Jonathan LaCour wrote: Michael Bayer wrote: the only change I would favor here would be to merge connect into MetaData, BoundMetaData and DynamicMetaData stay around for backwards compat for probably forever, and perhaps we add another method called connect_threadlocal() or something like that for people who want that behavior. i would like to have just one object that does the whole thing, now that some of the early FUD has subsided. To be clear: this is exactly what I want. I like the concept of metadata I just don't like the fact that we have two objects (BoundMetaData and DynamicMetaData) that are very different in terms of how you use them when they don't need to be. Putting 'connect' into MetaData will solve my complaint nicely, and I can safely ignore the craziness of DynamicMetaData and redundancy of BoundMetaData for the rest of time. OK, let me tell you what just happened the other day. Im dealing with a Pylons application, and Pylons provides the SA engine by binding it to the Session. but the application also had a DynamicMetaData stuck in there, and at some point they were creating their own engine and connecting it to the DMD. needless to say I quickly got uber-confused as the app was running with *two* engines, which happened to point to the same database, but still completely weird. So i fixed it. But by changing the DynamicMetaData to just plain MetaData, i was then *sure* that no other part of the app was trying to sneak a connect() on there. Whereas if we only had one kind MetaData I could not rely upon that. Not sure if that justifies the existence of DMD since its Python, things are dynamiclaly typed, theres an endless number of operations that you cant really guard against. But was just a moment that I felt thankful that there *were* two versions of MetaData. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: DynamicMetaData question
Michael Bayer wrote: OK, let me tell you what just happened the other day. Im dealing with a Pylons application, and Pylons provides the SA engine by binding it to the Session. but the application also had a DynamicMetaData stuck in there, and at some point they were creating their own engine and connecting it to the DMD. needless to say I quickly got uber-confused as the app was running with *two* engines, which happened to point to the same database, but still completely weird. So i fixed it. But by changing the DynamicMetaData to just plain MetaData, i was then *sure* that no other part of the app was trying to sneak a connect() on there. Whereas if we only had one kind MetaData I could not rely upon that. So you are saying you got uber-confused because of DynamicMetaData? Thats even more reason to not use it :) If it confused you, its sure to confuse me (as it already has before)! Just joking around... Not sure if that justifies the existence of DMD since its Python, things are dynamiclaly typed, theres an endless number of operations that you cant really guard against. But was just a moment that I felt thankful that there *were* two versions of MetaData. I see your point. I don't care what you do with DynamicMetaData, as long as I can do this one day: metadata = MetaData() engine = create_engine(...) metadata.connect(engine) ... preferably soon ;) -- Jonathan LaCour http://cleverdevil.org --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: DynamicMetaData question
On Mar 21, 2007, at 3:55 PM, Jonathan LaCour wrote: I see your point. I don't care what you do with DynamicMetaData, as long as I can do this one day: metadata = MetaData() engine = create_engine(...) metadata.connect(engine) ... preferably soon ;) right..but just my point is, since MetaData doesnt *have* a connect() on it, thats what gave me the warm and fuzzy in that scenario. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: DynamicMetaData question
Hey list, are you confused by the current system ? Please let me know. the only change I would favor here would be to merge connect into MetaData, BoundMetaData and DynamicMetaData stay around for backwards compat for probably forever. Not actually confused by the system. Since I need to define the connection parameter in a seperate file (config file), I *need* the DND.connect() method. So, DND is the only I use... The global_connect() is not useful for me since I need multiple datasources. Again, beeing able to define connection parameter later is crucial, so having a MetaData.connect() would be great. but I need to get at least 20 or 30 users on this list telling me how they are using metadata. A typical initialization of my models looks like: _metadata = DynamicMetaDate(I'll be bound later...) one_table = Table(blabla,_metadata, ...) _engine = None def init_model(dburi,**kwargs): global _engine if _engine: # Engine already defined # check if metadata bound if _metadata.is_bound(): # nothing to do pass else: # need to connect metadata. # occurs when starting new thread _metadata.connect(_engine) else: # first we access the model, init everything _engine = sa.create_engine(dburi,**kwargs) settlement_metadata.connect(_engine) BTW, I'd posted on the TG list to know how they handle multiple datasources with SA. Answer was for now, only one datasource [probably due to global_connect], we'll see next release). So maybe global_connect will not even be used anymore by TG... Cheers Seb -- Sébastien LELONG sebastien.lelong[at]sirloon.net http://www.sirloon.net --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: DynamicMetaData question
I think its more confusing for the API to break backwards-compatibility every 5 or 6 releases. Also i think adding a whole new class DelayedMetaData, which is literally just to avoid passing a flag, well i wont say insane but its a little obsessive. Agreed here. Hey list, are you confused by the current system ? Please let me know. Count me as one of the confused users. I must admit I only read through the docs quickly (but don't we all do?) and missed the mention of the threadlocal nature of the DynamicMetaData, which I thought only provided a way to delay connecting it to an engine, which was the feature I needed. the only change I would favor here would be to merge connect into MetaData, BoundMetaData and DynamicMetaData stay around for backwards compat for probably forever, and perhaps we add another method called connect_threadlocal() or something like that for people who want that behavior. i would like to have just one object that does the whole thing, now that some of the early FUD has subsided. +1 for the connect method on MetaData but I need to get at least 20 or 30 users on this list telling me how they are using metadata. -- Gaëtan de Menten http://openhex.org --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: DynamicMetaData question
can I have some feedback from the list who is using the thread-local capability of DynamicMetaData, and/or are using global_connect() ? (another feature i didnt like but it was me bowing to early pressure from TG users). Probably this will be no surprise, since I contributed most of the original ProxyEngine. ;) I use DynamicMetaData exactly as you described the main pylons/TG use case: I define tables at the module level, using a DynamicMetaData instance, and then call meta.connect(uri) at the start of each request. For me, this is the most sensible way to handle things in a WSGI-friendly web app. Minimal overhead, maximal flexibility. meta.connect(uri, threadlocal=True) or meta.connect_threadlocal(uri) would be ok, but I think worse as an API since they put too much responsibility on the caller and cause too much repetition. JP --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: DynamicMetaData question
On Mar 13, 11:59 am, Gaetan de Menten [EMAIL PROTECTED] wrote: I only discovered (or at least understood) this thread localness of DynamicMetaData, and honestly, I don't understand in what case it can be useful. It seems like the thread localness is limited to the engine connected to the metadata. So what I'd like to understand is when anyone wouldn't want to use a global engine? As long as the connections are thread-local, we are fine, right? Not me -- I have cases where the same schema is used with two different databases by two different apps (eg, admin uses a main postgres db and public uses a local sqlite cache), so the current API does what I need, but (if I understand what you're saying) only allowing the connect string of a global engine to vary per request, rather than allowing completely different engines, would not. u said different apps - or different threads? in anycase, would it be better if all metadata setup stays in a func with a parameter the metadata - thus u can call it to setup over diff. pre-made metadatas? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: DynamicMetaData question
On 3/14/07, JP [EMAIL PROTECTED] wrote: On Mar 13, 11:59 am, Gaetan de Menten [EMAIL PROTECTED] wrote: I only discovered (or at least understood) this thread localness of DynamicMetaData, and honestly, I don't understand in what case it can be useful. It seems like the thread localness is limited to the engine connected to the metadata. So what I'd like to understand is when anyone wouldn't want to use a global engine? As long as the connections are thread-local, we are fine, right? Not me -- I have cases where the same schema is used with two different databases by two different apps (eg, admin uses a main postgres db and public uses a local sqlite cache), so the current API does what I need, but (if I understand what you're saying) only allowing the connect string of a global engine to vary per request, rather than allowing completely different engines, would not. Yeah, I figured in the meantime that this case could cause problem. The point I have is that this case is _in my opinion_ much less common than the case where you need to delay the setup of your tables and use the DynamicMetaData for that. So I think it would have been better to have a DynamicMetaData with threadlocal=False by default. If that were the case, people would not have to connect the metadata to the engine in each and every thread. Now it's too late if we don't want to break backward compatibility but simply adding a connect method to the standard MetaData, as Michael proposed, would suit my needs just fine. Anyway, I've already rambled too much on this issue... Thanks for having taken the time to answer my question. -- Gaëtan de Menten http://openhex.org --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: DynamicMetaData question
[...] I couldn't use BoundMetaData because I don't know the connection parameters until much after import time, so I am using the only other option I know of, which is DynamicMetaData [...]. I have exactly the same problem. One option is to use global_connect and default_metadata (doc: Using the global Metadata object), but as my apps are essentially multi-sources, I need to define datasources separatly. And I definitly cannot use the BoundMetaData and hardcode the connection parameters... So I'm using the DynamicMetaData and its connect method. Seems to be two independant features in this DynamicMetaData... Seb -- Sébastien LELONG sebastien.lelong[at]sirloon.net http://www.sirloon.net --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: DynamicMetaData question
I only discovered (or at least understood) this thread localness of DynamicMetaData, and honestly, I don't understand in what case it can be useful. It seems like the thread localness is limited to the engine connected to the metadata. So what I'd like to understand is when anyone wouldn't want to use a global engine? As long as the connections are thread-local, we are fine, right? On 3/12/07, Michael Bayer [EMAIL PROTECTED] wrote: Er well the whole point of DynamicMetaData was to replace the old ProxyEngine, which was intended to emulate SQLObject's i dont know what its called object which is what TG was using for thread-local context. Also, it doesnt entirely make sense that the threadlocal engine strategy would conflict with dynamicmetadata's thread local-ness. if you have one engine per thread, and that engine is on the tlocal strategy, it should still be doing its thing within the one thread that its used within. so feel free to try to reproduce that in a ticket or something. but also, the threadlocal engine strategy is the thing here thats kind of like, you probably dont need to be using it...its basically SA 0.1's built in behavior kind of ripped out and put over to the side, where people that really want that sort of thing can use it. but ive downplayed it a lot since then, since its kind a confusing feature if youre not the person who wrote it (or read/understood the entire source to it). whereas DynamicMetaData i think is in pretty wide usage as a thread local construct and its pretty straightfoward. the non-threadlocal use case for it is not as obvious to me. On Mar 12, 2007, at 4:58 PM, Jonathan LaCour wrote: Random question for the list, and an idea. I have an application I am working on that needs to be able to dynamically bind its metadata to an engine based upon configuration. Logically, it seems I should use `DynamicMetaData` and just call metadata.connect(engine) after I have loaded my configuration. However, I had written all of my code depending upon a threadlocal strategy as defined by using `strategy='threadlocal'` in my `create_engine` call. It turns out that DynamicMetaData has threadlocal behavior by default as well, and somehow these two things conflict. My problem was solved by making sure to pass `threadlocal=False` to my DynamicMetaData constructor. Now, here is my question: why does DynamicMetaData have any threadlocal behavior at all? It seems like the primary reason one would use a DynamicMetaData would be for being able to delay the binding of your engine to your metadata. The fact that its threadlocal is easy to miss, and I don't see why it has any threadlocal behavior at all. Am I missing something? Wouldn't it be better to two separate MetaData types, one for dynamically binding your engine, and another for threadlocal metadata? -- Jonathan LaCour http://cleverdevil.org -- Gaëtan de Menten http://openhex.org --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: DynamicMetaData question
Michael Bayer wrote:
My controller actions don't worry about transactions or connections
at all, they just execute insert/update/delete/select actions and if
an exception is raised, their work is rolled back automatically.
well, thats exactly the pattern provided by strategy=threadlocal,
and its clear that you understand how it works, so there is no problem
at all doing that. its merely a pattern that allows a little less
explicitness (i.e. more magic).
Ah, okay, so I am not missing anything here, just appreciating this
particular piece of magic :)
people were quite confused by it when it was the built-in behavior,
which is because a lot of people dont really understand what
threadlocal means. so i made it all optional. no need to deprecate
it though, its still useful stuff.
I am surprised that people don't know what threadlocal means, since
it seems like a threaded application with a database pool is probably
one of the most familiar ways to do things. But, still, if people were
emailing the list in confusion, thats got to mean something :)
I must still not understand the appropriate usage pattern for
DynamicMetaData. I couldn't use BoundMetaData because I don't know
the connection parameters until much after import time, so I am
using the only other option I know of, which is DynamicMetaData.
The fact that it provides threadlocal behavior only caused me a
headache, because I would get errors unless it was disabled.
the main use case for DynamicMetaData is a single application, usually
a web application, that is actually many instances of the same
application running at once and talking to different databases for
each instance. so on each request, the DynamicMetaData gets pointed to
the official database connection for that request, within that thread.
Wow, does this ever actually happen? It seems like a very obscure use
case to me.
You are free to use it the way youre using it too. if i were writing
the app that didnt know the connection parameters until later, i
might just put my entire create the Tables + mappers logic within a
function call that gets called when we do know the actual connection
string...
... eek!
but then again using DMD with threadlocal turned off is yeah a lot
easier and cleaner since you can keep your declarations at the module
level.
Yes, this is a lot cleaner, however its confusing. Personally, I
think that there should be one and only one MetaData class that can be
told how to act. As far as I can tell, you could replace the existing
options in SQLAlchemy (BoundMetaData, DynamicMetaData) and simplify
things a bunch in the process:
metadata = MetaData()
engine = create_engine('connection_string')
metadata.connect(engine)
This would let you get rid of BoundMetaData and DynamicMetaData and just
have one simple, easy way to do things. Plus, it would give you the
ability to do what I am using DynamicMetaData to do without having to
pass in a weird threadlocal=False kwarg.
For people who want the insane threadlocal behavior of DynamicMetaData,
you could have the MetaData object take in the same kwarg which would be
defaulted to the saner False option (unlike the current DynamicMetaData)
metadata = MetaData(threadlocal=True)
...
I think that this would be a lot less confusing for users. I would
be more than willing to work on a patch to make this happen if you
like the idea. If not, I will at least supply a patch for yet another
MetaData subclass called DelayedMetaData that gives you the behavior of
DynamicMetaData without the threadlocal insanity :)
What do you think?
--
Jonathan LaCour
http://cleverdevil.org
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: DynamicMetaData question
What do you think? Hey list, are you confused by the current system ? Please let me know. the only change I would favor here would be to merge connect into MetaData, BoundMetaData and DynamicMetaData stay around for backwards compat for probably forever, and perhaps we add another method called connect_threadlocal() or something like that for people who want that behavior. i would like to have just one object that does the whole thing, now that some of the early FUD has subsided. but I need to get at least 20 or 30 users on this list telling me how they are using metadata. i am not doing a plain web app (although it will have http interface and behave like web app in some ways - but this is just _another_ remote UI), so i dont really know the difference of current *MetaData's. Even if well documented, the usage of these will go deep in the infrastructure and hence be forgotten and stay hidden - its not everyone's business to create db connections and alike. And for this, having 2 different strategies, which differ in some _hidden_ ways, and which usage is always invisible... i vote for having one type of object - or one factory if u want - with parameters. Only who should know about diffs would have to. so far i have only noted that certain cases work with premade BoundMetaData() and break with plain MetaData() being later bound. i can try dig them out if u want. btw theoreticaly, can u have a XXXMetadata obj that is used, closed and later turns into YYYMetaData? i can see a use case for this and it would not be easy possible with current system. In the new variant one can keep forever the metadata as made once, and just change the way it is connected, then disconnected. Hey, is this a sort of usageContext() ? i.e. u have a MetaData(), which .connect(args-here) returns some context-like object which hides the exact implemetation, and when dropped will close connection. ciao svilen --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: DynamicMetaData question
Er well the whole point of DynamicMetaData was to replace the old ProxyEngine, which was intended to emulate SQLObject's i dont know what its called object which is what TG was using for thread-local context. Also, it doesnt entirely make sense that the threadlocal engine strategy would conflict with dynamicmetadata's thread local-ness. if you have one engine per thread, and that engine is on the tlocal strategy, it should still be doing its thing within the one thread that its used within. so feel free to try to reproduce that in a ticket or something. but also, the threadlocal engine strategy is the thing here thats kind of like, you probably dont need to be using it...its basically SA 0.1's built in behavior kind of ripped out and put over to the side, where people that really want that sort of thing can use it. but ive downplayed it a lot since then, since its kind a confusing feature if youre not the person who wrote it (or read/understood the entire source to it). whereas DynamicMetaData i think is in pretty wide usage as a thread local construct and its pretty straightfoward. the non-threadlocal use case for it is not as obvious to me. On Mar 12, 2007, at 4:58 PM, Jonathan LaCour wrote: Random question for the list, and an idea. I have an application I am working on that needs to be able to dynamically bind its metadata to an engine based upon configuration. Logically, it seems I should use `DynamicMetaData` and just call metadata.connect(engine) after I have loaded my configuration. However, I had written all of my code depending upon a threadlocal strategy as defined by using `strategy='threadlocal'` in my `create_engine` call. It turns out that DynamicMetaData has threadlocal behavior by default as well, and somehow these two things conflict. My problem was solved by making sure to pass `threadlocal=False` to my DynamicMetaData constructor. Now, here is my question: why does DynamicMetaData have any threadlocal behavior at all? It seems like the primary reason one would use a DynamicMetaData would be for being able to delay the binding of your engine to your metadata. The fact that its threadlocal is easy to miss, and I don't see why it has any threadlocal behavior at all. Am I missing something? Wouldn't it be better to two separate MetaData types, one for dynamically binding your engine, and another for threadlocal metadata? -- Jonathan LaCour http://cleverdevil.org --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
