[sqlalchemy] Re: whats going to break in 0.4
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Jul 5, 2007, at 6:00 PM, Michael Bayer wrote: On Jul 5, 5:46 pm, Barry Warsaw [EMAIL PROTECTED] wrote: What I mean by that is that Mailman has at least three separate related collections of data: mailing lists, users, and messages. It should be possible to put each of those three in separate databases using three different engine urls. The classic use case is this: say my user database lived in my web application, but that web app was separate from the mailing list system, and the data for the lists lived in a separate database. It should be possible for Mailman to get list configuration data from database B and user data from the web app's database A. Similarly, you might want to put the message storage in database C, say the one that your archiver used. Of course, this means that you can't design your data model to have foreign keys across these three storages, but that's not hard, though you have to manage consistency at the application layer. I'm not sure such a design is actually feasible with SQLAlchemy though; I think it wasn't back when I actually tried to do this ages ago. its quite feasable, particularly since you are separating concerns at the table/class level (as opposed to the row level, which is the sharding thing ive been talking about). there are three general approaches to this: Thanks very much for the information Michael. I'm stashing this message away for a tasty later meal. :) - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRo588XEjvBPtnXfVAQLNfwP/QK1dhl4DmfE9C5jm49aHDn9rjkBc0f/e EiA6ZGF0189soweQHL80zBD+Ug5kIk7DsGKPtG3kZ+O4n25t3jGrdUmUezK2PUet RoxnftZx0sCVKYGWvrG4DdPpKfPi0VgIs3KpFVhCG6auCq6J2B6yw7PoNGa6Blpq g2gqfUy+lYs= =bRN1 -END PGP SIGNATURE- --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: whats going to break in 0.4
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Jul 3, 2007, at 8:14 PM, jason kirtland wrote: Barry wrote: I have exactly the same architecture and exactly the same question. In fact, my goal in Mailman 3 is to be able to let sites configure the system to use any supported database backend, just by tweaking the configuration variable that specifies the engine url. We'll ship with SQLite, but it would be awesome if I didn't have to do anything else to 'automatically' support PostgreSQL or MySQL, etc. Although I haven't tried it with these other backends, it currently works great with alternative SQLite database file locations (such as the tempfile one I use during a test suite run). The regular MetaData gives you this configuration flexibility for a given installation. Cool. If you want to support simultaneous and distinct 'configurations' within a threaded process, each with its own set of database tables (possibly mixing backends as well), then a DMD is perfect. Every thread connects the DMD to the engine of its choice before work starts. In a Mailman context I could imagine a single fat worker process at an ISP that serviced lots of domains, each owned by a different user with separate data storage. Possibly, although I'm not thinking about that level of division. Once thing I /would/ like to be able to do is to connect to different databases for each 'storage domain' within a single process (Mailman itself will continue to be single threaded). What I mean by that is that Mailman has at least three separate related collections of data: mailing lists, users, and messages. It should be possible to put each of those three in separate databases using three different engine urls. The classic use case is this: say my user database lived in my web application, but that web app was separate from the mailing list system, and the data for the lists lived in a separate database. It should be possible for Mailman to get list configuration data from database B and user data from the web app's database A. Similarly, you might want to put the message storage in database C, say the one that your archiver used. Of course, this means that you can't design your data model to have foreign keys across these three storages, but that's not hard, though you have to manage consistency at the application layer. I'm not sure such a design is actually feasible with SQLAlchemy though; I think it wasn't back when I actually tried to do this ages ago. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRo1mu3EjvBPtnXfVAQLI1AP9EaNt/rlYtcwix/PEPU2OyETBlIKGwIhv g/QeKoMonD0kYwGuF7Y1RS+3xGch7UQAqDE5z1bU7bKJEZoBUQaUkhtqVSRgB5rX Bii8icIX3iephIgSIcPixLJ+V3yv2FcUE1JOjvD8puudMXQvF8WmN046OqnRAuXb m1ACR5gxysQ= =i0ep -END PGP SIGNATURE- --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Concurrency problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Apr 3, 2007, at 9:35 AM, Michael Bayer wrote: OK, yes, im sorry about the lack of docs for isnew...sometimes i subconsciously want to see how long it takes for someone to ask me about something (bad habit) although in this case that flag is over a year old and i think youre the first :) Heh, um glad to help? :) so basically, isnew means populate attributes and if its false it means we already did this entity. Cool, thanks. That tells me what I need to know! - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRhKhIXEjvBPtnXfVAQIJAAP/XRTViMrga4tOECEJD0OkacjfcUCVQOV3 lDUTPtu09se0uiPlZvfW03w7BBniguupLFSUOPBQHlf/Zh3SWG585lF9HeNP0ocC viBepxS+2rm1Y3CZVnXvfNwZYUfpn1vwns2+cOFoHnwR8scdexK9UTAG7yIH/ICs IzNwbKB/g7U= =bDwL -END PGP SIGNATURE- --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Concurrency problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello SQLAlchemists, Over in the Mailman project, we've hit a concurrency problem that we don't know how to (properly) fix. Any suggestions you might have would be greatly appreciated. The full thread starts here: http://mail.python.org/pipermail/mailman-developers/2007-March/ 019437.html In summary, Mailman has several processes that all need to access the SA database layer. For example, we have a number of queues that process messages, we also have an LMTP process and a wsgi process. Each one of these will load a MailList object from the db. the MailList has a bunch of SA attributes, but it also has some non-SA attributes, such as __lock, which contains a pointer to a non- persistent LockFile object we use to serialize access to the MailList objects. The problem is that a change to a MailList in one process is not automatically viewable in another process. We've tried several approaches, including using the mapper attribute always_refresh, or calling session.refresh() or session.expire() at our reload boundary points. We've gotten things to work, but it's very fragile. Currently, we call session.expire(mlist) during our Lock() call and this does seem to work, but it's not the right place for us. Our Lock () will call our Load() method, and this is really where we want to ensure that the MailList object gets sync'd with its new state. Internally, we have MailList.Load() calls situated at the points where we want to update the MailList state, and we'll call that even if we only need read-only access to the MailList object. Currently we only call .Lock() if we need write access as well. The problem is that if we call session.expire(mlist) /after/ the MailList.__lock attribute gets set, it appears as though SA 'messes' with this attribute. I've proven that if the session.expire() gets called after MailList.Lock(), during the .Load() we lose our lock, but if we call session.expire() before we're locked, everything's fine. I'm not 100% sure yet why that happens, but it does appear to be a strange interaction between SA's identity map and non-SA controlled attributes. So, I'm looking for some advice on how to handle synchronization between processes in a situation like this. Ultimately I think I'd like to use mapper's always_refresh, but maybe that's not the right thing. The related issue is that MailList's may have other ORM objects hanging off of it and they'd all need to be re-sync'd with the database at the same time. Our MailList objects hang around for a long time, and so we already have the resync locations coded (essentially MailList.Load() calls). Those are the points where we'd like to expire our in-memory objects, though we can't lose the state of our non-SA controlled attributes. Thanks for any advice (or request for more information!) you can give. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRhFBcXEjvBPtnXfVAQJToAQAnQRHwsuCeWj+Brfnbtj/NDQmv7wblLJ+ lrppxHmv/1LAo1t7xqhVIzOTvdez+icBy+FtrzPkpvk8aXXKtc9hvP4BogrjmjOB H/VuQxhX2ivuR/gb6BPAR3WifMG3nF9T9aMz6ejbQH3Sn8fcyWrPlyn4brnq9P6y f8gGDN7N4Wk= =WK4w -END PGP SIGNATURE- --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] import logging
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Does sqlalchemy intend to expose a logging module that isn't Python's standard logging module when one does from sqlalchemy import *? AFAICT, this isn't supposed to be part of the public interface of the top level sqlalchemy package so it seems bad form to do this, but maybe I'm missing something. Cheers, - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRa0vpXEjvBPtnXfVAQJuQwP/Z2Z4JURBobp+ri7vt7gDZlIjRJaDf3yh 9WhVpRIfMxWUiXTBpXkYlFsB2Kz4bYKRQud1WQGURxNVu9kggbGfeUAEF8R9ds/1 QURXI9SIQbzZvf0WPyh6DBwZKLbV5cLX+JGYisIVBoxkdRKhk6H4PQe2Wlr7Hlrl 26XatqvuZiA= =N/vd -END PGP SIGNATURE- --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: import logging
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Jan 16, 2007, at 3:18 PM, Michael Bayer wrote: it does not intend to, nor does it intend to export types which may also be happening. im not exactly sure how logging is getting into the actual list of * exports. I haven't looked at the sqlalchemy code but it definitely seems to be. I tracebacked when I tried to call logging.getLogger(). Does the top level module use an __all__? but also, why are you using import * ? Probably because I think that's what the tutorial uses, but yah, it strikes me as icky. It's also unnecessary -- it was easy enough to restrict that to just BoundMetaData and create_session at least for the one file that was biting me. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRa1VM3EjvBPtnXfVAQJyJAQAofML5ZmXIa3jYQyiwpVZCFst4VEHb6XY WVJzCunGYSOYCDdJLyu94L6iFA24ZxRQ/8oQMPpEM0wmU41aj/Q7SzNYZ4YH00N7 su80iGU61WF9wVg0Nh8FNMZD9etwHvNoBNOHRuL8HrFoT2fu3MvO54gjyxBXOqsc StKxCINh3no= =Q9Ds -END PGP SIGNATURE- --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Mailman trunk now backed by SQLAlchemy
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi all, I thought I would let you know that I've committed a bunch of changes to the Mailman svn trunk that integrates a SQLAlchemy storage for all mailing list data. My goal was to be able to make this change without disrupting the basic architecture and current code base, and I think I've largely accomplished that. Our trunk is far from ready for prime time, but I've been very happy with how easy SQLAlchemy is to use so far. Right now I still use a bunch of PickleType columns for non-basic data types (e.g. attributes which contain lists, dicts, instances, etc.). I plan on incrementally moving each of those to their own separate tables. Having the pickle columns has really eased the transition though. Good stuff! - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRZbuTnEjvBPtnXfVAQIOzQP+ItkfrRhO50FEtv+OKSB1IGILpqKgDtcU Rbdod3Qv3b+hAAKtf32JSperDAltxj1fy6yoqKE8Uxkq5UIWVNTgtkye1GXIINM8 s32aqdOGmDS5LAGZtCUMakNaY33QnoQKSPZ6A+2hHrd4ZCGCQf2F10SdLqnzSY8J scG+aggm6GY= =5mUv -END PGP SIGNATURE- --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Mailman trunk now backed by SQLAlchemy
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Jan 1, 2007, at 3:24 PM, Michael Bayer wrote: I may put a note on the SQLAlchemy site about this, its big news ! Please feel free Mike, and thanks! - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRZlwgHEjvBPtnXfVAQJ8mAQAuhT29tlRdi+7pZxasWn/aSfUe29SS4s1 B+qvOtI/mV2smsu5yIsS1BX+zNi3xJn3jeK963I6joHKM2d3CzK5+V+zAFzO7Sr4 JNdrdBy5liVcay/E9C+hBCdEFzygPR+7TPAOxCDmVNjh+KEcFYiohf3fthisRY9N WDkyKaBEG+c= =GTnV -END PGP SIGNATURE- --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
