I'm curious why a special twisted server is needed to calculate things that should live very naturally within the relational database, i.e. votes, user stats, etc. If calculating these things via SQL is time consuming, those stats can be calculated on a periodic basis (either upon change in data, periodically, etc.) and placed in separate tables (i.e. somewhat like a materialized view but less formal).
My current application calculates some aggregate statistics whenever an end user or administrator submits new data - the logic for this is placed in after_flush() methods which are triggered by a SessionExtension. Regular database triggers can achieve this result as well. If lots of updates happen extremely frequently, a background queue can receive and process statistical updates without weighing down primary request time. The "pre-build the Python object from a DBM file" side of things sounds a little involved to me as well - is the DBM file actually doubling data which is already in the ORM ? If its ultimately just for caching, you'd get better performance just cpickling ORM-loaded objects into memcached. On Oct 5, 9:48 pm, Krishgy <[EMAIL PROTECTED]> wrote: > Hi All, > > We are using sqlalchemy with Pylons project. > > Currently, pylons controllers are directly using the sqlalchemy for > handling the request. Due to our bad query handling approach, contents > and algorithm computation, our application works very slow even for > single user :-(. I can use the indexing, lazy_loading to improve the > performance. > > All our logic are put into the controllers, so our controller code > base increasing rapidly. > > I want to have a separate data manager to manage our web site data. > The datamanager shall be a python module which can use sqlalchemy, > memcached, and/or ( hadoop in later stage). > > Primary responsibilities of the data manager are > > 1. Manage the sqlalchemy to store and retrieve the data, DB connection > pooling > 2. Store the data into the memcached whenever appropriate. > 3. Use our custom twisted server to calculate the site statistics such > as content votes, user reputation, content ranking, personalization > 4. Use our custom twisted server to pre-build the python object > whenever there is a data change in the particular model and save the > python object to the Berkeley database. So we will serve the data > mostly from berkeley database which is ready to be displayed. > > I will post my finding towards this approach here. > > My specific questions are, > > 1. The approach I have taken is good or bad? > 2. Is session = meta.Session() creates a heavy or light weight object? > 3. What is the best way to release the session object returned by > meta.Session() > 4. Can session object handle the connection polling in case our master > database died? Can it fallback to the slave database automatically? > 5. Do we really need a specific data manager? Can you recommend pros > and cons of having a specific data manager module? > > I am sorry, my post is too big... Thank you so much for your help.. > > Thanks & Regards, > > Krish --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~----------~----~----~----~------~----~------~--~---
