Hi Robert, I think it's great that you are focusing on performance. And while database used to be the main contention point in the past, my feeling is that it is not so anymore. I.e. we know how to fix DB performance issues, and many remain simply because of the lack of time to go through all of them. I might, of course, be mistaken, and people don't really have the same experience tackling performance problems like we do in Translations. :)
The discussion you are starting is very useful, because we should ultimately be able to design APIs we like while not sacrificing performance. Today, in Launchpad, you usually have to come up with ugly APIs to accommodate performance solutions you are employing. У чет, 12. 08 2010. у 15:34 +1200, Robert Collins пише: > I have some long term ideas about this: broadly, to separate out > 'looks like python' from 'looks like it does DB access' so that its > obvious when db access is going to happen, and structure things so > that: > - if it looks cheap, it is cheap - always > - no database access is needed during template rendering (as a safety > measure for the first point : templates look cheap always) > - set based, fast behaviour is easy and the default It would be very nice to come up with a sane approach for these kinds of things which doesn't include methods named like "getPOFilesAndPOTemplatesAndPersons", which is roughly the manual approach we are taking today (i.e. getting a ResultSet containing all objects we care about in one single query). My opinion is that we could just start with Free's work on Storm (cool stuff, Free!), expand it so it works well even with cases like ubuntu_coc (though, it seems to me it already supports a lot of it, and it does support more than one argument prejoin), and just come up with a coding convention in Launchpad to express pre-joins in a generic way. If that means starting with something simple like model API methods defined like: def getPOTemplates(populate=[POTemplate.sourcepackagename]) that's probably good enough for now (i.e. standardize on the argument name, whatever it is — "populate" is just an example — and always make it a column list or something). However, even when we use the optimal DB path, we have problems with the rendering time itself. Check out https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1685EB1355 for an example. That is 275ms of SQL time (and no, it's not a bug in the OOPS tools: that is the SQL time), and ~17s of rendering time, and we believe fmt:url for each object is one of big culprits there (well, removing it cuts the rendering time to 5s :). However, to avoid doing it, we'd have to do something like getPOTemplatesAndSourcePackageIfNeeded(), so generalizing "prejoins" using Storm would help here as well, but ultimately, we need to fix fmt:url to not take this long. Granted, this is rendering ~2000 objects and doing fmt:url on each one of them, and we can argue that it's not a good idea (there are reasons why we are putting it all on one page). We knew that DB could handle it just fine, but we didn't expect the Python code to be this slow (to be honest, initial version of this did even more fmt:urls and even used menu:navigation which caused the page rendering time to go into hundreds of seconds). I know there are plans to fix/look into many of these things (I've also talked it over with Gary during the Epic), however, I somehow feel that you are neglecting this entire area of Launchpad performance. I.e. even if we went and fixed the DB side of things for pages like https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1685EC1691 we'd still be left with >5s of generation time on the appserver. (If it was only pages like these, I would just say "let's worry about it later", but because of the pages like the first one, this is an actual problem that we hit right after we resolve all the DB issues on a single page) Cheers, Danilo _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
