Re: Using lp:udd beyond its original purpose
There are two big changes that aren't necessary for us, but would help a lot: 1. Decouple lp:udd from the package-import.canonical.com production deployment 2. Split the package scanning and the branch importing parts of lp:udd into separate projects That sounds good to me, especially #1. I would much rather have the splitting merged back upstream than to have the tree forked. -- Martin -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Using lp:udd beyond its original purpose
On Thu, Dec 8, 2011 at 7:47 AM, Vincent Ladeuil vila+...@canonical.com wrote: Jonathan Lange j...@mumak.net writes: snip/ and then split out the functionality you need. Does that make sense, or do you want to do it the other way around? Either way seems right to me. I guess if we split out the stuff we need, that means less work for you. You may even *import* the stuff you need. This way we should still be able to merge in both directions on the common parts if needed. Yeah, that's the plan. Split out something called, say, lp:package-scanner and both lp:udd lp:pkgme-binary would import that. But until I better understand which stuff you want to reuse, it's hard to say. Anything required to scan packages. This means mass-import, list-packages, add-import-jobs and any Python logic they depend on. In particular, the Launchpad connection management code and some the code to manage the database. Yes, the database. While pkgme-binary has its own database, that's for storing information about package symbols. lp:udd uses the database to manage jobs in the abstract, and thus lp:package-scanner will need to do something like that. We don't want import-package, and we *probably* don't want the failure categorization, or any of the logic for running a web service. ... Are you sure you don't want to define your own mass_import script ? The current design separate the subprocess/thread management (in udd/threads.py) from the db usage which is defined in mass_import itself. Quite sure. We'd only rewrite known working code if something else broke. In this case, you can simply *use* lp:udd and modifying it should be the exception rather than the norm. Well, that's kind of what we're doing. Except we've already had to modify lp:udd once or twice already, and have two more changes that we know we want to do now (split out the config; add a mode for scanning binary packages instead of source). Well, you'd have to add a dependency and manage deployment a little differently, but that's it. Care to elaborate on that ? The deployment is already pretty manual so one more step may not be an issue but I'm more concerned by the fallouts introduced by changes the package importer don't need at all. If we split the config into a separate branch, then you'll need to manage your production deployment differently. I honestly don't know how you do it now, but you'd be managing two branches rather than one, so there's going to have to be *some* difference. If we split out the package scanning code, then you'll have a dependency. Dependencies also make production deployment different. jml -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Using lp:udd beyond its original purpose
On Fri, Dec 2, 2011 at 2:21 PM, Jelmer Vernooij jel...@canonical.com wrote: On 12/02/2011 03:18 PM, Jonathan Lange wrote: We're in the process of deploying lp:udd to do something that has very little to do with Ubuntu distributed development. We're using it to maintain a local database mapping from object files to symbols, for our own nefarious purposes[1]. It works well for this, but there are some changes that we will want to make soon. There are two big changes that aren't necessary for us, but would help a lot: 1. Decouple lp:udd from the package-import.canonical.com production deployment 2. Split the package scanning and the branch importing parts of lp:udd into separate projects It sounds scary when I write it like that, but I don't think it's actually that much code. Neither of these things has to be done – we'll figure something out – but I reckon these changes will result in less code that's better tested. What say you? That seems reasonable to me. Can you explain a bit in what way you're using lp:udd ? Sure. We're using it to maintain a local database mapping from object files to symbols. Instead of running 'import-package', it runs a script in lp:pkgme-binary called 'fetch-symbol-files'. That script then fetches the package and updates our own database with its symbol information, if any. AIUI, we'll be using it with a cronjob set up much like the one on package-import.c.c. It seems to me like we should keep lp:udd about the package importer, and then split out the functionality you need. Does that make sense, or do you want to do it the other way around? Either way seems right to me. I guess if we split out the stuff we need, that means less work for you. Well, you'd have to add a dependency and manage deployment a little differently, but that's it. jml -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
Re: Using lp:udd beyond its original purpose
(re-send) On 8 December 2011 07:13, Martin Pool m...@sourcefrog.net wrote: There are two big changes that aren't necessary for us, but would help a lot: 1. Decouple lp:udd from the package-import.canonical.com production deployment 2. Split the package scanning and the branch importing parts of lp:udd into separate projects That sounds good to me, especially #1. I would much rather have the splitting merged back upstream than to have the tree forked. -- Martin -- ubuntu-distributed-devel mailing list ubuntu-distributed-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-distributed-devel
