> There is no way to get progress data out, AFAICS. I can feed the progress through pipe to yum, and handle it in getIdleProcess(). (multiplexing progress with errorlevels).
> The API both returns before what you've requested is downloaded _and_ can block for an indeterminate amount of time. Never thought of this as a problem. The blocking is necessary to have a simple 1-pass API + to limit the number of spawned downloaders. The progress display and proper timeout handling in the download helper should prevent being blocked for too long. > The downloaders are global, so one downloader might talk to rpmforge.org and redhat.com ... They're not global, each repo has a separate pool of downloaders. I think it makes sense to treat repositories independently, e.g. repo A should not care how many processes download from repo B. Since every downloader is started in particular repo's package directory and never chdirs, It could chroot later. > also means keepalive is going to be interesting. Keepalives and chroot should work fine with a proper download helper that doesn't spawn new process for each URL. > This uses select directly instead of poll, is there some reasoning? asyncore next (might be the best yet ?:). The select() loop only considers active downloaders for a particular repo. I guess the usual number of downloaders is going to be very small, (usually < 5) as large number of processes ruins connection: keep-alives. So using more efficient (but more complex to set up) interfaces as poll() or epoll() is (imho) not necessary. > If you are stuck trying to solve "the big problem" all at once, and are sending out this is an update of where you are atm. ... I can understand, but you'll probably go crazy trying to do it that way (and maybe take me with you ;). Hope not ;) > My suggestion would be to solve a small part of the problem fully. Eg. get an "almost 100%" patch for urlgrabber.grab() that spawns a single process, does the download and returns progress info. Then when we've That's doable, sure. We'd have downloads in a separate process, with 99% compatible semantics. But because of the blocking API, this can't be easily parallelized later. > got that, we can start from that base so it can run 2 procs. at once ... then ... eventually the pain needed to get it integrated into yum. You mean, define a new API (on top of the old one), that allows parallel downloading, and wrap it further up to the point when it's usable in rpms/drpm's download code, and in metadata download code? I considered taking the route the other way round. Anyway, all the stuff in-between must be reimplemented, and that's what bothers me. I can write a new, straightfdorward code, implementing only the necessary features. Or, I could try to reuse and patch the old code, keeping all the seemingly unused bits, and features-to-be. That's to be discussed I think.. -- Zdenek _______________________________________________ Yum-devel mailing list [email protected] http://lists.baseurl.org/mailman/listinfo/yum-devel
