Re: does aptitude really need to lock the status database when downloading?
Stanislav Maslovski stanislav.maslov...@gmail.com writes: On Fri, Feb 04, 2011 at 10:00:40AM +0100, Olaf van der Spek wrote: On Fri, Feb 4, 2011 at 9:57 AM, Stanislav Maslovski stanislav.maslov...@gmail.com wrote: This is possible, however, it is an extra busy work for a user. In any case, I think that holding a lock only for downloading is an overkill and this can be relaxed. What if you would launch two download-only ops at the same time? Isn't a lock needed in that case as well? That should a different lock. Currently, when _dowloading_ aptitude holds a lock that prevents _installing_. It also needs to hold apt and synaptic from downloading, at least from downloading the same file. The dpkg lock is universally used so that works (too well). A change to per file locks would need some carefull coordination. MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/8762sywqfe.fsf@frosties.localnet
Re: does aptitude really need to lock the status database when downloading?
Peter Samuelson pe...@p12n.org writes: [Simon Chopin] But I believe what Stanislas mean is to unpack while downloading the rest of the packages. I often wondered why it wasn't the case, but I've assumed so far that there was probably a reason I just could not think of :) I think it is because, in the general case, it is not at all obvious that applying _part_ of the scheduled upgrade will leave the system in a working state, and a state the user would approve of. For example: say you are upgrading package A, but the new version of A conflicts with package B1, so you are also replacing B1 with B2. So, what apt will do: Download A Download B2 Remove B1 Upgrade A Install B2 Now assume Download A has finished, but Download B2 is still in progress. You could upgrade A, which involves first removing B1 because of the conflict. Now assume B2 fails to download. You are left with a system where neither B1 nor B2 are installed. Is that what the user would want? What you actually have is Remove B1 Unpack A Unpack a million other packages Unpack B2 Configure A Configure B2 There already is a sometimes large window without B1. So I don't see that as a real issue. Anyway, there are many cases where you don't have a conflict/replaces. Also for large upgrades the space required in /var is huge. If apt could download and install in parallel it could also clean up its cache in parallel reducing the amount of space needed. One other thing that could happen in parallel: pre configuring. While apt downloads it could already trigger the /etc/apt/apt.conf.d/70debconf hook (DPkg::Pre-Install-Pkgs). That way the user could answere questions while waiting for downloads. That should actually be quite simple to implement. MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/871v3mwq15.fsf@frosties.localnet
Re: does aptitude really need to lock the status database when downloading?
On Sat, 2011-02-05 at 14:40 +0100, Goswin von Brederlow wrote: Stanislav Maslovski stanislav.maslov...@gmail.com writes: On Fri, Feb 04, 2011 at 10:00:40AM +0100, Olaf van der Spek wrote: On Fri, Feb 4, 2011 at 9:57 AM, Stanislav Maslovski stanislav.maslov...@gmail.com wrote: This is possible, however, it is an extra busy work for a user. In any case, I think that holding a lock only for downloading is an overkill and this can be relaxed. What if you would launch two download-only ops at the same time? Isn't a lock needed in that case as well? That should a different lock. Currently, when _dowloading_ aptitude holds a lock that prevents _installing_. It also needs to hold apt and synaptic from downloading, at least from downloading the same file. The dpkg lock is universally used so that works (too well). A change to per file locks would need some carefull coordination. No, all of these tools use the APT libraries and locking is handled there. Ben. -- Ben Hutchings Once a job is fouled up, anything done to improve it makes it worse. signature.asc Description: This is a digitally signed message part
Re: does aptitude really need to lock the status database when downloading?
On Fri, Feb 04, 2011 at 01:21:07AM +0100, Eduard Bloch wrote: #include hallo.h * Stanislav Maslovski [Fri, Feb 04 2011, 03:10:54AM]: Hi debian-devel, I wanted to ask this for quite a long time: Does aptitude (I think apt-get does the same) really have to lock the status database area when _downloading_ packages? For example, I am running an update on a slow connection and want to uninstall or install with dpkg a few packages while the others are being downloaded. Should not this be possible? I understand that there can be a situation that a dependency could be affected by such an action, but is not it easy to check for this right before unpacking? If you want to have that level of control, why don't you just check it manually? Use --download-only with apt-get (no dpkg locking this way) and when it's done, use apt-get without it to install the packages after making sure that there is no dpkg active anymore. This is possible, however, it is an extra busy work for a user. In any case, I think that holding a lock only for downloading is an overkill and this can be relaxed. As Julian Taylor mentioned, there is also another side of the same problem: aptitude itself can be improved so that it is able to download and unpack in parallel. If it were doing this then the lock would be justified. -- Stanislav -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110204085722.GA6573@kaiba.homelan
Re: does aptitude really need to lock the status database when downloading?
On Fri, Feb 4, 2011 at 9:57 AM, Stanislav Maslovski stanislav.maslov...@gmail.com wrote: This is possible, however, it is an extra busy work for a user. In any case, I think that holding a lock only for downloading is an overkill and this can be relaxed. What if you would launch two download-only ops at the same time? Isn't a lock needed in that case as well? -- Olaf -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/aanlktimgxj6vtx5qx8xdbzuaaoxb+jgbn0juz2tya...@mail.gmail.com
Re: does aptitude really need to lock the status database when downloading?
On Fri, Feb 04, 2011 at 03:29:14AM +0100, Julian Taylor wrote: On Fri, 2011-02-04 at 03:10 +0300, Stanislav Maslovski wrote: For example, I am running an update on a slow connection and want to uninstall or install with dpkg a few packages while the others are being downloaded. Should not this be possible? I understand that there can be a situation that a dependency could be affected by such an action, but is not it easy to check for this right before unpacking? FYI, There is a blueprint in ubuntu aimed for natty concerning this feature: https://blueprints.launchpad.net/ubuntu/+spec/other-foundations-n-update-manager-incremental-updates support downloading in parallel while upgrading the chunks: TODO While somewhat worse as it pauses the download, here's what I used to do in the days of slow links and no apt cache: apt-get xxx --no-download --fix-missing It does what it can do with already downloaded packages, silently ignoring the rest. -- 1KB // Microsoft corollary to Hanlon's razor: // Never attribute to stupidity what can be // adequately explained by malice. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110204083959.ga8...@angband.pl
Re: does aptitude really need to lock the status database when downloading?
On Fri, Feb 04, 2011 at 10:00:40AM +0100, Olaf van der Spek wrote: On Fri, Feb 4, 2011 at 9:57 AM, Stanislav Maslovski stanislav.maslov...@gmail.com wrote: This is possible, however, it is an extra busy work for a user. In any case, I think that holding a lock only for downloading is an overkill and this can be relaxed. What if you would launch two download-only ops at the same time? Isn't a lock needed in that case as well? A per-file lock, yeah. Nothing more. -- 1KB // Microsoft corollary to Hanlon's razor: // Never attribute to stupidity what can be // adequately explained by malice. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110204090537.ga8...@angband.pl
Re: does aptitude really need to lock the status database when downloading?
On Fri, Feb 04, 2011 at 10:00:40AM +0100, Olaf van der Spek wrote: On Fri, Feb 4, 2011 at 9:57 AM, Stanislav Maslovski stanislav.maslov...@gmail.com wrote: This is possible, however, it is an extra busy work for a user. In any case, I think that holding a lock only for downloading is an overkill and this can be relaxed. What if you would launch two download-only ops at the same time? Isn't a lock needed in that case as well? That should a different lock. Currently, when _dowloading_ aptitude holds a lock that prevents _installing_. -- Stanislav -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110204092307.GA10851@kaiba.homelan
Re: does aptitude really need to lock the status database when downloading?
On Fri, Feb 4, 2011 at 6:57 AM, Stanislav Maslovski stanislav.maslov...@gmail.com wrote: [...] If you want to have that level of control, why don't you just check it manually? Use --download-only with apt-get (no dpkg locking this way) and when it's done, use apt-get without it to install the packages after making sure that there is no dpkg active anymore. This is possible, however, it is an extra busy work for a user. In any case, I think that holding a lock only for downloading is an overkill and this can be relaxed. As far as I can tell (and please correct me if I'm wrong), when you do, say, an apt-get upgrade, apt prepares an upgrade plan that uses a given set of packages. If apt wouldn't lock and parallel to that you do an apt-get install, for example, the original plan might not be valid anymore (e.g., new Breaks or Conflicts were introduced). So a new plan would have to be created, the user would have to be asked for confirmation again. Doesn't sound that great. As Julian Taylor mentioned, there is also another side of the same problem: aptitude itself can be improved so that it is able to download and unpack in parallel. If it were doing this then the lock would be justified. As far as I know, apt-get already downloads in parallel. Not sure about aptitude. Regards, -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/AANLkTim31Hson4sEw-zVtkowQmyhkFAu=a+ummoem...@mail.gmail.com
Re: does aptitude really need to lock the status database when downloading?
On Fri, Feb 4, 2011 at 01:10, Stanislav Maslovski stanislav.maslov...@gmail.com wrote: For example, I am running an update on a slow connection and want to uninstall or install with dpkg a few packages while the others are being downloaded. Should not this be possible? I understand that there can be a situation that a dependency could be affected by such an action, but is not it easy to check for this right before unpacking? As always, the (small) APT team is happy to apply well crafted patches. (and i am sure that is true for aptitude as well) The usecase is so small that until now nobody cared to jump through the holes to ensure that the calculation done before the download is still valid after the download and if not recalculate it. So feel free to tackle it. :) Remember that a recalculation will require to check that nothing is changed in regards to the dependency tree: If we would need to use another path we also install/remove other packages so we need to re-ask if the solution is okay, downloading again, recalculating, … Which breaks the user-interface as nobody (no user and no program) expects a second (and third and so on) question round. But beside that: Yes, it is easy. Workarounds were already presented, so i just want to add that you can also disable locking completely with Debug::NoLocking in apt-get (i guess aptitude accepts it also, but i haven't test it) if you are feeling *extremely* lucky… Best regards David Kalnischkies -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/AANLkTinA=rrdoyexmwnukeomxdmw+7jor7wb96c1p...@mail.gmail.com
Re: does aptitude really need to lock the status database when downloading?
On Fri, Feb 4, 2011 at 12:47 PM, Fernando Lemos fernando...@gmail.com wrote: This is possible, however, it is an extra busy work for a user. In any case, I think that holding a lock only for downloading is an overkill and this can be relaxed. As far as I can tell (and please correct me if I'm wrong), when you do, say, an apt-get upgrade, apt prepares an upgrade plan that uses a given set of packages. If apt wouldn't lock and parallel to that you do an apt-get install, for example, the original plan might not be valid anymore (e.g., new Breaks or Conflicts were introduced). So a new plan would have to be created, the user would have to be asked for confirmation again. Doesn't sound that great. That's the choice of the user. Probably most of the time a change of plan wouldn't be needed. Olaf -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/aanlktikjto7gd44nuptrok9gfpd7k0qk6akaqzlcv...@mail.gmail.com
Re: does aptitude really need to lock the status database when downloading?
On Friday 04 February 2011 12.47:21 Fernando Lemos wrote: do, say, an apt-get upgrade, apt prepares an upgrade plan that uses a given set of packages. If apt wouldn't lock [...] new plan would have to be created, the user would have to be asked for confirmation again. Doesn't sound that great. If apt can (can it easily? Not sure) determine if the package db has not changed, this could be a noop for 99% of the users. Not sure if it's worth the complexity, thogh, I've only had this issue very occasionally. cheers -- vbi -- featured product: GNU Privacy Guard - http://gnupg.org signature.asc Description: This is a digitally signed message part.
Re: does aptitude really need to lock the status database when downloading?
On Fri, Feb 04, 2011 at 09:47:21AM -0200, Fernando Lemos wrote: On Fri, Feb 4, 2011 at 6:57 AM, Stanislav Maslovski stanislav.maslov...@gmail.com wrote: [...] If you want to have that level of control, why don't you just check it manually? Use --download-only with apt-get (no dpkg locking this way) and when it's done, use apt-get without it to install the packages after making sure that there is no dpkg active anymore. This is possible, however, it is an extra busy work for a user. In any case, I think that holding a lock only for downloading is an overkill and this can be relaxed. As far as I can tell (and please correct me if I'm wrong), when you do, say, an apt-get upgrade, apt prepares an upgrade plan that uses a given set of packages. If apt wouldn't lock and parallel to that you do an apt-get install, for example, the original plan might not be valid anymore (e.g., new Breaks or Conflicts were introduced). This I understand, as I mentioned in my original mail. So a new plan would have to be created, the user would have to be asked for confirmation again. Doesn't sound that great. Actually, it is the opposite. It sounds actually very great _for_the_target_user_. Because the user who wants this feature will be aware of what he is doing. For that user the _current_ situation with the global lock does not seem very great. Other users will not be affected at all if such a check is indroduced, because they do not use dpkg when doing an apt-get upgrade. Unfortunately, I do not know anything about the internal architecture of apt and friends, therefore I cannot promise any help with it and may only ask for other people that may be interested in implementing this. Maybe in a competing software (was there something called cupt?) As Julian Taylor mentioned, there is also another side of the same problem: aptitude itself can be improved so that it is able to download and unpack in parallel. If it were doing this then the lock would be justified. As far as I know, apt-get already downloads in parallel. Not sure about aptitude. A parallel download is not the same as a download superimposed with unpacking. -- Stanislav -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110204160931.GA13423@kaiba.homelan
Re: does aptitude really need to lock the status database when downloading?
Hi ! On Fri, Feb 04, 2011 at 09:47:21AM -0200, Fernando Lemos wrote: On Fri, Feb 4, 2011 at 6:57 AM, Stanislav Maslovski [...] As Julian Taylor mentioned, there is also another side of the same problem: aptitude itself can be improved so that it is able to download and unpack in parallel. If it were doing this then the lock would be justified. As far as I know, apt-get already downloads in parallel. Not sure about aptitude. I think it does, when on different servers IIRC. But I believe what Stanislas mean is to unpack while downloading the rest of the packages. I often wondered why it wasn't the case, but I've assumed so far that there was probably a reason I just could not think of :) Regards, Simon -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110204191045.gc25...@hebus.troy.local
Re: does aptitude really need to lock the status database when downloading?
On Fri, Feb 4, 2011 at 5:10 PM, Simon Chopin chopin.si...@googlemail.com wrote: [...] As Julian Taylor mentioned, there is also another side of the same problem: aptitude itself can be improved so that it is able to download and unpack in parallel. If it were doing this then the lock would be justified. As far as I know, apt-get already downloads in parallel. Not sure about aptitude. I think it does, when on different servers IIRC. But I believe what Stanislas mean is to unpack while downloading the rest of the packages. Ah yeah, indeed, I realized that when I read the sentence again... I often wondered why it wasn't the case, but I've assumed so far that there was probably a reason I just could not think of :) I can think one: autoremove and packages marked as automatically installed. When you install package B as a dependency of A, you install B as an automatically installed package, so that apt-get autoremove can get rid of it when A is no longer installed. If you lock to install package B, then unlock and lock again when you're about to install A, apt-get autoremove might have been invoked in the mean time and package B might not be present anymore. Sure, you can handle that too, but I really don't think it's worth the hassle. There might be a lot many more cases like this, and the benefit seems very insignificant. Anyways, the proper way to request this feature is through the bugtracker. Regards, -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/aanlktikd2nhvct43nqfywtf0nmwqkbmcgcjbrk-gr...@mail.gmail.com
Re: does aptitude really need to lock the status database when downloading?
[Simon Chopin] But I believe what Stanislas mean is to unpack while downloading the rest of the packages. I often wondered why it wasn't the case, but I've assumed so far that there was probably a reason I just could not think of :) I think it is because, in the general case, it is not at all obvious that applying _part_ of the scheduled upgrade will leave the system in a working state, and a state the user would approve of. For example: say you are upgrading package A, but the new version of A conflicts with package B1, so you are also replacing B1 with B2. So, what apt will do: Download A Download B2 Remove B1 Upgrade A Install B2 Now assume Download A has finished, but Download B2 is still in progress. You could upgrade A, which involves first removing B1 because of the conflict. Now assume B2 fails to download. You are left with a system where neither B1 nor B2 are installed. Is that what the user would want? -- Peter Samuelson | org-tld!p12n!peter | http://p12n.org/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110204200500.ga3...@p12n.org
Re: does aptitude really need to lock the status database when downloading?
#include hallo.h * Stanislav Maslovski [Fri, Feb 04 2011, 03:10:54AM]: Hi debian-devel, I wanted to ask this for quite a long time: Does aptitude (I think apt-get does the same) really have to lock the status database area when _downloading_ packages? For example, I am running an update on a slow connection and want to uninstall or install with dpkg a few packages while the others are being downloaded. Should not this be possible? I understand that there can be a situation that a dependency could be affected by such an action, but is not it easy to check for this right before unpacking? If you want to have that level of control, why don't you just check it manually? Use --download-only with apt-get (no dpkg locking this way) and when it's done, use apt-get without it to install the packages after making sure that there is no dpkg active anymore. Regards, Eduard. -- Wir lassen uns beide von unseren Frauen scheiden und ziehen zusammen. -- Toni Polster (über sein verbessertes Verhältnis zu Trainer Peter Neururer) -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20110204002107.ga14...@rotes76.wohnheim.uni-kl.de
Re: does aptitude really need to lock the status database when downloading?
On Fri, 2011-02-04 at 03:10 +0300, Stanislav Maslovski wrote: Hi debian-devel, I wanted to ask this for quite a long time: Does aptitude (I think apt-get does the same) really have to lock the status database area when _downloading_ packages? For example, I am running an update on a slow connection and want to uninstall or install with dpkg a few packages while the others are being downloaded. Should not this be possible? I understand that there can be a situation that a dependency could be affected by such an action, but is not it easy to check for this right before unpacking? -- Stanislav FYI, There is a blueprint in ubuntu aimed for natty concerning this feature: https://blueprints.launchpad.net/ubuntu/+spec/other-foundations-n-update-manager-incremental-updates support downloading in parallel while upgrading the chunks: TODO Regards, Julian signature.asc Description: This is a digitally signed message part
