Re: Optification breaks package on upgrade from non-optified older version
Marius Vollmer marius.voll...@nokia.com writes: I have pushed a new version of maemo-optify to gitorious that patches maintainer scripts to do late optification for stuff that dpkg didn't do. It's not very well tested yet, of course, so please dig in! http://maemo.gitorious.org/maemo-af/maemo-optify I have now released maemo-optify 0.1 to the fremantle buildbot with this change. It should now be possible to update a non-optified version of a package to a optified version. Everybody, please test! ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Optification breaks package on upgrade from non-optified older version
What about a slight variation: for each (symlinkname in the package) do if (symlinkname should point to /opt) and (symlinkname is a directory) then cp -a symlinkname/* to /opt/maemo/symlinkname/ rm -rf symlinkname ln -s /opt/maemo/symlinkname symlinkname endif endfor I hope no package ships with dotfiles in directories other packages want to optify. Yeah, well spotted! I have pushed a new version of maemo-optify to gitorious that patches maintainer scripts to do late optification for stuff that dpkg didn't do. It's not very well tested yet, of course, so please dig in! http://maemo.gitorious.org/maemo-af/maemo-optify I'd also like to bring up error handling: if /opt is almost full and you cannot migrate existing files, it would be bad to leave things in a broken state. Yes. It is unfortunately not possible to replace a directory with something else atomically. I think the best we can do is to move away the old directory and put a symlink in its place with two syscalls and hope that nothing happens in between. The old directory can then be removed afterwards. ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Optification breaks package on upgrade from non-optified older version
On Fri, Oct 16, 2009 at 01:43:12AM +0300, Mikko Vartiainen wrote: What's the practical difference between upgrading and removing-then-reinstalling? (It probably shows that I haven't read the maemo-optify and python apps thread.) Maybe you could fix the upgrade problem by writing a suitable preinst script? I have noticed with upgrading from non optified packages to maemo-optified packages that directory links are not created. I've read somewhere that dpkg is unable to replace directories with symlinks. For example if I first have installed a package with a directory /usr/share/games/crimson and upgrade to a package which has optified directory /usr/share/games/crimson - /opt/maemo/usr/share/games/crimson, the link is not created because dpkg never removes /usr/share/games/crimson directory. Result is that I have empty /usr/share/games/crimson directory and all of it's content in /opt directory. Perhaps a solution is to have maemo-optify keep the directory structure in place and create symlinks only for regular files. I think it's a bug in dpkg or then we are just using it wrong. Workaround for this would be to remove the directory in preinst script or to install to another directory. But currently when all users are more or less beta testers I don't care about it. Removing the directory is a bit scary: what if other packages have files in it? This is maybe not applicable to this particular example, but I could imagine crimson-extra-levels or crimson-updated-sound-pack putting stuff there. Marius Gedminas -- question = (to) ? be : !be; -- Shakespeare signature.asc Description: Digital signature ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Optification breaks package on upgrade from non-optified older version
On Fri, Oct 16, 2009 at 7:05 AM, Andrew Flegg and...@bleb.org wrote: On Fri, Oct 16, 2009 at 11:49, Mikko Vartiainen mvartiai...@gmail.com wrote: Perhaps a solution is to have maemo-optify keep the directory structure in place and create symlinks only for regular files. This would certainly work. maemo-optify should have option which prevents it from creating directory links. Why would/should it be an option? From the sounds of it, it shouldn't create directory symlinks; why would a package ever want them given the problems it can cause? Agreed. And I think the dpkg problem with symlinks replacing existing directories is a inherent to the way it unpacks the application. AFAIK dpkg upgrades packages in place, i.e. it first unpacks the new application and later removes the files that existed only in the old version (I might be wrong though). The files in the .deb (which is a simple archive file) are inside a data.tar.gz file, which is unpacked inplace on the root directory. You can see this by running ar t on any .deb file. It will list its raw contents. E.g.: $ ar t /var/cache/apt/archives/wget_1.10.2-3ubuntu1.1_i386.deb debian-binary control.tar.gz data.tar.gz I did an experiment that consisted on creating a tarball with a symlink called d and tried to unpack it on a directory with an existing d directory. If the directory is *empty*, tar simply removes the existing directory and unpacks the symlink. Now, if the directory is not empty, this is what happens: $ tar -xvf ../test.tar.gz ./ ./d tar: ./d: Cannot create symlink to `d2': File exists tar: Error exit delayed from previous errors Now why tar does not simply rm -rf the directory? I believe this is to avoid disasters if tarballs contains for instance a /usr symlink. Therefore, I think there is no way to guarantee that .deb packages with directory symlinks are installable UNLESS their preinst script explicitly removes non-empty directories with the same path as the symlinks which, again, is very dangerous IMHO. Regards, -- Anderson Lizardo OpenBossa Labs - INdT Manaus - Brazil ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Optification breaks package on upgrade from non-optified older version
On Fri, Oct 16, 2009 at 2:05 PM, Andrew Flegg and...@bleb.org wrote: On Fri, Oct 16, 2009 at 11:49, Mikko Vartiainen mvartiai...@gmail.com wrote: This would certainly work. maemo-optify should have option which prevents it from creating directory links. Why would/should it be an option? From the sounds of it, it shouldn't create directory symlinks; why would a package ever want them given the problems it can cause? Other way around then. Option to create directory symlinks. They are useful for large data directories with hundreds or maybe thousands of files. Currently maemo-optify doesn't create links for files which are smaller than 2KB, but there could be hundreds (or thousands) of small files. Of course it's possible to install to /opt without using maemo-optify but I find it very easy use. -- Mikko Vartiainen ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Optification breaks package on upgrade from non-optified older version
2009/10/15 Quim Gil quim@nokia.com: ext Thomas Perl wrote: In the latest version of gPodder (written in Python), I have started using maemo-optify. It saves around 1 MB according to maemo-optify's output. The problem is that now when a user upgrades from an older version to this version, it breaks (the user cannot start the application from the icon, and starting it from the command line tells me that Python can't find the gpodder module). Uninstalling and re-installing the package fixes this problem, but I assume there will be lots of users upgrading from the current version in Extras (non-optified) to some optified version in the future, and they won't just try to uninstall and re-install it and assume that the application is at fault :/ lots of users before sales start = actually not that many and all of them quite savvy and even probably reflashing at this point. Thanks for you report though. It's clearer that unless we find several solutions the time for optification is NOW, or whenever a developer is pushing a first release to Extras. Feel free to test the newer version and rate/comment on the package here: http://maemo.org/packages/package_instance/view/fremantle_extras-testing_free_armel/gpodder/2.0+git20091013-thp1/ The non-optified version from September is already in Extras, and it would be nice to get this updated, optified package into Extras before sales start to avoid having clueless users think that the app is broken :) Thanks, Thomas ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Optification breaks package on upgrade from non-optified older version
2009/10/16 Mikko Vartiainen mvartiai...@gmail.com: On Fri, Oct 16, 2009 at 2:05 PM, Andrew Flegg and...@bleb.org wrote: On Fri, Oct 16, 2009 at 11:49, Mikko Vartiainen mvartiai...@gmail.com wrote: This would certainly work. maemo-optify should have option which prevents it from creating directory links. Why would/should it be an option? From the sounds of it, it shouldn't create directory symlinks; why would a package ever want them given the problems it can cause? Other way around then. Option to create directory symlinks. They are useful for large data directories with hundreds or maybe thousands of files. Currently maemo-optify doesn't create links for files which are smaller than 2KB, but there could be hundreds (or thousands) of small files. Of course it's possible to install to /opt without using maemo-optify but I find it very easy use. (Is there a bug tracker for maemo-optify? Couldn't find it on bugs.m.o :/) Being able (as a packager) to tell maemo-optify would be very helpful when installing packages with lots of small data files (icon themes, ...). That said, the current problem (and the one I ran into, apparently) is the fact that existing directories in packages will not be replaced with symbolic links when upgrading packages. I don't really know much about dpkg internals, but maybe it would be possible for maemo-optify to add a small script to postinst that will do something like this (in pseudocode): for each (symlinkname in the package) do if (symlinkname points to /opt) and (symlinkname is a directory) then if (symlinkname is an empty directory) then remove directory, recreate correct symlink else create symlinks inside the directory endif endif endfor In the case where the directory is empty, this works as expected and should leave the same structure as a non-upgrade installation (essentially fixing the bug I was experiencing). In cases where there directory is non-empty, it will at least re-create a working symlink structure, but the on-disk structure will not match the package contents, which could possibly lead to problems with future upgrades. At least the small code above without the last else code path should be sane to add to maemo-optify to fix the upgrade directory/symlink bug? Thanks, Thomas ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Optification breaks package on upgrade from non-optified older version
ext Thomas Perl th.p...@gmail.com writes: I don't really know much about dpkg internals, but maybe it would be possible for maemo-optify to add a small script to postinst that will do something like this (in pseudocode): for each (symlinkname in the package) do if (symlinkname points to /opt) and (symlinkname is a directory) then if (symlinkname is an empty directory) then remove directory, recreate correct symlink else create symlinks inside the directory endif endif endfor What about a slight variation: for each (symlinkname in the package) do if (symlinkname should point to /opt) and (symlinkname is a directory) then cp -a symlinkname/* to /opt/maemo/symlinkname/ rm -rf symlinkname ln -s /opt/maemo/symlinkname symlinkname endif endfor This would migrate the existing files and would establish a symlink structure that matches what is in the package. (What a hack.) ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Optification breaks package on upgrade from non-optified older version
On Fri, Oct 16, 2009 at 14:58, Marius Vollmer marius.voll...@nokia.com wrote: ext Thomas Perl th.p...@gmail.com writes: I don't really know much about dpkg internals, but maybe it would be possible for maemo-optify to add a small script to postinst that will do something like this (in pseudocode): for each (symlinkname in the package) do if (symlinkname points to /opt) and (symlinkname is a directory) then if (symlinkname is an empty directory) then remove directory, recreate correct symlink else create symlinks inside the directory endif endif endfor So, the problem with this is removing content underneath that dir installed by something else. What about a slight variation: for each (symlinkname in the package) do if (symlinkname should point to /opt) and (symlinkname is a directory) then cp -a symlinkname/* to /opt/maemo/symlinkname/ rm -rf symlinkname ln -s /opt/maemo/symlinkname symlinkname endif endfor This would migrate the existing files and would establish a symlink structure that matches what is in the package. So, let's imagine: mygame_0.0.1has /usr/share/mygame/ mygame-extras_0.0.1 has /usr/share/mygame/extras.xml mygame_0.0.2has /usr/share/mygame - /opt/maemo/usr/share/mygame/ When mygame v0.0.2 is installed, what will happen when mygame-extras_0.0.1 is uninstalled? Is dpkg clever enough to follow the /usr/share/mygame symlink and delete /opt/maemo/usr/share/mygame/extras.xml? (What a hack.) Yeah, but not entirely unelegant ;-) Cheers, Andrew -- Andrew Flegg -- mailto:and...@bleb.org | http://www.bleb.org/ ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Optification breaks package on upgrade from non-optified older version
Hi Thomas, On Fri, Oct 16, 2009 at 9:27 AM, Thomas Perl th.p...@gmail.com wrote: Feel free to test the newer version and rate/comment on the package here: http://maemo.org/packages/package_instance/view/fremantle_extras-testing_free_armel/gpodder/2.0+git20091013-thp1/ The non-optified version from September is already in Extras, and it would be nice to get this updated, optified package into Extras before sales start to avoid having clueless users think that the app is broken :) I did a small experiment here with a dummy package that creates a single directory with a single file in it, and a newer version of it (0.2) that has a symlink instead of the directory. As expected, dpkg installed the new version of the package but left the old directory intact (so the symlink was not installed. Strangely, no warning or error was shown. But after putting some lines on debian/preinst to remove the directories that are being replaced (I used rmdir to ensure that it didn't accidentally removed files from other packages), the symlink was installed properly. So I think you can do a similar thing to your packages (remove the directories that are being replaced with symlinks on debian/preinst) but the problem is that you can't know in advance which directories are being optified. So I like better the suggestion given by someone else to have maemo-optify add extra lines to debian/*.preinst files (similar to how debhelper utilities do) to make this cleanup before upgrades, but taking care to not delete files from other packages, as Andrew pointed out. Regards, -- Anderson Lizardo OpenBossa Labs - INdT Manaus - Brazil ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Optification breaks package on upgrade from non-optified older version
Hello! In the latest version of gPodder (written in Python), I have started using maemo-optify. It saves around 1 MB according to maemo-optify's output. The problem is that now when a user upgrades from an older version to this version, it breaks (the user cannot start the application from the icon, and starting it from the command line tells me that Python can't find the gpodder module). Uninstalling and re-installing the package fixes this problem, but I assume there will be lots of users upgrading from the current version in Extras (non-optified) to some optified version in the future, and they won't just try to uninstall and re-install it and assume that the application is at fault :/ Is it possible to fix that problem somehow? Or should I just disable optification? Another problem is the slower startup speed, which is really noticeable now. That's another reason why I would like to go without optification. Thanks, Thomas ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Optification breaks package on upgrade from non-optified older version
On Thursday 15 October 2009 12:21:54 Thomas Perl wrote: Another problem is the slower startup speed, which is really noticeable now. That's another reason why I would like to go without optification. From what I heard that was also the reason why python (and Qt) are/were not optified, but apparently a choice will have to be made - speed or application space, but while speed is a 'should', application space is a 'must'. I wonder if some sort of optification could be done on the device itself, as in make it optional or a choice - this stuff I use often and I don't want it optified, while somehing else used rarely can be on /opt so it would not take up space on /. Not sure about the actual mechanism to do this, just thinking out loud. ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Optification breaks package on upgrade from non-optified older version
ext Thomas Perl wrote: Hello! In the latest version of gPodder (written in Python), I have started using maemo-optify. It saves around 1 MB according to maemo-optify's output. The problem is that now when a user upgrades from an older version to this version, it breaks (the user cannot start the application from the icon, and starting it from the command line tells me that Python can't find the gpodder module). Uninstalling and re-installing the package fixes this problem, but I assume there will be lots of users upgrading from the current version in Extras (non-optified) to some optified version in the future, and they won't just try to uninstall and re-install it and assume that the application is at fault :/ lots of users before sales start = actually not that many and all of them quite savvy and even probably reflashing at this point. Thanks for you report though. It's clearer that unless we find several solutions the time for optification is NOW, or whenever a developer is pushing a first release to Extras. Is it possible to fix that problem somehow? Or should I just disable optification? Another problem is the slower startup speed, which is really noticeable now. That's another reason why I would like to go without optification. Thanks, Thomas -- Quim Gil open source advocate Maemo Devices @ Nokia ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Optification breaks package on upgrade from non-optified older version
On Thu, 15 Oct 2009, Attila Csipa wrote: On Thursday 15 October 2009 12:21:54 Thomas Perl wrote: Another problem is the slower startup speed, which is really noticeable now. That's another reason why I would like to go without optification. From what I heard that was also the reason why python (and Qt) are/were not optified, but apparently a choice will have to be made - speed or application space, but while speed is a 'should', application space is a 'must'. I wonder if some sort of optification could be done on the device itself, as in make it optional or a choice - this stuff I use often and I don't want it optified, while somehing else used rarely can be on /opt so it would not take up space on /. Not sure about the actual mechanism to do this, just thinking out loud. Why not do something simple like leave all packages installing in the regular place, and running on the device something like cp -a /usr /home/ mv /usr /usr2 ln -s /home/usr /usr rm -rf /usr2 Every linux user who partitioned his disk wrongly has done something like this. All this requires is that bootup scripts do not use files under /usr before /home is mounted, but that's supposed to be true anyway. -- Matan. ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Optification breaks package on upgrade from non-optified older version
- Original message - Uninstalling and re-installing the package fixes this problem, but I assume there will be lots of users upgrading from the current version in Extras (non-optified) to some optified version in the future, and they won't just try to uninstall and re-install it and assume that the application is at fault :/ Not too many at the moment, I hope, but as this may be a source of problems in the future it is important that we minimise the need to upgrade from non-optified to optified by getting everything optified asap. Let's concentrate our testing on the optified versions. Is it possible to fix that problem somehow? Or should I just disable optification? Definitely not. There is too little space to allow non-optified apps to go into extras in the future. Another problem is the slower startup speed, which is really noticeable now. That's another reason why I would like to go without optification. If you could quantify this and/or do some investigation of what, in particular, is causing the slowdown it would be very useful. We have to find some way to move most of the app's storage to /home and the better we understand the impacts the better the solution can be. Graham ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Optification breaks package on upgrade from non-optified older version
On Thu, Oct 15, 2009 at 12:21:54PM +0200, Thomas Perl wrote: In the latest version of gPodder (written in Python), I have started using maemo-optify. It saves around 1 MB according to maemo-optify's output. The problem is that now when a user upgrades from an older version to this version, it breaks (the user cannot start the application from the icon, and starting it from the command line tells me that Python can't find the gpodder module). That's a packaging bug, from the user's perspective. Uninstalling and re-installing the package fixes this problem, but I assume there will be lots of users upgrading from the current version in Extras (non-optified) to some optified version in the future, and they won't just try to uninstall and re-install it and assume that the application is at fault :/ Is it possible to fix that problem somehow? Or should I just disable optification? What's the practical difference between upgrading and removing-then-reinstalling? (It probably shows that I haven't read the maemo-optify and python apps thread.) Maybe you could fix the upgrade problem by writing a suitable preinst script? Another problem is the slower startup speed, which is really noticeable now. That's another reason why I would like to go without optification. Ouch. Have you done any measurements? Marius Gedminas -- Bumper sticker: No radio - Already stolen. signature.asc Description: Digital signature ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
Re: Optification breaks package on upgrade from non-optified older version
ext Marius Gedminas mar...@pov.lt writes: Uninstalling and re-installing the package fixes this problem, but I assume there will be lots of users upgrading from the current version in Extras (non-optified) to some optified version in the future, and they won't just try to uninstall and re-install it and assume that the application is at fault :/ Is it possible to fix that problem somehow? Or should I just disable optification? What's the practical difference between upgrading and removing-then-reinstalling? (It probably shows that I haven't read the maemo-optify and python apps thread.) Maybe you could fix the upgrade problem by writing a suitable preinst script? I will have a close look at this soonish. Right now I assume that we need to 'fix' dpkg somehow to get over this, but hopefully not. In any case having upgrades break because of maemo-optify is of course quite severe and I/we need to fix this 'asap'. ___ maemo-developers mailing list maemo-developers@maemo.org https://lists.maemo.org/mailman/listinfo/maemo-developers
