Bug#464386: Solution proposal.
On 3/11/08, Mateusz Poszwa <[EMAIL PROTECTED]> wrote: > Symlinks in APT's cache do work. Anytime I run Finnix (a Debian-based > LiveDistro) I symlink all packages from both caches (lots of 'already > exists' messages) to ramdisk in order not to fetch them again. Good to know. > I have exaggerated, sorry... Some of package files (about 20% of mine) > do have different names though. > > Examples (from APT's cache): > anjuta_2%3a2.3.5-2_i386.deb [snip] > xorg_1%3a7.2-5_all.deb #and other xorg and xserver packages > > Difference between names is a consequence of /[:digit:]%3a/ strings just > after "pkgname_". I don't know where they come from, but i think APT's > developers know. I see now. The file names are urlencoded, which replaces everything but alpha-numerics, '.', '-', and '_', with '%' followed by the ascii code (in hex). The %3a is due to the ':' used to separate the epoch used in the versions of some packages. > Links of such packages would not work, I have just checked it. I assume by this that you mean using the DebTorrent file name as the link in apt's cache doesn't work. Presumably, knowing that apt's cache keeps urlencoded file names, the script you mentioned before could be modified to use the urlencoded file name for the link in apt's cache, which would then work fine. Alternatively, perhaps I should look into why apt stores the files urlencoded (there may be a security concern here), and modify DebTorrent to also store file names urlencoded. > Dnia 10-03-2008, pon o godzinie 15:14 -0700, Cameron Dale pisze: > > I don't believe there is, although as I mentioned in my previous mail > > to this bug, you can make apt not create a copy of the file in it's > > cache at all. Not ideal, but still a possible solution. > > Well, in my opinion this solution is DebTorrent only as APT would no > longer care about cache, and DebTorrent would care only about its part > of APT's cache (in this case entire APT's cache) and its own. :( > It would be great if there was a method to tell APT behave like that > only for DebTorrent origned packages (at least I don't know any). Actually, it's not an apt configuration or anything that would affect all files downloaded. Rather it's a parameter I can pass to apt with the file to let it know not to keep a copy in its cache. Therefore, it is possible to tell APT to behave like this only for DebTorrent originated packages. The downside is that apt then doesn't include the downloaded file in its display of statistics (download speed, time remaining), which isn't an issue now, but could be in the future. Cameron -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#464386: Solution proposal.
Dnia 10-03-2008, pon o godzinie 15:14 -0700, Cameron Dale pisze: > On 3/9/08, Mateusz Poszwa <[EMAIL PROTECTED]> wrote: > > Advantage is that it deletes&links only packages common for both caches. > > Unfortunately it runs rather slow, and requires additional dependency. > > What is more, it is not safe, because if files are on different > > filesystems or machines it produces symbolic links instead of hardlinks. > > That might still work, as long as the symlinks are in the apt cache. > Symlinks in the DebTorrent cache will definitely not work, but I'm not > sure how apt will behave when there are symlinks in its cache. Symlinks in APT's cache do work. Anytime I run Finnix (a Debian-based LiveDistro) I symlink all packages from both caches (lots of 'already exists' messages) to ramdisk in order not to fetch them again. Dnia 10-03-2008, pon o godzinie 15:14 -0700, Cameron Dale pisze: > That's pretty much what I was thinking of too. Except for the "names > of files are usually slightly different" part, which I don't see ever > happening. Why would the package file be named differently in apt's > cache than in DebTorrent's? I have exaggerated, sorry... Some of package files (about 20% of mine) do have different names though. Examples (from APT's cache): anjuta_2%3a2.3.5-2_i386.deb autogen_1%3a5.9.4-1_i386.deb automake1.4_1%3a1.4-p6-12_all.deb bsdutils_1%3a2.13.1-2_i386.deb busybox_1%3a1.1.3-5_i386.deb cpp_4%3a4.2.2-2_i386.deb gcc_4%3a4.2.2-2_i386.deb genisoimage_9%3a1.1.6-1_i386.deb gimp-data-extras_1%3a2.0.1-3_all.deb iceweasel-l10n-pl_1%3a2.0.0.12+debian-1_all.deb imagemagick_7%3a6.3.7.9.dfsg1-2_i386.deb login_1%3a4.0.18.1-7_i386.deb mkisofs_9%3a1.1.6-1_all.deb nfs-common_1%3a1.0.10-6+etch.1_i386.deb openoffice.org_1%3a2.3.1-5_i386.deb #and other OOo packages passwd_1%3a4.0.18.1-7_i386.deb pciutils_1%3a2.2.10-1_i386.deb procps_1%3a3.2.7-3_i386.deb wodim_9%3a1.1.6-1_i386.deb xorg_1%3a7.2-5_all.deb #and other xorg and xserver packages Difference between names is a consequence of /[:digit:]%3a/ strings just after "pkgname_". I don't know where they come from, but i think APT's developers know. Links of such packages would not work, I have just checked it. Dnia 10-03-2008, pon o godzinie 15:14 -0700, Cameron Dale pisze: > On 3/9/08, Mateusz Poszwa <[EMAIL PROTECTED]> wrote: > > I wonder if there is a method of making apt try creating hardlink before > > making actual copy of package. This would be (IMO) the best solution. > > I don't believe there is, although as I mentioned in my previous mail > to this bug, you can make apt not create a copy of the file in it's > cache at all. Not ideal, but still a possible solution. > > Cameron Well, in my opinion this solution is DebTorrent only as APT would no longer care about cache, and DebTorrent would care only about its part of APT's cache (in this case entire APT's cache) and its own. :( It would be great if there was a method to tell APT behave like that only for DebTorrent origned packages (at least I don't know any). -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#464386: Solution proposal.
On 3/9/08, Mateusz Poszwa <[EMAIL PROTECTED]> wrote:
> That's why I said:
>
> Dnia 08-03-2008, sob o godzinie 12:00 -0800, Cameron Dale pisze:
>
> > On 3/8/08, Mateusz Poszwa <[EMAIL PROTECTED]> wrote:
>
> > > If debtorrent is the only method of obtaining packages
Sorry, I missed that. :)
> I know a method of doing that. Package fslint is required.
>
> # /usr/share/fslint/fslint/findup \
> -m /var/cache/apt/archives /var/cache/debtorrent
Another great tip! Thanks again. :)
> Advantage is that it deletes&links only packages common for both caches.
> Unfortunately it runs rather slow, and requires additional dependency.
> What is more, it is not safe, because if files are on different
> filesystems or machines it produces symbolic links instead of hardlinks.
That might still work, as long as the symlinks are in the apt cache.
Symlinks in the DebTorrent cache will definitely not work, but I'm not
sure how apt will behave when there are symlinks in its cache.
> I think of writing a script which would work like that:
>
> if(caches are on the same machine and the same filesystem){
> for_each(package in debtorrent's cache){
> if(corresponding package in apt's cache exists){
>if(and is not a hardlink of package in debtorrents cache){
> delete it}
>else{
> break;}//in order not to let ln throw any error messages
>create link to package in debtorrent's cache named just like the\
> package deleted a moment ago (as names of files are usually slightly\
> different)}}}
That's pretty much what I was thinking of too. Except for the "names
of files are usually slightly different" part, which I don't see ever
happening. Why would the package file be named differently in apt's
cache than in DebTorrent's?
> This would solve the problem of packages which are in apt's cache only.
> A side effect of this solution is that packages, which were previously
> only in debtorrent's cache are in both caches after executing the
> script, but i think it is not a big problem (it is even desirable ;).
I agree.
> I wonder if there is a method of making apt try creating hardlink before
> making actual copy of package. This would be (IMO) the best solution.
I don't believe there is, although as I mentioned in my previous mail
to this bug, you can make apt not create a copy of the file in it's
cache at all. Not ideal, but still a possible solution.
Cameron
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#464386: Solution proposal.
Dnia 08-03-2008, sob o godzinie 12:00 -0800, Cameron Dale pisze:
> This would clean out ALL packages files though, and replace them with
> only the ones downloaded by DebTorrent, so if you have 'deb' lines in
> your sources.list file that aren't pointing to DebTorrent you may want
> to think twice before doing this.
That's why I said:
Dnia 08-03-2008, sob o godzinie 12:00 -0800, Cameron Dale pisze:
> On 3/8/08, Mateusz Poszwa <[EMAIL PROTECTED]> wrote:
> > If debtorrent is the only method of obtaining packages
Dnia 08-03-2008, sob o godzinie 12:00 -0800, Cameron Dale pisze:
> On 3/8/08, Mateusz Poszwa <[EMAIL PROTECTED]> wrote:
> > In case the caches are on different slices ln throws error messages,
> > so there is ability to catch them and fall back.
>
> I guess this adds another possible solution to this bug, in addition
> to the one I already mentioned, so I'll have to decide which one is
> better. Probably this method is better, especially if I make a script
> that removes and links only the files in apt's cache that debtorrent
> has available.
I know a method of doing that. Package fslint is required.
# /usr/share/fslint/fslint/findup \
-m /var/cache/apt/archives /var/cache/debtorrent
Advantage is that it deletes&links only packages common for both caches.
Unfortunately it runs rather slow, and requires additional dependency.
What is more, it is not safe, because if files are on different
filesystems or machines it produces symbolic links instead of hardlinks.
I think of writing a script which would work like that:
if(caches are on the same machine and the same filesystem){
for_each(package in debtorrent's cache){
if(corresponding package in apt's cache exists){
if(and is not a hardlink of package in debtorrents cache){
delete it}
else{
break;}//in order not to let ln throw any error messages
create link to package in debtorrent's cache named just like the\
package deleted a moment ago (as names of files are usually slightly\
different)}}}
This would solve the problem of packages which are in apt's cache only.
A side effect of this solution is that packages, which were previously
only in debtorrent's cache are in both caches after executing the
script, but i think it is not a big problem (it is even desirable ;).
I wonder if there is a method of making apt try creating hardlink before
making actual copy of package. This would be (IMO) the best solution.
PS: Thank you for a dpkg::post-invoke tip. ;)
--
Sorry for my English.
Regards from Poland
Mateusz "Old" Poszwa
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#464386: Solution proposal.
On 3/8/08, Mateusz Poszwa <[EMAIL PROTECTED]> wrote:
> Or maybe even better it would be if apt used DebTorrent's cache.
> It can be easily achieved by merging both caches
> if they are on the same slice (usually they are).
> If debtorrent is the only method of obtaining packages
> #apt-get clean
> #ln /var/cache/debtorrent/*/pool/*/*/*/*.deb /var/cache/apt/archives
Thanks for the suggestion, that seems to work very well. :)
If you wanted to automate this setup, you could try adding the
following to your apt.conf file:
DPkg::Post-Invoke { "apt-get clean && ln
/var/cache/debtorrent/*/pool/*/*/*/*.deb /var/cache/apt/archives"; };
This would clean out ALL packages files though, and replace them with
only the ones downloaded by DebTorrent, so if you have 'deb' lines in
your sources.list file that aren't pointing to DebTorrent you may want
to think twice before doing this.
> should be safe unless package files are written to (I think they're not).
Once they're downloaded they aren't written again, so as long as
DebTorrent isn't downloading anything, running this should be fine.
> In case the caches are on different slices ln throws error messages,
> so there is ability to catch them and fall back.
I guess this adds another possible solution to this bug, in addition
to the one I already mentioned, so I'll have to decide which one is
better. Probably this method is better, especially if I make a script
that removes and links only the files in apt's cache that debtorrent
has available.
Thanks,
Cameron
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#464386: Solution proposal.
>It would be nice if debtorrent used apt's package cache instead of a separate >one. The main advantage would be less used space. Or maybe even better it would be if apt used DebTorrent's cache. It can be easily achieved by merging both caches if they are on the same slice (usually they are). If debtorrent is the only method of obtaining packages #apt-get clean #ln /var/cache/debtorrent/*/pool/*/*/*/*.deb /var/cache/apt/archives should be safe unless package files are written to (I think they're not). In case the caches are on different slices ln throws error messages, so there is ability to catch them and fall back. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
