Re: a simple script for restoring downloaded debs
Jörg-Volker Peetz wrote:
songbird wrote on 07/19/2014 16:23:
Jörg-Volker Peetz wrote:
snip
Why not copy all files back and order something like apt-get autoclean to
get
rid of the older packages?
it's a rather huge archive (i think about 18000
debs), but yes that would be another way of doing
it. i think the flaw is that you would lose the
local, obsolete or debs from unofficial sources
that you no longer keep in the sources.list (but
i would have to verify this).
Yes, you would lose them.
i've set up a archive for those too.
the script looks to have gotten the job mostly
done. i've added a bit to copy my local, obsolete
and unofficial debs back from another archive and
i think what remains are those debs that i somehow
missed archiving. sometimes i have to get files
via the library and USB stick.
A note on your scipt:
If you build a pipe of grep, sed, and awk, it can often be done just with awk.
E.g., your line
debname=`apt-cache show $pkgname=$version | egrep '^Filename: ' | sed -e
's/^Filename: //' | awk --field-separator='/' -e '{print \$NF}'`
could be written as something like
debname=$(apt-cache show $pkgname=$version | awk '/^Filename: /
{sub(.*/,, $2) ; print $2}')
saving the call of egrep and sed.
sure, i could do that, usually i build up my
statements a bit at a time to make sure each step
works before adding the next chunk. this isn't a
script run that often i wouldn't worry about
optimizing it.
the last bug i had to figure out was that some debs
are treated differently in their version names as
they translate to the filename stored in the pool
(console-data, libreoffice and several others).
i'm not sure why or where the difference exists but it
was 42 debs (could it be the ghost of Douglas Adams
having a bit of a prank? :) ). i added a check for
the alternate version.
there may be other bugs yet to find, but for my
own apt archive of 2375 installed packages selecting
from the backed up archive of 10980 debs it now is
complete (my wheezy archive was at 18000+ debs).
the current script version now looks like:
=
#!/bin/sh
#
#
# run as root
#
# this script attempts to restore the debs to
# /var/cache/apt/archives for all of the packages
# you currently have installed from a local archive
# (see localarch).
#
# if you have obsolete, local or unofficial debs from
# other sources they can be kept in another place as a
# backup and also copied by this script (see deblocal).
#
# when finished if there is a downloadlist file in the
# current directory those were the debs missing from
# the local archive (or something strange is going on
# with the version numbers).
# your local archive
#
localarch=/archives/debian
# for local, obsolete or unofficial debs that are
# still used apt-cache show will not have a filename.
#
deblocal=$localarch/local/current
# the archive of debs downloaded
#
debarch=$localarch/jessie
# the apt archive
#
dest=/var/cache/apt/archives
# copy the local, obsolete or unofficial debs
#
cp --preserve=all $deblocal/*.deb $dest
# we will generate a list of wget statements to fill in the gaps
#
rm -f downloadlist 21 /dev/null
pkglist=`dpkg -l | egrep '^ii ' | cut --delimiter=' ' -f3,3 | cut
--delimiter=':' -f1,1`
versionlist=`dpkg -l | egrep '^ii ' | sed -e 's/ */ /g' | cut -d ' ' -f3,3`
for pkgname in $pkglist; do
version=`echo $versionlist | cut -d ' ' -f1,1`
versionlist=`echo $versionlist | cut -d ' ' -f2-`
debname=`apt-cache show $pkgname=$version | egrep '^Filename: ' | sed -e
's/^Filename: //' | awk --field-separator='/' -e '{print \$NF}'`
if test -z $debname ; then
echo Package $pkgname Version --$version-- may be obsolete, local or
from a non-official source.
continue
fi
versionplay=`echo $version | grep ':' | wc -c`
if test ! $versionplay = 0 ; then
#for some debs we need an alternate version
altdebname=`echo $debname | sed -e 's/.://'`
#echo We have a funny version number, let's fix it in the filename
versionhead=`echo $version | cut -d ':' -f1,1 | sed -e 's/$/%3a/'`
#echo $pkgname $version $versionhead $debname
debname=`echo $debname | sed -e s/_/_$versionhead/`
#echo $pkgname $version $versionhead $debname
fi
if test -e $debarch/$debname ; then
#echo $debarch/$debname $dest
if test ! -e $dest/$debname ; then
cp --preserve=all $debarch/$debname $dest
fi
elif test -e $debarch/$altdebname ; then
#echo $debarch/$altdebname $dest
if test ! -e $dest/$altdebname ; then
cp --preserve=all $debarch/$altdebname $dest
fi
else
filename=`apt-cache show $pkgname=$version | egrep '^Filename: ' | sed -e
's/^Filename: //'`
echo wget -c http://ftp.us.debian.org/debian/$filename; downloadlist
echo $pkgname $debname missing? version -$version-
fi
done
chown root $dest/*
chgrp root $dest/*
sync
=
songbird
--
To UNSUBSCRIBE, email to
Re: a simple script for restoring downloaded debs
Curt wrote: Chris Bannister wrote: Are you aware of snapshot.debian.org? Save yourself time and anxiety. :) yes aware. no anxiety. I believe he wants to keep things local due to a slow and/or problematic internet connection. right. songbird -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/g4st9b-kqd@id-306963.user.uni-berlin.de
Re: a simple script for restoring downloaded debs
On Sat, Jul 19, 2014 at 10:23:15AM -0400, songbird wrote: songbird wrote: the other night i accidentally deleted the debs i normally keep in the /var/cache/apt/archives directory and while i do keep a backup of them in another directory it is along with all the previous versions [...] it's a rather huge archive (i think about 18000 debs), but yes that would be another way of doing it. i think the flaw is that you would lose the local, obsolete or debs from unofficial sources that you no longer keep in the sources.list (but i would have to verify this). Are you aware of snapshot.debian.org? Save yourself time and anxiety. :) -- If you're not careful, the newspapers will have you hating the people who are being oppressed, and loving the people who are doing the oppressing. --- Malcolm X -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140720123637.GK12789@tal
Re: a simple script for restoring downloaded debs
On 2014-07-20, Chris Bannister cbannis...@slingshot.co.nz wrote: Are you aware of snapshot.debian.org? Save yourself time and anxiety. :) I believe he wants to keep things local due to a slow and/or problematic internet connection. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/slrnlsnfje.21v.cu...@einstein.electron.org
Re: a simple script for restoring downloaded debs
songbird wrote on 07/18/2014 06:02: the other night i accidentally deleted the debs i normally keep in the /var/cache/apt/archives directory and while i do keep a backup of them in another directory it is along with all the previous versions too, so it isn't as easy as just copying them. instead i have to figure out which is the installed version (because that is the one i am most interested in having in there in case i do have to do a reinstall). snip Why not copy all files back and order something like apt-get autoclean to get rid of the older packages? -- Regards, jvp. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/lqd9gi$5uo$1...@ger.gmane.org
Re: a simple script for restoring downloaded debs
Jörg-Volker Peetz wrote: songbird wrote: the other night i accidentally deleted the debs i normally keep in the /var/cache/apt/archives directory and while i do keep a backup of them in another directory it is along with all the previous versions too, so it isn't as easy as just copying them. instead i have to figure out which is the installed version (because that is the one i am most interested in having in there in case i do have to do a reinstall). snip Why not copy all files back and order something like apt-get autoclean to get rid of the older packages? it's a rather huge archive (i think about 18000 debs), but yes that would be another way of doing it. i think the flaw is that you would lose the local, obsolete or debs from unofficial sources that you no longer keep in the sources.list (but i would have to verify this). the script looks to have gotten the job mostly done. i've added a bit to copy my local, obsolete and unofficial debs back from another archive and i think what remains are those debs that i somehow missed archiving. sometimes i have to get files via the library and USB stick. songbird -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/jjbq9b-ng1@id-306963.user.uni-berlin.de
Re: a simple script for restoring downloaded debs
Andrei POPESCU wrote: songbird wrote: Andrei POPESCU wrote: songbird wrote: the other night i accidentally deleted the debs i normally keep in the /var/cache/apt/archives directory =3D2E..=3D20 so here is round 1 of the script (which gets 90% of the packages back for me from my backup archive at /myarchive/debian/jessie).=3D20 A few suggestions: - to find packages you might want to use dpkg-query instead of parsing= =3D20 the output of 'dpkg -l' =20 hmm, looked at: =20 me@ant(7)~$ dpkg-query -l | egrep ^ii | wc -l 2377 I was rather thinking about --showformat ... Kind regards, Andrei i see what you mean now. :) probably will leave it as is for the moment, i'm close enough now that i have to check what is left by hand to see why they're missing, i may have just not gotten those ones copied to the archive. as it stands the script is fairly done with a few added parts. i'll probably do that the next few days to add a bit to generate a download script for the missing debs. think that will pretty much take care of it. songbird -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/krbq9b-ng1@id-306963.user.uni-berlin.de
Re: a simple script for restoring downloaded debs
songbird wrote on 07/19/2014 16:23:
Jörg-Volker Peetz wrote:
snip
Why not copy all files back and order something like apt-get autoclean to
get
rid of the older packages?
it's a rather huge archive (i think about 18000
debs), but yes that would be another way of doing
it. i think the flaw is that you would lose the
local, obsolete or debs from unofficial sources
that you no longer keep in the sources.list (but
i would have to verify this).
Yes, you would lose them.
the script looks to have gotten the job mostly
done. i've added a bit to copy my local, obsolete
and unofficial debs back from another archive and
i think what remains are those debs that i somehow
missed archiving. sometimes i have to get files
via the library and USB stick.
songbird
A note on your scipt:
If you build a pipe of grep, sed, and awk, it can often be done just with awk.
E.g., your line
debname=`apt-cache show $pkgname=$version | egrep '^Filename: ' | sed -e
's/^Filename: //' | awk --field-separator='/' -e '{print \$NF}'`
could be written as something like
debname=$(apt-cache show $pkgname=$version | awk '/^Filename: /
{sub(.*/,, $2) ; print $2}')
saving the call of egrep and sed.
--
Regards,
jvp.
--
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/lqebst$hvr$1...@ger.gmane.org
Re: a simple script for restoring downloaded debs
On Vi, 18 iul 14, 00:02:43, songbird wrote: the other night i accidentally deleted the debs i normally keep in the /var/cache/apt/archives directory ... so here is round 1 of the script (which gets 90% of the packages back for me from my backup archive at /myarchive/debian/jessie). A few suggestions: - to find packages you might want to use dpkg-query instead of parsing the output of 'dpkg -l' - to download the .deb if not available in your backup you could use something like 'aptitude reinstall --download-only' Kind regards, Andrei -- http://wiki.debian.org/FAQsFromDebianUser Offtopic discussions among Debian users and developers: http://lists.alioth.debian.org/mailman/listinfo/d-community-offtopic http://nuvreauspam.ro/gpg-transition.txt signature.asc Description: Digital signature
Re: a simple script for restoring downloaded debs
Andrei POPESCU wrote: songbird wrote: the other night i accidentally deleted the debs i normally keep in the /var/cache/apt/archives directory =2E..=20 so here is round 1 of the script (which gets 90% of the packages back for me from my backup archive at /myarchive/debian/jessie).=20 A few suggestions: - to find packages you might want to use dpkg-query instead of parsing=20 the output of 'dpkg -l' hmm, looked at: me@ant(7)~$ dpkg-query -l | egrep ^ii | wc -l 2377 me@ant(8)~$ dpkg -l | egrep ^ii | wc -l 2377 me@ant(9)~$ dpkg -l | egrep ^ii | wc -c 331166 me@ant(10)~$ dpkg-query -l | egrep ^ii | wc -c 331166 me@ant(11)~$ dpkg-query -l | egrep ^iiaa me@ant(12)~$ dpkg -l | egrep ^iibb me@ant(13)~$ diff aa bb no difference between those lists... - to download the .deb if not available in your backup you could use=20 something like 'aptitude reinstall --download-only' yes, i am aware of those options if needed, so far i'm just trying to repopulate from the local archive as much as possible before downloading anything again. on this connection it can take quite a while to get some of the larger debs. using debdelta saves me quite a bit sometimes. getting there... :) thanks for your reply, cheers, songbird -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/9oon9b-lo1@id-306963.user.uni-berlin.de
Re: a simple script for restoring downloaded debs
On Vi, 18 iul 14, 10:49:13, songbird wrote: Andrei POPESCU wrote: songbird wrote: the other night i accidentally deleted the debs i normally keep in the /var/cache/apt/archives directory =2E..=20 so here is round 1 of the script (which gets 90% of the packages back for me from my backup archive at /myarchive/debian/jessie).=20 A few suggestions: - to find packages you might want to use dpkg-query instead of parsing=20 the output of 'dpkg -l' hmm, looked at: me@ant(7)~$ dpkg-query -l | egrep ^ii | wc -l 2377 I was rather thinking about --showformat ... Kind regards, Andrei -- http://wiki.debian.org/FAQsFromDebianUser Offtopic discussions among Debian users and developers: http://lists.alioth.debian.org/mailman/listinfo/d-community-offtopic http://nuvreauspam.ro/gpg-transition.txt signature.asc Description: Digital signature
Re: a simple script for restoring downloaded debs
songbird wrote:
ok, figured the four that are missing and generating
null file names are local or obsolete packages that i
have installed... that i can work around.
new version of script (with some simple changes to
make things go faster and to make it more flexible
for other uses):
=
#!/bin/sh
#
#
debarch=/archive/debian/jessie
dest=/var/cache/apt/archives
pkglist=`dpkg -l | egrep '^ii ' | cut --delimiter=' ' -f3,3 | cut
--delimiter=':' -f1,1`
versionlist=`dpkg -l | egrep '^ii ' | sed -e 's/ */ /g' | cut -d ' ' -f3,3`
for pkgname in $pkglist; do
version=`echo $versionlist | cut -d ' ' -f1,1`
versionlist=`echo $versionlist | cut -d ' ' -f2-`
debname=`apt-cache show $pkgname=$version | egrep '^Filename: ' | sed -e
's/^Filename: //' | awk --field-separator='/' -e '{print \$NF}'`
if test -e $debarch/$debname ; then
echo $debarch/$debname $dest /dev/null
if test ! -e $dest/$debname ; then
cp -a $debarch/$debname $dest
fi
else
echo $debname missing? version -$version-
fi
done
chown root $dest/*
chgrp root $dest/*
sync
--
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/umkm9b-nba@id-306963.user.uni-berlin.de
Re: a simple script for restoring downloaded debs
songbird wrote:
...
and the next version, which gets me down
to 61 missing files.
it is getting late so i'll have to take
another look at this again later...
=
#!/bin/sh
#
#
debarch=/archives/debian/jessie
dest=/var/cache/apt/archives
pkglist=`dpkg -l | egrep '^ii ' | cut --delimiter=' ' -f3,3 | cut
--delimiter=':' -f1,1`
versionlist=`dpkg -l | egrep '^ii ' | sed -e 's/ */ /g' | cut -d ' ' -f3,3`
for pkgname in $pkglist; do
version=`echo $versionlist | cut -d ' ' -f1,1`
debname=`apt-cache show $pkgname=$version | egrep '^Filename: ' | sed -e
's/^Filename: //' | awk --field-separator='/' -e '{print \$NF}'`
versionplay=`echo $version | grep ':' | wc -c`
if test ! $versionplay = 0 ; then
#echo We have a funny version number, let's fix it in the filename
versionhead=`echo $version | cut -d ':' -f1,1 | sed -e 's/$/%3a/'`
#echo $pkgname $version $versionhead $debname
debname=`echo $debname | sed -e s/_/_$versionhead/`
#echo $pkgname $version $versionhead $debname
fi
versionlist=`echo $versionlist | cut -d ' ' -f2-`
if test -e $debarch/$debname ; then
echo $debarch/$debname $dest /dev/null
if test ! -e $dest/$debname ; then
cp -a $debarch/$debname $dest
fi
else
echo $debname missing? version -$version-
fi
done
chown root $dest/*
chgrp root $dest/*
sync
=
songbird
--
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/8kpm9b-6cq@id-306963.user.uni-berlin.de
