Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
On Wed, Jun 24, 2015 at 11:25:54PM +0200, Johannes Schauer wrote: Hi, Quoting David Kalnischkies (2015-06-24 23:18:12) So, long story short, the solution I would like to propose is: $ apt-get files Codename: sid Trusted: yes Created-By: Packages […] so the arguments to apt-get files are fields and values and the command will output all paragraphs where all those fields match? Can one omit these field:value pairs to get everything? This would allow doing a custom selection with the more powerful grep-dctrl tool for example. Sure, but I realize now the doc isn't very clear about that… In fact, it is pretty much expected to us another tool to do the heavy lifting as I really don't want to implement any kind of fancy filtering, hence the deb822 format as in a Debian context that should be easy to deal with. This crude line-filtering is mostly there for times in which you don't want/can't use anything more powerful (like apt testcases where piping is hard™ compared to just calling a single command and performing (additional) tests on its exit status and output automatically). I will add some explicit mentioning of grep-dctrl and an example for good measure. Thanks for the quick feedback! Best regards David Kalnischkies signature.asc Description: Digital signature
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
On Thu, Jun 26, 2014 at 10:58:38PM +0200, David Kalnischkies wrote:
On Thu, Jun 26, 2014 at 10:30:24PM +0200, Johannes Schauer wrote:
Quoting David Kalnischkies (2014-06-26 21:48:42)
I think it is a bad idea to use our lib-directory directly.
if apt developers agree that it is a bad idea to allow 3rd party software
access /var/lib/apt/lists, then please close this bug report.
Just to be clear: I think it is incorrect to use /var/lib/apt/lists
directly as you can never be sure what is in there. If you get the
filename from the API on the other hand, that is okay.
So, in the light of acquiring additional files apt itself has no use
for, but other frontends want without implementing all the boring
security sensitive download of it, I had to think about this problem
a bit again.
As I don't want the frontends to control the storage location (that
would just mean having e.g. the big Contents files in three different
places just because frontends can't decide on a common location) I have
to store them in lists/ and expose information about them somehow
– especially also about the Release file they came from as I really
don't want them to parse these… (detached sig, clear sig, different
names, unsigned content in clearsig files, …).
The upside to this is that I can handle Packages, Sources and
Translation-* files in the very same way so they are exposed over the
same API and hence deal with your problem too, even through these files
hardly qualify as additional.
So, long story short, the solution I would like to propose is:
$ apt-get files Codename: sid Trusted: yes Created-By: Packages
[…]
MetaKey: main/binary-amd64/Packages
ShortDesc: Packages
Description: http://http.debian.net/debian sid/main amd64 Packages
URI: http://http.debian.net/debian/dists/sid/main/binary-amd64/Packages
Filename:
/var/lib/apt/lists/http.debian.net_debian_dists_sid_main_binary-amd64_Packages
Optional: no
Codename: sid
Label: Debian
Origin: Debian
Suite: unstable
Trusted: yes
Architecture: amd64
Base-URI: http://http.debian.net/debian/dists/sid/
Component: main
Created-By: Packages
Release: sid
Repo-URI: http://http.debian.net/debian/
Site: http://http.debian.net/debian
Target-Of: deb
[…]
So basically our trusted deb822 format we see everywhere else and most
people who will want to use it probably already have a parser for. Or
add a --format=$(FILENAME) and you have just the filename(s) of
authenticated Packages files from a Release with the codename 'sid'.
(In both cases the filename given is already checked for existance and
if it is compressed you get the filename of the compressed file of
course)
The C++ interface is pretty much what it used to be with reading
sources.list via pkgSourceList, iterating over metaIndex's this creates
(~ Release files) to ask them which files will be (tried to be)
requested from here: IndexTarget, which is new in this context and
contains most of the data you asked for.
You can also now 'map' a metaIndex to a (new) ReleaseFile struct in the
Cache, which will actually contain codename and co – metaIndex has no
concept of codename, it just knows what was given in the sources.list,
while ReleaseFile actually contains the information parsed from an
{In,}Release file. (This used to be possible only for debPackagesIndex
as the information belonging to a Release file was stored as if it
belonged to the Packages file – which used to be correct for the old
flat-style repos) Both combined, you should have all the information you
ever wanted…
Sounds complicated, but should amount to only a few lines of code on
your end, just see the 'apt-get file' implementation for an example.
Some more detailed 'user' documentation can be found in
./doc/acquire-additional-files.txt [0].
I would consider this bug done if this code reaches a release (sofar it
is sitting in the debian/experimental branch in git as this is part of
the acquire system rework in general and one giant API break…), but
would be very happy if you and/or anyone else could comment on the
usefulness (or not) of it in the mean time. There are some things
still missing and I am not too sure if I will stick to the very generic
files command or invent something else, but that is at least the
general direction… Tech-preview status I would say.
Best regards
David Kalnischkies
[0]
https://anonscm.debian.org/cgit/apt/apt.git/tree/doc/acquire-additional-files.txt?h=debian/experimental
signature.asc
Description: Digital signature
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
Hi, Quoting David Kalnischkies (2015-06-24 23:18:12) So, long story short, the solution I would like to propose is: $ apt-get files Codename: sid Trusted: yes Created-By: Packages […] so the arguments to apt-get files are fields and values and the command will output all paragraphs where all those fields match? Can one omit these field:value pairs to get everything? This would allow doing a custom selection with the more powerful grep-dctrl tool for example. I would consider this bug done if this code reaches a release (sofar it is sitting in the debian/experimental branch in git as this is part of the acquire system rework in general and one giant API break…), but would be very happy if you and/or anyone else could comment on the usefulness (or not) of it in the mean time. There are some things still missing and I am not too sure if I will stick to the very generic files command or invent something else, but that is at least the general direction… Tech-preview status I would say. works for me, thanks! cheers, josch signature.asc Description: signature
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
On Thu, Jun 26, 2014 at 11:13:28PM +0200, Johannes Schauer wrote:
Hi,
Hi,
[..]
I was thinking that it would nice if apt would provide an API to retrieve the
location and properties of Packages and Sources files in /var/lib/apt/lists
If apt had such an interface, then third party applications which make use of
Packages and Sources files like dose3, ben and botch could directly make use
of
those files and the user would not have to retrieve them from somewhere else.
`apt-cache dumpavail` doesnt work well here because it prints all available
binary packages and doesnt allow to select a suite or distribution. It also
doesnt allow listing source packages.
[..]
Ok, that is certainly doable, attached is a very simple patch that
adds a local filename (abi break). I guess we probably want to return
a struct instead that describes it a bit more. But as David points
out, there are flat archives that are really just a Packages file
with mixed architectures so its not always meaningful.
Cheers,
Michael
You mentioned creating a mapping between sources.list and files in
/var/lib/apt/lists. This could be one way to solve this but it would not be
the
only way.
Here more detail about he use case: a 3rd party application (like botch or
dose3 or ben) work on Packages and Sources files. If the user now wants to let
Debian sid amd64 be analyzed by any of these utilities, then they first have
to
require Packages and Sources files for Debian sid amd64. They'd have to do
that
even though apt might already have usable ones in /var/lib/apt/lists.
Unfortunately, as you already pointed out it is not safe to use anything in
/var/lib/apt/lists yet. It would be nice if apt could be queried about the
content of /var/lib/apt/lists so that 3rd party applications can then decide
whether they can make use of those contents and can thus avoid extra
downloads.
I hope this makes more sense now?
cheers, josch
--
To UNSUBSCRIBE, email to deity-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/20140626211328.3886.46854@hoothoot
diff --git a/apt-pkg/deb/debindexfile.h b/apt-pkg/deb/debindexfile.h
index 18322dc..266a1b7 100644
--- a/apt-pkg/deb/debindexfile.h
+++ b/apt-pkg/deb/debindexfile.h
@@ -51,6 +51,8 @@ class debStatusIndex : public pkgIndexFile
bool Merge(pkgCacheGenerator Gen,OpProgress *Prog, unsigned long const Flag) const;
virtual pkgCache::PkgFileIterator FindInCache(pkgCache Cache) const;
+ virtual std::string LocalFileName() const {return File;};
+
debStatusIndex(std::string File);
virtual ~debStatusIndex() {};
};
@@ -87,6 +89,8 @@ class debPackagesIndex : public pkgIndexFile
virtual bool Merge(pkgCacheGenerator Gen,OpProgress *Prog) const;
virtual pkgCache::PkgFileIterator FindInCache(pkgCache Cache) const;
+ virtual std::string LocalFileName() const {return IndexFile(Packages);};
+
debPackagesIndex(std::string const URI, std::string const Dist, std::string const Section,
bool const Trusted, std::string const Arch = native);
virtual ~debPackagesIndex() {};
@@ -123,6 +127,8 @@ class debTranslationsIndex : public pkgIndexFile
virtual bool Merge(pkgCacheGenerator Gen,OpProgress *Prog) const;
virtual pkgCache::PkgFileIterator FindInCache(pkgCache Cache) const;
+ virtual std::string LocalFileName() const {return IndexFile(Language);};
+
debTranslationsIndex(std::string URI,std::string Dist,std::string Section, char const * const Language);
virtual ~debTranslationsIndex() {};
};
@@ -160,6 +166,8 @@ class debSourcesIndex : public pkgIndexFile
virtual bool HasPackages() const {return false;};
virtual unsigned long Size() const;
+ virtual std::string LocalFileName() const {return IndexFile(Sources);};
+
debSourcesIndex(std::string URI,std::string Dist,std::string Section,bool Trusted);
virtual ~debSourcesIndex() {};
};
@@ -190,6 +198,8 @@ class debDebPkgFileIndex : public pkgIndexFile
// Interface for acquire
virtual std::string ArchiveURI(std::string /*File*/) const;
+ virtual std::string LocalFileName() const {return DebFile;};
+
debDebPkgFileIndex(std::string DebFile);
virtual ~debDebPkgFileIndex() {};
};
@@ -207,6 +217,7 @@ class debDscFileIndex : public pkgIndexFile
virtual std::string Describe(bool /*Short*/) const {
return DscFile;
};
+ virtual std::string LocalFileName() const {return DscFile;};
debDscFileIndex(std::string DscFile);
virtual ~debDscFileIndex() {};
diff --git a/apt-pkg/indexfile.h b/apt-pkg/indexfile.h
index 817165f..9a95725 100644
--- a/apt-pkg/indexfile.h
+++ b/apt-pkg/indexfile.h
@@ -95,6 +95,9 @@ class pkgIndexFile
static std::string LanguageCode();
bool IsTrusted() const { return Trusted; };
+
+ // returns the path of the local file (or if its not available)
+ virtual std::string LocalFileName() const {return ;};
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
On Thu, Jun 26, 2014 at 11:13:28PM +0200, Johannes Schauer wrote: Quoting David Kalnischkies (2014-06-26 22:58:38) If apt had such an interface, then third party applications which make use of Packages and Sources files like dose3, ben and botch could directly make use of those files and the user would not have to retrieve them from somewhere else. `apt-cache dumpavail` doesnt work well here because it prints all available binary packages and doesnt allow to select a suite or distribution. It also doesnt allow listing source packages. Well, I still kinda like my workaround presented in my first mail as it tries to solve this usecase at a higher level, which you haven't commented on so far. Regardless if you want to use the files apt already has or not, you still have to get those files if you can't find them (e.g. stable machine trying to make an anaylse for unstable). So you need acquire code to do all that and all these tools probably don't want to reimplement what apt already does in all its glory with multiple methods, proxy detection, security and what not (or, they at least should not). On the other hand, you could let apt do all this for you by creating a few directories, a sources.list and set some config options. You get file reuse for free then if you copy the lists directory over… This way a 'dumpavail' would include only packages you care about (as the sources.list comes from you), up-to-date files already downloaded are reused (instead of potentially reusing months old files if user has no way of updating as she isn't root) and you get told if one of the files is not as trustworthy as you might want (as apt-get update or similar carries a warning/error instead of you guessing based on other files (not) present in lists). [We would need a 'dumpavailsource' (or similar such) of course] (This thinking is shell-based as I am most familiar with it thanks to our tests, but I assume it is even better/simpler with python-apt) Best regards David Kalnischkies signature.asc Description: Digital signature
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
Hi, Quoting David Kalnischkies (2014-07-08 19:24:33) Well, I still kinda like my workaround presented in my first mail as it tries to solve this usecase at a higher level, which you haven't commented on so far. Sorry, let me do that now: If you like REALLY want to go the extra mile, you could write your own sources.list file, copy our lists directory to somewhere else and run apts acquire code against this lists directory and this sources. It will download the files missing, reuse uptodate files or update them otherwise and delete files via list-cleanup which are not referred to by your sources.list. This should be relatively sane and gives you all the features (like auto-proxy detection) you could possibly ask for for free as well. What do you mean by the apts acquire code? Otherwise it sounds like a good idea but it still leaves me with somehow having to interpret which file in my new lists directory represents what. And there I'm still at the mercy of apt not changing the filename pattern (which is a bad idea). So I would still need a way to know which file represents which of my results. After all, for bootstrapping for example one needs at least two files, one for Packages and one for Sources and more often than not it's even more files because one needs more than one architecture. Regardless if you want to use the files apt already has or not, you still have to get those files if you can't find them (e.g. stable machine trying to make an anaylse for unstable). So you need acquire code to do all that and all these tools probably don't want to reimplement what apt already does in all its glory with multiple methods, proxy detection, security and what not (or, they at least should not). You are right. It would be great if I could let the apt code do all that for me instead of hacking something together as I have done it until now. On the other hand, you could let apt do all this for you by creating a few directories, a sources.list and set some config options. You get file reuse for free then if you copy the lists directory over… I know how easy it is to let apt work with a completely different set of directories (I implemented my own minimal multistrap-like tool) but same argument as above: I still need to know which of the downloaded files in my lists directory is which. This way a 'dumpavail' would include only packages you care about (as the sources.list comes from you), up-to-date files already downloaded are reused (instead of potentially reusing months old files if user has no way of updating as she isn't root) and you get told if one of the files is not as trustworthy as you might want (as apt-get update or similar carries a warning/error instead of you guessing based on other files (not) present in lists). But `apt-cache dumpavail` does only give me binary packages and not source packages. Is there a way to get source packages too? [We would need a 'dumpavailsource' (or similar such) of course] Oh right, yes :) (This thinking is shell-based as I am most familiar with it thanks to our tests, but I assume it is even better/simpler with python-apt) I guess your method would work if there was a dumpavailsrc. Then one would use apt to download or update all the archives one wants and use two calls to get all binary and source packages. How hard is it to create a dumpavailsrc? cheers, josch -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
On Wed, Jun 25, 2014 at 10:16:57PM +0200, Johannes Schauer wrote: some software in Debian works on Packages and Sources files. Instead of retrieving those from mirrors they could re-use the copies which are likely present in /var/lib/apt/lists/. But there exists no way to ask apt which mirror, suite or architecture any of these files belong to, short of parsing the description string returned by debPackagesIndex.Describe() I think it is a bad idea to use our lib-directory directly. A lot of strange stuff happens in there, from InRelease to Release renames as Julian mentioned, over gzip compressed indexes (Acquire::GzipIndexes) to files which are not authenticated. Sometimes even unused stuff is still there (List-Cleanup disabled – aptitude did it for a while if I remember correctly, if it isn't still doing it). I probably forgot the other more obscure half and nobody knows what the future will bring aka: This is a really horrible interface to work with so much that I would agrue that every user is a bug (but I guess this is mostly a we have no other choice at the moment bug…). I would treat it similar to the control files dpkg stores in its own lib directory: You know they are there, but you still should ask for the content via dpkg interfaces and not access/parse the files directly yourself as occasionally filenames/orga-structure changes and your tool/package would be one more thing standing in the way of progress. For binary packages Julian already mentioned the 'correct' API. I am pretty sure it could be improved though. Maybe EDSP(-like) could be a handy format for you as well as it bundles quite a few data files. Or just plain old 'apt-cache dumpavail'. Note btw that there are also flat-repositories (still) in existance which do not have suite, architecture and all that by design, so all you can really work with is the filename if you really want filename resolution instead of all. For source, we have only a find the (next) record for this package in all files at the moment. We could change that -- one less program parsing our lib-directory might be a good motiviation to do that, if you tell us what you need exactly… It sounds a bit like you are trying to match sources.list entries to your configured sources to indentify which files you could reuse. If so I think you are trying too hard. Other tools like debootstrap and stuff do not do this either. If users really care about the wasted download, they will have a proxy configured anyway, non-root users might be suprised if days-old data from lists is used instead of fresh stuff acquired from the net, … If you like REALLY want to go the extra mile, you could write your own sources.list file, copy our lists directory to somewhere else and run apts acquire code against this lists directory and this sources. It will download the files missing, reuse uptodate files or update them otherwise and delete files via list-cleanup which are not referred to by your sources.list. This should be relatively sane and gives you all the features (like auto-proxy detection) you could possibly ask for for free as well. Best regards David Kalnischkies signature.asc Description: Digital signature
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
Hi David, Quoting David Kalnischkies (2014-06-26 21:48:42) I think it is a bad idea to use our lib-directory directly. if apt developers agree that it is a bad idea to allow 3rd party software access /var/lib/apt/lists, then please close this bug report. I will then redirect those who want botch to make use of /var/lib/apt/lists to this bug report. Thank you! cheers, josch -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
On Thu, Jun 26, 2014 at 10:30:24PM +0200, Johannes Schauer wrote: Quoting David Kalnischkies (2014-06-26 21:48:42) I think it is a bad idea to use our lib-directory directly. if apt developers agree that it is a bad idea to allow 3rd party software access /var/lib/apt/lists, then please close this bug report. Just to be clear: I think it is incorrect to use /var/lib/apt/lists directly as you can never be sure what is in there. If you get the filename from the API on the other hand, that is okay. Compare it to cat /var/lib/dpkg/info/apt.preinst vs dpkg-query --control-show apt preinst to get the content of preinst script. For binary packages this exists already, but not for sources packages. There is (also) no (simple) mapping from sources.list to Packages/Sources files and back. Both could surely be added if we have a compelling reason/user for this. apt-* tools so far just never needed it so it doesn't exist. That said, I don't want to promise that apt would instantly grow support for this or that. I still haven't understood what you are actually trying to achieve and why things like the workaround I mentioned last do not work for you so I am 'happily' shooting into the dark here and not close it just yet. Best regards David Kalnischkies signature.asc Description: Digital signature
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
Hi, Quoting David Kalnischkies (2014-06-26 22:58:38) For binary packages this exists already, but not for sources packages. There is (also) no (simple) mapping from sources.list to Packages/Sources files and back. Both could surely be added if we have a compelling reason/user for this. apt-* tools so far just never needed it so it doesn't exist. That said, I don't want to promise that apt would instantly grow support for this or that. I still haven't understood what you are actually trying to achieve and why things like the workaround I mentioned last do not work for you so I am 'happily' shooting into the dark here and not close it just yet. thanks a lot for having patience with me :) I was thinking that it would nice if apt would provide an API to retrieve the location and properties of Packages and Sources files in /var/lib/apt/lists If apt had such an interface, then third party applications which make use of Packages and Sources files like dose3, ben and botch could directly make use of those files and the user would not have to retrieve them from somewhere else. `apt-cache dumpavail` doesnt work well here because it prints all available binary packages and doesnt allow to select a suite or distribution. It also doesnt allow listing source packages. You mentioned creating a mapping between sources.list and files in /var/lib/apt/lists. This could be one way to solve this but it would not be the only way. Here more detail about he use case: a 3rd party application (like botch or dose3 or ben) work on Packages and Sources files. If the user now wants to let Debian sid amd64 be analyzed by any of these utilities, then they first have to require Packages and Sources files for Debian sid amd64. They'd have to do that even though apt might already have usable ones in /var/lib/apt/lists. Unfortunately, as you already pointed out it is not safe to use anything in /var/lib/apt/lists yet. It would be nice if apt could be queried about the content of /var/lib/apt/lists so that 3rd party applications can then decide whether they can make use of those contents and can thus avoid extra downloads. I hope this makes more sense now? cheers, josch -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
Package: apt
Version: 1.0.1
Severity: wishlist
Hi,
some software in Debian works on Packages and Sources files. Instead of
retrieving those from mirrors they could re-use the copies which are
likely present in /var/lib/apt/lists/. But there exists no way to ask
apt which mirror, suite or architecture any of these files belong to,
short of parsing the description string returned by
debPackagesIndex.Describe()
Consider the following code example:
--%---
#include apt-pkg/init.h
#include apt-pkg/sourcelist.h
#include apt-pkg/debindexfile.h
#include stdio.h
#include string.h
int main() {
pkgInitConfig(*_config);
pkgInitSystem(*_config,_system);
vectormetaIndex *::const_iterator I;
pkgSourceList list;
list.ReadMainList();
for (I=list.begin(); I != list.end(); ++I) {
string diststr = (*I)-GetDist();
string uristr = (*I)-GetURI();
std::cout dist: diststr uristr: uristr \n;
vectorpkgIndexFile *::const_iterator J;
vectorpkgIndexFile * *indexes = (*I)-GetIndexFiles();
for (J=indexes-begin(); J != indexes-end(); ++J) {
if (strcmp((*J)-GetType()-Label, Debian Package Index) == 0) {
debPackagesIndex* i = (debPackagesIndex*)(*J);
std::cout i-Describe(false) \n;
}
}
}
}
--%---
it will print something like:
dist: jessie uristr: http://ftp.debian.org/debian/
http://ftp.debian.org/debian/ jessie/main amd64 Packages
(/var/lib/apt/lists/ftp.debian.org_debian_dists_jessie_main_binary-amd64_Packages)
http://ftp.debian.org/debian/ jessie/main i386 Packages
(/var/lib/apt/lists/ftp.debian.org_debian_dists_jessie_main_binary-i386_Packages)
dist: sid uristr: http://ftp.debian.org/debian/
http://ftp.debian.org/debian/ sid/main amd64 Packages
(/var/lib/apt/lists/ftp.debian.org_debian_dists_sid_main_binary-amd64_Packages)
http://ftp.debian.org/debian/ sid/main i386 Packages
(/var/lib/apt/lists/ftp.debian.org_debian_dists_sid_main_binary-i386_Packages)
dist: experimental uristr: http://ftp.debian.org/debian/
http://ftp.debian.org/debian/ experimental/main amd64 Packages
(/var/lib/apt/lists/ftp.debian.org_debian_dists_experimental_main_binary-amd64_Packages)
http://ftp.debian.org/debian/ experimental/main i386 Packages
(/var/lib/apt/lists/ftp.debian.org_debian_dists_experimental_main_binary-i386_Packages)
So I can find the distribution and the uri but I cannot find out the
architecture or suite except if I start relying on the Description
string format staying the same which is probably not a good idea.
Having looked at the code, it would probably be easy to add a few more
getter functions which return the individual private members that
Describe() uses to produce the description string?
cheers, josch
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
On Wed, Jun 25, 2014 at 10:16 PM, Johannes Schauer j.scha...@email.de wrote: Package: apt Version: 1.0.1 Severity: wishlist Hi, some software in Debian works on Packages and Sources files. Instead of retrieving those from mirrors they could re-use the copies which are likely present in /var/lib/apt/lists/. But there exists no way to ask apt which mirror, suite or architecture any of these files belong to, short of parsing the description string returned by debPackagesIndex.Describe() That software should use the API provided by APT to access APT files. The layout of /var/lib/apt/lists and the contents of the files is an implementation detail, we already broke the expectations of others once when we renamed InRelease files to Release files if we could not trust them. Furthermore, APT might store indices compressed in various formats. With future archive changes, things might even become more different. -- Julian Andres Klode - Debian Developer, Ubuntu Member See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
Hi Julian, Quoting Julian Andres Klode (2014-06-25 22:28:51) That software should use the API provided by APT to access APT files. The layout of /var/lib/apt/lists and the contents of the files is an implementation detail, we already broke the expectations of others once when we renamed InRelease files to Release files if we could not trust them. Furthermore, APT might store indices compressed in various formats. With future archive changes, things might even become more different. maybe I am misreading your message but That software should use the API provided by APT sounds as if there is an API to retrieve the information that is otherwise given by the string returned by debPackagesIndex.Describe(). Which function am I overlooking? cheers, josch -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
On Wed, Jun 25, 2014 at 10:34 PM, Johannes Schauer j.scha...@email.de wrote: Hi Julian, Quoting Julian Andres Klode (2014-06-25 22:28:51) That software should use the API provided by APT to access APT files. The layout of /var/lib/apt/lists and the contents of the files is an implementation detail, we already broke the expectations of others once when we renamed InRelease files to Release files if we could not trust them. Furthermore, APT might store indices compressed in various formats. With future archive changes, things might even become more different. maybe I am misreading your message but That software should use the API provided by APT sounds as if there is an API to retrieve the information that is otherwise given by the string returned by debPackagesIndex.Describe(). Which function am I overlooking? Now I understand what you mean. I thought you meant the location of the files on disk. Do you mean the location on the mirror instead? Stuff like architecture, etc. is exported in the cache in pkgCache::PackageFile. -- Julian Andres Klode - Debian Developer, Ubuntu Member See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
Hi,
Quoting Julian Andres Klode (2014-06-25 22:40:40)
Now I understand what you mean. I thought you meant the location of the files
on disk. Do you mean the location on the mirror instead? Stuff like
architecture, etc. is exported in the cache in pkgCache::PackageFile.
All of it. The location on disk, the arch, the suite, the mirror url, the
component...
Ideal would be a query interface like: give me the binary package file on disk
for debian sid, main, amd64. But the other way round would probably be much
easier to implement: extend the existing interface to provide this information
for all files in the main list.
Back to the code snippet I posted:
01for (I=list.begin(); I != list.end(); ++I) {
02string diststr = (*I)-GetDist();
03string uristr = (*I)-GetURI();
04std::cout dist: diststr uristr: uristr \n;
05vectorpkgIndexFile *::const_iterator J;
06vectorpkgIndexFile * *indexes = (*I)-GetIndexFiles();
07for (J=indexes-begin(); J != indexes-end(); ++J) {
08if (strcmp((*J)-GetType()-Label, Debian Package Index) == 0) {
09debPackagesIndex* i = (debPackagesIndex*)(*J);
10std::cout i-Describe(false) \n;
11}
12}
13}
the distribution and uristring can already be accessed (see lines 2 and 3) but
I do not see a way to get the architecture, the suite, or the exact filename
for example even though you say it's possible through pkgCache::PackageFile. If
yes - how?
The function debPackagesIndex-Describe() gives information about architecture,
suite and filename but that information is encoded in a string which I do not
think is safe to parse as it cannot be guaranteed that its format will stay the
same in the future, as it is meant for human consumption, no?
So what I'm looking for are functions like debPackagesIndex-getArch(),
debPackagesIndex-getSuite() and debPackagesIndex-getFilename(). Having these
functions is what I want to make this bugreport about.
Some minor things are, that I apparently have to forcefully cast the
pkgIndexFile to a debPackagesIndex in line 9 based on a string comparison in
line 8. There does not seem to be a better way to determine the type of a
pkgIndexFile object than to call GetType() and do a string comparison. It would
be better to have an enum or similar for this, no?
cheers, josch
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
On Wed, Jun 25, 2014 at 10:56 PM, Johannes Schauer j.scha...@email.de wrote:
Hi,
Quoting Julian Andres Klode (2014-06-25 22:40:40)
Now I understand what you mean. I thought you meant the location of the files
on disk. Do you mean the location on the mirror instead? Stuff like
architecture, etc. is exported in the cache in pkgCache::PackageFile.
All of it. The location on disk, the arch, the suite, the mirror url, the
component...
Well, as I said WRT on disk. We expose this, but you should most
likely not use it without APT parsers, as it may not contain data you
expect (for example, compressed indices). Using APT's parsers for
those files is the best idea.
Ideal would be a query interface like: give me the binary package file on disk
for debian sid, main, amd64. But the other way round would probably be much
easier to implement: extend the existing interface to provide this information
for all files in the main list.
Back to the code snippet I posted:
01for (I=list.begin(); I != list.end(); ++I) {
02string diststr = (*I)-GetDist();
03string uristr = (*I)-GetURI();
04std::cout dist: diststr uristr: uristr \n;
05vectorpkgIndexFile *::const_iterator J;
06vectorpkgIndexFile * *indexes = (*I)-GetIndexFiles();
07for (J=indexes-begin(); J != indexes-end(); ++J) {
08if (strcmp((*J)-GetType()-Label, Debian Package Index) ==
0) {
09debPackagesIndex* i = (debPackagesIndex*)(*J);
10std::cout i-Describe(false) \n;
11}
12}
13}
the distribution and uristring can already be accessed (see lines 2 and 3) but
I do not see a way to get the architecture, the suite, or the exact filename
for example even though you say it's possible through pkgCache::PackageFile.
If
yes - how?
You are reading the wrong list. You need to go through the cache, not
the pkgSourceList. pkgSourceList give you IndexFiles, the cache gives
you PackageFiles you can iterate over. You need to loop through the
files in the cache starting at Cache-FileBegin() and ending at
Cache-FileEnd(). This gives you pkgCache::PkgFileIterator() with
stuff like FileName, Archive, Component, Version, Architecture.
--
Julian Andres Klode - Debian Developer, Ubuntu Member
See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
Hi, Quoting Julian Andres Klode (2014-06-25 23:02:33) Well, as I said WRT on disk. We expose this, but you should most likely not use it without APT parsers, as it may not contain data you expect (for example, compressed indices). Using APT's parsers for those files is the best idea. what are compressed indices? I expected files that identify as Debian Package Index in my code snipped to always be deb822 format files? I never encountered a situation in which they were not. You are reading the wrong list. You need to go through the cache, not the pkgSourceList. pkgSourceList give you IndexFiles, the cache gives you PackageFiles you can iterate over. You need to loop through the files in the cache starting at Cache-FileBegin() and ending at Cache-FileEnd(). This gives you pkgCache::PkgFileIterator() with stuff like FileName, Archive, Component, Version, Architecture. but it gives me only binary packages and not source packages, right? To come back to my original problem: /var/lib/apt/lists/ contains Packages and Sources files and I want to know details about them. So far I only see how to get the information in a way that isnt reliable because a string has to be parsed or which limits me to binary packages. If some files in /var/lib/apt/lists/ are not in deb822 format, there should be a way to find that out too. cheers, josch -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
On Wed, Jun 25, 2014 at 11:30 PM, Johannes Schauer j.scha...@email.de wrote: Hi, Quoting Julian Andres Klode (2014-06-25 23:02:33) Well, as I said WRT on disk. We expose this, but you should most likely not use it without APT parsers, as it may not contain data you expect (for example, compressed indices). Using APT's parsers for those files is the best idea. what are compressed indices? I expected files that identify as Debian Package Index in my code snipped to always be deb822 format files? I never encountered a situation in which they were not. You can enable storing them in compressed form without uncompressing them first. APT transparently handles compressed files. We do not need to store that information anywhere, because of that. As said, the files in there are an implementation details and other packages using them are not checked for breakage if a need for a change arises. The same applies to /var/lib/dpkg as well I've been told. You are reading the wrong list. You need to go through the cache, not the pkgSourceList. pkgSourceList give you IndexFiles, the cache gives you PackageFiles you can iterate over. You need to loop through the files in the cache starting at Cache-FileBegin() and ending at Cache-FileEnd(). This gives you pkgCache::PkgFileIterator() with stuff like FileName, Archive, Component, Version, Architecture. but it gives me only binary packages and not source packages, right? Right, yes. David wanted to do some sort of sources in cache in the future IIRC. The accessor things seems like the best idea then. -- Julian Andres Klode - Debian Developer, Ubuntu Member See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#752702: apt: please add a way to to retrieve the location of Packages/Sources files in /var/lib/apt/lists/
Hi, Quoting Julian Andres Klode (2014-06-26 00:20:44) You can enable storing them in compressed form without uncompressing them first. APT transparently handles compressed files. We do not need to store that information anywhere, because of that. As said, the files in there are an implementation details and other packages using them are not checked for breakage if a need for a change arises. my use case is botch (itp bug: #748102) which needs a pair of Packages and Sources files. Since it is tedious to manually grab a pair from a mirror, I thought it would make sense for the user to be able to use the local ones that apt stored in /var/lib/apt/lists/. Botch transparently decompresses gzip, bz2 and xz so that would not be a problem (in contrast to apt it uses the file magic instead of the filename extension). If the files in /var/lib/apt/lists/ are indeed deemed just an implementation detail then there should not be any API functions to retrieve their location and associated information. It's of course your decision to make. With this bugreport I just want to raise the issue that others may want to make use of the Packages/Sources files apt already downloaded and that if apt wants to allow that, then it needs some additional functions to retrieve more detailed information about the Packages/Sources files it retrieves. Thank you for your consideration! cheers, josch -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
