Re: On adding size info to Packages files [very long]
Manoj Srivastava wrote: Brederlow The size file should be generated by the server. Here are Brederlow the reasons: I am (perhaps unnecessarily) worried about time requirements Considering that we plan to eventually have lintian run automatically on packages before they get moved into the debian archives, and lintian unpacks each package itself, there's not much reason to worry about time. Running lintian on master is going to take longer than generating the du files. And if time does become a big concern, we could generate the du files from the lintian lab or something. Brederlow 3. Maintainer will forget to include a du index or have different Brederlowblock sizes. This can be made policy and a lintian check. We have to trust the maintainers a trifle ;-) How is it possible to check for block sizes with lintian? And what do you expect a maintainer to do if they use a different block size and lintian dislikes that? Reformat? -- see shy jo I'm on a long trip, pardon any delays in my reply. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: On adding size info to Packages files [very long]
Joey Hess [EMAIL PROTECTED] wrote: How is it possible to check for block sizes with lintian? And what do you expect a maintainer to do if they use a different block size and lintian dislikes that? Reformat? To deal with block sizes we'll need to abandon (or upgrade) du. To find out what block sizes are for a file system, use statfs (or fstatfs). This mostly matters on the target system, not where we're running lintian (which perhaps was Joey's point). A brutally simplistic mechanism for representing the data would provide a different sizes file for each different block size we support. Optimizations are possible but may not be worth the effort. -- Raul -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: On adding size info to Packages files [very long]
Raul Miller [EMAIL PROTECTED] wrote: To deal with block sizes we'll need to abandon (or upgrade) du. Argh.. please ignore this sentence, it makes no sense. -- Raul -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: On adding size info to Packages files [very long]
On Sat, 6 Jun 1998, Raul Miller wrote: A brutally simplistic mechanism for representing the data would provide a different sizes file for each different block size we support. Optimizations are possible but may not be worth the effort. This is getting out of hand, do we really need to consider slack space when calculating if the user has enough room to install!? Jason -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: On adding size info to Packages files [very long]
Jason Gunthorpe [EMAIL PROTECTED] wrote: This is getting out of hand, do we really need to consider slack space when calculating if the user has enough room to install!? No, what we mostly need is an estimate. -- Raul -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: On adding size info to Packages files [very long]
On Sat, 6 Jun 1998, Raul Miller wrote: Joey Hess [EMAIL PROTECTED] wrote: How is it possible to check for block sizes with lintian? And what do you expect a maintainer to do if they use a different block size and lintian dislikes that? Reformat? To deal with block sizes we'll need to abandon (or upgrade) du. To find out what block sizes are for a file system, use statfs (or fstatfs). This mostly matters on the target system, not where we're running lintian (which perhaps was Joey's point). A brutally simplistic mechanism for representing the data would provide a different sizes file for each different block size we support. Optimizations are possible but may not be worth the effort. Is du -k not the answer? Or am I misunderstanding you? Jules /+---+-\ | Jelibean aka | [EMAIL PROTECTED] | 6 Evelyn Rd| | Jules aka | | Richmond, Surrey | | Julian Bean | [EMAIL PROTECTED]| TW9 2TF *UK* | ++---+-+ | War doesn't demonstrate who's right... just who's left. | | When privacy is outlawed... only the outlaws have privacy. | \--/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: On adding size info to Packages files [very long]
Jules Bean [EMAIL PROTECTED] wrote: Is du -k not the answer? du -S, but you need to know how many files are in each directory to estimate block-size overhead -- assume that each file requires two thirds of a block of unused space and you won't be too far off. -- Raul -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: On adding size info to Packages files [very long]
Manoj Srivastava [EMAIL PROTECTED] writes: Hi, Brederlow == Brederlow [EMAIL PROTECTED] writes: Brederlow I mean, that when a package is installed, that the Brederlow recorded du tree (which is needed to calculate the size Brederlow increase/decrease for updates) could be trimmed to what Brederlow the users system reflects. The users system setup should Brederlow be scanned and kept in a status file for speed reasons Brederlow (trying all dirs for symlinks takes time) and then Brederlow trimming should be fairly easy and save a lot of space. It Brederlow would save the more the smaler (less partitions) the Brederlow system is. You have a point. Hmm. The sizes file I haegv been talking about is analogous to the Packages/available file; we also need the analogue of the Status file, and it may make sense to coalesce the data down in the Sizes.installed file. However, that would make the handling of newly created partitions impossible (I created /usr/lib when my /usr partition was in danger of running out of space). Once coalesced down, there is no easy way of recreating the data; and since the installed Sizes file is of the order of 100k compressed, and the savings are unlikly to be more than 30-40k, I still think we should not discard data. Correct operation is worth more than 40k ;-) In case you changed something, you have to run the symlink check again. When you update a Package it will from now on trimm the tree to the correct dirs. The calculation of space needed/gained for an update would then be wrong once. Since the data is only 100K, lets keep it all. May the Source be with you. Mrvn -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: On adding size info to Packages files [very long]
Manoj Srivastava [EMAIL PROTECTED] writes: Hi, Brederlow == Brederlow [EMAIL PROTECTED] writes: Brederlow Would that already be a correct Packages file or would dpkg and Brederlow similar scream about wrong entries? Could old dpkg's handle the new Brederlow entries? As I mentioned elsewhere, there is no reason for this to be included in the Packages file. The size estimation should be *OPTIONAL*, and people should not have to download all the data unless they want the estimate. That was more a question of would and not of should. Would it be a correct Package file with something like that (or similar) in it, or would it be rejected by dpkg? Moerever, the available file (and the status file) are created from the Packages files, and there is no reason to bloat them further with duplicated information, and further increase the startup time for programs that rread them and cache them in memory. Brederlow Lets trimm those to reasonable size. Directories that are Brederlow package intern will hardly be on separate Brederlow partition. I see little reason to go through these hueristics; I prefer the original proposal of creating separate files based on the depth; however, the raw data is there for people who want it. Some Packages have a big du tree with very deep directory levels. Each directory contains only a few blocks. It should be save to assume that directories that are package intern and are less than n blocks (n small, say 100 or 50) won't be on seperate partitions. The work to do to link (or mount) those directories elsewhere is just not worth the gain (of max 100K), so it won't be done. (And if its a matter of 100K being somewhere else, the person should kick the du index all together to save another 100K). Brederlow Theres another possibility: Normal users wont backup their Brederlow System, repartition differently and restore the backup (at Brederlow least not often). Also they wont move directories around Brederlow and link them often. We could thus trimm the du tree down Brederlow to what the current system reflects. Huh? what does the current system reflect? For whose machine? Oh, you mean everytime we download the data, we spend time mucking around with it and trimming it down? I think that would be very prohibitive on slower machines, especially since the data is only useful for a short time. A better approach is to download the proper sized size file (If all my partitions are at the top level or at the second level, like /usr/lib; I just download Size.2.gz; and never do local processing on the data). I mean, that when a package is installed, that the recorded du tree (which is needed to calculate the size increase/decrease for updates) could be trimmed to what the users system reflects. The users system setup should be scanned and kept in a status file for speed reasons (trying all dirs for symlinks takes time) and then trimming should be fairly easy and save a lot of space. It would save the more the smaler (less partitions) the system is. May the Source be with you. Mrvn -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: On adding size info to Packages files [very long]
Manoj Srivastava [EMAIL PROTECTED] writes: Hi, Well, now all we have to do is decide how this information should be gathered. Does the package maintainer stick it into the deb file? Should one upload the file separately (possibly modifying dpkg-genchanges to also pgp stamp the sizes file, though I think a hacked sizes file is not a major security breach, as long as the package itself is intact). If we go with a separate file kernel-package_4.08-1_i386.sizes.gz; then all we need is a modified dupload, and a modification of Guys script to handle the sizes file, even stashing it some place on the archive (debian/dists/unstable/sizes-i386/misc/kernel-package_4.08-1.gz) though that location maybe suboptimal cause of the mirrors. dinsall can just zcat the files sa needed. The size file should be generated by the server. Here are the reasons: 1. The upload should be unpacked to check if the gz is not corrupted. (Far to often some Package is broken) 2. The du indexes should be gathered and pipe into one file to gain far better compression. 3. Maintainer will forget to include a du index or have different block sizes. The Server should hold a unpacked Version of each Package's du index and after processing all uploads it should combine and pack them. May the Source be with you. Mrvn -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: On adding size info to Packages files [very long]
Hi, Brederlow == Brederlow [EMAIL PROTECTED] writes: Brederlow The size file should be generated by the server. Here are Brederlow the reasons: I am (perhaps unnecessarily) worried about time requirements Brederlow 1. The upload should be unpacked to check if the gz is not Brederlowcorrupted. (Far to often some Package is broken) This would be a lintian check. It is better done on the maintainer machine. Brederlow 2. The du indexes should be gathered and pipe into one file to gain Brederlowfar better compression. They shall be, for the Sizes.gz files (analogous to Packages.gz file). The reason to have them in a separate dir is that if one package changes, it is easier to replace a sizes file, zcat all sizes files into a big one, re-gzip that, and get a new Sizes.gz file. Brederlow 3. Maintainer will forget to include a du index or have different Brederlowblock sizes. This can be made policy and a lintian check. We have to trust the maintainers a trifle ;-) Brederlow The Server should hold a unpacked Version of each Package's du index Brederlow and after processing all uploads it should combine and pack them. Quite so. manoj -- Knowing that one is dear to oneself, one should guard oneself well. For one out of the three watches of the night a wise man should keep watch. 157 Manoj Srivastava [EMAIL PROTECTED] http://www.datasync.com/%7Esrivasta/ Key C7261095 fingerprint = CB D9 F4 12 68 07 E4 05 CC 2D 27 12 1D F5 E8 6E -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: On adding size info to Packages files [very long]
In message [EMAIL PROTECTED], Brederlow writes: Would that already be a correct Packages file or would dpkg and similar scream about wrong entries? Could old dpkg's handle the new entries? It seems to work for me. Any entries that are not recognised are ignored or passed through unchanged. Theres another possibility: Normal users wont backup their System, repartition differently and restore the backup (at least not often). Also they wont move directories around and link them often. We could thus trimm the du tree down to what the current system reflects. In case the user does I don't think this will gain much... It's probably easier to separate the size information from the packages file; after all, it isn't needed either during normal operation or when removing packages, only when you're installing new packages. The rest of the time it needn't be kept locally. -- Charles Briscoe-Smith White pages entry, with PGP key: URL:http://alethea.ukc.ac.uk/wp?95cpb4 PGP public keyprint: 74 68 AB 2E 1C 60 22 94 B8 21 2D 01 DE 66 13 E2 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
