Re: Improving your archive and package system for small package

[Guillem Jover]

Hi!

On Thu, 2015-09-03 at 13:26:12 -0700, Josh Triplett wrote:
> Jonas Smedegaard wrote:
> > Seems Osamu Aoki is working on at least part of the puzzle:
> > https://bugs.debian.org/797045
> 
> Merging multiple sources *really* shouldn't be necessary.  And the
> metadata for those sources will vary, so that likely won't save that
> much space.

Well, there seems to be different kinds of overhead when it comes to
extremely tiny packages (those with dozens or hundreds of lines of code).

Metadata is one, amount of packages on the distribution, installed systems
and files on the mirrors is another one.

All the above involve in one way or another some overhead on at least
the amount or size of source packages, binary packages, Sources and
Packages indices, package manager databases and possibly increased
dependency complexity, usage on disk after installation, inodes used
on mirrors or installed systems, number of source VCS, etc.

This can have a cost on the mirror network, buildds, on any team doing
distribution wide work, such as the ftp-masters, release, porter, QA or
reproducible teams, tools like lintian, autopkgtest, DUCK, VCS or watch
checkers, britney, botch, etc. On maintainers having to maintain hundreds
of similar tiny packages.


Doing package collections in Debian might reduce part of the above
overhead, but *if* this needs fixing, ideally it should be fixed
upstream. Having to package 100 new upstream release updates instead
of one is significant work, and that cannot be easily skipped if
upstreams do not do the conglomeration themselves.

> Perhaps we should add a few more things to common-licenses, or figure
> out if our packaging metadata could be further reduced or de-duplicated.
> It should be possible to package a 1kB library without several kB of
> overhead.

There are certain things that we could do to reduce overhead in some
places, I don't think we can easily reduce most of the overhead
anyway. For example each source and binary package contain a
changelog, that's usually what takes most space. Even if we went
with my proposal to store that and the copyright files in the dpkg
database, that might only reduce some overhead on installed systems.

> But even if we have to pay that overhead, so be it; we have
> tens of thousands of packages already, what's a few hundred more tiny
> JavaScript packages as long as they're actually used?

If we were talking about few hundred packages, I don't think anyone
would have much of an issue, I guess what people are worried about is
this setting precedent and opening the flood gates. That's probably
one of the reasons people have not tried to inject much of CPAN or CRAN
or similar upstream archives into Debian even if I don't think those
are as tiny as the ones proposed here, and most of it could be automated
for example.

Thanks,
Guillem

Re: Improving your archive and package system for small package

[Guillem Jover]

Hi!

On Thu, 2015-09-03 at 17:28:38 +0200, Adrien CLERC wrote:
> This may have been discussed before, but could we achieve something like
> this with virtual package?

With the implemented version Provides, which means having versioned
virtual packages, this is actually probably a good solution, either
to pack a collection of different upstream projects or a collection
of versions for the same upstream project.

Thanks,
Guillem

Re: Improving your archive and package system for small package

[Jose-Luis Rivas]

On 03/09/15, 03:13pm, Bastien ROUCARIES wrote:
 
> I was thinking generally, perl latex python have a lot of small
> package. Each language could not come with its own solution. Maybe
> creating a tool agregating small debian package in a big one. But
> doing something only for javascript is not a solution.
> 
> Bastien

The use of small packages in nodejs is different to that on perl, latex
and python. They are very used and they are way too much. There are
several solutions for the same problem and everyone uses a different
one.

I use nodejs everyday in my work and opensource projects and I refuse to
package them into debian precisely by this reason.

-- 
⨳ PGP 0x13EC43EEB9AC8C43 ⨳ https://ghostbar.co


signature.asc
Description: Digital signature

Re: Improving your archive and package system for small package

[Adrien CLERC]

Le 03/09/2015 13:36, Bastien ROUCARIES a écrit :
> Hi,
>
> In order to improve node situation we need to improve the small
> packages problems.
>
> What are the main bottlenet ? What could be done to improve the situation ?
>
> The node small package does not change often so it could be a win to
> your archive size.
> Moreover if we could solve this problem we could think about small
> perl package or even tex package.
>
Hi,

This may have been discussed before, but could we achieve something like
this with virtual package?
For example, define virtual packages for every small package
(nodejs-myfunkyoneliner) that is only provided by a bigger package (e.g.
nodejs-libraries1, so we can split those bigger packages), containing a
compilation of those librairies.
Then, another packaged software can just depends on those virtual
packages. The real files will be installed with potentially uneeded
other ones, but it will reduce overhead.

This may be a bad idea, though. I'm not a dpkg expert.

Adrien

Re: Improving your archive and package system for small package

[Jonas Smedegaard]

Quoting Bastien ROUCARIES (2015-09-03 13:36:15)
> In order to improve node situation we need to improve the small 
> packages problems.
> 
> What are the main bottlenet ? What could be done to improve the 
> situation ?
> 
> The node small package does not change often so it could be a win to 
> your archive size.
> Moreover if we could solve this problem we could think about small 
> perl package or even tex package.

Seems Osamu Aoki is working on at least part of the puzzle: 
https://bugs.debian.org/797045


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: signature

Re: Improving your archive and package system for small package

[Josh Triplett]

Jonas Smedegaard wrote:
> Seems Osamu Aoki is working on at least part of the puzzle:
> https://bugs.debian.org/797045

Merging multiple sources *really* shouldn't be necessary.  And the
metadata for those sources will vary, so that likely won't save that
much space.

Perhaps we should add a few more things to common-licenses, or figure
out if our packaging metadata could be further reduced or de-duplicated.
It should be possible to package a 1kB library without several kB of
overhead.  But even if we have to pay that overhead, so be it; we have
tens of thousands of packages already, what's a few hundred more tiny
JavaScript packages as long as they're actually used?

- Josh Triplett

Re: Improving your archive and package system for small package

[Jonas Smedegaard]

Quoting Josh Triplett (2015-09-03 22:26:12)
> Jonas Smedegaard wrote:
>> Seems Osamu Aoki is working on at least part of the puzzle:
>> https://bugs.debian.org/797045
>
> Merging multiple sources *really* shouldn't be necessary.  And the 
> metadata for those sources will vary, so that likely won't save that 
> much space.
>
> Perhaps we should add a few more things to common-licenses, or figure 
> out if our packaging metadata could be further reduced or 
> de-duplicated. It should be possible to package a 1kB library without 
> several kB of overhead.  But even if we have to pay that overhead, so 
> be it; we have tens of thousands of packages already, what's a few 
> hundred more tiny JavaScript packages as long as they're actually 
> used?

For the record: I totally agree with above.  I have never seen any 
measures of actual problems caused by tiny-content packages - and if 
that really is a problem then I suspect metapackages and transitional 
packages are even bigger problem...

...that said, I responded to someone indicating interest in improving 
something in Debian, and I want to support that initiative because even 
though this *shouldn't* be necessary, it currently is - because some in 
Debian, including ftpmsters, consider it necessary.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: signature

Improving your archive and package system for small package

[Bastien ROUCARIES]

Hi,

In order to improve node situation we need to improve the small
packages problems.

What are the main bottlenet ? What could be done to improve the situation ?

The node small package does not change often so it could be a win to
your archive size.
Moreover if we could solve this problem we could think about small
perl package or even tex package.

Regards

Bastien

PS: no flame war please. I am juste trying to package stuff for grunt
and I have a lot of package in the kB range.

Re: Improving your archive and package system for small package

[Jérémy Lal]

2015-09-03 13:36 GMT+02:00 Bastien ROUCARIES :

> Hi,
>
> In order to improve node situation we need to improve the small
> packages problems.
>
> What are the main bottlenet ? What could be done to improve the situation ?
>
> The node small package does not change often so it could be a win to
> your archive size.
> Moreover if we could solve this problem we could think about small
> perl package or even tex package.
>
> Regards
>
> Bastien
>
> PS: no flame war please. I am juste trying to package stuff for grunt
> and I have a lot of package in the kB range.
>

Hi Bastien,

please read
https://lists.alioth.debian.org/pipermail/pkg-javascript-devel/2015-June/010693.html
and what follows.

At this point anyone can help by writing some python code...

Jérémy

Re: Improving your archive and package system for small package

[Bastien ROUCARIES]

On Thu, Sep 3, 2015 at 2:26 PM, Jérémy Lal  wrote:
>
>
> 2015-09-03 13:36 GMT+02:00 Bastien ROUCARIES :
>>
>> Hi,
>>
>> In order to improve node situation we need to improve the small
>> packages problems.
>>
>> What are the main bottlenet ? What could be done to improve the situation
>> ?
>>
>> The node small package does not change often so it could be a win to
>> your archive size.
>> Moreover if we could solve this problem we could think about small
>> perl package or even tex package.
>>
>> Regards
>>
>> Bastien
>>
>> PS: no flame war please. I am juste trying to package stuff for grunt
>> and I have a lot of package in the kB range.
>
>
> Hi Bastien,
>
> please read
> https://lists.alioth.debian.org/pipermail/pkg-javascript-devel/2015-June/010693.html
> and what follows.
>
> At this point anyone can help by writing some python code...

I was thinking generally, perl latex python have a lot of small
package. Each language could not come with its own solution. Maybe
creating a tool agregating small debian package in a big one. But
doing something only for javascript is not a solution.

Bastien

> Jérémy
>