Re: third-party packages adding apt sources

[Andrew McGlashan]

On 21/05/2016 8:03 AM, Hakan Peker wrote:
> You looking for a technical solution to a social problem. sources.list
> exist for the very purpose that repositories can be added to the system.
> A system where this facility don't exist or restricted is a form of
> walled garden.
> 
> Adding an update repository for the very same package the user has
> deliberately installed is a *convenience*. It is disrespectful of admin
> modification only if the program keeps reverting the admin modification
> and doesn't provide an option to disable this behavior. At that point
> you would be better contacting upstream that you are not comfortable
> with the behavior and you want such an option to disable the repository.

The fact that a deb can run arbitrary code, then it MUST be trusted and
there are limits to how much trust I can give to outside third parties;
it is too great a risk as far as I am concerned.

Cheers
A.



signature.asc
Description: OpenPGP digital signature

Re: third-party packages adding apt sources

[Hakan Peker]

On 05/20/2016 10:35 PM, Vincent Danjean wrote:

Le 19/05/2016 19:20, Hakan Peker a écrit :

On 05/19/2016 06:18 PM, Daniel Pocock wrote:

  From a technical perspective, can we do more to prevent users being
surprised by packages putting new entries in /etc/apt/sources.list.d?


Please no. The system is working as intended. I don't think anybody is 
surprised after installing a 3rd party deb package and seeing it adds its apt 
repository to the system, actually they would be pleased to know that they 
won't have to manually update it later. And anybody who is uncomfortable with 
it would either disable it or not install the deb in the first place.


   Please yes.

   I built some (local) packages to easily install the same sources.list
on several machines, so I agree the feature in useful and interesting.
I named my packages like *-sources and the description is explicit.

   But I've been very suprised when I discovered that the google-earth
package installed a sources.list on my parents' computer after I
downloaded and install an application deb with dpkg.
   And even more when I see that the source is silently reinstalled when
you remove it (no respect of admin modification)

   Would it be possible to have a package that install a trigger and
advertise (debconf/mail/...) when such a change occurs? I would install
it anywhere personally.



You looking for a technical solution to a social problem. sources.list 
exist for the very purpose that repositories can be added to the system. 
A system where this facility don't exist or restricted is a form of 
walled garden.


Adding an update repository for the very same package the user has 
deliberately installed is a *convenience*. It is disrespectful of admin 
modification only if the program keeps reverting the admin modification 
and doesn't provide an option to disable this behavior. At that point 
you would be better contacting upstream that you are not comfortable 
with the behavior and you want such an option to disable the repository.


-

ONLY AT VFEmail! - Use our Metadata Mitigator to keep your email out of the 
NSA's hands!
$24.95 ONETIME Lifetime accounts with Privacy Features!  
15GB disk! No bandwidth quotas!
Commercial and Bulk Mail Options!  

Re: third-party packages adding apt sources

[Paul Wise]

On Sat, May 21, 2016 at 8:32 PM, Adam Borowski wrote:

> This looks wrong to me: a vast majority of machines these days have a single
> user, thus pwning root gives you little additional gain.

Getting further into a system (user -> root -> GRUB -> MBR -> boot
firmware -> peripheral firmware) gives a successful attack much more
persistence. This is why the TLAs go as deep as they can.

> So, for running untrusted code you should execute it solely in a special
> environment of some kind.  And if you're not executing those binaries
> directly, what's the point in putting them into the standard paths?

No idea what anarcat's thoughts were but I can think of two reasons:

Prevents those binaries from being modified by the binaries themselves
or other programs run by the user.

Makes integration with the rest of the system easier.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: third-party packages adding apt sources

[Adam Borowski]

On Sat, May 21, 2016 at 01:47:41PM +0800, Paul Wise wrote:
> On Thu, May 19, 2016 at 11:18 PM, Daniel Pocock wrote:
> 
> > More and more frequently I'm encountering systems where third-party
> > repositories have been added into /etc/apt/sources.list or
> > /etc/apt/sources.list.d, usually put there by some .deb package that a
> > user installed from some third party site.
> 
> This discussion reminds me of this wiki page:
> 
> https://wiki.debian.org/UntrustedDebs

This looks wrong to me: a vast majority of machines these days have a single
user, thus pwning root gives you little additional gain.

So, for running untrusted code you should execute it solely in a special
environment of some kind.  And if you're not executing those binaries
directly, what's the point in putting them into the standard paths?

-- 
An imaginary friend squared is a real enemy.

Re: third-party packages adding apt sources

[Martin Steigerwald]

On Samstag, 21. Mai 2016 10:53:34 CEST Vincent Bernat wrote:
>  ❦ 21 mai 2016 10:24 +0200, Martin Steigerwald  :
> > Still, the turn around time between upstream and debian release would be
> > quite high for Debian stable users, but maybe part of such a
> > collaboration could be to also provide newer releases via backports.
> > Also… if upstream wants to release the built packages even quicker to
> > testers or adventurous people, why not allow them to put newer versions
> > of the official packages into their own repo while still integrating them
> > with the official repo? For Owncloud for example this could lead at least
> > to *compatible* packaging. Right now switching between Debian packages
> > and upstream packages basically destroys a working Owncloud installation
> > and requires quite some manual interaction to get things working again.
> > But with compatible packages people could easily switch between "I use
> > stable packages", "I use backport packages" and "I don´t care I want the
> > latest I add the upstream repo".
> 
> Owncloud upstream seems quite hostile towards Debian. But your
> proposition works with some other upstreams. For example, we are doing

Yeah, with Owncloud I still hope for changes, but I think it needs to happen 
on both sides and I saw some willingness on side of upstream, but also quite 
some concern. Also upstream project seems changing quite a bit with people 
leaving Owncloud Inc.

> all the packaging work for HAProxy, both official and unofficial
> packages (more backports, backports to Ubuntu) and upstream is quite
> happy (while in the past, upstream asked us to not ship HAProxy in
> Debian because it would be too old).
> 
>  http://mozilla.debian.net/
>  http://haproxy.debian.net/
>  http://ganeti.debian.net/
> 
> I think those packages are ideal to keep everyone happy. People can
> choose whatever they want and bear with the consequences. And the
> packages are "top" quality because they are derived from the packages in
> unstable.

Okay. I was aware of mozilla.debian.net, but not the others, well, that would 
be also a nice approach for upstream which might be nice to mention on 
upstream landing page.

I never thought this is available on a more general base.

> However, the examples above are compatible with our way of
> packaging. Would we want spend time on packaging stuff that would never
> go to the Debian archive due to excessive vendoring or unwilling from
> upstream to be in a stable release?

No. I don´t think that is a good idea and so I agree with David´s decision 
regarding Owncloud. He spend *a lot* of work which will not end up in Debian 
Stretch, at least not with the current situation.

But in some case I am not sure whether there have been any serious attempts to 
talk and find solutions that work. Maybe currently its not feasible with 
Owncloud, but I do think the current situation is a loss for both upstream and 
Debian.

> Now, I usually ask upstream if they would be interested to have their
> software in Debian and then, I propose for them to maintain it or
> comaintain it. Many are happy with that but some just say no. I don't
> keep tabs, but here is one example (not my own request):
> 
>  https://github.com/jordansissel/fpm/issues/409

*sigh*, not the first time I heard that Jordan is not fond of Debian packaging 
guidelines and more of a Fedora guy.

-- 
Martin

signature.asc
Description: This is a digitally signed message part.

Re: third-party packages adding apt sources

[Martin Steigerwald]

On Samstag, 21. Mai 2016 11:13:41 CEST Lars Wirzenius wrote:
> On Sat, May 21, 2016 at 10:07:43AM +0200, Martin Steigerwald wrote:
> > I wonder about a landing page for upstreams interested in working with the
> > Debian project to provide packages within the official Debian repos.
> 
> Is https://wiki.debian.org/UpstreamGuide the kind of page you mean? It
> is not necessarily well known.

Oh, its all there already, *including* a mailing list.

Ok, so maybe all whats needed for now is some promo work on this? And probably 
updating here and there.

I was totally unaware of this.

Thanks,
-- 
Martin

signature.asc
Description: This is a digitally signed message part.

Re: third-party packages adding apt sources

[Vincent Bernat]

 ❦ 21 mai 2016 09:40 +0200, Ole Streicher  :

>>> Providing a proper Debian source package is also a lot more work than
>>> writing some kind of ad-hoc build system that spits out a .deb or
>>> three.
>>
>> Totally agree. Our standards are far too high for many upstreams.
>
> which is a Good Thing.
>
> I usually put quite much effort to bring the packages to the Debian
> level, and at the end many upstreams are convinced that our standards
> help them as well on the long run.

The problem is with upstreams that (sometimes loudly) complains that our
standards are useless. Users will still install their packages and
rumble against us. Of course, I am not saying that we should lower our
standards, but it is a difficult world.
-- 
Use self-identifying input.  Allow defaults.  Echo both on output.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: third-party packages adding apt sources

[Vincent Bernat]

 ❦ 21 mai 2016 10:24 +0200, Martin Steigerwald  :

> Still, the turn around time between upstream and debian release would be 
> quite 
> high for Debian stable users, but maybe part of such a collaboration could be 
> to also provide newer releases via backports. Also… if upstream wants to 
> release the built packages even quicker to testers or adventurous people, why 
> not allow them to put newer versions of the official packages into their own 
> repo while still integrating them with the official repo? For Owncloud for 
> example this could lead at least to *compatible* packaging. Right now 
> switching between Debian packages and upstream packages basically destroys a 
> working Owncloud installation and requires quite some manual interaction to 
> get things working again. But with compatible packages people could easily 
> switch between "I use stable packages", "I use backport packages" and "I 
> don´t 
> care I want the latest I add the upstream repo".

Owncloud upstream seems quite hostile towards Debian. But your
proposition works with some other upstreams. For example, we are doing
all the packaging work for HAProxy, both official and unofficial
packages (more backports, backports to Ubuntu) and upstream is quite
happy (while in the past, upstream asked us to not ship HAProxy in
Debian because it would be too old).

 http://mozilla.debian.net/
 http://haproxy.debian.net/
 http://ganeti.debian.net/

I think those packages are ideal to keep everyone happy. People can
choose whatever they want and bear with the consequences. And the
packages are "top" quality because they are derived from the packages in
unstable.

In some cases, upstream accepts/wants to host packaging in its own git
repository as well and is happy to help (I have librdkafka/kafkacat and
ExaBGP as examples). Therefore, they can do their own releases as well.

Maybe once we have PPA, we could mainstream a bit the wizard and propose
more options.

However, the examples above are compatible with our way of
packaging. Would we want spend time on packaging stuff that would never
go to the Debian archive due to excessive vendoring or unwilling from
upstream to be in a stable release?

Now, I usually ask upstream if they would be interested to have their
software in Debian and then, I propose for them to maintain it or
comaintain it. Many are happy with that but some just say no. I don't
keep tabs, but here is one example (not my own request):

 https://github.com/jordansissel/fpm/issues/409
-- 
If you laid all of our laws end to end, there would be no end.
-- Mark Twain


signature.asc
Description: PGP signature

Re: third-party packages adding apt sources

[Lars Wirzenius]

On Sat, May 21, 2016 at 10:07:43AM +0200, Martin Steigerwald wrote:
> I wonder about a landing page for upstreams interested in working with the 
> Debian project to provide packages within the official Debian repos.

Is https://wiki.debian.org/UpstreamGuide the kind of page you mean? It
is not necessarily well known.

-- 
Schrödinger's backup hypothesis: the condition of any backup is
undefined until a restore is attempted. -- andrewsh


signature.asc
Description: PGP signature

Re: third-party packages adding apt sources

[Martin Steigerwald]

On Samstag, 21. Mai 2016 10:24:22 CEST Martin Steigerwald wrote:
> I wonder about some kind of adopt an upstream within a Debian team kind of 
> approach. A landing page and mailing list where upstream can write in for 
> getting help and advice and voicing their needs. And when there are people
> in  Debian willing to work with upstream they can build a team.
[…]
> I do think spelling out an invitation to work together officially can help
> to  attract more upstreams to work closer with Debian.

Also explaining on that page to upstream why Debian has the quality and 
stability requirements it has and what benefits this can create for upstreams 
including with offers on how to meet different upstream needs where that would 
be possible.

-- 
Martin

signature.asc
Description: This is a digitally signed message part.

Re: third-party packages adding apt sources

[Martin Steigerwald]

On Samstag, 21. Mai 2016 10:24:06 CEST Lars Wirzenius wrote:
> Et cetera. Debian has one set of quality factors it particularly cares
> about, and some upstreams think differently.

Yes, I seen all those reasons you mentioned.

I just wonder how about if upstreams can learn easily how to work together 
with Debian developers and maintainers which basically would mean offloading 
some of the work involved?

Still, the turn around time between upstream and debian release would be quite 
high for Debian stable users, but maybe part of such a collaboration could be 
to also provide newer releases via backports. Also… if upstream wants to 
release the built packages even quicker to testers or adventurous people, why 
not allow them to put newer versions of the official packages into their own 
repo while still integrating them with the official repo? For Owncloud for 
example this could lead at least to *compatible* packaging. Right now 
switching between Debian packages and upstream packages basically destroys a 
working Owncloud installation and requires quite some manual interaction to 
get things working again. But with compatible packages people could easily 
switch between "I use stable packages", "I use backport packages" and "I don´t 
care I want the latest I add the upstream repo".


I wonder about some kind of adopt an upstream within a Debian team kind of 
approach. A landing page and mailing list where upstream can write in for 
getting help and advice and voicing their needs. And when there are people in 
Debian willing to work with upstream they can build a team.

May still not be suitable for some upstreams or some upstreams not be suitable 
for Debian, but I got the impression that Debian may be seen as elitist and 
basically unapproachable by some upstreams – and that in quite some cases like 
with the Owncloud packaging a different approach would be possible. (Actually 
that is what I thought for a long time as well, until I found sponsors for 
some package I wanted in the archive and well, until Debconf 2015 in 
Heidelberg.)

I do think spelling out an invitation to work together officially can help to 
attract more upstreams to work closer with Debian.

-- 
Martin

signature.asc
Description: This is a digitally signed message part.

Re: third-party packages adding apt sources

[Martin Steigerwald]

Hello Paul,

On Samstag, 21. Mai 2016 14:07:53 CEST Paul Wise wrote:
> On Fri, May 20, 2016 at 1:34 PM, Vincent Bernat wrote:
> > Totally agree. Our standards are far too high for many upstreams.
> 
> I don't understand the disconnect here. Are upstreams not interested
> in software quality to the extent we are?

I know one quite popular example where some upstream people even objected 
having the software packaged in Debian:

Unfit upstream / RM: owncloud -- ROM; PHP 7.0 Transition
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=816376

I think similar can be said for many web applications that are just developed 
at a very high pace. Combine that with an upstream policy not to support 
skipping updates between major versions… by supporting redoing older update 
steps and you have something that is compatible with Debian stable standards.

Now I know other web apps where this works better, like Wordpress, that now 
even has a wonderfully working backport. Still this has brought up a principal 
issue that is still unsolved.

I know of some other upstreams that provided packages themselves for… whatever 
reasons, might be a good idea to ask them.

So for example basically all log analysis and indexing software that seems to 
be so popular in the last time:

- Elasticsearch: newer versions than in unstable
- Logstash
- Kibana
- Fluentd
- Graylog

Also things like OpenNebula where I have been involved a bit failed in Debian 
due to lack of manpower combined with that some effort to work together did 
not play out probably due to lack of manpower on upstream side as well. 
Instead they just do a stuff it all in one or a few packages no matter what. 
Which is similar to how Owncloud upstream does packages. It is basically like 
a huge tarball, all just stuff in /var/html, including config files and be 
done with it. Logstash packages at least have their config files in /etc, yet 
all else in /opt/logstash or so.

I suspect one of the reasons is the perceived simplicity of just doing it 
yourself without having to abide to all the rules, to have the package lintian 
clean (I never checked those against lintian tough), and to generally get more 
work done in less time, even if that work has a lower quality than what is 
usually accepted into Debian.

> > I am always flabestered by the popularity of fpm to build Debian
> > packages (and by the increasing popularity of pleaserun by the same
> > author on the same concepts). It provides a way to easily build a Debian
> > package from a directory but produces somewhat crippled/incomplete
> > packages and is no help to us since it's completely outside of any of
> > our tools. It also handles RPM (and now other package formats), but I
> > don't think this would explain its popularity alone.
> 
> I think we probably need to get dpkg-buildpackage to automatically run
> some of these:
> 
> https://wiki.debian.org/AutomaticPackagingTools

I am very happy about Maxy´s work to automate Plasma 5 and KDE Frameworks 5 
packaging. Here there is a similar issue:

Upstream has way faster release cycles and additionally they splitted up their 
stuff into a lot more single projects. With KDE 3, a bit less with KDE SC 4 it 
was a number of larger packages that you could even still count easily enough 
with your fingers. With Plasma 5  and KDE Frameworks 5 upstream provides 
literally hundreds of tarballs and git repos to package in order to get a full 
desktop experience. According to cd /usr/share/doc/libkf I have 213 
library packages installed on my system.

Maxy went to a great extent of automating things, which in some cases can lead 
to this:

karchive (5.22.0-1) unstable; urgency=medium

  [ Automatic packaging ]
  * Update symbols files from buildds logs (5.21.0-1).
  * Update symbols files.
  * Update build-deps and deps with the info from cmake

 -- Maximiliano Curia <[…]>  Thu, 19 May 2016 00:19:49 +0200

In other cases it is a mix of manual and automatic work:

karchive (5.21.0-1) experimental; urgency=medium

  [ Maximiliano Curia ]
  * Replace the "Historical name" ddeb-migration by its "Modern, clearer" 
replacement dbgsym-migration.
  * Add upstream metadata (DEP-12)
  * debian/control: Update Vcs-Browser and Vcs-Git fields
  * Update acc test script
  * uscan no longer supports more than one main upstream tarball being listed

  [ Automatic packaging ]
  * Update build-deps and deps with the info from cmake
  * Update symbols files.
  * Bump Standards-Version to 3.9.8

 -- Maximiliano Curia <[…]>  Thu, 05 May 2016 15:04:24 +0200


I am aware of similar efforts for a lot of different things like haskell 
packages, where Joachim Breitner and other teams upload literally hundreds of 
packages in a single day. I don´t know whether someone has an overview on 
whats available and what is used in the different packaging teams. But I do 
think the wiki page you linked is not complete.


I wonder about a landing page for upstreams interested in working with the 
Debian proj

Re: third-party packages adding apt sources

[Ole Streicher]

Vincent Bernat  writes:
>  ❦ 19 mai 2016 18:04 +0100, Ian Jackson  :
>>> b) many upstreams appear frustrated about getting their package
>>> officially supported in Debian.  Sometimes there is good reason their
>>> package doesn't belong in Debian but sometimes it is more about inertia
>>> in Debian or the upstream isn't aware about backports and thinks their
>>> package will be stuck at a particular version forever
>>
>> Providing a proper Debian source package is also a lot more work than
>> writing some kind of ad-hoc build system that spits out a .deb or
>> three.
>
> Totally agree. Our standards are far too high for many upstreams.

which is a Good Thing.

I usually put quite much effort to bring the packages to the Debian
level, and at the end many upstreams are convinced that our standards
help them as well on the long run.

Best

Ole

Re: third-party packages adding apt sources

[Lars Wirzenius]

On Sat, May 21, 2016 at 02:07:53PM +0800, Paul Wise wrote:
> On Fri, May 20, 2016 at 1:34 PM, Vincent Bernat wrote:
> 
> > Totally agree. Our standards are far too high for many upstreams.
> 
> I don't understand the disconnect here. Are upstreams not interested
> in software quality to the extent we are?

I don't think it's that upstreams aren't interested in quality, but
that Debian and (some) upstreams have different opinions on what
aspects of quality are more important.

* Debian: don't embed copies of libraries you use. It makes it harder
  to do security updates in the libraries, makes it harder to use the
  libraries on their own, and makes the Debian package archive
  unnecessarily larger.

  Some upstreams: we embed copies of libraries. It makes it easier to
  install our software, and guarantees us that the library doesn't
  change from underneath us, and that means we don't need to support
  many versions of the library. We're an active project, and if a
  library needs a security update, we do it quickly.

* Debian: it's important to follow Debian Policy and the Debian
  workflow of uploading to unstable and letting packages flow from
  there to testing and stable, if they don't have bad bugs. There's
  thousands of people making packages and things will break if they
  all do the same thing differently.

  Some upstreams: it's OK to cut some corners and do things simply. We
  care about getting the software into the hands of its users as soon
  as possible, and we also don't have a lot of time to spend on
  packaging. From our point of view, packaging is a necessary evil
  that is much too difficult and takes much too much time from us.
  That's effort we could spend on making the software better instead.

* Debian: it's important to have package versions that can be
  supported for many years. We produce a release every two years, and
  support it for at least three, and more if one counts the LTS
  project. Software that changes a lot, or that has an API that
  changes a lot, or that doesn't separate security updates and
  backport those to the version included in a Debian release, make
  this harder for us. We can't generally update to a new upstream
  release whenever there is a security problem, as it would negate the
  point of making Debian releases. For example, the new upstream
  version might require entirely new forests of dependencies, or newer
  versions of dependencies, all the way down to the kernel. For some
  packages that we deem sufficiently important to our users, we deal
  with that, but it is not something that generalises to all packages.

  Some upstreams: we don't support our old releases. We have only so
  much time to spend on this software, and supporting old releases
  would take a lot of effort we don't have time for. That's why we
  embed most of our dependencies into the installation packages we
  make, so that they can be installed onto the Debian releases, even
  if we decide to require more dependencies, or newer versions of
  them.

Et cetera. Debian has one set of quality factors it particularly cares
about, and some upstreams think differently.

-- 
Schrödinger's backup hypothesis: the condition of any backup is
undefined until a restore is attempted. -- andrewsh


signature.asc
Description: PGP signature

Re: third-party packages adding apt sources

[Vincent Bernat]

 ❦ 21 mai 2016 14:55 +0800, Paul Wise  :

>> For some languages, embedded copies are a pattern. Notably Go. But there
>> is also the omnibus stance: the embedded copy could not be in the
>> source, but could be in the shipped artifact. This includes Go, JS and
>> Java (when using uberjars). For some other ecosystems, the embedded copy
>> is more the exception than the rule (C, C++, Python).
>
> By shipped artifact, it sounds like you are talking about source
> packages? or do you mean binary packages?

I meant binary packages but wanted to be more general (not limited to Debian).

>> If upstream is using embedded copies, they are quite unlikely to make
>> any effort tu undo this aspect.
>
> I see upstreams doing that on debian-mentors reasonably often,
> especially when the upstream is the one doing the RFS. For upstreams
> who aren't interested in getting their software into Debian, that is
> definitely the case.

I shouldn't have made a generality. Some upstreams are more receptive
than others to this problematic.
-- 
Each module should do one thing well.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: third-party packages adding apt sources

[Paul Wise]

On Sat, May 21, 2016 at 2:46 PM, Vincent Bernat wrote:

> A meta tool "package me this" would be interesting.

There is debdry but it got orphaned.

> many of those tools are too complex for many upstreams because they
> don't want to package each dependency one by one. For example,
> dh-make-golang is fully automatic but you need to run it on each
> dependency. That's fine because those tools are for building packages
> that should be fit to go into our archive.

ISTR A recursive mode is what some automatic packaging tools do there.

> For some languages, embedded copies are a pattern. Notably Go. But there
> is also the omnibus stance: the embedded copy could not be in the
> source, but could be in the shipped artifact. This includes Go, JS and
> Java (when using uberjars). For some other ecosystems, the embedded copy
> is more the exception than the rule (C, C++, Python).

By shipped artifact, it sounds like you are talking about source
packages? or do you mean binary packages?

> If upstream is using embedded copies, they are quite unlikely to make
> any effort tu undo this aspect.

I see upstreams doing that on debian-mentors reasonably often,
especially when the upstream is the one doing the RFS. For upstreams
who aren't interested in getting their software into Debian, that is
definitely the case.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: third-party packages adding apt sources

[Vincent Bernat]

 ❦ 21 mai 2016 14:07 +0800, Paul Wise  :

>> Totally agree. Our standards are far too high for many upstreams.
>
> I don't understand the disconnect here. Are upstreams not interested
> in software quality to the extent we are?

Many of them don't consider packaging quality as important. As long as
the package does its job, they are fine. Notably, packages generated
with fpm have many downside but they mostly work and people are
happy. The main feature of fpm is being easy, not producing good quality
packages. In our cases, good quality packages are the main feature we
want. Being easy is a plus.

>> I am always flabestered by the popularity of fpm to build Debian
>> packages (and by the increasing popularity of pleaserun by the same
>> author on the same concepts). It provides a way to easily build a Debian
>> package from a directory but produces somewhat crippled/incomplete
>> packages and is no help to us since it's completely outside of any of
>> our tools. It also handles RPM (and now other package formats), but I
>> don't think this would explain its popularity alone.
>
> I think we probably need to get dpkg-buildpackage to automatically run
> some of these:
>
> https://wiki.debian.org/AutomaticPackagingTools

A meta tool "package me this" would be interesting. However note that
many of those tools are too complex for many upstreams because they
don't want to package each dependency one by one. For example,
dh-make-golang is fully automatic but you need to run it on each
dependency. That's fine because those tools are for building packages
that should be fit to go into our archive.

>> And there is also the lost cause of vendoring that gained even more
>> traction with Go but that was already a problem with the Java ecosystem.
>> I don't think there is much to do about this one.
>
> Embedded code/data copies have been a problem since as long as I've
> been a Debian user, they aren't specific to any particular language
> community.
>
> https://wiki.debian.org/EmbeddedCodeCopies

For some languages, embedded copies are a pattern. Notably Go. But there
is also the omnibus stance: the embedded copy could not be in the
source, but could be in the shipped artifact. This includes Go, JS and
Java (when using uberjars). For some other ecosystems, the embedded copy
is more the exception than the rule (C, C++, Python).

If upstream is using embedded copies, they are quite unlikely to make
any effort tu undo this aspect.
-- 
Indent to show the logical structure of a program.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: third-party packages adding apt sources

[Paul Wise]

On Fri, May 20, 2016 at 1:34 PM, Vincent Bernat wrote:

> Totally agree. Our standards are far too high for many upstreams.

I don't understand the disconnect here. Are upstreams not interested
in software quality to the extent we are?

> I am always flabestered by the popularity of fpm to build Debian
> packages (and by the increasing popularity of pleaserun by the same
> author on the same concepts). It provides a way to easily build a Debian
> package from a directory but produces somewhat crippled/incomplete
> packages and is no help to us since it's completely outside of any of
> our tools. It also handles RPM (and now other package formats), but I
> don't think this would explain its popularity alone.

I think we probably need to get dpkg-buildpackage to automatically run
some of these:

https://wiki.debian.org/AutomaticPackagingTools

> And there is also the lost cause of vendoring that gained even more
> traction with Go but that was already a problem with the Java ecosystem.
> I don't think there is much to do about this one.

Embedded code/data copies have been a problem since as long as I've
been a Debian user, they aren't specific to any particular language
community.

https://wiki.debian.org/EmbeddedCodeCopies

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: third-party packages adding apt sources

[Paul Wise]

On Fri, May 20, 2016 at 1:26 PM, Vincent Bernat wrote:

> testing is not suitable for most people because:
>
>  1. no security support

This can be mitigated by adding unstable to your sources.list and
using a wrapper around debsecan to automatically pull in packages from
unstable when
there are security fixes available:

https://bugs.debian.org/725934

>  2. packages can disappear at any time

This can be mitigated by adding unstable to your sources.list. The
same is true for unstable however and that can be mitigated by running
how-can-i-help frequently and migrating to new packages or putting in
some work when you see packages that are important to you are to be
removed.

These mitigations may not be suitable for everyone though.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: third-party packages adding apt sources

[Paul Wise]

On Thu, May 19, 2016 at 11:18 PM, Daniel Pocock wrote:

> More and more frequently I'm encountering systems where third-party
> repositories have been added into /etc/apt/sources.list or
> /etc/apt/sources.list.d, usually put there by some .deb package that a
> user installed from some third party site.

This discussion reminds me of this wiki page:

https://wiki.debian.org/UntrustedDebs

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: third-party packages adding apt sources

[Charles Plessy]

Hi Daniel,

Le Thu, May 19, 2016 at 05:18:28PM +0200, Daniel Pocock a écrit :
> 
> From a technical perspective, can we do more to prevent users being
> surprised by packages putting new entries in /etc/apt/sources.list.d?

maybe you are looking for an Apt option that would only install a package if it
comes from repository signed with a key that itself is signed by a trusted key ?

This way, from inside or outside Debian, chains of trust could be used to
certify the compliance of third-party repositories with good practices, or
other rules.

As suggested in this thread, dpkg triggers or other kinds of hooks could check
that packages installed directly without Apt would not change the trust keys
without the user being aware of this.

Have a nice day,

-- 
Charles Plessy
Tsurumi, Kanagawa, Japan

Re: third-party packages adding apt sources

[Vincent Danjean]

Le 19/05/2016 19:20, Hakan Peker a écrit :
> On 05/19/2016 06:18 PM, Daniel Pocock wrote:
>>  From a technical perspective, can we do more to prevent users being
>> surprised by packages putting new entries in /etc/apt/sources.list.d?
>>
> Please no. The system is working as intended. I don't think anybody is 
> surprised after installing a 3rd party deb package and seeing it adds its apt 
> repository to the system, actually they would be pleased to know that they 
> won't have to manually update it later. And anybody who is uncomfortable with 
> it would either disable it or not install the deb in the first place.

  Please yes.

  I built some (local) packages to easily install the same sources.list
on several machines, so I agree the feature in useful and interesting.
I named my packages like *-sources and the description is explicit.

  But I've been very suprised when I discovered that the google-earth
package installed a sources.list on my parents' computer after I
downloaded and install an application deb with dpkg.
  And even more when I see that the source is silently reinstalled when
you remove it (no respect of admin modification)

  Would it be possible to have a package that install a trigger and
advertise (debconf/mail/...) when such a change occurs? I would install
it anywhere personally.

  Regards,
Vincent

-- 
Vincent Danjean   GPG key ID 0xD17897FA vdanj...@debian.org
GPG key fingerprint: 621E 3509 654D D77C 43F5  CA4A F6AE F2AF D178 97FA
Unofficial pkgs: http://moais.imag.fr/membres/vincent.danjean/deb.html
APT repo:  deb http://people.debian.org/~vdanjean/debian unstable main

Re: third-party packages adding apt sources

[Vincent Bernat]

 ❦ 20 mai 2016 08:59 -0300, Antonio Terceiro  :

>> testing is not suitable for most people because:
>> 
>>  1. no security support
>
> That's not true. Proper security fixes will get into testing after 2
> days in unstable if everything goes right as long as the maintainer, or
> something that cares, does the work.

The fix could be blocked by a transition, by an unrelated RC bug, by a
failure to build on some architecture. And nobody will care that much
since testing is not that important until the freeze is near.

>>  2. packages can disappear at any time
>
> If they are broken. In my book that a feature and not a bug.

They can disappear due to any RC bug, even something that's not
noticeable by the user, like a FTBFS of a dependency. Would you postpone
the installation of a server until the package is available?
-- 
Instrument your programs.  Measure before making "efficiency" changes.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

testing is a tool for the release team (Re: third-party packages adding apt sources

[Holger Levsen]

On Fri, May 20, 2016 at 02:40:56PM +0200, Ole Streicher wrote:
> This behavious may be useful for a development platform, but for an end
> user this is just inacceptable.
 
This is why we keep saying that testing is a tool for the release team
and not a suite ment for users. 

Despite that it is suitable for many users most of the time ;-)

Unstable is another suite which is very usuable for many users most of
the time, yet we only advertise it to be used by developers who can live
with breakage.

The release team needs testing. If anybody else needs a similar suite but
slightly different, go create it. It's not that this cannot be done, it
"just" needs *someone* to *do* it.

I wouldn't want us to break the release teams tool, releasing Debian is
hard enough already. 


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: third-party packages adding apt sources

[Ole Streicher]

Antonio Terceiro  writes:
> On Fri, May 20, 2016 at 07:26:28AM +0200, Vincent Bernat wrote:
>>  2. packages can disappear at any time
>
> If they are broken. In my book that a feature and not a bug.

>From the user's perspective, they are also often *not* broken. Just take
the "pandas" package as an example. The package was removed from testing
due to some RC bugs, which are now resolved. However, now the packages
do not build on many platforms, so the migration to testing fails.

For an amd64 machine (which most of the users have), the package works
fine, and from an user's view the package is just not broken. It
disappeared and he can (as long as he has no good knowledge about the
package management) not solve this. The situation gets even more
complicated when the user is not using "python3-pandas" directly, but as
a dependency of another package.

This behavious may be useful for a development platform, but for an end
user this is just inacceptable. Just think if this happens with the only
email program or web browser the user knows about.

Best regards

Ole

Re: third-party packages adding apt sources

[Tollef Fog Heen]

]] Bas Wijnen 

> Debian stable is for users who want a rock solid system.  It is out of date by
> the nature of how it is built.  Users who want to get the newest versions of
> their software should not be running stable; testing is probably better for
> them.

This often isn't what users want, though.  They want the newer versions
of some packages: On my server, I want a new weechat (since it has
bugfixes and features I use), I want a newer grafana (ditto).  I don't
care about having a newer version of emacs, the version in stable is
fine.  Ditto for, say, postgres.

Other users are going to care about different sets of packages, but we
all have the same «I want stable, but for $list_of_packages».

Maybe backports is what we decide is the best solution for those cases,
but maybe we can do better than that.

> > We have zero procedure in place for the following:
> > 
> >   - Totally unsupported very old version of ${FOO} in stable, maintainer
> > isn't patching bugs, bugs are going to upstream, and upstream is
> > annoyed Debian has out of date, perhaps insecure thing X.
> 
> Yes, this is a different problem.  We may want to shield upstream from
> bugreports for those packages somehow.

I'm not at all convinced we can shield upstreams, since users will show
up on mailing lists and IRC upstream and go «I'm trying to do $X, but it
doesn't work, and according to the online docs, it should» and after a
bit of a back and forth, it's discovered that they're running an old
version which either lacks the feature or where it's broken.

> IMO it would be good to recommend our users to file all bugs with us,
> and leave it to the maintainer to pass them to upstream.  I have
> upstreams that follow our bug tracker, which is great.  But if they
> don't, it should be up to me to decide whether the report is worth
> their time.

A lot of user interaction isn't bug reports, but rather closer to user
support.  I don't think it's reasonable for maintainers to be asked to
do all the user support that upstream does.  Even just handling bugs
well (triaging and in some cases fixing and sending patches upstream) is
a lot of work.

> > Hell, teams packaging Mozilla-soft and PostgreSQL are DDs maintaining
> > *external archives* because it's easier.
> 
> This indicates that our procedures are too hard.  That needs to be fixed.
> Maybe people from those teams are reading this and can comment on it?

No, it doesn't merely indicate that our procedures are «too hard».  I
think apt.postgresql.org exists because PostgreSQL.org themselves want
to provide a service to our common users.  We don't want to have all the
supported postgresql versions in our distro, but it's super useful for
users who need a particular version.

> > Making it hard to install a new archive will only lead to more
> > workarounds, more FAQs telling users to dismiss warnings, and more
> > upstreams hell-bent on working against us, because we keep making their
> > lives harder.
> 
> Yes.  But I'm reluctant to throw away our stable release process.  If it's not
> for everyone (and it isn't), we should be more clear about that.

I haven't seen anybody advocating throwing away our stable release
process.  So far it's mostly pointing out problems, not yet trying to
come up with solutions.  (Which is fine, we need to find out what the
problems and the pain points are before it's useful to come up with
solutions.)

-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are

Re: third-party packages adding apt sources

[The Wanderer]

On 2016-05-20 at 07:59, Antonio Terceiro wrote:

> On Fri, May 20, 2016 at 07:26:28AM +0200, Vincent Bernat wrote:
> 
>> ❦ 19 mai 2016 16:39 GMT, Bas Wijnen  :
>> 
>>> Debian stable is for users who want a rock solid system.  It is
>>> out of date by the nature of how it is built.  Users who want to
>>> get the newest versions of their software should not be running
>>> stable; testing is probably better for them.
>> 
>> testing is not suitable for most people because:

>> 2. packages can disappear at any time
> 
> If they are broken. In my book that a feature and not a bug.

From a user perspective, it can be a problem, e.g. in the following
scenario:

* Package is in testing, works fine. User installs it on one or more
machines.

* Package in testing gets updated to a newer version, which is broken.

* Package gets removed from testing, because it's broken.

* User, who quite possibly never saw the broken version, wants to
install package on another machine.

From the user's perspective, what should have happened at step 3 is for
the package version available in testing to be reverted back to its last
non-broken version (whether verbatim or with a version bump over the
broken one), not for the package to have been removed entirely.

This may well not be practical as a policy, and I'm not trying to
suggest it as one, but it's an example of how broken-package removal can
be a bug rather than a feature.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man. -- George Bernard Shaw



signature.asc
Description: OpenPGP digital signature

Re: third-party packages adding apt sources

[Antonio Terceiro]

On Fri, May 20, 2016 at 07:26:28AM +0200, Vincent Bernat wrote:
>  ❦ 19 mai 2016 16:39 GMT, Bas Wijnen  :
> 
> > Debian stable is for users who want a rock solid system.  It is out of date 
> > by
> > the nature of how it is built.  Users who want to get the newest versions of
> > their software should not be running stable; testing is probably better for
> > them.
> 
> testing is not suitable for most people because:
> 
>  1. no security support

That's not true. Proper security fixes will get into testing after 2
days in unstable if everything goes right as long as the maintainer, or
something that cares, does the work.

>  2. packages can disappear at any time

If they are broken. In my book that a feature and not a bug.

> There was some discussion in the past to turn testing into a rolling
> release but nothing was done. Such a project would be interesting to
> check its adoption. People complaining that stuff is too old could be
> directed to this rolling release version and we will likely get people
> complaining that stuff moves too fast, but at least, we can't be blamed
> to ship too old softwares, that would be a user choice.

If you assume that freezes take more or less the last 25% of the
development cycle, as far as I am concerned testing _is_ already a
rolling release 75% of the time. And TBH having those last 25% of the
time as a stabilization period does not hurt anyone, on the contrary.


signature.asc
Description: PGP signature

Re: third-party packages adding apt sources

[Charles Plessy]

Le Fri, May 20, 2016 at 07:34:59AM +0200, Vincent Bernat a écrit :
> 
> I am always flabestered by the popularity of fpm to build Debian
> packages (and by the increasing popularity of pleaserun by the same
> author on the same concepts). It provides a way to easily build a Debian
> package from a directory but produces somewhat crippled/incomplete
> packages and is no help to us since it's completely outside of any of
> our tools. It also handles RPM (and now other package formats), but I
> don't think this would explain its popularity alone.

For software using CMake, there are also CPackDeb and CPackRPM.

https://cmake.org/cmake/help/latest/module/CPackDeb.html

Were I an upstream developer, I would definitely be interested by tools like
this that leverage the build system to build installation packages for various
platforms.

Have a nice week-end,

Charles

-- 
Charles Plessy
Debian Med packaging team,
http://www.debian.org/devel/debian-med
Tsurumi, Kanagawa, Japan

Re: third-party packages adding apt sources

[Vincent Bernat]

 ❦ 19 mai 2016 18:04 +0100, Ian Jackson  :

>> b) many upstreams appear frustrated about getting their package
>> officially supported in Debian.  Sometimes there is good reason their
>> package doesn't belong in Debian but sometimes it is more about inertia
>> in Debian or the upstream isn't aware about backports and thinks their
>> package will be stuck at a particular version forever
>
> Providing a proper Debian source package is also a lot more work than
> writing some kind of ad-hoc build system that spits out a .deb or
> three.

Totally agree. Our standards are far too high for many upstreams.

I am always flabestered by the popularity of fpm to build Debian
packages (and by the increasing popularity of pleaserun by the same
author on the same concepts). It provides a way to easily build a Debian
package from a directory but produces somewhat crippled/incomplete
packages and is no help to us since it's completely outside of any of
our tools. It also handles RPM (and now other package formats), but I
don't think this would explain its popularity alone.

And there is also the lost cause of vendoring that gained even more
traction with Go but that was already a problem with the Java ecosystem.
I don't think there is much to do about this one.
-- 
Don't sacrifice clarity for small gains in "efficiency".
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: third-party packages adding apt sources

[Vincent Bernat]

 ❦ 19 mai 2016 16:39 GMT, Bas Wijnen  :

> Debian stable is for users who want a rock solid system.  It is out of date by
> the nature of how it is built.  Users who want to get the newest versions of
> their software should not be running stable; testing is probably better for
> them.

testing is not suitable for most people because:

 1. no security support
 2. packages can disappear at any time

There was some discussion in the past to turn testing into a rolling
release but nothing was done. Such a project would be interesting to
check its adoption. People complaining that stuff is too old could be
directed to this rolling release version and we will likely get people
complaining that stuff moves too fast, but at least, we can't be blamed
to ship too old softwares, that would be a user choice.
-- 
As flies to wanton boys are we to the gods; they kill us for their sport.
-- Shakespeare, "King Lear"


signature.asc
Description: PGP signature

Re: third-party packages adding apt sources

[Mike Hommey]

On Thu, May 19, 2016 at 04:39:24PM +, Bas Wijnen wrote:
> > Hell, teams packaging Mozilla-soft and PostgreSQL are DDs maintaining
> > *external archives* because it's easier.
> 
> This indicates that our procedures are too hard.  That needs to be fixed.
> Maybe people from those teams are reading this and can comment on it?

FWIW, I'm maintaining mozilla.debian.net because we still don't have no
freaking PPAs despite the announcement a few years ago that we would have
them soon. I'm not holding my breath. Also because the process for
backports doesn't allow things that are not in testing to be in
backports, which means it is not possible to have a non ESR firefox
there.

Mike

Re: third-party packages adding apt sources

[Ian Jackson]

Paul Tagliamonte writes ("Re: third-party packages adding apt sources"):
> [cc'ing devel, since this is a rant that involves technical topics, and
>  god knows I only go on so many rants a year these days]

I think you may have only BCC'd -devel, or something.

> > Sometimes there is good reason their
> > package doesn't belong in Debian but sometimes it is more about inertia
> > in Debian or the upstream isn't aware about backports and thinks their
> > package will be stuck at a particular version forever
> 
> Frankly, I have a hell of a lot of sympathy for this.

I just wanted to say that while my own messages have been addressing a
rather different set of third-party repos, I don't really disagree
with the picture you paint.

> We have zero procedure in place for the following:

And I definitely agree with this complaint.

> Go to any mature project, they have a way to bypas the archive, and get
> the latest stable from upstream. This is a huge failure. Upstreams
> aren't becoming DDs and updating packages, dispite the fact they can
> package and maintain things.
> 
> Hell, teams packaging Mozilla-soft and PostgreSQL are DDs maintaining
> *external archives* because it's easier.

Yes.

Ian.

Re: third-party packages adding apt sources

[Ian Jackson]

Bas Wijnen writes ("Re: third-party packages adding apt sources"):
> On Thu, May 19, 2016 at 07:15:01PM +0200, Daniel Pocock wrote:
> > Another thing comes to mind: making sure that even if the user
> > explicitly allows some other repository, they are protected from package
> > updates that come along and replace other things like apt itself, libc,
> > bash, gnupg, ...
> 
> I don't think we want to prevent that.  If they want to install a
> package that does that, they can.  However, I think it is reasonable
> to warn them that they should get ready for trouble when installing
> a package that isn't from Debian, and especially if they install a
> new entry into sources.list from an external source.
> 
> I don't see how to technically do such a thing though; the problem is that
> these kind of upstreams often don't care about our (or their user's) systems
> and will inject any code in their package that makes the warnings go away.

If we provided a better, more official, way, that gave the relevant
software provider some kind of semi-approval, then we could probably
persuade the upstreams to start using it.

But I agree that playing core wars against the third party repo
packages is a really really bad idea.  It won't work.  It's a recipe
for craziness.  And it's unethical because it also amounts to playing
core wars against our users - who have, after all, probably decided
that this is what they want.

Ian.

Re: third-party packages adding apt sources

[Russ Allbery]

Daniel Pocock  writes:

> Another thing comes to mind: making sure that even if the user
> explicitly allows some other repository, they are protected from package
> updates that come along and replace other things like apt itself, libc,
> bash, gnupg, ...

While this would be nice to prevent accidents, it's not clear that you can
really establish any security guarantees.  You can protect against some
very obvious things, such as wholesale replacing core packages, but
postinst scripts still run as root and can do anything they want.  So you
don't get any real security benefit here that I can see.

-- 
Russ Allbery (r...@debian.org)   

Re: third-party packages adding apt sources

[Ian Jackson]

Daniel Pocock writes ("Re: third-party packages adding apt sources"):
> On 19/05/16 19:04, Ian Jackson wrote:
> > Debian proper has a very high bar for inclusion.  Obviously there are
> > perhaps some packages which are close to suitable for inclusion, but
> > the vast majority of things that aren't in Debian proper are outside
> > it for real, nontrivial reasons (whether of technical quality of the
> > binaries, technical quality of the source, or political/ethical
> > reasons).
> 
> Do you think that if these upstreams became involved in other ways - for
> example, if we proactively invited them to MiniDebConfs and other events
> - we might bridge the gap to help them understand our way of thinking,
> whether it is technical or otherwise?

I think you are missing my point.  You seem to have the assumption
that for many of the people providing packages in third-party repos,
they don't have what I'm calling real and nontrivial reasons.

Hell, I even have .debs I distribute myself outside Debian (although I
haven't gone to the trouble of setting up reprepro).

> Sure, some of them will never change, some of them have no capacity to
> think long-term but there are others who simply don't quite understand
> and may go the extra mile if they get to know us a little better.

The problem is not one of lack of understanding.  These out-of-Debian
repos are not going to be abolished by an outreach and education
programme.


Let me go into more detail about a few of the kinds of real,
nontrivial reasons, I'm talking about:


1. Proper source format.  I wrote:

> > Providing a proper Debian source package is also a lot more work than
> > writing some kind of ad-hoc build system that spits out a .deb or
> > three.

Packages with ad-hoc source build systems are never going to be in
Debian proper, for obvious reasons.  They wouldn't fit.  That's not
something we can fix.

But writing a proper build system for an ad-hoc Debian source package
is a fair amount of work.  I did it recently for a moderately easy
package I encountered in Raspbian, and it took /me/ most of a day.

That upstream didn't do it is quite understandable.  Doing so was not
in their `long-term' interests, as you put it: the effort for them to
learn how to do it, and then do it, and then debug it, and then deal
with the transitional pain, was simply not worth it to them.  And
probably not to their userbase, either.

We in Debian can improve this situation by making proper Debian source
packages easier to build.  For example, we can:
 * Provide build tools which require less boilerplate, are less
   fragile, and are more automatic.  Yay dh(1).
 * Provide _shorter_ intro documentation.
 * Provide better source code management systems - that are less
   prehistoric than .dsc 1.0 (non-native) and less "omg wtf bbq"
   than .dsc "3.0 (quilt)".  (non-crazy git integration, dgit)

But there are still going to be plenty of people for whom it's a
better use of their time to do something else but tart up the source
for their .deb builds.


2. ABI/API

The package I mention above didn't take the care with API/ABI, split
library packaging, and so on, that would be expected in Debian.  This
was tolerable in the original Raspbian context.

I sent the upstream patches to fix this but there are compatibility
implications with changing the packaging arrangements.  Thinking about
this stuff, and planning for it, and anticipating the needs of the
users, is nontrivial.

For quite a few packages, the levels of compatibility and
upgradeability we would expect in Debian are simply not worth the
effort.  I don't think in Debian we would want to relax this, do we ?


3. Freedom

This is the killer reason to provide a third-party apt repository.

A software provider who provides packages outside Debian does not need
our permission to do so; they do not need our help to provide new
versions; they do not need to confirm to our release rules.

Of course in Debian our focus is freedom for our users, so there are
limits to how much freedom we want to make available to upstreams and
third-party packagers: we want to avoid becoming complicit in efforts
to control users.  But users often choose (for reasons that seem good
to those users - and, often, reasons that /are/ good for those users)
to want to run non-free software.  So even when it concerns non-free
software we need to respect and support that decision, while avoiding
encouraging it.

I don't think an outreach campaign by Debian, encouraging people to do
work in Debian rather than outside, is going to significantly change
the way software providers make decisions about licensing, release
schedules, or whatever.


> Another thing comes to mind: making sure that even if the user
> explicitly allows some other repository, they are protected from package
> updates that come along and replace other things like apt itself, libc,
> bash, gnupg, ...

This has to be an optional feature.  The user might specifically want
that !

Ian.

Re: third-party packages adding apt sources

[Bas Wijnen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Thu, May 19, 2016 at 07:15:01PM +0200, Daniel Pocock wrote:
> Another thing comes to mind: making sure that even if the user
> explicitly allows some other repository, they are protected from package
> updates that come along and replace other things like apt itself, libc,
> bash, gnupg, ...

I don't think we want to prevent that.  If they want to install a package that
does that, they can.  However, I think it is reasonable to warn them that they
should get ready for trouble when installing a package that isn't from Debian,
and especially if they install a new entry into sources.list from an external
source.

I don't see how to technically do such a thing though; the problem is that
these kind of upstreams often don't care about our (or their user's) systems
and will inject any code in their package that makes the warnings go away.

Thanks,
Bas
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQIcBAEBAgAGBQJXPfugAAoJEJzRfVgHwHE61mcQAKQzjwht59J73zievxLTlqoz
QNMnmorJJb5vvx1PDwJzfqqL4rqRPu5/h9wiHjYhi+O3YM2J8W4phKxyMNIDYR1R
p+BA1CwSUuGX3/4je1QWaNpHa6IpgU9HxlUtrLNnjhJvtAuRR4PXfv15tPxsGgxi
AT5770XMSuCjNSehpC5nhp2l9HFiaRnaTa8tIENf6Bj1NrXrH/Em2/3CKbZiTkTf
S5C0IHWPTJyIYGqRALub+DiVvYd/d2ZNdFAwRW3/8nyJeBLwEkQ9BO8mNOdBHZWF
Fbvi0WJ5bBo1mcIVc9vO/4QvrFGPqxXYo8Sf+dI2O/NmzKdQ6XXwcy30HL8R/DhD
gZzhOJLnJFmUTpvqUAv1ywt9mfqNE4ed7/9ccN+4nTVHNSbxqJZyEimIi6x7dZop
dYZvdjoDgHRBFG7cBaGGH0Dqb+r0fSkP05Foxxy3ShITMzYQRPDzRPmxRxaU6ojB
Y5+GLQ3wlEMmiNsK34y1pQcJYKI5B7d+LYS1B/K5/Enkv7Z+4n8CX2AHtRMDmpHt
wQYKKaHjOktGZQonE1fF0vb4WE4otoidAyyN0jnlQDlq9aTp4OHDFcx5o8u6ppgG
LmKw7YTosFwzd37kHmH7icwvXiPHwIvHwR+9Vt/0wra/juD9Xktom2TtuA02MwIY
nPAD/aVlO95tD43+9EQP
=54DY
-END PGP SIGNATURE-

Re: third-party packages adding apt sources

[Hakan Peker]

On 05/19/2016 06:18 PM, Daniel Pocock wrote:


More and more frequently I'm encountering systems where third-party
repositories have been added into /etc/apt/sources.list or
/etc/apt/sources.list.d, usually put there by some .deb package that a
user installed from some third party site.



Hey, This is a good thing™! I'm glad some upstreams care enough for 
Debian that they build deb packages of their releases and I'm even more 
delighted they are doing it properly by serving them in an apt repository.


This doesn't mean there is no benefit in discussing problematic areas or 
potential improvements in providing them also in Debian repository, but 
please don't pressure the poor upstreams too much making them feel like 
they are doing a disservice by providing a deb package to their users 
comfort.



 From a technical perspective, can we do more to prevent users being
surprised by packages putting new entries in /etc/apt/sources.list.d?

Please no. The system is working as intended. I don't think anybody is 
surprised after installing a 3rd party deb package and seeing it adds 
its apt repository to the system, actually they would be pleased to know 
that they won't have to manually update it later. And anybody who is 
uncomfortable with it would either disable it or not install the deb in 
the first place.




-

ONLY AT VFEmail! - Use our Metadata Mitigator to keep your email out of the 
NSA's hands!
$24.95 ONETIME Lifetime accounts with Privacy Features!  
15GB disk! No bandwidth quotas!
Commercial and Bulk Mail Options!  

Re: third-party packages adding apt sources

[Daniel Pocock]

On 19/05/16 19:04, Ian Jackson wrote:
> Daniel Pocock writes ("third-party packages adding apt sources"):
>> b) many upstreams appear frustrated about getting their package
>> officially supported in Debian.  Sometimes there is good reason their
>> package doesn't belong in Debian but sometimes it is more about inertia
>> in Debian or the upstream isn't aware about backports and thinks their
>> package will be stuck at a particular version forever
> 
> Providing a proper Debian source package is also a lot more work than
> writing some kind of ad-hoc build system that spits out a .deb or
> three.
> 
>> From a technical perspective, can we do more to prevent users being
>> surprised by packages putting new entries in /etc/apt/sources.list.d?
> 
> IMO we should set up a registry of such organisations, and their
> cryptographic keys, and at least document promises made by the
> organisation about its behaviour with respect to various principles
> that we might care about.
> 
> (For example, "this repo only contains packages which are dfsg-free
> and come with source code"; "this repo contains packages which do not
> themselves phone home"; ...)
> 
>> From an organizational perspective, can we do more to make contact with
>> such upstreams and try to find ways to involve them in releasing their
>> packages through official channels?  Is there any way we could gather
>> data about how many upstreams do this without compromising user privacy?
> 
> Debian proper has a very high bar for inclusion.  Obviously there are
> perhaps some packages which are close to suitable for inclusion, but
> the vast majority of things that aren't in Debian proper are outside
> it for real, nontrivial reasons (whether of technical quality of the
> binaries, technical quality of the source, or political/ethical
> reasons).
> 

Do you think that if these upstreams became involved in other ways - for
example, if we proactively invited them to MiniDebConfs and other events
- we might bridge the gap to help them understand our way of thinking,
whether it is technical or otherwise?

Sure, some of them will never change, some of them have no capacity to
think long-term but there are others who simply don't quite understand
and may go the extra mile if they get to know us a little better.

> What we need to do is provide an easier and better way for unofficial
> repositories.  That means an easy way for third party software
> providers to publish repositories which it is then easy for users to
> use, if the user chooses to do so.
> 
> Importantly, we need:
> 
> 1. A way for the user to get good, trustworthy (ie, coming in some
>sense from Debian), information about the repository.  Including
>the identity of the organisation providing it; and some
>classification of Debian's opinion about the software in it.
> 
> 2. A way for the user to reliably get the public keys on their system,
>that doesn't involve them clicking on a .deb on the public
>internet.
> 


Another thing comes to mind: making sure that even if the user
explicitly allows some other repository, they are protected from package
updates that come along and replace other things like apt itself, libc,
bash, gnupg, ...

Re: third-party packages adding apt sources

[Ian Jackson]

Daniel Pocock writes ("third-party packages adding apt sources"):
> b) many upstreams appear frustrated about getting their package
> officially supported in Debian.  Sometimes there is good reason their
> package doesn't belong in Debian but sometimes it is more about inertia
> in Debian or the upstream isn't aware about backports and thinks their
> package will be stuck at a particular version forever

Providing a proper Debian source package is also a lot more work than
writing some kind of ad-hoc build system that spits out a .deb or
three.

> From a technical perspective, can we do more to prevent users being
> surprised by packages putting new entries in /etc/apt/sources.list.d?

IMO we should set up a registry of such organisations, and their
cryptographic keys, and at least document promises made by the
organisation about its behaviour with respect to various principles
that we might care about.

(For example, "this repo only contains packages which are dfsg-free
and come with source code"; "this repo contains packages which do not
themselves phone home"; ...)

> From an organizational perspective, can we do more to make contact with
> such upstreams and try to find ways to involve them in releasing their
> packages through official channels?  Is there any way we could gather
> data about how many upstreams do this without compromising user privacy?

Debian proper has a very high bar for inclusion.  Obviously there are
perhaps some packages which are close to suitable for inclusion, but
the vast majority of things that aren't in Debian proper are outside
it for real, nontrivial reasons (whether of technical quality of the
binaries, technical quality of the source, or political/ethical
reasons).

What we need to do is provide an easier and better way for unofficial
repositories.  That means an easy way for third party software
providers to publish repositories which it is then easy for users to
use, if the user chooses to do so.

Importantly, we need:

1. A way for the user to get good, trustworthy (ie, coming in some
   sense from Debian), information about the repository.  Including
   the identity of the organisation providing it; and some
   classification of Debian's opinion about the software in it.

2. A way for the user to reliably get the public keys on their system,
   that doesn't involve them clicking on a .deb on the public
   internet.

If we had such a thing, dealing with the admin of it would be a
nontrivial task.  It would have to be distributed.

Ian.

Re: third-party packages adding apt sources

[Adam D. Barratt]

On 2016-05-19 17:39, Bas Wijnen wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Thu, May 19, 2016 at 11:46:53AM -0400, Paul Tagliamonte wrote:
[cc'ing devel, since this is a rant that involves technical topics, 
and

 god knows I only go on so many rants a year these days]


You didn't actually do this.


https://lists.debian.org/debian-devel/2016/05/msg00277.html disagrees 
(modulo quibbles on naming).


Regards,

Adam

Re: third-party packages adding apt sources

[Bas Wijnen]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Thu, May 19, 2016 at 11:46:53AM -0400, Paul Tagliamonte wrote:
> [cc'ing devel, since this is a rant that involves technical topics, and
>  god knows I only go on so many rants a year these days]

You didn't actually do this.

> > Sometimes there is good reason their
> > package doesn't belong in Debian but sometimes it is more about inertia
> > in Debian or the upstream isn't aware about backports and thinks their
> > package will be stuck at a particular version forever
> 
> Frankly, I have a hell of a lot of sympathy for this.

So do I, but I disagree with your solution.  Here's how I see it:

Debian stable is for users who want a rock solid system.  It is out of date by
the nature of how it is built.  Users who want to get the newest versions of
their software should not be running stable; testing is probably better for
them.

When programs get packaged outside of unstable, we should look at why that
happens, and try to convince the packager to do it inside Debian instead.
(This convincing may take the form of adding features to our system that solve
their problems.)

When users or upstreams complain that the version in stable isn't up to date,
we should explain to the users that they probably want to use testing.  If this
is a common problem for upstreams, our user documentation about this is not
sufficiently clear.

When users want to run stable with one or two packages cherry-picked from
backports, that is something which we may want to make easier for them.

> Backports are a whole thing. People have to be actively aware of them.
> Users have to be told to add a new thing in the sources by hand, and
> install something explicitly. It's calories, and explaining a Debian
> process to a user isn't fun. Why would upstreams want to do this?

Agreed, we should make this easier; allow DDs and the package uploader to
trigger building a backport for any package that is in testing by sending an
e-mail, perhaps?

And aside from that, I agree that it should be easier to install a system that
includes backports in the sources.list.

> My claim, as I'll outline below, is, if the upstream wants to give the
> user an up-to-date software package, and they have to teach them how to
> add a new archive, they'll give them an archive *they control*, because
> they're now on the hook for delivering through that channel. Upstream
> wants to spend as little time as they can with this, so they make it
> easy - they make a deb.

Yes.  I do this myself for some experimental packages that I don't consider
ready for the general public yet.  That is IMO a valid reason for having a
private repository.  "Doing it right is too hard" is not; that is something we
should fix.

> Backports are present when a package is in testing, and backports are a
> single channel. Backports are not for upstream's releases, whenever they
> want to ship a thing.

If we make it easy for them to become DDs or DMs and for uploaders to trigger
backports builds, backports can be what upstream wants.

> We have zero procedure in place for the following:
> 
>   - Totally unsupported very old version of ${FOO} in stable, maintainer
> isn't patching bugs, bugs are going to upstream, and upstream is
> annoyed Debian has out of date, perhaps insecure thing X.

Yes, this is a different problem.  We may want to shield upstream from
bugreports for those packages somehow.

IMO it would be good to recommend our users to file all bugs with us, and leave
it to the maintainer to pass them to upstream.  I have upstreams that follow
our bug tracker, which is great.  But if they don't, it should be up to me to
decide whether the report is worth their time.

>   - Leaf package ${BAR} has a robust upstream community, where releases
> are very well tested, with a mature stable/unstable release cycle.
> Our stable release freeze was off by a few months, so we've been
> shipping their 'oldstable' in our 'stable' for years. The
> maintainers are annoyed we don't use the latest stable in our
> stable.

If the bugreports go through us, they shouldn't have a problem with this.  It
is up to the users to choose who they trust.  If they trust upstream not to
break things (which for some upstreams is totally justified, but for others it
isn't), they can get the new package from backports (assuming that we made that
easy, so the new version is in there).  If they only trust us, they use the
older version and there is nothing wrong with that.  If upstream doesn't want
that to happen, tough luck.  It's our job to give users what they want, not to
force onto them what upstream wants.

> Largely, I think the first situation is a common one that our culture
> has forced people to group-think "Well, that's bad and the system is
> working as intended". We can't let software change on our stable
> installs, so this situation is bad, but the intent of stable.

I think it is the intent of stable, and it's good.  There

Re: third-party packages adding apt sources

[Holger Levsen]

On Thu, May 19, 2016 at 08:45:09AM -0700, Russ Allbery wrote:
> I don't think we can provide that inside Debian, at least without some
> pretty significant changes to how we handle stable releases that are
> contrary to some of our goals for stable.

I think I heard someone saying "PPA" or such…

;-)

(But I don't think those would be suitable for Chrome from Google.)


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: third-party packages adding apt sources

[Paul Tagliamonte]

[cc'ing devel, since this is a rant that involves technical topics, and
 god knows I only go on so many rants a year these days]

On Thu, May 19, 2016 at 05:18:28PM +0200, Daniel Pocock wrote:
> b) many upstreams appear frustrated about getting their package
> officially supported in Debian.

Yeah, I don't think that's it. "officially supported" is burrying a lot
of really important discussion we're not having.

> Sometimes there is good reason their
> package doesn't belong in Debian but sometimes it is more about inertia
> in Debian or the upstream isn't aware about backports and thinks their
> package will be stuck at a particular version forever

Frankly, I have a hell of a lot of sympathy for this.

Backports are a whole thing. People have to be actively aware of them.
Users have to be told to add a new thing in the sources by hand, and
install something explicitly. It's calories, and explaining a Debian
process to a user isn't fun. Why would upstreams want to do this?


My claim, as I'll outline below, is, if the upstream wants to give the
user an up-to-date software package, and they have to teach them how to
add a new archive, they'll give them an archive *they control*, because
they're now on the hook for delivering through that channel. Upstream
wants to spend as little time as they can with this, so they make it
easy - they make a deb.


Now, for the rant I promised.


Backports are present when a package is in testing, and backports are a
single channel. Backports are not for upstream's releases, whenever they
want to ship a thing.

We have zero procedure in place for the following:

  - Totally unsupported very old version of ${FOO} in stable, maintainer
isn't patching bugs, bugs are going to upstream, and upstream is
annoyed Debian has out of date, perhaps insecure thing X.

  - Leaf package ${BAR} has a robust upstream community, where releases
are very well tested, with a mature stable/unstable release cycle.
Our stable release freeze was off by a few months, so we've been
shipping their 'oldstable' in our 'stable' for years. The
maintainers are annoyed we don't use the latest stable in our
stable.


We can talk about what is an isn't right all day long, or about how PPAs
are going to solve all this one day, but I've become more and more
worried that we're failing to serve users in this way.

Largely, I think the first situation is a common one that our culture
has forced people to group-think "Well, that's bad and the system is
working as intended". We can't let software change on our stable
installs, so this situation is bad, but the intent of stable.

The second one is harder to say that with, since upstream is making
assertions (just as strong as us) on some things. Be it protocol
stability, API stability, or whatever.

We're mostly approximating #2 by stacking up patches from their next
stable, and applying them to our stable. We're basically shipping the
new version with the old version number, without as much testing as the
real version, and only confusing ourselves (patches are a bitch), users
(I have version 1.2), and upstreams (why doesn't Debian trust the
release process), causing tension everywhere. Look at OpenSSL, it's
nuts. (God bless the OpenSSL team for doing this, and finding a way to
keep DDs happy -- or rather -- merely quiet, as well as upstreams and
users).

So, your question, why do people try to make it easy to get the latest
stable software is answered simply with "because we're not". We are the
problem. No one wants to do this. Maintianing an archive sucks. No one
wants to maintain a Debian archive. It's just the least work to deliver
something supportable and maintainable to users.

Go to any mature project, they have a way to bypas the archive, and get
the latest stable from upstream. This is a huge failure. Upstreams
aren't becoming DDs and updating packages, dispite the fact they can
package and maintain things.

Hell, teams packaging Mozilla-soft and PostgreSQL are DDs maintaining
*external archives* because it's easier.

The issue is, we have a model of software delivery that's slowly growing
more and more distant from the realities of shipping software today. Why
is this? What can we do? What do our users want? What do our users
*expect*?

Making it hard to install a new archive will only lead to more
workarounds, more FAQs telling users to dismiss warnings, and more
upstreams hell-bent on working against us, because we keep making their
lives harder.


This is a 100% larger conversation, and it's not about a hacky deb, it's
about how our place in the software ecosystem has been evolving, and we
need to evolve with it, or we'll find ourselves part of the problem we
were trying to solve in the first place.


signature.asc
Description: PGP signature

Re: third-party packages adding apt sources

[Russ Allbery]

Daniel Pocock  writes:

> b) many upstreams appear frustrated about getting their package
> officially supported in Debian.  Sometimes there is good reason their
> package doesn't belong in Debian but sometimes it is more about inertia
> in Debian or the upstream isn't aware about backports and thinks their
> package will be stuck at a particular version forever

One of the big reasons why organizations do this is because they provide
packages for stable users and want control over exactly when they ship new
packages to those users without meeting the standards for stable update
review (which are very strict).  Consider Google Chrome's packages, for
example, which are one set that use this method.

I don't think we can provide that inside Debian, at least without some
pretty significant changes to how we handle stable releases that are
contrary to some of our goals for stable.

-- 
Russ Allbery (r...@debian.org)