Bug#971515: CTTE decision on "kubernetes: excessive vendoring (private libraries)"

2021-01-22 Thread Dmitry Smirnov 
On Saturday, 23 January 2021 3:18:52 AM AEDT Shengjing Zhu wrote:
> The real complex things are, dealing license and copyright and *NEW* queue.
> If this TC decision is that we just trust what upstream say, then why not
> just unvendor them. Then many pieces of libraries can be reused by others.

Even without packaging new libraries, we could do so much better un-vendoring
some of already packaged ones, even if maintainer focuses only on those that
are _easy_ to un-vendor. Vendoring _everything_ is crazy, unnecessary,
irrational...

-- 
Cheers,
 Dmitry Smirnov
 GPG key : 4096R/52B6BBD953968D1B

---

People are rarely grateful for a demonstration of their credulity.
-- Carl Sagan


signature.asc
Description: This is a digitally signed message part.

Bug#971515: CTTE decision on "kubernetes: excessive vendoring (private libraries)"

2021-01-22 Thread Shengjing Zhu 
On Wed, Jan 20, 2021 at 12:28:46PM -0700, Sean Whitton wrote:
> Our consensus is that Kubernetes ought to be considered special in the
> same way that Firefox is considered special -- we treat the package
> differently from most other source packages because (i) it is very large
> and complex, and (ii) upstream has significantly more resources to keep
> all those moving parts up-to-date than Debian does.
>

Sigh, You seems to misunderstand why packaging Kubernetes is complex.
Package every dependency and using dpkg dependency relationships are
possible and easy. Creating a Go package has automated tool, it won't
take much time to package all. We can also have more than one version
of one library if Kubernetes has particular need.

The real complex things are, dealing license and copyright and *NEW* queue.
If this TC decision is that we just trust what upstream say, then why not
just unvendor them. Then many pieces of libraries can be reused by others.

Shengjing Zhu


signature.asc
Description: PGP signature

Bug#971515: marked as done (kubernetes: excessive vendoring (private libraries))

2021-01-21 Thread Dmitry Smirnov 
I'm disappointed with TC's decision. IMHO it is lacking depth of judgement.

The original problem I reported is absolutism - "vendor everything".
And TC apparently considered only that without trying to find the balance.

What are the reasons for vendoring stable well maintained library like
Logrus? None. That system library can by used easily with hardly any burden
to maintenance. Same can be said probably about ~100 other libraries.

We could do so much better with recommending at least to un-vendor trivial
cases of 3rd party code...



> The Committee declines to overrule the maintainer and accepts the level
> of vendoring used in the 'kubernetes' source package.  We also decline
> to intervene in bugs requesting that vendored components in the
> 'kubernetes' source package be installed in binary packages in such a
> way that other packages can make use of them.

> Our consensus is that Kubernetes ought to be considered special in the
> same way that Firefox is considered special -- we treat the package
> differently from most other source packages because (i) it is very large
> and complex, and (ii) upstream has significantly more resources to keep
> all those moving parts up-to-date than Debian does.

That is based on misleading Elana's comparison that I've answered
(regretfully too late) in the other email...


> Our most basic reason for this point of view is that given how much
> fewer resources we have than upstream Kubernetes, it is not feasible for
> Kubernetes to make it into a stable release of Debian unless we take an
> approach like this one.

IMHO Kubernetes is not suitable for "stable" either way.


> The same goes for the possibility of providing
> security support.  And given that a strategy for security support is
> available, we do not see any reason why having the Kubernetes bundle in
> a stable release alongside other copies of its vendored dependencies is
> worse than not having Kubernetes in a stable release at all.

It is not suppose to be "all or nothing" approach. We are talking about 3rd
party code, not maintained by Kubernetes. At least some of it is perfectly 
feasible to take from packaged libraries instead of messy upstream bundle.



> It should be noted that we think the greater resources of upstream

Let's not praise greater resources unnecessarily. Upstream don't care
about good versioning practices and even automatically closes bugs for
inactivity without addressing them (e.g. https://github.com/kubernetes/
kubernetes/issues/17077).


> is
> relevant not only to keeping on top of patches and security fixes, but
> also to license compliance.  It is our belief that there is no reason at
> the present time to be concerned that non-DFSG material would find its
> way into the package, because Kubernetes upstream care about ensuring
> that all vendored dependencies are free software, and they have the
> resources to check.

Since when we are delegating DFSG compliance to upstream based on faith??

FYI in case of "cadvisor" - a required Kubernetes component, DFSG compliance 
was difficult to achieve due to pre-minified source-less 3rd party javascript 
files...

It was a hell of an effort to get Kubernetes through NEW process. Apparently
I should not have bothered with all that because upstream is so "good" (it 
wasn't).
Did you ask ftp-masters opinion/position on that?? Are they OK with volume
of needless garbage in vendored bundle? Would they allow that before TC's 
conclusion?


> This could change in the future and it may not be
> true for other upstreams.

I've spent about a year of effort for introducing Kubernetes to Debian - yes, 
I had that much faith in Kubernetes but ultimately the effort lead to
disappointment in the technology and loss of confidence in upstream's
ability to maintain the project. But in the course of the effort many new
libraries were introduced as packages, the Debian Golang ecosystem was 
stabilised and we've gained a valuable experience with maintaining a complex 
Golang software. The hard work was accomplished and it was only the matter or 
maintaining Kubernetes properly...
The TC's decision tells me that the effort was largely wasted because it was 
"too hard" to do things this way... Naturally the sloppy way is so much 
easier...

:(

-- 
Best wishes,
 Dmitry Smirnov
 GPG key : 4096R/52B6BBD953968D1B

---

There are occasions when it pays better to fight and be beaten than not to
fight at all.
-- George Orwell

---

"Increased Risk of Noninfluenza Respiratory Virus Infections Associated
With Receipt of Inactivated Influenza Vaccine".
-- https://pubmed.ncbi.nlm.nih.gov/22423139/


signature.asc
Description: This is a digitally signed message part.

Bug#971515: Status as of last tech-ctte meeting

2021-01-21 Thread Dmitry Smirnov 
On Friday, 20 November 2020 4:35:23 PM AEDT Elana Hashman wrote:
> Kubernetes is a very large and active project. It has about 600
> members,[0] 1000 voters,[1] 100 committers (which I define as members of
> the milestone-maintainers team),[2] and over 50,000 contributors.[3] The
> project has its own security[4] and release teams,[5] and the release
> team includes a software licensing team.
> [...]
> As such, the scale of Kubernetes is similar to that of Debian itself.

I was too slow to reply to this email but I'll leave some comments for the
record.

Comparison of Kubernetes' team size to Debian is misleading and the
comparison is not in favour of Kubernetes.

Debian is compartmentalised into relatively small (mostly) well coordinated
teams. Kubernetes - the monolithic project - suffers from well known
problem with coordinating large teams. Frederick Brooks described the
problem well in his "The Mythical Man-Month" book. 
Regarding number of contributors, what is strength for Debian
is a weakness for Kubernetes.

Kubernetes bug tracker shows the scale of the problem and I don't have
impression that the team is big enough to handle bug reports effectively or
have them under control.

It was my observation that vendoring problems in Kubernetes are worse than
everything I've seen in other projects.
Is that despite the size of the team or because of it?

The particular example that I did not mention enough [1] (and it is not the 
only one) show how little that remarkably large team cares about dependency
hygiene: a trivial library update with a patch was not done in a few years
time and all that was produced by the large team are excuses for not doing
the update, only to finally perform it in the end as advised.
Who can have confidence in the Kubernetes dependency management after that?

[1]: https://github.com/kubernetes/kubernetes/issues/27543

Of course the problem have little to do with Kubernetes. The "use world"
situation with over-dependency on large number of 3rd party libraries is a
challenging one especially with historically poor Golang approach to
dependency management a.k.a. "vendoring". I'm just saying that Kubernetes
is not all that special in regards to vendoring because of the team size.



> Kubernetes as an upstream project is much better resourced than the
> single downstream maintainer in Debian.

Nobody argues with that. But it is only "single maintainer" with monolitic
packaging - a case that I'm arguing against.
Golang team is strong and its capacity is impressive.
Packages that use dependency libraries are always team-maintained.

It is the same situation as with Kubernetes upstream where
much of the 3rd party code is not maintained by Kubernetes team.


> The resourcing and scale of the Kubernetes project gives us better
> assurances that upstream has done due diligence for dependency
> management.

IMHO upstream demonstrated bad and terrible attitude with dependency
management. But apparently nobody here considered that argument... :(

-- 
Kind regards,
 Dmitry Smirnov
 GPG key : 4096R/52B6BBD953968D1B

---

We often refuse to accept an idea merely because the way in which it has
been expressed is unsympathetic to us.
-- Friedrich Nietzsche

---

COVID-19: The Trouble With PCR Tests
https://swprs.org/the-trouble-with-pcr-tests/


signature.asc
Description: This is a digitally signed message part.

Bug#971515: Status as of last tech-ctte meeting

2020-11-22 Thread Tollef Fog Heen 
]] Philip Hands 

> Tollef Fog Heen  writes:
> 
> > ]] Shengjing Zhu 
> >
> >> Firefox is special, since for Debian desktop users, they need a browser. Is
> >> kubernetes same here?
> >
> > FWIW, the lack of Kubernetes or a similar orchestration platform (mesos,
> > nomad, docker swarm) in stable has been keeping back development of the
>   ^^
> 
> Do you really mean stable here?  I had gained the impression that there
> was no prospect of Kubernetes getting into stable, regardless of the
> details of how it ends up being packaged.  Have I misunderstood?

I do mean stable.  I was making a narrow comment about how special or
not Kubernetes is, I don't know if the goal is for it to reach stable or
not.

(My comment above was also slightly imprecise, I've since learned that
Docker Swarm is in Debian, in the docker.io package, thanks to Shengjing
Zhu who pointed it out in private email.)

-- 
Tollef Fog Heen

Bug#971515: Status as of last tech-ctte meeting

2020-11-21 Thread Philip Hands 
Tollef Fog Heen  writes:

> ]] Shengjing Zhu 
>
>> Firefox is special, since for Debian desktop users, they need a browser. Is
>> kubernetes same here?
>
> FWIW, the lack of Kubernetes or a similar orchestration platform (mesos,
> nomad, docker swarm) in stable has been keeping back development of the
  ^^

Do you really mean stable here?  I had gained the impression that there
was no prospect of Kubernetes getting into stable, regardless of the
details of how it ends up being packaged.  Have I misunderstood?

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Bug#971515: Status as of last tech-ctte meeting

2020-11-21 Thread Tollef Fog Heen 
]] Shengjing Zhu 

> Firefox is special, since for Debian desktop users, they need a browser. Is
> kubernetes same here?

FWIW, the lack of Kubernetes or a similar orchestration platform (mesos,
nomad, docker swarm) in stable has been keeping back development of the
next generation way of handling Debian services, since that will need a
reasonable container orchestation platform to build on.  The lack of a
platform is not the only reason for the delay, but it surely hasn't
helped either.

-- 
Tollef Fog Heen, speaking for himself, but as a DSA member

Bug#971515:

2020-11-21 Thread Janos LENART 
Dear interested parties,

I would like to express my appreciation for everyone's valuable input; the
time and effort put into this matter. I doubt there is much left to be
added as both sides argued at length.

Speaking from a technological point of view, I want to restate that I also
do not like the current extreme vendoring situation (which plagues Go,
especially, but not only). In the particular case of Kubernetes though,
while it is not ideal, I still do not see a better way for now. I have
tried to carefully weigh this in March, and have reevaluated it again over
the past few months. After the TC decides I am planning carry on with the
work to make Kubernetes happen in Debian, in some form or another. For
example. at the moment it is quite some work to set up a workable cluster
(I have set up test clusters passing the integration tests with every
binary I have uploaded).

Speaking personally, as a DD since 2001, I have read copious amounts of
bickering on debian-private and debian-devel. So I was pretty sure that a
heated discussion is all but unavoidable. I was not looking forward to it,
but I wanted progress. I have attempted to convey this in README.Debian
too, but in retrospect those were not the most cooling words either, and I
am sorry for that (reworded in 1.19.4-2, linking to this bug). The amount
of personal insults I have received, both publicly and in private, is truly
alarming :-( Debian will blaze ahead with or without Kubernetes in stable
(or at all), it won't however work at all without people.

Thank you.

-- 
LÉNÁRT, János

Bug#971515: Status as of last tech-ctte meeting

2020-11-20 Thread Jonathan Carter 
Hi Josh

On 2020/11/20 01:30, Josh Triplett wrote:
> In particular, with my upstream Rust/Cargo hats on, I would love to work
> with you and others on questions of what software packaging could look
> like, and how to maintain the quality and curation *and* package
> availability of Debian in collaboration with other ecosystems of package
> and dependency management.
> 
> Other potentially interesting questions: what are the assumptions that
> go into our current tradeoffs about shared libraries vs static
> libraries, and are those still the correct tradeoffs in all cases?

There is already a lot of text to read on this bug report, please try to
refrain from adding more off-topic content to it.

thanks,

-Jonathan

Bug#971515: Status as of last tech-ctte meeting

2020-11-19 Thread Elana Hashman 
On Wed, Nov 18, 2020 at 11:36:09AM -0600, Gunnar Wolf wrote:
> ...
> I'm pasting here a bit of the discussion that happened later during
> the meeting: having this discussion K8S-specific, Elana mentioned that
> "that is a big part of the tension. sometimes, you have an upstream
> where the authors are less resourced and responsive than the
> downstream. this case is almost certainly the opposite".

To better illustrate this quote, I agreed to provide more context on the
scale of the Kubernetes project.

Kubernetes is a very large and active project. It has about 600
members,[0] 1000 voters,[1] 100 committers (which I define as members of
the milestone-maintainers team),[2] and over 50,000 contributors.[3] The
project has its own security[4] and release teams,[5] and the release
team includes a software licensing team. I am a SIG Chair and milestone
maintainer in the upstream Kubernetes project, which is comparable to
being a team lead and uploading developer in Debian.

As such, the scale of Kubernetes is similar to that of Debian itself.
Kubernetes as an upstream project is much better resourced than the
single downstream maintainer in Debian.

[0]: https://github.com/orgs/kubernetes/people
[1]: 
https://github.com/kubernetes/community/blob/8bdeb0a4d6e7a3fc9afdb874aa2cefa2ba88bc9c/events/elections/2020/voters.md
[2]: 
https://github.com/kubernetes/org/blob/adc0166a081ec7a613ebc8422d9676ff4035fc31/config/kubernetes/sig-release/teams.yaml#L17-L141
[3]: https://k8s.devstats.cncf.io/d/24/overall-project-statistics?orgId=1
[4]: https://github.com/kubernetes/security
[5]: https://github.com/kubernetes/community/tree/master/sig-release

> At this point, we found some friction as to _what_ we are discussing
> on: Is this bug specifically on Kubernetes, which should be taken as a
> special case? Or would we try to rule as the Technical Committee on
> vendoring in general in the Debian ecosystem? Elana repeated her
> assurance that the Kubernetes team is thorough in their
> freeness-checking and security practices; I insisted on "not
> discussing about K8S, but about a bundling practice to which K8S
> subscribes".

The scope of the bug as submitted is limited to the Kubernetes package.
I think it is better to try to limit our decision to that scope, as
Kubernetes is not comparable to a single-maintainer Golang project that
might have a similar number of vendored dependencies.

The resourcing and scale of the Kubernetes project gives us better
assurances that upstream has done due diligence for dependency
management. I don't think we could assume this for an arbitrary package.

Per yesterday's TC discussion,[6] I think it makes sense to handle
Kubernetes with consideration to its size and importance to users,
similar to how we special-case Firefox.

- e

[6]: 
http://meetbot.debian.net/debian-ctte/2020/debian-ctte.2020-11-18-17.58.html


signature.asc
Description: PGP signature

Bug#971515: Status as of last tech-ctte meeting

2020-11-19 Thread Josh Triplett 
On Wed, Nov 18, 2020 at 10:26:21AM -0800, Russ Allbery wrote:
> There are a lot of fascinating edge cases and precedents and philosophical
> questions about the function of a distribution here, and I think they
> rightfully attract a lot of energy, but I'm worried this is at the cost of
> losing sight of the core functionality provided by Debian.  Because that
> functionality is currently working, the people who are benefiting from it
> don't know to weigh in.
[...]
> As a Debian developer, I appreciate the arguments about vendoring and the
> desire to maintain Go libraries and the pride that we take in our work of
> really understanding software and packaging it properly.
> 
> However, as a Debian *user* of the kubernetes-client, I have to say that I
> do not care in the slightest.  The older, unvendored kubernetes-client
> package worked great.  The new, heavily vendored kubernetes-client package
> works great.  Both do exactly what I want, and I don't care at all, as a
> user, which is in Debian.  But if the package were removed from Debian, I
> would be really unhappy.  And if Helm and Argo CD were packaged for
> Debian, I would be much happier, so if unvendoring them is the obstacle,
> as a user I'm opposed to unvendoring!
> 
> I want to push back a bit against the feeling that some things are
> ill-suited for how Debian has historically packaged software and therefore
> maybe Debian isn't the place for them, and we should instead ask people to
> manage them outside of Debian (but somehow make this easy).
> 
> First, while I appreciate and cheer Phil and Sean's optimism, there have
> been a lot of plans in Debian historically to make something that isn't a
> package easy to build and install, and they have basically never worked.
> The way Debian makes something easy to build and install is by making it a
> package.  That's what our entire project is designed around, and I'm
> dubious that we're going to be able to reproduce those properties with
> something that isn't a package.
> 
> Second, the point of Debian at the end of the day is that I can install it
> on a computer and use it to get things done.  The details we're discussing
> here are important and work towards making Debian maintainable and
> sustainable and to ensure that a quality bar has been met in terms of
> security and legality and licensing, but I think it's important that they
> are also means to an end and the end is not security and licensing per se.
> We're not making a random collection of relatively secure software; we're
> giving people programs that they can run while keeping them secure.  We're
> not just a classification system for what software is free; we're giving
> people software they can use while ensuring that all of it is free.  I
> think it's worth occasionally stepping back and dwelling on that thought
> for a moment and making sure we're picking the right strategy for meeting
> our quality bar that allows us to still achieve the mission.
> 
> This is particularly true for something like vendoring or the avoidance of
> vendoring, which is not a core mission.  It's not in the social contract,
> and it's not a DFSG principle.  It's a hard-won and very valuable piece of
> experience with how best to sustainably make a distribution... but one
> that was hard-won in the era of C programs and shared libraries and that
> has generalized admirably to Perl and Python.  It may generalize to other
> languages and other mechanisms for distributing software.  It may not!  If
> it doesn't, that's significant because it's such a deeply-engrained part
> of our culture, but it's *not* a breach of our fundamental project goals.
> We can consider new approaches here without becoming untethered, and
> indeed may be able to achieve our primary goal *better* by expanding the
> scope of software that we can include in the distribution and that can
> therefore just work.
> 
> I think there's a bit of a crisis of confidence in Debian because of how
> much larger the free software ecosystem is now than it was when the
> project started, how far away from doing everything through a distribution
> a lot of developers have moved, and how many resources are flooding into
> other areas that we have difficulty keeping pace with.  One of the natural
> reactions to that crisis of confidence is to pull back from these new and
> difficult and unfamiliar areas and decrease scope to the things we know
> we're really good at: packaging primarily C, C++, Perl, and Python code.
> And that is a valid strategy; it's okay to just keep being very good at
> something one is already very good at.  But I think in the long term that
> means Debian becomes something much different than it historically has
> been.  It means Debian would become a base on which other people would
> build things, rather than a comprehensive collection of tools that covers
> all the incidental things you need from a computer, and where people need
> only reach outside of Debian for

Bug#971515: Status as of last tech-ctte meeting

2020-11-19 Thread Shengjing Zhu 
On Wed, Nov 18, 2020 at 10:26:21AM -0800, Russ Allbery wrote:
> 
> I maintain a bunch of Kubernetes clusters as part of my job.  Those
> clusters are run by other people (cloud providers, data centers, etc.).  I
> need clients to talk to those clusters.  When I first started working with
> Kubernetes, I realized I needed a client, worried about how much of a pain
> that would be, did an apt-cache search for kubernetes, found
> kubernetes-client, breathed a sigh of relief, and ran apt install
> kubernetes-client.  I have subsequently not given Kubernetes clients a
> second thought.
> 

In priciple, the kubectl compatibility matrix says it only supports one minor
version (older or newer). When you working with many clusters, the cluster
version may not be compatible with the kubectl version in Debian stable.

PS, in practice, the basic function of kubectl doesn't change much, and new
kubectl with old cluster seems to work.

As a user of upstream kubectl package user, I'm quite happy with it. I can
always install the latest, or any old version. They keep all old versions in
their apt repo. And with the nature of static link, their repo works well
other parts of the system, regardless I'm running Debian oldstable, stable,
or unstable.

IMO, we should admit that current Debian way doesn't fit well for such
softwares.


signature.asc
Description: PGP signature

Bug#971515: Status as of last tech-ctte meeting

2020-11-19 Thread Ian Jackson 
Russ Allbery writes ("Bug#971515: Status as of last tech-ctte meeting"):
> [much stuff]

Oh, wow, Russ.  Thank you very much.  What an excellent piece of
writing.  I agree entirely with all of it.

Ian.

(And I speak as someone who thinks that this newfangled "vendor
everything" and "just download that shit from the internet" approach
is hideously bad engineering practice.)

-- 
Ian JacksonThese opinions are my own.  

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

Bug#971515: Status as of last tech-ctte meeting

2020-11-18 Thread Russ Allbery 
This is an excellent discussion and a good summary.  Thank you, Gunnar!

Gunnar Wolf  writes:

> There is a course of action that's unlikely to leave everybody happy,
> but is worth considering: Phil said:

> Phil>
> that makes it seem like what we actually need is a decent way of
> letting our users install the upstream binaries, rather than
> trying to force those packages into Debian as a special case

> Sean agreed with him; probably something as K8S due to its nature is
> not suited for Debian inclusion (but we'd still have to ensure it's
> easy to build and install, not from packages). But the quesion is
> where to draw the line:

I want to explicitly state my concern with this approach, speaking as one
of a community that I think may have been a bit underrepresented in this
discussion to date: a user of the kubernetes-client package in Debian.

There are a lot of fascinating edge cases and precedents and philosophical
questions about the function of a distribution here, and I think they
rightfully attract a lot of energy, but I'm worried this is at the cost of
losing sight of the core functionality provided by Debian.  Because that
functionality is currently working, the people who are benefiting from it
don't know to weigh in.

I maintain a bunch of Kubernetes clusters as part of my job.  Those
clusters are run by other people (cloud providers, data centers, etc.).  I
need clients to talk to those clusters.  When I first started working with
Kubernetes, I realized I needed a client, worried about how much of a pain
that would be, did an apt-cache search for kubernetes, found
kubernetes-client, breathed a sigh of relief, and ran apt install
kubernetes-client.  I have subsequently not given Kubernetes clients a
second thought.

Not having to think about things like that is exactly the point of Debian.
It's possibly the highest standard of success for a distribution.

By comparison, I also use Helm and Argo CD as part of my job.  Helm has a
snap package that's kind of maintained and sort of works but kept being
irritating plus I had to remember to upgrade a different thing.  Argo CD
is available only as a binary download from its release page and therefore
I never remember to upgrade it and run into weird problems and then have
to go download new versions.  The experience is annoying and irritating
and reminds me of when my primary working system was running Solaris.

As a Debian developer, I appreciate the arguments about vendoring and the
desire to maintain Go libraries and the pride that we take in our work of
really understanding software and packaging it properly.

However, as a Debian *user* of the kubernetes-client, I have to say that I
do not care in the slightest.  The older, unvendored kubernetes-client
package worked great.  The new, heavily vendored kubernetes-client package
works great.  Both do exactly what I want, and I don't care at all, as a
user, which is in Debian.  But if the package were removed from Debian, I
would be really unhappy.  And if Helm and Argo CD were packaged for
Debian, I would be much happier, so if unvendoring them is the obstacle,
as a user I'm opposed to unvendoring!

I want to push back a bit against the feeling that some things are
ill-suited for how Debian has historically packaged software and therefore
maybe Debian isn't the place for them, and we should instead ask people to
manage them outside of Debian (but somehow make this easy).

First, while I appreciate and cheer Phil and Sean's optimism, there have
been a lot of plans in Debian historically to make something that isn't a
package easy to build and install, and they have basically never worked.
The way Debian makes something easy to build and install is by making it a
package.  That's what our entire project is designed around, and I'm
dubious that we're going to be able to reproduce those properties with
something that isn't a package.

Second, the point of Debian at the end of the day is that I can install it
on a computer and use it to get things done.  The details we're discussing
here are important and work towards making Debian maintainable and
sustainable and to ensure that a quality bar has been met in terms of
security and legality and licensing, but I think it's important that they
are also means to an end and the end is not security and licensing per se.
We're not making a random collection of relatively secure software; we're
giving people programs that they can run while keeping them secure.  We're
not just a classification system for what software is free; we're giving
people software they can use while ensuring that all of it is free.  I
think it's worth occasionally stepping back and dwelling on that thought
for a moment and making sure we're picking the right strategy for meeting
our quality bar that allows us to still achieve the mission.

This is particularly true for something like vendoring or the avoidance of
vendoring, which is not a core mission.  It's not in

Bug#971515: Status as of last tech-ctte meeting

2020-11-18 Thread Sean Whitton 
Hello all,

On Wed 18 Nov 2020 at 11:36AM -06, Gunnar Wolf wrote:

> So... It's not like we reached a conclusion, but I do feel the meeting
> was interesting and the discussion very much worthy. Yes, this will
> surely end up in reinforcing the notion that the Technical Committee
> is a slow and bureaucratic beast :-) But... well, I'll stop
> apologizing. Only some minutes to go before the meeting, and I want to
> give the rest of the Committee the opportunity to check if I didn't
> misrepresent them or skip too much of their opinion.

Thank you for this summary.

At today's meeting one point which we thought was missing from this
summary was that no-one on the committee has any appetite for overruling
the package maintainer, so it is very unlikely that will be the outcome
of this bug.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Bug#971515: Status as of last tech-ctte meeting

2020-11-18 Thread Gunnar Wolf 
Hello world,

When we had our last tech-ctte meeting, 2020.10.21¹, I volunteered to
write up a summary of our positions regarding this bug. Then... Well,
life happened, and I have not had the time to sit down and write until
today -- A couple of hours before our next meeting. Several other
discussions have happened as well, most notably, the article by
Jonathan Corbet on this issue in LWN².

¹ http://meetbot.debian.net/debian-ctte/2020/debian-ctte.2020-10-21-17.58.html
  
http://meetbot.debian.net/debian-ctte/2020/debian-ctte.2020-10-21-17.58.log.html
² https://lwn.net/Articles/835599/

# Vendoring: Impedance mismatch with our long-established
# practices/traditions
##

I believe the issue of vendoring to be central to the discussion of
what Debian is, and what its role should be. We are very lucky to have
proponents of both sides of this issue in the Committee, and I'll try
to keep my ideas as centered as possible (of course, disclaimer: I do
hold a position, I really do not like vendoring; we have the luck of
having Elana in the ctte, as she is a developer of the affected
package, and Sean, who is a Debian Policy editor).

Elana started off with a very important and true point:

Elana>
I think this bug was sort of inevitable. Debian policy cruft is
clashing with how people actually build and distribute software
these days. we have an appeal to override a maintainer on the
basis of "policy" and the "Debian way being superior" without any
real technical examination of the merits of "the Debian way"

We are quite far from 1996, yes, and many languages have for a long
time delivered their own packaging systems, "freeing" the users from
the need of installing a myriad of little packages, and "freeing" the
distribution caregivers (and I don't only mean the developers, but
also the ftp-masters) and infrastructure from carrying this myriad of
little packages.

Elana and Sean seem to share the thought that each language ecosystem
could work with somewhat different rules:

Sean>
It might be reasonable to vendor like mad for something written in
Go but not for something written in Haskell, say

Elana>
Debian policy is specifically built around the distribution and
execution of dynamically linked C/C++ applications and
libraries. distros in general were. but modern software above that
base does not necessarily operate under the same assumptions and
it is silly to apply policy that was designed for applications
that are dynamically linked against programs that are statically
linked and are impossible to dynamically link

Simon mentioned that "our identity should be about shipping
high-quality packages, whatever that means", and mentioning that "our
packaging policy is designed for medium-sized packages", refereed back
to the discussions had long time ago over tiny Javascript snippets
packaged as packages, back in the starting days of the nodejs growth.

# DFSG-freeness checking
##

Sean and I shared the concern that excessive vendoring (say, having
tens to hundreds of libraries shipped as part of a single package)
place a very heavy burden on our ftp-masters when checking
DFSG-freeness; coupling this with the rapid change rate that
"new-wave" software development usually has, if adding / dropping
dependencies from already packaged software is usual practice,
DFSG-checks would not just have to be made in NEW, but as a continuous
process. Far from sustainable, and impossible to do by our ftp-master
team practices.

# Security issues
###

The issue of security support was also mentioned: If many packages
start shipping tens or hundreds of vendored libraries, how can we
ensure security support? This has long been a pride point for Debian
and, in general, for the Linux distributions model. We package
independent libraries, dynamically link them, and security supports
"just happens" for the users. Now, what happens in languages as Go or
Rust, where linking is done statically? They would have to be at least
recompiled when any of their constitutive libraries is updated. But
what if the libraries are vendored-in? Security issues are prone to be
hidden from our sight.

I'm pasting here a bit of the discussion that happened later during
the meeting: having this discussion K8S-specific, Elana mentioned that
"that is a big part of the tension. sometimes, you have an upstream
where the authors are less resourced and responsive than the
downstream. this case is almost certainly the opposite".

At this point, we found some friction as to _what_ we are discussing
on: Is this bug specifically on Kubernetes, which should be taken as a
special case? Or would we try to rule as the Technical Committee on
vendoring in general in the Debian ecosystem? Elana repeated her
assurance that the Kubernetes team is thorough in their
freeness-checking and security practices; I insisted on "not
discussing about K8S, but about a bundling practice to which K8S

Bug#971515: Request for security team input on kubernetes TC bug

2020-11-17 Thread Florian Weimer 
* Moritz Mühlenhoff:

> On Sun, Nov 08, 2020 at 10:49:31PM +0100, Florian Weimer wrote:
>> * Moritz Mühlenhoff:
>> 
>> > * Follow a scheme similar to Firefox ESR where in case of a security
>> >   the update either happens to the latest minor release of
>> >   the current branch or if that has stopped, happens to the next
>> >   major release. To map this to specific k8s releases: Let's assume 
>> > bullseye
>> >   were already stable and would ship 19.3. In three months a security issue
>> >   arises which is severe enough to warrant an update and we ship 19.5
>> >   in a DSA. Fast forward 6 months and 19.x is EOLed, but some severe
>> >   security issue needs to be fixed; then the bullseye update would move
>> >   to 20.1 or so. That sounds unusual for a Debian release, but it's
>> >   the status quo for _any_ Kubernetes user after all (whether deployed
>> >   on premises or at some "cloud vendor").
>> 
>> Another thing to consider: Kubernetes rebases will likely require Go
>> rebases, if I read this table correctly:
>> 
>> 
>
> I can't tell how strict these requirements are in practice.

Let's just say that some Kubernetes developers are very eager to get
their hands on the most recent Go toolchain even if there are
practical issues with choices in the run-time library, such as the
changes to certification validation.  Not sure if anyone would want to
suffer these toolchain rebases if there was a clean way around them. 8-)

> It's similar to Firefox requiring more recent versions of rustc and
> golang packages are co-installable, so when it comes to that, shipping
> a new golang-x.y package might also be an option.

I see.

>> Since Go has a bit of spotty history in terms of kernel backwards
>> compatibility, this could cause further challenges.
>
> Do you have a concrete example of such a change? I see that 1.14 is
> available in backports, so this seems to work out so far.

I think that's it: 
If I recall correctly, there was a kernel version check (!) that
managed to reject kernels which did not have the bug.  And combined
with the workaround failing, this led to non-functional programs.

Bug#971515: Request for security team input on kubernetes TC bug

2020-11-17 Thread Moritz Mühlenhoff 
On Sun, Nov 08, 2020 at 10:49:31PM +0100, Florian Weimer wrote:
> * Moritz Mühlenhoff:
> 
> > * Follow a scheme similar to Firefox ESR where in case of a security
> >   the update either happens to the latest minor release of
> >   the current branch or if that has stopped, happens to the next
> >   major release. To map this to specific k8s releases: Let's assume bullseye
> >   were already stable and would ship 19.3. In three months a security issue
> >   arises which is severe enough to warrant an update and we ship 19.5
> >   in a DSA. Fast forward 6 months and 19.x is EOLed, but some severe
> >   security issue needs to be fixed; then the bullseye update would move
> >   to 20.1 or so. That sounds unusual for a Debian release, but it's
> >   the status quo for _any_ Kubernetes user after all (whether deployed
> >   on premises or at some "cloud vendor").
> 
> Another thing to consider: Kubernetes rebases will likely require Go
> rebases, if I read this table correctly:
> 
> 

I can't tell how strict these requirements are in practice.

It's similar to Firefox requiring more recent versions of rustc and
golang packages are co-installable, so when it comes to that, shipping
a new golang-x.y package might also be an option.

> Since Go has a bit of spotty history in terms of kernel backwards
> compatibility, this could cause further challenges.

Do you have a concrete example of such a change? I see that 1.14 is available
in backports, so this seems to work out so far.

Cheers,
Moritz

Bug#971515: Request for security team input on kubernetes TC bug

2020-11-17 Thread Moritz Mühlenhoff 


Catching up on this...

> > This leaves Debian with two options:
> > * Keep it out of a stable release and accept that it's good enough
> >   if people just install whatever deb they currently find in testing/sid
> >   (works out well enough for most given that blob nature of Go!)
> 
> IMHO this is the most reasonable option and perhaps the only viable one.

Ultimately that's for the release team and/or CTTE to make a call on.

And this while discussion also needs the input of Janos as the current 
kubernetes
maintainer.

> > * Follow a scheme similar to Firefox ESR where in case of a security
> >   the update either happens to the latest minor release of
> >   the current branch or if that has stopped, happens to the next
> >   major release.
> 
> I think Kubernetes have many more vendored 3rd party libraries than Firefox.
> IMHO we can not expect the same level of confidence for Kubernetes...

But each of their releases constitutes a bundle of third party libraries
they've vetted to work together, so this seems to work in practice? (as
can be seen by the 1.19 upload from today). Or maybe I'm missing the specific
concern of yours, is this about them missing fixes in their bundled libs?

Cheers,
Moritz

Bug#971515: Request for security team input on kubernetes TC bug

2020-11-08 Thread Florian Weimer 
* Moritz Mühlenhoff:

> * Follow a scheme similar to Firefox ESR where in case of a security
>   the update either happens to the latest minor release of
>   the current branch or if that has stopped, happens to the next
>   major release. To map this to specific k8s releases: Let's assume bullseye
>   were already stable and would ship 19.3. In three months a security issue
>   arises which is severe enough to warrant an update and we ship 19.5
>   in a DSA. Fast forward 6 months and 19.x is EOLed, but some severe
>   security issue needs to be fixed; then the bullseye update would move
>   to 20.1 or so. That sounds unusual for a Debian release, but it's
>   the status quo for _any_ Kubernetes user after all (whether deployed
>   on premises or at some "cloud vendor").

Another thing to consider: Kubernetes rebases will likely require Go
rebases, if I read this table correctly:



Since Go has a bit of spotty history in terms of kernel backwards
compatibility, this could cause further challenges.

Bug#971515: Request for security team input on kubernetes TC bug

2020-10-27 Thread Dmitry Smirnov 
On Wednesday, 28 October 2020 6:13:41 AM AEDT Moritz Mühlenhoff wrote:
> The bigger issue here (independent of the whole vendoring aspect) is
> how kubernetes can be supported in a stable release to begin with.
> This was raised by Shengjing Zhu in #959685 before.

If Kubernetes can be supported then such support will be done by upstream,
but with extraordinary amount of dependencies (and upstream reluctance to 
manage them), I have very low confidence and low expectations for quality
of such support. The problem primarily is that Kubernetes vendors hundreds
of dependencies representing a large support surface. Effectively it is
"#include world" (or vendor world) situation. And when it comes to problems
in 3rd party vendored libraries, it iw worth remembering that Kubernetes 
don't own them.


> This leaves Debian with two options:
> * Keep it out of a stable release and accept that it's good enough
>   if people just install whatever deb they currently find in testing/sid
>   (works out well enough for most given that blob nature of Go!)

IMHO this is the most reasonable option and perhaps the only viable one.


> * Follow a scheme similar to Firefox ESR where in case of a security
>   the update either happens to the latest minor release of
>   the current branch or if that has stopped, happens to the next
>   major release.

I think Kubernetes have many more vendored 3rd party libraries than Firefox.
IMHO we can not expect the same level of confidence for Kubernetes...

-- 
Best wishes,
 Dmitry Smirnov
 GPG key : 4096R/52B6BBD953968D1B

---

You have to start with the truth. The truth is the only way that we can
get anywhere. Because any decision-making that is based upon lies or
ignorance can't lead to a good conclusion.
-- Julian Assange, 2010


signature.asc
Description: This is a digitally signed message part.

Bug#971515: Request for security team input on kubernetes TC bug

2020-10-27 Thread Moritz Mühlenhoff 
On Wed, Oct 21, 2020 at 08:22:11AM -0700, Sean Whitton wrote:
> Hello security team,
> 
> The TC are being asked about src:kubernetes, and it would be good to
> hear from you about whether and how security support is a relevant
> consideration in determining whether the level of vendoring in that
> package is acceptable.  Could you take a look at #971515 and perhaps
> give us some input, please?

I don't currently have the time to read through all the discussion backlog,
but let me try to "zoom out a bit":

The bigger issue here (independent of the whole vendoring aspect) is
how kubernetes can be supported in a stable release to begin with.
This was raised by Shengjing Zhu in #959685 before.

Upstream support only lasts for up to a year and it would be
presumptuous to pretend that we can seriously commit to fix
security issues in k8s for longer than upstream.

This leaves Debian with two options:
* Keep it out of a stable release and accept that it's good enough
  if people just install whatever deb they currently find in testing/sid
  (works out well enough for most given that blob nature of Go!)
* Follow a scheme similar to Firefox ESR where in case of a security
  the update either happens to the latest minor release of
  the current branch or if that has stopped, happens to the next
  major release. To map this to specific k8s releases: Let's assume bullseye
  were already stable and would ship 19.3. In three months a security issue
  arises which is severe enough to warrant an update and we ship 19.5
  in a DSA. Fast forward 6 months and 19.x is EOLed, but some severe
  security issue needs to be fixed; then the bullseye update would move
  to 20.1 or so. That sounds unusual for a Debian release, but it's
  the status quo for _any_ Kubernetes user after all (whether deployed
  on premises or at some "cloud vendor").

If we follow the latter approach, then the current way k8s is packaged
is the only viable option (as otherwise the run time deps shipped
for 19.x would be incompatible with 20.x etc.)

Cheers,
Moritz

Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-22 Thread Dmitry Smirnov 
On Friday, 23 October 2020 4:20:37 AM AEDT Shengjing Zhu wrote:
> However kubernetes is also a library, and the maintainer doesn't provide
> it. It becomes headache for other maintainers.

Yes we have to vendor "k8s.io/" all the time in multiple packages but I doubt 
it would be of help if Kubernetes provide a "-dev" library because
it is notoriously unstable with frequent breaking changes.

Ideally that's how it should be done but, pragmatically speaking, Kubernetes 
can never migrate to "testing" hence everything that depends on it will be 
unsuitable for release.


> The way the kubernetes maintainer chooses, is making others difficult.
> Every package that uses kubernetes as a library, has to keep a local copy
> of the source.

The partial solution might be to introduce another source package only with 
portion of Kubernetes that provides a shared library but that will require 
proper dependency management which is being discussed here.


> But I hope Janos can cooperate with the Go packaging team to make Debian,
> especially the Go packages in Debian more maintainable.

Yes indeed. We should not need to argue that cooperation is important.
Cooperation should always be a default mode of operation.

-- 
Kind regards,
 Dmitry Smirnov
 GPG key : 4096R/52B6BBD953968D1B

---

Hay smells different to lovers and horses.
-- Stanisław Jerzy Lec


signature.asc
Description: This is a digitally signed message part.

Bug#971515: Request for security team input on kubernetes TC bug

2020-10-22 Thread Dmitry Smirnov 
On Thursday, 22 October 2020 2:22:11 AM AEDT Sean Whitton wrote:
> The TC are being asked about src:kubernetes, and it would be good to
> hear from you about whether and how security support is a relevant
> consideration in determining whether the level of vendoring in that
> package is acceptable.  Could you take a look at #971515 and perhaps
> give us some input, please?

I'm not a member of security team but I want to share my prospective on this.

To us what matters is what we can do about vulnerability in situation where
the problem has been identified and fixed upstream, in the particular 
library.

When a library is packaged separately, this is a matter of patching/updating 
a particular library then re-building depending packages.

With vendored libraries, we have no control as patching arbitrary versions of 
all instances of vendored library is a very bad option that is practically 
unsustainable. Even tracking of vendored libraries is difficult.

So in case of vendored libraries we can rely on "Oracle model" when upstream 
releases an update of a software where vendored library is patched so we ship 
the whole bundle. This is the worst option because we have no control and 
because we have to rely on faith in good upstream maintenance.

But here lies the problem. Kubernetes have bad dependency hygiene and poor 
abolity to control their enormous vendor mess (burden). With hundreds of 
vendored libraries they have large surface for problems and limited ability 
to track and respond. Just like we know from experience, an attempt to update 
a vendored library may lead to FTBFS and that's why upstream is reluctant to 
make changes to vendored libraries, unless when they absolutely have to.
Kubernetes can not effectively address issues in "vendor/" space.

So my conclusion about security support of upstream Kubernetes is that it is 
can not be done effectively in either case. There are greater chances for 
meaningful security maintenance when most dependency libraries are un-
vendored. With most libraries vendored, there is less security maintenance 
burden to us, but the outcome is no better.

-- 
All the best,
 Dmitry Smirnov
 GPG key : 4096R/52B6BBD953968D1B

---

No snowflake in an avalanche ever feels responsible.
-- Stanisław Jerzy Lec


signature.asc
Description: This is a digitally signed message part.

Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-22 Thread Shengjing Zhu 
On Tue, Oct 20, 2020 at 12:16:03PM -0700, Sean Whitton wrote:
> 
> - are people trying to do cross-archive work going to find the packaging
>   of kubernetes getting in their way?  (e.g. other Go team members
>   trying to update things, improve our binNMU techniques and machinery,
>   etc.)
> 

The kubernetes maintainer is not in the Go packaging team, so doesn't follow
the team practice. But as kubernetes vendor everything, it doesn't need to
work with all other Go packages.

However kubernetes is also a library, and the maintainer doesn't provide it.
It becomes headache for other maintainers.
We (the Go packaging team) need to vendor the kubernetes in our packages,
for example:

+ 
https://sources.debian.org/src/golang-github-openshift-api/4.0+git20190508.81d064c-4/vendor/k8s.io/
+ 
https://sources.debian.org/src/gitlab-ci-multi-runner/13.3.1+dfsg-1/vendor/k8s.io/
+ https://sources.debian.org/src/flannel/0.9.1~ds1-1/vendor/k8s.io/
+ https://sources.debian.org/src/consul/1.7.4+dfsg1-1/vendor/k8s.io/

And many, they can be found with:

https://codesearch.debian.net/search?q=%22k8s.io%2F+filetype%3Ago=1

It has been asked at #970331, without response from maintainer.

If kubernetes package want to make other package which Build-Depends on it
to build successfully, it needs to decouple the dependencies from vendor dir.
When building other packages, the Go toolchain can't use the vendor in
kubernetes, since the Go toolchain only supports the top level vendor dir.

The way the kubernetes maintainer chooses, is making others difficult.
Every package that uses kubernetes as a library, has to keep a local copy of
the source.

It's not Janos fault, as he doesn't make things worse. Kubernetes package is
in a bad state for a long time.
But I hope Janos can cooperate with the Go packaging team to make Debian,
especially the Go packages in Debian more maintainable.

-- 
Shengjing Zhu


signature.asc
Description: PGP signature

Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-21 Thread Dmitry Smirnov 
On Thursday, 22 October 2020 2:16:16 AM AEDT Sean Whitton wrote:
> I think that we can all agree with everything you've written about the
> reasons why packaging components separately is better. 

Thank you.


> The problem is
> that in this case the choice seems to be between not having recent
> Kubernetes in Debian at all, or giving up on some of those advantages.

Ultimately it is about maintainers comfort then. Because there is a 
compromise to un-vendor most libraries (one by one) and keep some/few 
strategically vendored, when a system library can not be used (determined 
case by case).

Examples of this approach are numerous: syncthing, consul, nomad, vault, 
docker.io, runc, gitlab-runner, libpod, buildah, singularity-container
and _most_ Golang packages.

Kubernetes had no maintainer - that was the problem. Having maintainer who
does not care for shared libraries because he does not want to be bothered
is a different problem.

-- 
Kind regards,
 Dmitry Smirnov
 GPG key : 4096R/52B6BBD953968D1B

---

In times of universal deceit, telling the truth becomes a revolucionary
act.
-- George Orwell


signature.asc
Description: This is a digitally signed message part.

Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-21 Thread Dmitry Smirnov 
On Wednesday, 21 October 2020 8:56:53 PM AEDT Felix Lechner wrote:
> How is the second one inferior, please? I think it includes a crucial
> missing feature (co-installable versions).

To reduce maintainers burden, you want maintainers to look after not one but 
multiple versions of libraries. This is a contradiction (oxymoron).
How many releases (current + obsolete ones) one can meaningfully support?

-- 
Best wishes,
 Dmitry Smirnov
 GPG key : 4096R/52B6BBD953968D1B

---

The cure for the evils of democracy is more democracy.
-- H. L. Mencken


signature.asc
Description: This is a digitally signed message part.

Bug#971515: Request for security team input on kubernetes TC bug

2020-10-21 Thread Sean Whitton 
Hello security team,

The TC are being asked about src:kubernetes, and it would be good to
hear from you about whether and how security support is a relevant
consideration in determining whether the level of vendoring in that
package is acceptable.  Could you take a look at #971515 and perhaps
give us some input, please?

Thanks.

-- 
Sean Whitton

Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-21 Thread Sean Whitton 
Hello,

On Tue 20 Oct 2020 at 06:52pm -07, Felix Lechner wrote:

>  I think our response to the vendoring explosion is at odds with the
> trends in many languages.
>
> It's time to retool. At the two ends of the solution spectrum, I see
>
> 1. Fully vendored source packages; or
> 2. A packaging system that allows different vendor versions to co-exist.
>
> Either one allows dependent sources to consume whichever versions they
> require, but in my view solution (2) is otherwise superior---provided
> that the packaging process is automated. (A language's build system
> also has to distinguish the installed versions.) For each language so
> affected, could we make (2) our goal, and allow (1) until then?

This is a very general (but of course interesting) topic.  Could I ask
that it be kept out of this TC bug, please?  We have to figure out what
to do about this package given our present tooling.

-- 
Sean Whitton

Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-21 Thread Sean Whitton 
Hello Dmitry,

On Wed 21 Oct 2020 at 11:21am +11, Dmitry Smirnov wrote:

> On Wednesday, 21 October 2020 6:16:03 AM AEDT Sean Whitton wrote:
>> I think that my message [1] is what makes you think that the package
>> would not have got through NEW?
>
> It was not your message but my own experience with introducing of 100+ 
> packages through NEW, especially those ones with large burden of vendored 
> libraries, including Kubernetes. The main hassle usually is to convince FTP-
> masters when some vendoring is _necessary_ (case by case) and the usual 
> request is to package all vendored libraries separately. With rare exceptions 
> some (few) vendored libraries are allowed like when a library is a fork, 
> customised for the particular project and therefore not re-usable by other 
> software. Another example is when vendored library is an obsolete software 
> phasing out in future releases. Few micro-libraries might be tolerated when 
> vendored, especially when they are not widely used. Also vendoring might be 
> acceptable when software components with mutual/circular dependencies are 
> shipped in one or several name spaces - in other words when a software code 
> base is not from one name space but from several. None of those cases applies 
> to Kubernetes.
>
> A specific example (libpod/podman) is mentioned in 
>
>   https://lists.debian.org/debian-devel/2020/03/msg00441.html
>
> "Podman was rejected due to "many embedded packages in vendor/" with only
>  6 or 7 private libs versus 120 libraries removed in favour of packaged
>  ones."

Fair enough, but again, this is about NEW as a review by experienced
packagers rather than NEW as a blocker for inclusion.

> If your concern is about security support then IMHO Kubernetes can not be 
> meaningfully supported from security prospective, with or without vendored 
> dependencies.

Fair enough.  In the -devel thread the security team indicated that they
thought they could do security support in the case where there is
vendoring.  It would be good to get more input from them in this bug.

> If Debian only cared about maintainers' convenience or reducing maintainers 
> efforts then Debian would not be what it is now. We favour technical elegance 
> often in expense of maintainers' comfort.

I don't think maintainers' comfort is part of Janos' motivation -- it's
the issue of not having recent Kubernetes in Debian at all.

>> Are there issues the TC should think about which do not fall under this
>> way of looking at things?  I.e., weighing the impact on people other
>> than Janos who want to work on the package, vs. the impact on people who
>> want recent kubernetes to be part in the archive at all?
>
> Is Debian ecosystem of packaged reusable libraries worth caring about?
>
> If so then why grant exception to one particular package? We have several (or 
> more) sophisticated Golang packages using hundreds of packaged libraries.
>
> In the early days of packaging Kubernetes we did not even have most 
> components packaged and I've been spending most effort on packaging, 
> introducing and stabilising dependency libraries.
> These days the major work has already been done and the argument for 
> monolithic vendoring is much weaker.
>
> [...]

I think that we can all agree with everything you've written about the
reasons why packaging components separately is better.  The problem is
that in this case the choice seems to be between not having recent
Kubernetes in Debian at all, or giving up on some of those advantages.

-- 
Sean Whitton

Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-21 Thread Felix Lechner 
Hi Dmitry,

On Wed, Oct 21, 2020 at 12:50 AM Dmitry Smirnov  wrote:
>
> Yes, at first there will
> be a significant effort but then it will become easier.

I know you put a lot of effort in (as did Michael Stapelberg, with
whom I spent some time before he left), but I don't think our current
approach makes anything easy. It is also why the world is moving in
another direction.

> You are advocating for disruptive
> changes therefore your proposed theoretical solutions have to be available as
> a proof of concept for review.

Did you catch the part about different versions being co-installable?
It would be similar to the freedom we grant to major numbers of shared
object libraries. My point is theoretical only because we do not
presently have it.

> Personally I'm not satisfied with either of those inferior proposals.

How is the second one inferior, please? I think it includes a crucial
missing feature (co-installable versions).

> Look how many packages we already have:
>
>   
> https://qa.debian.org/developer.php?login=pkg-go-maintainers%40lists.alioth.debian.org+team%2Bpkg-go%40tracker.debian.org

It's an impressive list; thank you for forwarding it. I am proud to be
on the Golang team. :)

> In the meantime you could follow the established practice that is
> demonstrated to be working on several packaged heavy Golang applications.

Not sure about heavy Golang applications, but our established practice
does work well for the relatively lightweight 'gocryptfs', which I
maintain. That source moves very fast. I often have problems finding
the proper go-fuse or xattr prerequisite required for a new version.
Sometimes they are incompatible with the needs of other packages in
the archive.

As a side note, several "library" packages that gocryptfs relies on
should really be marked "Architecture: any" to exercise their test
suites properly. It is another example of the impedance mismatch in
our current approach. We are confusing sources and libraries, and our
method of shoehorning one into the other ought to be rethought.

> Besides un-vendoring libraries can prevent some CVE issues as well.

Packages could declare vendored sources (or Lintian could scan for
them) for an effective match with their security status.

> If we tolerate full vendoring now, because "there is no better way" yet, then
> there will be no better way for sure.

Please do not despair. I offered full vendoring only in the interest
of compromise, which I believe is a worthwhile goal just like the
consensus we are working on. As a project, we are looking for a better
endpoint together.

Kind regards
Felix Lechner

Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-21 Thread Dmitry Smirnov 
Hi Felix,

On Wednesday, 21 October 2020 12:52:40 PM AEDT Felix Lechner wrote:
> > We favour technical elegance often in expense of maintainers' comfort.
> 
> Is our approach really either one of those? I think our response to
> the vendoring explosion is at odds with the trends in many languages.

IMHO we are managing quite admirably. Basically, to me it looks like you 
don't want to maintain Kubernetes the way we maintain heavy Golang packages.
You would have to learn to un-vendor many libraries. Yes, at first there will 
be a significant effort but then it will become easier. "Too many vendored 
libraries to use packaged libs" is a poor excuse.

We have been dealing with "explosion" for years already. Tools like "dh-make-
golang" are helpful to generate initial packaging for new Golang libraries in 
a semi-automatic manner. FTP-masters are usually quite effective with 
processing of NEW packages. Look how many packages we already have:

  
https://qa.debian.org/developer.php?login=pkg-go-maintainers%40lists.alioth.debian.org+team%2Bpkg-go%40tracker.debian.org


> It's time to retool. At the two ends of the solution spectrum, I see
> 
> 1. Fully vendored source packages; or
> 2. A packaging system that allows different vendor versions to
> co-exist.

Personally I'm not satisfied with either of those inferior proposals.

Besides un-vendoring libraries can prevent some CVE issues as well.


> Either one allows dependent sources to consume whichever versions they
> require, but in my view solution (2) is otherwise superior---provided
> that the packaging process is automated. (A language's build system
> also has to distinguish the installed versions.) For each language so
> affected, could we make (2) our goal, and allow (1) until then?

IMHO tools have to come first (if ever). You are advocating for disruptive 
changes therefore your proposed theoretical solutions have to be available as 
a proof of concept for review.

In the meantime you could follow the established practice that is 
demonstrated to be working on several packaged heavy Golang applications.

If we tolerate full vendoring now, because "there is no better way" yet, then 
there will be no better way for sure. For now using packaged system libraries 
whenever possible is the best way.

-- 
Kind regards,
 Dmitry Smirnov
 GPG key : 4096R/52B6BBD953968D1B

---

Those who disdain wealth as a worthy goal for an individual or a society
seem not to realize that wealth is the only thing that can prevent poverty.
-- Thomas Sowell


signature.asc
Description: This is a digitally signed message part.

Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-20 Thread Felix Lechner 
Hi Dmitry,

On Tue, Oct 20, 2020 at 5:24 PM Dmitry Smirnov  wrote:
>
> Let's not attempt to fabricate perception of consensus please.

Consensus is a worthy goal and perhaps even possible, per below.

> We favour technical elegance
> often in expense of maintainers' comfort.

Is our approach really either one of those? I think our response to
the vendoring explosion is at odds with the trends in many languages.

It's time to retool. At the two ends of the solution spectrum, I see

1. Fully vendored source packages; or
2. A packaging system that allows different vendor versions to co-exist.

Either one allows dependent sources to consume whichever versions they
require, but in my view solution (2) is otherwise superior---provided
that the packaging process is automated. (A language's build system
also has to distinguish the installed versions.) For each language so
affected, could we make (2) our goal, and allow (1) until then?

Kind regards
Felix Lechner

Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-20 Thread Dmitry Smirnov 
On Wednesday, 21 October 2020 6:16:03 AM AEDT Sean Whitton wrote:
> I think that my message [1] is what makes you think that the package
> would not have got through NEW?

It was not your message but my own experience with introducing of 100+ 
packages through NEW, especially those ones with large burden of vendored 
libraries, including Kubernetes. The main hassle usually is to convince FTP-
masters when some vendoring is _necessary_ (case by case) and the usual 
request is to package all vendored libraries separately. With rare exceptions 
some (few) vendored libraries are allowed like when a library is a fork, 
customised for the particular project and therefore not re-usable by other 
software. Another example is when vendored library is an obsolete software 
phasing out in future releases. Few micro-libraries might be tolerated when 
vendored, especially when they are not widely used. Also vendoring might be 
acceptable when software components with mutual/circular dependencies are 
shipped in one or several name spaces - in other words when a software code 
base is not from one name space but from several. None of those cases applies 
to Kubernetes.

A specific example (libpod/podman) is mentioned in 

  https://lists.debian.org/debian-devel/2020/03/msg00441.html

"Podman was rejected due to "many embedded packages in vendor/" with only
 6 or 7 private libs versus 120 libraries removed in favour of packaged
 ones."


> There are a few issues tangled together here.

IMHO it is really one issue of how we maintain Debian packages. If you want 
to distinguish few issues then they are all closely related.


> NEW is mainly about license and DFSG compliance, and secondarily about
> the idea that we want to avoid accepting packages where doing so would
> make Debian worse, even if it would also make Debian better along other
> dimensions.  As a simple example, we try to avoid accepting a package
> that is already packaged under a slightly different name, because in
> most cases it is worse for both users and contributors to have the same
> thing in the archive twice (not talking about vendoring here).

It is also about preserving integrity of Debian identity. We try to prevent 
monolithic bundles like Kubernetes in favour of maintaining ecosystem of re-
usable libraries, packaged individually.


> In this case, the reason I wrote in [1] that I would probably have
> rejected the package, had I come across it in NEW, is that it seemed to
> me that having this package in Debian would make the archive less
> maintainable by contributors other than Janos who might need to work
> with the package.  (After the discussion on -devel, I'm no longer so
> sure about that opinion of mine.)

  https://wiki.debian.org/UpstreamGuide#No_inclusion_of_third_party_code

If your concern is about security support then IMHO Kubernetes can not be 
meaningfully supported from security prospective, with or without vendored 
dependencies.

Also I must point out that Kubernetes upstream have the worst management of 
vendored libraries that I have ever seen. Examples include vendoring multiple 
versions of the same library, etc.
A particular case when upstream failed to update a problematic vendored 
library for years(!) practically destroys faith in upstream care for good 
hygiene of vendored dependencies:

  https://github.com/kubernetes/kubernetes/issues/27543

Note the expressed _resistance_ to upgrading a vendored library...

With the above example, how can anyone have confidence in upstream security 
patching? After all Kubernetes have more vendored code than its own.



> It's not correct to say, however, that the package "is in violation of
> ftpmaster's policy for inclusion of new packages".  That could only be
> true is if the package met one of the "serious violations" listed in the
> REJECT-FAQ, which is basically DFSG and licensing issues, and a few
> obvious clangers.  Instead, what we have is a situation in which there
> is reason to be worried about the way the package is put together, but
> the opinion of one FTP team member at one particular point in time
> carries about the same weight as the opinion of any experienced
> packager.

There is an established practice, a tradition if you wish, that is followed 
all the time even if not explicitly described in REJECT-FAQ. Debian clearly
tries to be modular whenever possible.


> In other words, I suggest that we ignore the NEW issue entirely, and
> just consider whether the way the package is currently put together
> imposes an unreasonable burden on Debian contributors other than Janos
> who want to work on the package, or users who want to patch it, etc.
> The sorts of questions we should try to answer:
> 
> - does the vendoring make Debian security support harder (discussion on
>   -devel suggests it makes it easier)
> 
> - everyone seems to think the level of vendoring is at best a necessary
>   evil;

Let's not attempt to fabricate perception of

Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-10-20 Thread Sean Whitton 
Hello Dmitry, others,

On Thu 01 Oct 2020 at 11:15am +10, Dmitry Smirnov wrote:

> I seek your judgement regarding excessive, unnecessary and unjustifiable
> vendoring of private libraries in Kubernetes package:
>
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970717
>
> Some relevant argumentation can be found in
>
>   https://lists.debian.org/debian-devel/2020/03/msg00359.html
>   https://lists.debian.org/debian-devel/2020/03/msg00400.html
>   https://lists.debian.org/debian-devel/2020/03/msg00441.html
>
> In short, myself and Golang packaging team spent years on stabilising
> Golang ecosystem of packaged libraries for re-use by Golang software.
>
> To the best of my knowledge, all packaged Golang software, regardless of its 
> sophistication, use some packages libraries.
> Except Kubernetes, that disconnected itself from cooperation by not using any 
> packaged libraries, instead exclusively using only private libraries in 
> numbers greater than any Debian package ever used.
>
> As a former Kubernetes maintainer and a developer who originally introduced
> Kubernetes to Debian, I know that it could be maintained with only few (or 
> several) private libraries at most.
>
> Current state of Kubernetes package is in violation of ftp-master's policy 
> for inclusion of new packages to Debian.

I think that my message [1] is what makes you think that the package
would not have got through NEW?  There are a few issues tangled together
here.

NEW is mainly about license and DFSG compliance, and secondarily about
the idea that we want to avoid accepting packages where doing so would
make Debian worse, even if it would also make Debian better along other
dimensions.  As a simple example, we try to avoid accepting a package
that is already packaged under a slightly different name, because in
most cases it is worse for both users and contributors to have the same
thing in the archive twice (not talking about vendoring here).

In this case, the reason I wrote in [1] that I would probably have
rejected the package, had I come across it in NEW, is that it seemed to
me that having this package in Debian would make the archive less
maintainable by contributors other than Janos who might need to work
with the package.  (After the discussion on -devel, I'm no longer so
sure about that opinion of mine.)

It's not correct to say, however, that the package "is in violation of
ftpmaster's policy for inclusion of new packages".  That could only be
true is if the package met one of the "serious violations" listed in the
REJECT-FAQ, which is basically DFSG and licensing issues, and a few
obvious clangers.  Instead, what we have is a situation in which there
is reason to be worried about the way the package is put together, but
the opinion of one FTP team member at one particular point in time
carries about the same weight as the opinion of any experienced
packager.

In other words, I suggest that we ignore the NEW issue entirely, and
just consider whether the way the package is currently put together
imposes an unreasonable burden on Debian contributors other than Janos
who want to work on the package, or users who want to patch it, etc.
The sorts of questions we should try to answer:

- does the vendoring make Debian security support harder (discussion on
  -devel suggests it makes it easier)

- everyone seems to think the level of vendoring is at best a necessary
  evil; if someone wants to try to reduce the level of vendoring (as
  Dmitry did when he was maintainer), is the current way the package is
  built going to make it harder for people to work on making that sort
  of contribution?

- are people trying to do cross-archive work going to find the packaging
  of kubernetes getting in their way?  (e.g. other Go team members
  trying to update things, improve our binNMU techniques and machinery,
  etc.)

... and this is to be weighed against the negative impact of having
kubernetes in Debian lag so seriously behind upstream such that almost
no-one would want to bother installing it.

Are there issues the TC should think about which do not fall under this
way of looking at things?  I.e., weighing the impact on people other
than Janos who want to work on the package, vs. the impact on people who
want recent kubernetes to be part in the archive at all?

If I'm right about what the question for the TC is, I hope that Janos
and Dmitry can both help us discuss it in a way which sets aside the
heat which characterised the -devel thread.  It is completely
understandable to (a) feel very frustrated at Debian not including
recent versions of a useful piece of free software; and also (b) feel
very frustrated when someone chooses to accept a less-than-ideal
approach as necessary when one has put a lot of time into trying to find
workable alternatives.

The thing is, both (a) and (b) are motivated by the same basic desire to
make Debian better and more useful, so perhaps we can focus on that
point of commonality.

[1]

Bug#971515: kubernetes: excessive vendoring (private libraries)

2020-09-30 Thread Dmitry Smirnov 
Package: tech-ctte
Severity: normal
X-Debbugs-CC: o...@debian.org
Control: block 970717 by -1

Dear Technical Committee,

Apologies that we were not able to resolve the problem without your help.

I seek your judgement regarding excessive, unnecessary and unjustifiable
vendoring of private libraries in Kubernetes package:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970717

Some relevant argumentation can be found in

  https://lists.debian.org/debian-devel/2020/03/msg00359.html
  https://lists.debian.org/debian-devel/2020/03/msg00400.html
  https://lists.debian.org/debian-devel/2020/03/msg00441.html

In short, myself and Golang packaging team spent years on stabilising
Golang ecosystem of packaged libraries for re-use by Golang software.

To the best of my knowledge, all packaged Golang software, regardless of its 
sophistication, use some packages libraries.
Except Kubernetes, that disconnected itself from cooperation by not using any 
packaged libraries, instead exclusively using only private libraries in 
numbers greater than any Debian package ever used.

As a former Kubernetes maintainer and a developer who originally introduced
Kubernetes to Debian, I know that it could be maintained with only few (or 
several) private libraries at most.

Current state of Kubernetes package is in violation of ftp-master's policy 
for inclusion of new packages to Debian.

Thank you.

This matter is not urgent.

-- 
Regards,
 Dmitry Smirnov

---

We do our friends no favors by pretending not to notice flaws in their
work, especially when those who are not their friends are bound to notice
these same flaws.
-- Sam Harris


signature.asc
Description: This is a digitally signed message part.