Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Sun, Sep 21, 2003 at 08:27:49PM -0500, Manoj Srivastava wrote:

>   I am always willing to improve my packages; the constraints
>  are ability (I would need to grok the details of the current
>  implementation), time, and collaboration (I would need to find out
>  how to get a hook into the current patch dependency setup).
> 
>   I would also like to incorporate conflict mechanisms -- so
>  patch developers can give a hint about patches that are not
>  compatible (the swsusp patches are incompatible with most other
>  patches I wanted to apply).
> 
>   Where do I start?

I suppose a good place to start would be to talk to Yann Dirson, and look at
dh-kpatches.  That is how all of my kernel-patch packages work now.

-- 
 - mdz

Re: Maintaining kernel source in sarge

[Russell Coker]

On Mon, 22 Sep 2003 11:27, Manoj Srivastava wrote:
> > The scripts handle ordering by testing each dependency, and if it is
> > not already applied, invoking the corresponding apply script.  In
> > this way, all dependencies should be applied in proper order.
> > Rollback could presumably be implemented by unapplying all patches
> > if any patch fails (dh-kpatches should now implement correct
> > ordering for unapplication as well).
>
> Well, if I have asked for 5 patches to be applied (preempt,
>  lowlatency, skas, device-mapper, lsm2, and debianlogo, in a recent
>  invocation), the lsm2 script indeed failed -- but only after preempt,
>  lowlatency, skas,  and device-mapper patches were applied (I did not
>  have acl kernel-patch on the machine). It would have been nice to
>  know about that before all the patches were applied.

I think that's a bug.  The next upload of the kernel-patch-2.4-lsm package 
will depend on the ACL patch package.

Also I will split the package and put the patch for the old SE Linux in a 
separate package on my web site.

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

Re: Maintaining kernel source in sarge

[Manoj Srivastava]

On Sun, 21 Sep 2003 20:12:26 -0400, Matt Zimmerman <[EMAIL PROTECTED]> said: 

> On Sun, Sep 21, 2003 at 06:36:25PM -0500, Manoj Srivastava wrote:
>> On Sun, 18 May 2003 20:32:05 -0400, Matt Zimmerman <[EMAIL PROTECTED]>
>> said:
>> > dh-kpatches provides a dependency/ordering facility which has
>> > worked well for me in my packages.  It also provides
>> > /usr/share/doc/kernel-image-foo/applied-patches documenting the
>> > package and version for each patch that is applied.  I think this
>> > would be a good starting point for such a policy, since it is
>> > already being applied in Debian.
>>
>> Is this dependency information easily accessible to external
>> scripts? It is nice that the patch scripts themselves realize when
>> a required patch has been installed or not, but it would work much
>> better if the order in which these patches were applied could also
>> be ordered nicely (so patches are applied in dependency order), and
>> so that we can abort early (before anything was modified in the
>> local sources) in case some dependencies have not been met.
>>
>> So, is there a way that kernel-package can interface with this
>> dependency/ordering facility?

> As far as I know, there is no external interface (the dependency
> information is stored in shell variables inside the script).

> The scripts handle ordering by testing each dependency, and if it is
> not already applied, invoking the corresponding apply script.  In
> this way, all dependencies should be applied in proper order.
> Rollback could presumably be implemented by unapplying all patches
> if any patch fails (dh-kpatches should now implement correct
> ordering for unapplication as well).

Well, if I have asked for 5 patches to be applied (preempt,
 lowlatency, skas, device-mapper, lsm2, and debianlogo, in a recent
 invocation), the lsm2 script indeed failed -- but only after preempt,
 lowlatency, skas,  and device-mapper patches were applied (I did not
 have acl kernel-patch on the machine). It would have been nice to
 know about that before all the patches were applied. 


> It may indeed make sense to move some of this logic into
> kernel-package, if you are willing to do it.

I am always willing to improve my packages; the constraints
 are ability (I would need to grok the details of the current
 implementation), time, and collaboration (I would need to find out
 how to get a hook into the current patch dependency setup).

I would also like to incorporate conflict mechanisms -- so
 patch developers can give a hint about patches that are not
 compatible (the swsusp patches are incompatible with most other
 patches I wanted to apply).

Where do I start?

manoj
-- 
No proper program contains an indication which as an operator-applied
occurrence identifies an operator-defining occurrence which as an
indication-applied occurrence identifies an indication-defining
occurrence different from the one identified by the given indication
as an indication-applied occurrence. ALGOL 68 Report
Manoj Srivastava   <[EMAIL PROTECTED]>  
1024R/C7261095 print CB D9 F4 12 68 07 E4 05  CC 2D 27 12 1D F5 E8 6E
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Sun, Sep 21, 2003 at 06:36:25PM -0500, Manoj Srivastava wrote:

> On Sun, 18 May 2003 20:32:05 -0400, Matt Zimmerman <[EMAIL PROTECTED]> said: 
> > dh-kpatches provides a dependency/ordering facility which has worked
> > well for me in my packages.  It also provides
> > /usr/share/doc/kernel-image-foo/applied-patches documenting the
> > package and version for each patch that is applied.  I think this
> > would be a good starting point for such a policy, since it is
> > already being applied in Debian.
> 
>   Is this dependency information easily accessible to external
>  scripts? It is nice that the patch scripts themselves realize when a
>  required patch has been installed or not, but it would work much
>  better if the order in which these patches were applied could also be
>  ordered nicely (so patches are applied in dependency order), and so
>  that we can abort early (before anything was modified in the local
>  sources) in case some dependencies have not been met.
> 
>   So, is there a way that kernel-package can interface with this
>  dependency/ordering facility? 

As far as I know, there is no external interface (the dependency information
is stored in shell variables inside the script).

The scripts handle ordering by testing each dependency, and if it is not
already applied, invoking the corresponding apply script.  In this way, all
dependencies should be applied in proper order.  Rollback could presumably
be implemented by unapplying all patches if any patch fails (dh-kpatches
should now implement correct ordering for unapplication as well).

It may indeed make sense to move some of this logic into kernel-package, if
you are willing to do it.

-- 
 - mdz

Re: Maintaining kernel source in sarge

[Manoj Srivastava]

On Sun, 18 May 2003 20:32:05 -0400, Matt Zimmerman <[EMAIL PROTECTED]> said: 

> On Sun, May 18, 2003 at 12:06:21PM -0500, Manoj Srivastava wrote:
>> There is also a mechanism to order the order in which
>> kernel-patches are applied -- so if, say, a m68k kernel image
>> maintainer wanted to create a patch relative to the i386 patches,
>> they could depend on that patch, and order the m68k kernel-pacth to
>> be applied later in the chain than the i386 patch.
>>
>> This dependency-and-ordering mechanism could be extended to third
>> party modules.
>>
>> People interested in hammering out details of this mechanism, and
>> kernel image maintainers, please contact me; perhaps it is time to
>> create policy for kernel patches.

> dh-kpatches provides a dependency/ordering facility which has worked
> well for me in my packages.  It also provides
> /usr/share/doc/kernel-image-foo/applied-patches documenting the
> package and version for each patch that is applied.  I think this
> would be a good starting point for such a policy, since it is
> already being applied in Debian.

Is this dependency information easily accessible to external
 scripts? It is nice that the patch scripts themselves realize when a
 required patch has been installed or not, but it would work much
 better if the order in which these patches were applied could also be
 ordered nicely (so patches are applied in dependency order), and so
 that we can abort early (before anything was modified in the local
 sources) in case some dependencies have not been met.

So, is there a way that kernel-package can interface with this
 dependency/ordering facility? 

manoj
-- 
love, n.: When it's growing, you don't mind watering it with a few
tears.
Manoj Srivastava   <[EMAIL PROTECTED]>  
1024R/C7261095 print CB D9 F4 12 68 07 E4 05  CC 2D 27 12 1D F5 E8 6E
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C

Re: Maintaining kernel source in sarge

[Manoj Srivastava]

On Tue, 27 May 2003 11:04:07 +0200, Arnd Bergmann <[EMAIL PROTECTED]> said: 

> The order in which the patches are applied should in general not be
> significant. If it is, it should be stated in the patch
> description. I assumed that the 'Depends' tag is semantically more a
> 'Pre-Depends', right?

> In my example, if the isdnbonding patch has to be applied after
> ethernetpadding, it would have to say so in its description.
> patch-2.4.21-pre7 can contain the same patch and should consequently
> be applied at the same time (e.g. after binfmtmisc, but before
> isdnbonding).

>> That's only fine _if_ the replaced patches are similar enough so
>> that any patch in the debian set, that would depend on those, can
>> still apply atop the new one.  That is, if there are several
>> revisions of a given fix, we'll have to use versionned Provides:
>> and Depends:, or we're doomed.  Not that it's impossible, but I'm
>> not sure it's exactly trivial to implement...

> Yes, versioned patches are a problem that I did not think of. I
> assumed that one patch providing the functionality of another one is
> always a superset. It may be possible to require versioned patches
> to be made as a sequence of diffs between the versions.  Of course
> that creates new problems.

I am thinking of implementing something like this in
 kernel-package. Ideally, I would like to support things like:

  a) Specify a range or a single kernel version that the patch applies  to
  b) Specify a patch version
  c) specify patches (with versions) that one depends on, these
 patches need be present, and shall be applied before the current
 patch
  
User should be able to specify a list of patches that they
 want to apply; and have all the dependent patches pulled in, or warn
 that patches could not be applied (not-present; failed-depends); etc.

The patches applied would be unapplied in reverse order on
 clean up. 

I am thinking of a generic run-parts replacement with
 versioned depends; so that this work be useful beyond
 kernel-package. 

manoj
-- 
I wish a robot would get elected president.  That way, when he came to
town, we could all take a shot at him and not feel too bad. Jack
Handley
Manoj Srivastava   <[EMAIL PROTECTED]>  
1024R/C7261095 print CB D9 F4 12 68 07 E4 05  CC 2D 27 12 1D F5 E8 6E
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C

Re: Maintaining kernel source in sarge

[Herbert Xu]

On Sat, May 31, 2003 at 12:14:31PM -0500, Christian T. Steigies wrote:
>
> I think this is contracticting to what you said just a few messages earlier.
> For 2.4.21 will there be a pristine kernel-source and seperate i386-patch or
> not? It seems I am the m68k maintainer, and I am for the seperate patch

It does.  The current plan is outlined in

http://lists.debian.org/debian-devel/2003/debian-devel-200305/msg01805.html

Basically it means you can continue doing what you're doing now if you
wish, and if so you don't have to worry about the kernel-source package
from changing under you since the kernel-patch-debian package will always
give you the same tree until you change it.

> our kernel-patch packages? On the other hand it seems if I create the diff
> the right way, it is becoming pretty small and you might be able to merge it
> completely into your source, so you still want that?

What I was saying there is that any elements in your diff that is completely
within arch/m68k or include/asm-m68k can be sent to me and I will merge it.

> Another thing that is unclear to me is the naming of the packages. I was
> suprised to see so many different 2.4.20 packages exist for i386, and I
> still have problems figuring out the differences between them, and the
> related kernel-headers packages. Are there some rules for this? I don't care

OK, let me give an explanation for the nomenclature.  Firstly there are
the kernel-image packgaes.  They are named as

kernel-image---

The modules ABI is changed whenever another release is made within the
same upstream version that is incompatible with previous modules.

There is one generic kernel-headers package bulting using Linus's default
config file.  For each kernel-image package, there is also a corresponding
kernel-headers package with the same naming scheme as the kernel-image.
This can be used to build modules for that particular kernel-image package.

There is also a generic kernel-build package which depends on all
kernel-headers packages.  This is meant to be used module builders.

Finally there are the kernel-image meta-packages that always depend on
the latest kernel-image packages.  They're called

kernel-image-2.4-

I would recommend that arch-maintainers adopt the modules ABI if at
all possible as otherwise whenever an incompatible package of the
same upstream version is installed, a reboot usually has to be done
immediately.
-- 
Debian GNU/Linux 3.0 is out! ( http://www.debian.org/ )
Email:  Herbert Xu ~{PmV>HI~} <[EMAIL PROTECTED]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Maintaining kernel source in sarge

[Christian T. Steigies]

On Sun, May 25, 2003 at 08:12:00AM +1000, Herbert Xu wrote:
> Christoph Hellwig <[EMAIL PROTECTED]> wrote:
> > 
> > Why can't Debian have just one tree for multiple architectures like
> > SuSE and RedHat (sometimes) do.  Okay suse supports 'only' i386,
> > x86_64,ppc,ppc64,s390,s390x,ia64 but their kernel also has patches
> > for sparc,sparc64,mips and m68k although I can't guarantee that these
> > really work in the relased tree (but last time I visted their office
> > people were playing with those ports in their spare time).
> 
> I don't think we can go all the way yet, but let's make a start.  If
> the architecture maintainers send me patches which clearly don't affect
> other archs or otherwise cause build problems, I will merge them.

I found this thread only yesterday, it would be nice if Joey could summarize
again, I don't read d-d very often.

I think this is contracticting to what you said just a few messages earlier.
For 2.4.21 will there be a pristine kernel-source and seperate i386-patch or
not? It seems I am the m68k maintainer, and I am for the seperate patch
solution. Reading this thread, I think the m68k kernel-patch has to be
improved, but if you leave the kernel-source as it is now, this means I have
to diff linux-m68k against linux-i386 to generate my m68k diff. Then I have
to create the i386 diff from your kernel-source package (IIRC it does not
ship as a seperate patch, it is included when I unpack kernel-source), then
put those two diffs in the m68k kernel-patch package and hope they both
apply. I think creating the i386 diff, or the security-patches diff is your
job, not mine or of the maintainers of the other 10 arches. Especially since
you know much better which patches are security related, which are i386
specific, and which fix some stupid typo which you can not stand to read
anymore, but might break compatibility with all the other arches. Has this
been decided yet? Maybe even if you continue to pack everything into the
kernel-source package, you could create this security diff with every new
version just for us port maintainers so it will be easier for us to update
our kernel-patch packages? On the other hand it seems if I create the diff
the right way, it is becoming pretty small and you might be able to merge it
completely into your source, so you still want that?

Another thing that is unclear to me is the naming of the packages. I was
suprised to see so many different 2.4.20 packages exist for i386, and I
still have problems figuring out the differences between them, and the
related kernel-headers packages. Are there some rules for this? I don't care
much about the i386 package names (except for which to use on my own box),
and I think nobody minds the kernel-image-2.4.20-[amiga|atari|mac|*vme*]
names, but there seems to be some discrepancy about the kernel-headers. m68k
used to have: kernel-headers-2.4.9_2.4.9-2_m68k.deb (which BTW is obsolete,
together with all 2.4.x < 2.4.20). I changed that for the latest version to:
kernel-headers-2.4.20-m68k_2.4.20-1_m68k.deb since I did not want to
conflict with other kernel-headers which might exist for m68k. Is this
good/bad/doesn't matter and what is the recommended naming? It seems every
maintainer does it as she likes (some random packages):

arch-in-name_all.deb:
kernel-headers-2.4.20-sparc_26.potato.1_all.deb

arch-in-name_arch.deb:
kernel-headers-2.4.18-686_2.4.18-5_i386.deb

something-in-name_arch.deb:
kernel-headers-2.4.20-1-wildfire_2.4.20-6_alpha.deb

nothing-in-name_arch.deb:
kernel-headers-2.4.18_2.4.18-5_i386.deb

something-and-arch-in-name_arch.deb:
kernel-headers-2.4.20-1-686_2.4.20-7_i386.deb

Same for kernel-patch, but most of them are arch-in-name_all.deb.
Any guidelines on this or are we free to do as we like?

Fortunately there are no kernel-modules for m68k (that I know of), so I am
ignoring the major part of this thread.

Christian

Re: Maintaining kernel source in sarge

[Anthony Towns]

On Sun, May 25, 2003 at 08:26:08PM +1000, Anthony Towns wrote:
> As far as I could tell, we don't have any other examples at present,
> than alsa and pcmcia.

Since no one seems to have replied correcting me, there's also apparently
linux-wlan-ng-modules-*, i2c-*, and lm-sensors-*.

Cheers,
aj

-- 
Anthony Towns <[EMAIL PROTECTED]> 
I don't speak for anyone save myself. GPG signed mail preferred.

  ``Dear Anthony Towns: [...] Congratulations -- 
you are now certified as a Red Hat Certified Engineer!''

Re: Maintaining kernel source in sarge

[Josselin Mouette]

I'm answering a bit late, but after rethinking, I don't see what's that
wrong with that approach.

Le dim 25/05/2003 à 11:33, Herbert Xu a écrit :
> That is an advantage.  However, it also means that any update to the
> modules source package cannot be built until another entire
> kernel-image set is built.

True. But currently, there are only 4 modules packages in sid, and apart
from ALSA (which is stabilizing), they don't need frequent updates.

> But what really makes it impossible for me is that if there is a build
> problem in one of the modules, then the entire kernel-image has to be
> delayed or the module dropped.  If the module build problem is then
> fixed, the entire kernel-image has to be rebuilt again.
> 
> So IMHO, the cost outweighs the benefit for now.

I strongly disagree here. Waiting the new kernel image for a week until
a build problem is solved is nothing compared to the time between 2
kernel releases. And it is a worthwile loss if we don't have anymore
issues with modules being out of sync.

> In the long term, we should have as few binary module packages as
> possible.  They should either be integrated into our kernel-source
> if it is popular enough or made source-only so that the people who
> really need them can build them privately.  I would see alsa in the
> former category (it is already integrated into 2.5) and pcmcia-cs in
> the latter (the built-in pcmcia works for most people).

I wholeheartedly agree here. And that's exactly why all modules we
provide should be provided as a patch for the kernel packages, one way
or another.

This would also be a good generic approach to separate some modules not
needed everywhere, such as multimedia, usb, leading to smaller kernel
packages.
-- 
 .''`.   Josselin Mouette/\./\
: :' :   [EMAIL PROTECTED]
`. `'[EMAIL PROTECTED]
  `-  Debian GNU/Linux -- The power of freedom


signature.asc
Description: PGP signature

Re: Maintaining kernel source in sarge

[Gunnar Wolf]

Yann Dirson dijo [Tue, May 27, 2003 at 10:38:54AM +0200]:
> That's more or less what I'd think of as well.  We can start with an
> empty security patch, and have this one grow as needed.  This way, apt
> will show people they have an outdated security patch - which, BTW,
> may be more of an incentive to upgrade than just an outdated
> kernel-source package.
> 
> That does not mean the user will rebuild his kernel at once with the
> new patch, but well, I don't think we can do much more here :)

I like the idea... And maybe we should alert the user that he needs to
rebuild the kernel using a debconf dialog or by mailing him...

-- 
Gunnar Wolf - [EMAIL PROTECTED] - (+52-55)5630-9700 ext. 1366
PGP key 1024D/8BB527AF 2001-10-23
Fingerprint: 0C79 D2D1 2C4E 9CE4 5973  F800 D80E F35A 8BB5 27AF

Re: Maintaining kernel source in sarge

[Yann Dirson]

Herbert wrote:
> Yes that isn't easy to check apart from the fact that if there isn't
> an arch update after a security update to kernel-source, then that arch
> is probably vulnerable.  If you've got an idea on how this can improved,
> please let us know.

A possibility would be to define a versionning scheme on the arch-dep
kernel-patches.  You issue version 2.4.20-8, and others release a new
version as 2.4.20-8, and then possibly further -8.1 and such, and
don't bump to -9 before you do.

That steps into the NMU numbering-space, but I suppose we can expect
NMUs on arch kernel-patches to be quite rare anyway.  And we could
give a NMU numbering-space with 2 dots, by making the 1st revision of
an arch patch to be -8.0 at first.

Does it make sense ?
-- 
Yann Dirson<[EMAIL PROTECTED]> |Why make M$-Bill richer & richer ?
Debian-related: <[EMAIL PROTECTED]> |   Support Debian GNU/Linux:
Pro:<[EMAIL PROTECTED]> |  Freedom, Power, Stability, Gratuity
 http://ydirson.free.fr/| Check 

Re: Maintaining kernel source in sarge

[Yann Dirson]

Arnd wrote:
> > Let's look at your example:
> > | Patch-name: Debian base patch
> > | Patch-id: debian
> > | Architecture: all
> > | Kernel-version: 2.4.20
> > | Depends: ptrace, isdnbonding, binfmtmisc, ethernetpadding, ...
> > |
> > | Patch-name: Pre-patch 2.4.21-pre7
> > | Patch-id: patch-2.4.21-pre7
> > | Architecture: all
> > | Kernel-version: 2.4.20
> > | Provides: ptrace, ethernetpadding
> >
> > Here I suppose the pre-patch is supposed to be applied first, and then
> > the application of the debian patch would only trigger application of
> > those dependant patches not provided by the pre-patch.
[...]
> In my example, if the isdnbonding patch has to be applied after
> ethernetpadding, it would have to say so in its description.
> patch-2.4.21-pre7 can contain the same patch and should
> consequently be applied at the same time (e.g. after
> binfmtmisc, but before isdnbonding).

OK, let's assume a moment that isdnbonding declares a dependency on
ethernetpadding:

Let's apply the debian patch first, since that's what you suggested
later.  This patch application will trigger, among others, the
(pre-)application of isdnbonding, which itself will trigger the
(pre-)application of ethernetpadding.

Now the "Provides" logic will have to cause the application of
patch-2.4.21-pre7 instead.  That seems to settle reasonably.

But now suppose that a new version of the ptrace patch is issued.
Either we just rely on the current version of the debian patch, and
the new ptrace one won't even be considered by the mechanism here
described, if you ask for the application of patch-2.4.21-pre7.  Or we
have to issue a new revision of the debian patch, either depending on
a new ptrace2 patch, declaring a conflict with old ptrace (in which
case we have to handle conflicts as well), or with a versionned
depends (which has to be implemented as well) on the new version of
ptrace.

Basically, this means bring a full dpkg/apt-like dependency model into
patch application.  This overall sounds to my ears a bit like
overkill...


> The debian patch set should generally come first, except for the parts
> in it that depend on patches 'provided' by some other patch. The maintainer
> of patch-2.4.21-pre7 would have to make sure it applies on top of or mixed
> with the debian patch set.

I don't know what you mean with "mixed with".  It is already some work
when we have to provide, in addition to the upstream version, a
version of the patch(es) that applies to the (current) debian kernel
(some people may have noted that dh-kpatches supports declaring such
patches since recently).  I hope you're not suggesting all mixes of
sub-patches of the debian patch should be supported - this idea would
have no chance to survive long :)

Regards,
-- 
Yann Dirson<[EMAIL PROTECTED]> |Why make M$-Bill richer & richer ?
Debian-related: <[EMAIL PROTECTED]> |   Support Debian GNU/Linux:
Pro:<[EMAIL PROTECTED]> |  Freedom, Power, Stability, Gratuity
 http://ydirson.free.fr/| Check 

Re: Maintaining kernel source in sarge

[Yann Dirson]

Sven wrote:
> Why don't we use a scheme similar to what xfree86 use for its patches.
> Sure we would need to adapt it as the patches are distributed, but we
> could well do it. 

As I understand it, the xfree86 package uses (some derivative of) dbs,
in which the package maintainer has to order the patches by hand,
instead of declaring explicit dependencies (much like sysvinit and
others do, and like the patch-ordering facility in make-kpkg).

If the above is correct, I'd see that as a step backwards.

Regards,
-- 
Yann Dirson<[EMAIL PROTECTED]> |Why make M$-Bill richer & richer ?
Debian-related: <[EMAIL PROTECTED]> |   Support Debian GNU/Linux:
Pro:<[EMAIL PROTECTED]> |  Freedom, Power, Stability, Gratuity
 http://ydirson.free.fr/| Check 

Re: Maintaining kernel source in sarge

[Sven Luther]

On Tue, May 27, 2003 at 08:16:59PM +1000, Russell Coker wrote:
> On Tue, 27 May 2003 19:04, Arnd Bergmann wrote:
> > > Here I suppose the pre-patch is supposed to be applied first, and then
> > > the application of the debian patch would only trigger application of
> > > those dependant patches not provided by the pre-patch.
> >
> > The order in which the patches are applied should in general not be
> > significant. If it is, it should be stated in the patch description. I
> > assumed that the 'Depends' tag is semantically more a 'Pre-Depends',
> > right?
> 
> It's fairly rare for a patch to depend on another patch without any order 
> dependency.

Why don't we use a scheme similar to what xfree86 use for its patches.
Sure we would need to adapt it as the patches are distributed, but we
could well do it. 

Friendly,

Sven Luther

Re: Maintaining kernel source in sarge

[Russell Coker]

On Tue, 27 May 2003 19:04, Arnd Bergmann wrote:
> > Here I suppose the pre-patch is supposed to be applied first, and then
> > the application of the debian patch would only trigger application of
> > those dependant patches not provided by the pre-patch.
>
> The order in which the patches are applied should in general not be
> significant. If it is, it should be stated in the patch description. I
> assumed that the 'Depends' tag is semantically more a 'Pre-Depends',
> right?

It's fairly rare for a patch to depend on another patch without any order 
dependency.

When a patch depends (in the English language not the Debian package meaning) 
on another patch it usually changes files that were created or changed by the 
other patch.

Patches tend to be independant (IE you can use either of them on it's own or 
apply them in any order) or there is a dependency which also requires an 
ordering of the application.

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

Re: Maintaining kernel source in sarge

[Arnd Bergmann]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Tuesday 27 May 2003 10:27, Yann Dirson wrote:

> Let's look at your example:
> | Patch-name: Debian base patch
> | Patch-id: debian
> | Architecture: all
> | Kernel-version: 2.4.20
> | Depends: ptrace, isdnbonding, binfmtmisc, ethernetpadding, ...
> |
> | Patch-name: Pre-patch 2.4.21-pre7
> | Patch-id: patch-2.4.21-pre7
> | Architecture: all
> | Kernel-version: 2.4.20
> | Provides: ptrace, ethernetpadding
>
> Here I suppose the pre-patch is supposed to be applied first, and then
> the application of the debian patch would only trigger application of
> those dependant patches not provided by the pre-patch.

The order in which the patches are applied should in general not be
significant. If it is, it should be stated in the patch description. I
assumed that the 'Depends' tag is semantically more a 'Pre-Depends',
right?

In my example, if the isdnbonding patch has to be applied after
ethernetpadding, it would have to say so in its description.
patch-2.4.21-pre7 can contain the same patch and should
consequently be applied at the same time (e.g. after
binfmtmisc, but before isdnbonding).

> That's only fine _if_ the replaced patches are similar enough so that
> any patch in the debian set, that would depend on those, can still
> apply atop the new one.  That is, if there are several revisions of a
> given fix, we'll have to use versionned Provides: and Depends:, or
> we're doomed.  Not that it's impossible, but I'm not sure it's exactly
> trivial to implement...

Yes, versioned patches are a problem that I did not think of. I
assumed that one patch providing the functionality of another
one is always a superset. It may be possible to require versioned
patches to be made as a sequence of diffs between the versions.
Of course that creates new problems.

> | Patch-name: AMD64 CVS snapshot
> | Patch-id: amd64-20030417
> | Architecture: i386, amd64
> | Kernel-version: 2.4.20
> | Depends: debian, patch-2.4.21-pre7, simicsfs
> | Provides: aic7xxx
>
> Here I suppose you should have swapped debian and patch-2.4.21-pre7,
> or that simply wouldn't apply.

No. The debian patch set should generally come first, except for the parts
in it that depend on patches 'provided' by some other patch. The maintainer
of patch-2.4.21-pre7 would have to make sure it applies on top of or mixed
with the debian patch set.

Arnd <><
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+0yoI5t5GS2LDRf4RAki3AJ9s5cnZLy3daFOzJ1tVrJLd4vOzlQCfc0t/
+v6dXGmliMSueQWE5Hxw1xI=
=Fkob
-END PGP SIGNATURE-

Re: Maintaining kernel source in sarge

[Herbert Xu]

Matt Zimmerman <[EMAIL PROTECTED]> wrote:
> 
> All definite benefits.  The one thing which seems to be missing is to ensure
> that the arch-specific kernels do not miss out on important fixes (such as
> security) to the main kernel source tree.

Yes that isn't easy to check apart from the fact that if there isn't
an arch update after a security update to kernel-source, then that arch
is probably vulnerable.  If you've got an idea on how this can improved,
please let us know.
-- 
Debian GNU/Linux 3.0 is out! ( http://www.debian.org/ )
Email:  Herbert Xu ~{PmV>HI~} <[EMAIL PROTECTED]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Maintaining kernel source in sarge

[Yann Dirson]

On Tue, May 27, 2003 at 07:37:42AM +0200, Sven Luther wrote:
> On Tue, May 27, 2003 at 07:23:27AM +1000, Herbert Xu wrote:
> > On Mon, May 26, 2003 at 10:00:06PM +0200, Yann Dirson wrote:
> > > 
> > > We could get around Guido's point mentionned above by having a list of
> > > default patches to apply, which would by default contain the debian
> > > patch.
> > 
> > Yes, but then the problem is that unsuspecting users could be
> > building kernels using the kernel-source package thinking that
> > it contained all the security fixes.
> 
> Have it depend on a kernel-source-security-fixes or something
> such ?

That's more or less what I'd think of as well.  We can start with an
empty security patch, and have this one grow as needed.  This way, apt
will show people they have an outdated security patch - which, BTW,
may be more of an incentive to upgrade than just an outdated
kernel-source package.

That does not mean the user will rebuild his kernel at once with the
new patch, but well, I don't think we can do much more here :)


> And have make-kpkg issue a big warning if it detects that the
> sources were not patched ?

That could be easy to do.  Just have the security patch create a
debian/APPLIED_security stamp, and have make-kpkg look at that...

Regards,
-- 
Yann Dirson<[EMAIL PROTECTED]> |Why make M$-Bill richer & richer ?
Debian-related: <[EMAIL PROTECTED]> |   Support Debian GNU/Linux:
Pro:<[EMAIL PROTECTED]> |  Freedom, Power, Stability, Gratuity
 http://ydirson.free.fr/| Check 

Re: Maintaining kernel source in sarge

[Yann Dirson]

Arnd wrote:
> Actually, I was thinking of a different concept with a 'Replaces: tag,

Hm.  As I understand it, it would be more something like a "Provides:"
declaration, it seems.  Such a feature does not seem useless to me at
first glance (we already see aglomerations of patches, like FOLK,
which coule have make use of this), but I'm not sure it would be
something we can work with.

Let's look at your example:

| Patch-name: Debian base patch
| Patch-id: debian
| Architecture: all
| Kernel-version: 2.4.20
| Depends: ptrace, isdnbonding, binfmtmisc, ethernetpadding, ...
|
| Patch-name: Pre-patch 2.4.21-pre7
| Patch-id: patch-2.4.21-pre7
| Architecture: all
| Kernel-version: 2.4.20
| Provides: ptrace, ethernetpadding

Here I suppose the pre-patch is supposed to be applied first, and then
the application of the debian patch would only trigger application of
those dependant patches not provided by the pre-patch.

That's only fine _if_ the replaced patches are similar enough so that
any patch in the debian set, that would depend on those, can still
apply atop the new one.  That is, if there are several revisions of a
given fix, we'll have to use versionned Provides: and Depends:, or
we're doomed.  Not that it's impossible, but I'm not sure it's exactly
trivial to implement...


| Patch-name: AMD64 CVS snapshot
| Patch-id: amd64-20030417
| Architecture: i386, amd64
| Kernel-version: 2.4.20
| Depends: debian, patch-2.4.21-pre7, simicsfs
| Provides: aic7xxx

Here I suppose you should have swapped debian and patch-2.4.21-pre7,
or that simply wouldn't apply.


> Do you think that make-kpkg and dh-kpatches could/should be merged,
> making the dh-kpatches information evaluated by make-kpkg if available, 
> or do we need changes beyond that?

They cannot be completely merged, since they work on complementary
stages of the kernel build.  OTOH, gradually the kernel-patch-scripts
package is gradually going to factorise things that currently get
duplicated in all kernel-patch packages, and since this part of
dh-kpatches will work in the same stage as make-kpkg does, it may make
sense at some point to have a look at something like a merge of this
part into make-kpkg.

But since this is all about things yet to be written, we'll see later.

Regards,
-- 
Yann Dirson<[EMAIL PROTECTED]> |Why make M$-Bill richer & richer ?
Debian-related: <[EMAIL PROTECTED]> |   Support Debian GNU/Linux:
Pro:<[EMAIL PROTECTED]> |  Freedom, Power, Stability, Gratuity
 http://ydirson.free.fr/| Check 

Re: Maintaining kernel source in sarge

[Sven Luther]

On Tue, May 27, 2003 at 07:23:27AM +1000, Herbert Xu wrote:
> On Mon, May 26, 2003 at 10:00:06PM +0200, Yann Dirson wrote:
> > 
> > We could get around Guido's point mentionned above by having a list of
> > default patches to apply, which would by default contain the debian
> > patch.
> 
> Yes, but then the problem is that unsuspecting users could be
> building kernels using the kernel-source package thinking that
> it contained all the security fixes.

Have it depend on a kernel-source-security-fixes or something
such ? And have make-kpkg issue a big warning if it detects that the
sources were not patched ?
> 
> I believe that distributing a binary package that may contain
> known security problems is a very serious problem.
> -- 
> Debian GNU/Linux 3.0 is out! ( http://www.debian.org/ )
> Email:  Herbert Xu ~{PmV>HI~} <[EMAIL PROTECTED]>
> Home Page: http://gondor.apana.org.au/~herbert/
> PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
> 
> 
> -- 
> To UNSUBSCRIBE, email to [EMAIL PROTECTED]
> with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: Maintaining kernel source in sarge

[Herbert Xu]

On Mon, May 26, 2003 at 10:00:06PM +0200, Yann Dirson wrote:
> 
> We could get around Guido's point mentionned above by having a list of
> default patches to apply, which would by default contain the debian
> patch.

Yes, but then the problem is that unsuspecting users could be
building kernels using the kernel-source package thinking that
it contained all the security fixes.

I believe that distributing a binary package that may contain
known security problems is a very serious problem.
-- 
Debian GNU/Linux 3.0 is out! ( http://www.debian.org/ )
Email:  Herbert Xu ~{PmV>HI~} <[EMAIL PROTECTED]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Maintaining kernel source in sarge

[Herbert Xu]

Yann Dirson <[EMAIL PROTECTED]> wrote:
> 
> Since recently there are also kernel-build packages, which appear to
> be made precisely for that.  I take it to mean the kernel-headers
> packages have proven deficient in some way, but I probably missed many
> things - I am even under the impression their construction is not even
> done by make-kpkg itself.
> 
> Could someone elaborate on that ? (Herbert ?  Manoj ?)

The kernel-build package is simply an aggregation of all flavoured
kernel-headers packages so that module builders can have a single
package to depend on.
-- 
Debian GNU/Linux 3.0 is out! ( http://www.debian.org/ )
Email:  Herbert Xu ~{PmV>HI~} <[EMAIL PROTECTED]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Maintaining kernel source in sarge

[Arnd Bergmann]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Monday 26 May 2003 22:20, Yann Dirson wrote:

> If you mean, whether it can handle something like "Architecture:
> !ia64, !hppa", well, not yet, although it could be done.  But that
> would mean stopping the use of make-kpkg-level architecture support,
> just like it does not use make-kpkg-level kernel-version support.

Actually, I was thinking of a different concept with a 'Replaces: tag,
something like:

| Patch-name: Debian base patch
| Patch-id: debian
| Architecture: all
| Kernel-version: 2.4.20
| Depends: ptrace, isdnbonding, binfmtmisc, ethernetpadding, ...

| Patch-name: Pre-patch 2.4.21-pre7
| Patch-id: patch-2.4.21-pre7
| Architecture: all
| Kernel-version: 2.4.20
| Replaces: ptrace, ethernetpadding

| Patch-name: AMD64 CVS snapshot
| Patch-id: amd64-20030417
| Architecture: i386, amd64
| Kernel-version: 2.4.20
| Depends: debian, patch-2.4.21-pre7, simicsfs
| Replaces: aic7xxx

> Although that may not look like a big deal, that seems to show that at
> some time a redesign of the interface between make-kpkg and the
> patches themselves would be a good idea.
Yes, in my amd64 kernel-patch package, the first thing I did was introducing
an additional layer to make that interface more flexible. I'll probably
change it to use dh-kpatches, but I wasn't aware of it when I made the
package.

Do you think that make-kpkg and dh-kpatches could/should be merged,
making the dh-kpatches information evaluated by make-kpkg if available, 
or do we need changes beyond that?

Arnd <><
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+0oLg5t5GS2LDRf4RAk9sAJ9mXkdo1Ecktj5vjtN+0xsjY8LXTgCdExGU
kwPFctma5TMf2Qv4anlB854=
=uGo9
-END PGP SIGNATURE-

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Sun, May 25, 2003 at 11:00:34AM +1000, Herbert Xu wrote:

> But the pristine kernel source and the Debian patch are already available
> to the architecture maintainers:
> 
> apt-get --tar-only source kernel-source-2.4.xx
> apt-get --diff-only source kernel-source-2.4.xx
> 
> So I don't think having a kernel-patch-debian by itself is of any value
> to the merging process.  It also doesn't solve the problem that a new
> kernel-patch-debian may break the building of old kernel-image packages.

The diff for kernel-source-xx contains more than just the Debian-specific
changes to the source; for example, it contains the debian/ directory.
Also, source packages are much less straightforward to work with than binary
packages.  For example, they cannot be used in dependency relationships.

If the current system works fine for the kernel package maintainers, it is,
of course, not my place to say how it should be done.  I'll leave this to
the people doing the work.

> There is also the suggestion of a complete break down of all Debian
> changes should be made available.  Unfortunately that is simply
> unmaintainable and unusable as the number of changes grows as the
> dependencies between the patches are complex.

A breakdown meaning separate patches, right?  Some maintainers manage to do
this, but I agree that it is definitely more work than maintaining a unified
tree.

> I think perhaps we could find a middle ground.  We can keep kernel-source
> as it is with all the patches applied.  In addition to that, we can have a
> kernel-patch-debian package which depends on the kernel-source of the same
> upstream version containing patches that will take the kernel-source to
> the following source trees:
>[...]
> This approach has the following benefits:
> 
> * The kernel-source binary contains all bug fixes as is.  Guido raised a
> good point that if we separated the patches from the kernel-source, then
> users may miss out on the bug fixes.  This is especially important in
> light of the current speed of upstream releases.

I agree; simply untarring /usr/src/kernel-source... should give most users
what they expect.

> * The pristine kernel-source is available in the binary archive.  Many
> people have asked for this in the past and this makes it available in the
> form of a source tar ball plus a patch.
>
> * Per-arch kernel-image will no longer be made unbuildable by new
> kernel-source updates.  This has always been a problem for those
> architectures using the main kernel-source archive.
>
> * The incremental patches should ease the merging process of those
> architectures that wish to incoporate Debian changes.
>
> * The incremental patches should also allow people to get only the updated
> kernel-patch packages if they wish.

All definite benefits.  The one thing which seems to be missing is to ensure
that the arch-specific kernels do not miss out on important fixes (such as
security) to the main kernel source tree.

-- 
 - mdz

Re: Maintaining kernel source in sarge

[Yann Dirson]

Matt wrote:
> The part you seem to have missed is the distinction between a source package
> and a binary package in what I wrote above.

Not completely, although the time between reading and answering did
not help me to be 100% clear with this :)

> I do not think this is a practical idea to work toward at this time,
> however.

Yes, a new generation of source packages is needed, but if we could
settle the kernel-packaging issue without this, that would at least
help to get it running for sarge...

Regards,
-- 
Yann Dirson<[EMAIL PROTECTED]> |Why make M$-Bill richer & richer ?
Debian-related: <[EMAIL PROTECTED]> |   Support Debian GNU/Linux:
Pro:<[EMAIL PROTECTED]> |  Freedom, Power, Stability, Gratuity
 http://ydirson.free.fr/| Check 

Re: Maintaining kernel source in sarge

[Yann Dirson]

Arnd wrote:
> Ok, but I still would love to see single patches instead of one big
> patch containing all the common stuff. You can't really avoid
> situations where you want a patch on all architectures except one or
> two. This may be either because a patch breaks on one architecture
> (which should be of course be fixed, but you might not want to 
> rebuild all kernels and modules on all other architectures because of
> it), or because the same fix is contained both in the debian collection
> and in the patch set published by the upstream arch kernel maintainer.
> 
> I'm not sure if dh-kpatches already solves this problem, but it should
> be possible to add.

If you mean, whether it can handle something like "Architecture:
!ia64, !hppa", well, not yet, although it could be done.  But that
would mean stopping the use of make-kpkg-level architecture support,
just like it does not use make-kpkg-level kernel-version support.
Although that may not look like a big deal, that seems to show that at
some time a redesign of the interface between make-kpkg and the
patches themselves would be a good idea.

Regards,
-- 
Yann Dirson<[EMAIL PROTECTED]> |Why make M$-Bill richer & richer ?
Debian-related: <[EMAIL PROTECTED]> |   Support Debian GNU/Linux:
Pro:<[EMAIL PROTECTED]> |  Freedom, Power, Stability, Gratuity
 http://ydirson.free.fr/| Check 

Re: Maintaining kernel source in sarge

[Yann Dirson]

Herbert wrote:
> * The kernel-source binary contains all bug fixes as is.  Guido raised
> a good point that if we separated the patches from the kernel-source, then
> users may miss out on the bug fixes.  This is especially important in light
> of the current speed of upstream releases.
> 
> * The pristine kernel-source is available in the binary archive.  Many people
> have asked for this in the past and this makes it available in the form of
> a source tar ball plus a patch.

On the other hand, I would find it somewhat non-intuitive that the
pristine source would only be available as the result of the
application of a patch - and I tend to see that as an indication that
it'd be more logical the other way round, with a pristine tarball and
a real diff.

We could get around Guido's point mentionned above by having a list of
default patches to apply, which would by default contain the debian
patch.


Additionally, with things done this way, we're not even forced to
update the whole kernel-source-2.4.20 seven times or more, we just
have to update kernel-source-debian-2.4.20, which I believe would be
good for the health of our network pipes.

Regards,
-- 
Yann Dirson<[EMAIL PROTECTED]> |Why make M$-Bill richer & richer ?
Debian-related: <[EMAIL PROTECTED]> |   Support Debian GNU/Linux:
Pro:<[EMAIL PROTECTED]> |  Freedom, Power, Stability, Gratuity
 http://ydirson.free.fr/| Check 

Re: Maintaining kernel source in sarge

[Yann Dirson]

Matt wrote:
> It would be a significant gain if kernel modules could always be built
> against kernel-headers, without requiring full kernel-source.

Since recently there are also kernel-build packages, which appear to
be made precisely for that.  I take it to mean the kernel-headers
packages have proven deficient in some way, but I probably missed many
things - I am even under the impression their construction is not even
done by make-kpkg itself.

Could someone elaborate on that ? (Herbert ?  Manoj ?)

Regards,
-- 
Yann Dirson<[EMAIL PROTECTED]> |Why make M$-Bill richer & richer ?
Debian-related: <[EMAIL PROTECTED]> |   Support Debian GNU/Linux:
Pro:<[EMAIL PROTECTED]> |  Freedom, Power, Stability, Gratuity
 http://ydirson.free.fr/| Check 

Re: Maintaining kernel source in sarge

[Yann Dirson]

Matt wrote:
> The ideal solution would be to be able to share tarballs between source
> packages.  Then, all of the kernel-image packages could be built as if they
> had a complete kernel source tree as their source package (which simplifies
> things a lot), and yet we would only need one such tarball in the archive.
> Of course, I have little idea how much work this would be to implement,
> except that it would touch a lot of different tools.

The way I understand it was intended to work would be to have one
single kernel-source-X-Y-Z package somewhat like what we have today,
and having the various kernel-image-whatever build-depend on this
kernel-source and any necessary kernel-patch packages.

If this understanding is correct, I admit I don't see why the practice
has diverged from this idea.


-- 
Yann Dirson<[EMAIL PROTECTED]> |Why make M$-Bill richer & richer ?
Debian-related: <[EMAIL PROTECTED]> |   Support Debian GNU/Linux:
Pro:<[EMAIL PROTECTED]> |  Freedom, Power, Stability, Gratuity
 http://ydirson.free.fr/| Check 

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Mon, May 26, 2003 at 09:25:42PM +0200, Yann Dirson wrote:

> Matt wrote:
> > The ideal solution would be to be able to share tarballs between source
> > packages.  Then, all of the kernel-image packages could be built as if
> > they had a complete kernel source tree as their source package (which
> > simplifies things a lot), and yet we would only need one such tarball in
> > the archive.  Of course, I have little idea how much work this would be
> > to implement, except that it would touch a lot of different tools.
> 
> The way I understand it was intended to work would be to have one single
> kernel-source-X-Y-Z package somewhat like what we have today, and having
> the various kernel-image-whatever build-depend on this kernel-source and
> any necessary kernel-patch packages.
> 
> If this understanding is correct, I admit I don't see why the practice has
> diverged from this idea.

The part you seem to have missed is the distinction between a source package
and a binary package in what I wrote above.

I do not think this is a practical idea to work toward at this time,
however.

-- 
 - mdz

Re: Maintaining kernel source in sarge

[Christoph Hellwig]

On Sun, May 25, 2003 at 02:34:50PM +0200, Arnd Bergmann wrote:
> > > When building kernel-image-s390, make-kpkg would first apply
> > > the arch specific patches and the the arch independent ones that
> > > have not been superceded by an arch specific one.
> >
> > Again that's a very bad idea.  Arch-independent patches should
> > _always_ be applied before Arch-dependent ones, otherwiese applying
> > MI patches will get absolute horror.
> 
> Ok, but I still would love to see single patches instead of one big
> patch containing all the common stuff.

The one big patch is very bad, I agree.

> You can't really avoid
> situations where you want a patch on all architectures except one or
> two. This may be either because a patch breaks on one architecture
> (which should be of course be fixed, but you might not want to 
> rebuild all kernels and modules on all other architectures because of
> it),

Right.

> or because the same fix is contained both in the debian collection
> and in the patch set published by the upstream arch kernel maintainer.

No, for that case it shouldn't.  Fix the upstream maintainers patch instead.

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Sun, May 25, 2003 at 04:09:51PM +1000, Russell Coker wrote:

> On Sun, 25 May 2003 15:11, Matt Zimmerman wrote:
> > This approach does not scale.  I cannot personally review the diffs for
> > every upstream release of all the software in Debian, nor can any other
> > individual or even a small group.
> 
> It does not scale to all software in Debian.  But most software does not
> need much in the way of security auditing.

Any software that a user interacts with is trusted to some extent, and could
contain significant vulnerabilities.  We rely on external sources for this
information, such as upstream authors, individual security researchers and
organizations.

> A small group of people could review all kernel patches that make it into
> the official tree.  Of course getting even a small group of people who
> have the skill to do such work properly and the time to do it continually
> may not be easy.

Indeed, it would be even more difficult to find people who would be willing
to waste time searching for things that others discarded, apparently knowing
that they were important.  I certainly don't want that job.

-- 
 - mdz

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Sun, May 25, 2003 at 09:23:44AM +0200, Christoph Hellwig wrote:

> On Sun, May 25, 2003 at 01:11:44AM -0400, Matt Zimmerman wrote:
> > > Then read through the prepatch diffs, everything adding checks to
> > > ioctl methods or similar is likely one them.  
> > 
> > This approach does not scale.
> 
> Right, you got it.  Similarly it doesn't scale to announce all these bits.
> Just take the latest upstream if you want these kinds of fixes.

No, that is not similar.  All those bits are changed by many different
people, not one, and even if they weren't, it is easier (by a HUGE measure)
for the person who has made the change to announce it to others, as they are
already aware and do not need to sift through a single diff, much less the
entire kernel tree as you suggested that I do.

This is analogous to someone dropping a particular leaf into a huge pile of
leaves, and suggesting that it makes more sense for me to search the pile
than for them to tell me where they dropped it (or show it to me in the
first place).

This is not a question of what I want.  Either our users need these fixes to
maintain the security of their systems (in which case Debian needs these
fixes, and they are important enough to be announced publicly), or they do
not, and they are not worth talking about.

> This is how every bigger upstream (and other projects like OpenBSD) work.

Apache?  XFree86?  KDE?  Mozilla?  OpenWall?  glibc?  All of these projects
manage to enumerate security fixes.

OpenBSD fails miserably in this respect, and makes for an example of how NOT
to work with the community on security issues.  Their approach is, roughly,
"we fixed this a while ago but didn't tell anyone, so you're vulnerable and
we're not, ha-ha-ha".

-- 
 - mdz

Re: Maintaining kernel source in sarge

[Arnd Bergmann]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Sunday 25 May 2003 06:19, Christoph Hellwig wrote:
> On Sun, May 25, 2003 at 01:51:05AM +0200, Arnd Bergmann wrote:

> > As a real-world example, kernel-patch-s390 can provide
> > the ptrace bug fix from Martin Schwidefsky, while
> > kernel-patch-debian contains the generic solution from Alan Cox.
>
> This sounds like a really bad idea (okay, not worse than the
> current situation but still bad).  Martin should diff his patch ontop
> of Alan, if it has other core changes he should really post it to
> lkml and explain those.  Now this unfortunately hasn't happend and
> if it has incompatible core changes (I haven't checked) that's one
> of those rare cases where you want to appy a patch condtionally.

Martins patch only has one additional change for s390 specific code
and he published both this diff alone as well as the combined patch.
The trouble was that the sid kernel-source included Alan's patch
while the woody kernel-source didn't (it does now), so now you 
could no longer use kernel-patch-s390 from one tree for the other.

> > When building kernel-image-s390, make-kpkg would first apply
> > the arch specific patches and the the arch independent ones that
> > have not been superceded by an arch specific one.
>
> Again that's a very bad idea.  Arch-independent patches should
> _always_ be applied before Arch-dependent ones, otherwiese applying
> MI patches will get absolute horror.

Ok, but I still would love to see single patches instead of one big
patch containing all the common stuff. You can't really avoid
situations where you want a patch on all architectures except one or
two. This may be either because a patch breaks on one architecture
(which should be of course be fixed, but you might not want to 
rebuild all kernels and modules on all other architectures because of
it), or because the same fix is contained both in the debian collection
and in the patch set published by the upstream arch kernel maintainer.

I'm not sure if dh-kpatches already solves this problem, but it should
be possible to add.

Arnd <><
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+0Lhs5t5GS2LDRf4RAkuFAJ4o2XdC1rVcnW/Bp0S4AMaB8ZQm4wCgpWuc
hDExBmIDeANBbt/jVWmlthw=
=j3ZW
-END PGP SIGNATURE-

Re: Maintaining kernel source in sarge

[Christoph Hellwig]

On Sun, May 25, 2003 at 08:20:39PM +1000, Russell Coker wrote:
> What is the status of the pcmcia support anyway?

Seems to work fine.  Red Hat uses inkernel pcmcia at least.

There's some pcmcia drivers not (yet?) merged in the kernel but patching
them in is rather easy.

Re: Maintaining kernel source in sarge

[Russell Coker]

On Sun, 25 May 2003 19:33, Herbert Xu wrote:
> In the long term, we should have as few binary module packages as
> possible.  They should either be integrated into our kernel-source
> if it is popular enough or made source-only so that the people who
> really need them can build them privately.  I would see alsa in the
> former category (it is already integrated into 2.5) and pcmcia-cs in
> the latter (the built-in pcmcia works for most people).

This sounds very sensible.

What is the status of the pcmcia support anyway?  What is broken in the 
in-kernel support?  Is there anything we can do to make the in-kernel code 
work better and get more people to use it?

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

Re: Maintaining kernel source in sarge

[Anthony Towns]

On Sun, May 25, 2003 at 07:33:04PM +1000, Herbert Xu wrote:
> Anthony Towns  wrote:
> > Having kernel modules associated with the kernel source package they're
> > built for makes it a bunch easier to make sure they're deleted from
> > the archive along with the corresponding kernel images, and makes sure
> > that when someone uploads a new kernel image, new module images get
> > uploaded too.
> That is an advantage.  However, it also means that any update to the
> modules source package cannot be built until another entire
> kernel-image set is built.

Is that really that big a problem?

In unstable for i386, eg, we have pcmcia-modules packages for 2.2.20
(reiser and udma100-ext3), 2.2.25 (vanilla, compact and idepci)
and 2.4.18-bf2.4. For alsa-modules we have seven flavours for
2.4.20-1. 2.4.20-3 seems entirely unsupported.

> But what really makes it impossible for me is that if there is a build
> problem in one of the modules, then the entire kernel-image has to be
> delayed or the module dropped.

Well, what we have now seems to be that the modules are all dropped
everytime a new kernel is uploaded. :-/

> In the long term, we should have as few binary module packages as
> possible.  They should either be integrated into our kernel-source
> if it is popular enough or made source-only so that the people who
> really need them can build them privately.  I would see alsa in the
> former category (it is already integrated into 2.5) and pcmcia-cs in
> the latter (the built-in pcmcia works for most people).

As far as I could tell, we don't have any other examples at present,
than alsa and pcmcia.

Cheers,
aj

-- 
Anthony Towns <[EMAIL PROTECTED]> 
I don't speak for anyone save myself. GPG signed mail preferred.

  ``Dear Anthony Towns: [...] Congratulations -- 
you are now certified as a Red Hat Certified Engineer!''


pgpKBoQT807U5.pgp
Description: PGP signature

Re: Maintaining kernel source in sarge

[Herbert Xu]

Anthony Towns  wrote:
> 
> Having kernel modules associated with the kernel source package they're
> built for makes it a bunch easier to make sure they're deleted from
> the archive along with the corresponding kernel images, and makes sure
> that when someone uploads a new kernel image, new module images get
> uploaded too.

That is an advantage.  However, it also means that any update to the
modules source package cannot be built until another entire
kernel-image set is built.

But what really makes it impossible for me is that if there is a build
problem in one of the modules, then the entire kernel-image has to be
delayed or the module dropped.  If the module build problem is then
fixed, the entire kernel-image has to be rebuilt again.

So IMHO, the cost outweighs the benefit for now.

In the long term, we should have as few binary module packages as
possible.  They should either be integrated into our kernel-source
if it is popular enough or made source-only so that the people who
really need them can build them privately.  I would see alsa in the
former category (it is already integrated into 2.5) and pcmcia-cs in
the latter (the built-in pcmcia works for most people).
-- 
Debian GNU/Linux 3.0 is out! ( http://www.debian.org/ )
Email:  Herbert Xu ~{PmV>HI~} <[EMAIL PROTECTED]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Maintaining kernel source in sarge

[Christoph Hellwig]

On Sun, May 25, 2003 at 01:11:44AM -0400, Matt Zimmerman wrote:
> > Then read through the prepatch diffs, everything adding checks to
> > ioctl methods or similar is likely one them.  
> 
> This approach does not scale.

Right, you got it.  Similarly it doesn't scale to announce all these
bits.  Just take the latest upstream if you want these kinds of fixes.

This is how every bigger upstream (and other projects like OpenBSD) work.

Re: Maintaining kernel source in sarge

[Russell Coker]

On Sun, 25 May 2003 15:11, Matt Zimmerman wrote:
> On Sun, May 25, 2003 at 06:21:00AM +0200, Christoph Hellwig wrote:
> > On Sat, May 24, 2003 at 06:32:26PM -0400, Matt Zimmerman wrote:
> > > It's not noise at all when it's something that we and others
> > > (desperately!) want to know about.
> >
> > Then read through the prepatch diffs, everything adding checks to
> > ioctl methods or similar is likely one them.
>
> This approach does not scale.  I cannot personally review the diffs for
> every upstream release of all the software in Debian, nor can any other
> individual or even a small group.

It does not scale to all software in Debian.  But most software does not need 
much in the way of security auditing.

A small group of people could review all kernel patches that make it into the 
official tree.  Of course getting even a small group of people who have the 
skill to do such work properly and the time to do it continually may not be 
easy.

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

Re: Maintaining kernel source in sarge

[Anthony Towns]

On Sat, May 24, 2003 at 09:24:16PM +1000, Herbert Xu wrote:
> OK, barring any major objections, that's how it will be for 2.4.21.

Is there any possibility of having the various foo-modules-2.4.xx packages
be built concurrently with kernel-image-2.4.xx too?

Something like:

kernel-i386_2.4.21-3_i386.dsc:
Build-Depends: linus-kernel (=2.4.21),
pcmcia-source, alsa-modules-source
Binaries: kernel-image-2.4.21, pcmcia-modules-2.4.21, 
alsa-modules-2.4.21

Having kernel modules associated with the kernel source package they're
built for makes it a bunch easier to make sure they're deleted from
the archive along with the corresponding kernel images, and makes sure
that when someone uploads a new kernel image, new module images get
uploaded too.

Pretty please?

Cheers,
aj

-- 
Anthony Towns <[EMAIL PROTECTED]> 
I don't speak for anyone save myself. GPG signed mail preferred.

  ``Dear Anthony Towns: [...] Congratulations -- 
you are now certified as a Red Hat Certified Engineer!''

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Sun, May 25, 2003 at 06:21:00AM +0200, Christoph Hellwig wrote:

> On Sat, May 24, 2003 at 06:32:26PM -0400, Matt Zimmerman wrote:
> > It's not noise at all when it's something that we and others (desperately!)
> > want to know about.
> 
> Then read through the prepatch diffs, everything adding checks to
> ioctl methods or similar is likely one them.  

This approach does not scale.  I cannot personally review the diffs for
every upstream release of all the software in Debian, nor can any other
individual or even a small group.

For the most part, this is not necessary, because upstream announces
security problems.

-- 
 - mdz

Re: Maintaining kernel source in sarge

[Christoph Hellwig]

On Sat, May 24, 2003 at 06:32:26PM -0400, Matt Zimmerman wrote:
> It's not noise at all when it's something that we and others (desperately!)
> want to know about.

Then read through the prepatch diffs, everything adding checks to
ioctl methods or similar is likely one them.  

Re: Maintaining kernel source in sarge

[Christoph Hellwig]

On Sat, May 24, 2003 at 08:10:20PM -0400, Matt Zimmerman wrote:
> > in task_struct then perhaps so assuming that we care about it enough to do
> > it in such a way.  Otherwise I don't see your point.
> 
> Are task_struct and mm_struct exposed to modules?

Yes.

> they should need to be, but I am no expert in the kernel.  If this is meant
> to be this way, then shouldn't the struct have some amount of padding to
> allow for changes like this without breaking compatibility?

At least not for upstream.  If you think it's valueable for debian
to provide this do it.

Re: Maintaining kernel source in sarge

[Christoph Hellwig]

On Sun, May 25, 2003 at 01:51:05AM +0200, Arnd Bergmann wrote:
> SuSE don't have a single kernel source either. They have a set of
> a few hundred common patches plus some more patches (e.g. 200
> for s390) that are used only for one architecture, usually both 32 and 
> 64 bit. Single patches can be enabled or excluded per architecture
> there.

Right.  Still they have a single source package for all the achitectures
which is a good start.  And I talked to people involved in their kernel
maintaince and they say they're trying to cut down these conditional
applies patches as much as possible because they are a maintaince
nightmare

> As a real-world example, kernel-patch-s390 can provide 
> the ptrace bug fix from Martin Schwidefsky, while
> kernel-patch-debian contains the generic solution from Alan Cox.

This sounds like a really bad idea (okay, not worse than the
current situation but still bad).  Martin should diff his patch ontop
of Alan, if it has other core changes he should really post it to
lkml and explain those.  Now this unfortunately hasn't happend and
if it has incompatible core changes (I haven't checked) that's one
of those rare cases where you want to appy a patch condtionally.

> When building kernel-image-s390, make-kpkg would first apply
> the arch specific patches and the the arch independent ones that
> have not been superceded by an arch specific one.

Again that's a very bad idea.  Arch-independent patches should
_always_ be applied before Arch-dependent ones, otherwiese applying
MI patches will get absolute horror.

Re: Maintaining kernel source in sarge

[Herbert Xu]

Matt Zimmerman <[EMAIL PROTECTED]> wrote:
> 
> In general, this is not a problem.  The exception is coordinated disclosure,
> where it is important that fixes be available for all architectures in order
> to minimize exposure.  In those cases, it would be helpful to coordinate
> with all of the kernel package maintainers ahead of time.

Agreed.
 
> Of course, we cannot expect anyone to tell us things that they do not
> realize themselves.  I thought you were referring to changes with known
> security impact.

Well I do know at least some of those problems.  But it'll require someone
with a fulltime job to sit down and list them all.  Since IMHO it doesn't
make sense to just provide a partial fix to what is a wide-spread problem,
especially when you're just taking fixes from a new version which has
been available for a long time, I must say that it is not worth it.

> We cannot justifiably be too liberal with new releases in the hope of
> getting unknown bugfixes, as we are as likely to receive unknown bugs.

It's not just bug fixes.  There are also new drivers or updates to
existing drivers that are required to make the system work on new systems.
Of course this also brings the possibility of breaking existing systems
when old drivers are updated.  But that is hopefully what testing will
eliminiate or at least minimise.

We must remember that most of our concerns in this area is shared by
the upstream.

> Are task_struct and mm_struct exposed to modules?  It does not seem like

Yes.
 
>> Firstly I'm not suggesting the uploading new versions in direct response
>> to a security alert.  I'm saying that we should be keeping up with new
>> versions so that we can be better prepared for subsequent security alerts.
> 
> That is, you are saying that we should try to run the most recent kernels so
> that we can patch them more easily?  I'm wary of introducing churn in stable
> for problems which we don't yet know about.

Unfortunately it is my opinion that we have no choice given our current
release schedules.

> One example which comes to mind immediately is the grossly incompatible
> TUN/TAP changes around kernel 2.4.5.  The UML skas breakage from the ptrace
> fix is another, but I don't know if that was possible to avoid.

Well first 2.4.5 is fair game since any stable Linux release in single
digits should be treated as fluid.

The ptrace breakage is not the fault of the upstream since they did not
actually release a new version.  In fact, the next release will contain
the fixed patch.

> We also have to deal with the fact that miscellaneous changes between kernel
> releases can and do break third-party patches (EVMS 1.2.x, device-mapper,
> freeswan, etc.) that we ship and our users depend on to run their systes.
> If the only fix we provide is an updated kernel, then users can no longer
> use the patches that we supply.

I agree that we should try to maintain compatibility.  But it should not
override the need to include new stable kernel releases from upstream.
-- 
Debian GNU/Linux 3.0 is out! ( http://www.debian.org/ )
Email:  Herbert Xu ~{PmV>HI~} <[EMAIL PROTECTED]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Maintaining kernel source in sarge

[Herbert Xu]

Matt Zimmerman <[EMAIL PROTECTED]> wrote:
> On Sun, May 25, 2003 at 07:37:10AM +1000, Herbert Xu wrote:
>> 
>> How does this automate the arch-specific merges where conflicts arise?
> 
> 1. unpack pristine kernel source
> 2. apply Debian patch
> 3. dry-run arch-specific patch
> 4. if no conflicts, test and release, otherwise:
> 5. apply arch-specific patch to pristine kernel source
> 6. 3-way merge between pristine kernel source, Debian tree, and
>   arch-specific tree
> 7. resolve marked conflicts by hand
> 
> Obviously conflicts must be resolved manually, hence the "mostly" above.
> This is relatively straightforward to do within a source package, given
> direct access to the (separate) pristine kernel source and Debian patch.

But the pristine kernel source and the Debian patch are already available
to the architecture maintainers:

apt-get --tar-only source kernel-source-2.4.xx
apt-get --diff-only source kernel-source-2.4.xx

So I don't think having a kernel-patch-debian by itself is of any value
to the merging process.  It also doesn't solve the problem that a new
kernel-patch-debian may break the building of old kernel-image packages.

There is also the suggestion of a complete break down of all Debian
changes should be made available.  Unfortunately that is simply
unmaintainable and unusable as the number of changes grows as the
dependencies between the patches are complex.

I think perhaps we could find a middle ground.  We can keep kernel-source
as it is with all the patches applied.  In addition to that, we can have a
kernel-patch-debian package which depends on the kernel-source of the same
upstream version containing patches that will take the kernel-source to
the following source trees:

1. The pristine upstream except for non-free removals.
2. All previous kernel-source revisions of that release.  For example,
kernel-patch-debian-2.4.20 will contain patches that will give you the
kernel-source-2.4.20-[1-7] packages respectively.  These patches will
be presented in incremental form with wrapper scripts around them that
will preserve compatibility with previous kernel-patch-debian packages
of the same version.

The per-architecture kernel images can then depend on kernel-patch-debian
and use the wrapper script that is most appropriate at that particular
time.  If a new kernel-source/kernel-patch-debian pair of the same version
is released then nothing will be broken as there will be a patch with the
same name in kernel-patch-debian that takes you to exactly the same source
tree used to build previous kernel-image packages.

This approach has the following benefits:

* The kernel-source binary contains all bug fixes as is.  Guido raised
a good point that if we separated the patches from the kernel-source, then
users may miss out on the bug fixes.  This is especially important in light
of the current speed of upstream releases.

* The pristine kernel-source is available in the binary archive.  Many people
have asked for this in the past and this makes it available in the form of
a source tar ball plus a patch.

* Per-arch kernel-image will no longer be made unbuildable by new
kernel-source updates.  This has always been a problem for those
architectures using the main kernel-source archive.

* The incremental patches should ease the merging process of those
architectures that wish to incoporate Debian changes.

* The incremental patches should also allow people to get only the updated
kernel-patch packages if they wish.

I'd like to hear from the architecture maintainers if this proposal
poses any problems to them.

Cheers,
-- 
Debian GNU/Linux 3.0 is out! ( http://www.debian.org/ )
Email:  Herbert Xu ~{PmV>HI~} <[EMAIL PROTECTED]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Maintaining kernel source in sarge

[Arnd Bergmann]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

> Christoph Hellwig <[EMAIL PROTECTED]> wrote:
> > Why can't Debian have just one tree for multiple architectures like
> > SuSE and RedHat (sometimes) do.  Okay suse supports 'only' i386,
> > x86_64,ppc,ppc64,s390,s390x,ia64 but their kernel also has patches
> > for sparc,sparc64,mips and m68k although I can't guarantee that these
> > really work in the relased tree (but last time I visted their office
> > people were playing with those ports in their spare time).

SuSE don't have a single kernel source either. They have a set of
a few hundred common patches plus some more patches (e.g. 200
for s390) that are used only for one architecture, usually both 32 and 
64 bit. Single patches can be enabled or excluded per architecture
there.

On Sunday 25 May 2003 00:12, Herbert Xu wrote:
> I don't think we can go all the way yet, but let's make a start.  If
> the architecture maintainers send me patches which clearly don't affect
> other archs or otherwise cause build problems, I will merge them.
>
> I don't think we can guarantee a tree that builds on all or most
> architectures, but we should be able to keep the difference to a
> minimum.

IMHO the ability to easily override base patches is important, which
would require not a patched kernel-source package but a 
kernel-patch-debian package containing all changes as single
diffs. AFAICS, dh-kpatches allows creating versioned patches.

As a real-world example, kernel-patch-s390 can provide 
the ptrace bug fix from Martin Schwidefsky, while
kernel-patch-debian contains the generic solution from Alan Cox.
When building kernel-image-s390, make-kpkg would first apply
the arch specific patches and the the arch independent ones that
have not been superceded by an arch specific one.

The same scheme allows creating a kernel-patch-2.4.21-rc3
package for kernel-source-2.4.20 that simply overrides all
fixes that have been backported from Marcellos tree to
kernel-patch-debian. This is very helpful for architectures
like amd64, where the official kernel tree only contains changes
against the latest prepatch.

Arnd <><
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+0AVt5t5GS2LDRf4RAhRjAJ9mzGRGp4uA5P4xBI7CBNFW/wb/kwCff6z8
uNPf7MfMLgNoXq8PQRG0IkI=
=a4wk
-END PGP SIGNATURE-

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Sun, May 25, 2003 at 07:58:09AM +1000, Herbert Xu wrote:

> Matt Zimmerman <[EMAIL PROTECTED]> wrote:
> > By 'independent packages', do you mean that we should have separate
> > kernel-source source packages for each architecture?  This would seem to
> > be a step backward.
> 
> No, they are independent packages already in that each architecture is
> built from a different Debian source package.  I'm just saying that the
> Security Team should treat it as different packages for the purposes of
> release DSAs.

In general, this is not a problem.  The exception is coordinated disclosure,
where it is important that fixes be available for all architectures in order
to minimize exposure.  In those cases, it would be helpful to coordinate
with all of the kernel package maintainers ahead of time.

> > I believe that we should not do anything to encourage this kind of
> > irresponsible behaviour.  If an upstream is not willing to share
> > information about security fixes with us, we should not reward them by
> > doing extra integration work to try to push a new upstream release into
> > stable.
> 
> Well it's not a question of irresponsibility.  It's simply impossible to
> for either the upstream or us to vet every single patch that passes
> through the system and decide whether it has security implications and
> raise an alert as a result.

Of course, we cannot expect anyone to tell us things that they do not
realize themselves.  I thought you were referring to changes with known
security impact.

We cannot justifiably be too liberal with new releases in the hope of
getting unknown bugfixes, as we are as likely to receive unknown bugs.

> > It is infortunate if this must sometimes happen, but hopefully it is an
> > exception, and in those cases we will need to rebuild modules and
> > provide for both kernel images to be installed at once.
> 
> It's going to be more common than you think due to the way the Linux
> module ABI works.  A small change in any structure that is either directly
> or indirectly referred to by a commonly used one will result in a change.

If so, then we must be prepared to rebuild kernel module packages, and this
is a fact of life.  Fortunately, we don't provide many of these.

> > I don't see why the ptrace fix necessitated a change in the module ABI
> > however, only that it was implemented that way upstream.
> 
> If you mean that it could have been fixed without changing the bitfields
> in task_struct then perhaps so assuming that we care about it enough to do
> it in such a way.  Otherwise I don't see your point.

Are task_struct and mm_struct exposed to modules?  It does not seem like
they should need to be, but I am no expert in the kernel.  If this is meant
to be this way, then shouldn't the struct have some amount of padding to
allow for changes like this without breaking compatibility?

> > Yes, on that note: could you send any separate patches that you have for
> > these issues to [EMAIL PROTECTED]
> > 
> > - CAN-2003-0001: etherleak
> > - CAN-2003-0127: ptrace
> > - CAN-2003-0244: ip route cache hash
> > - CAN-2003-0246: ioperm issue
> > - CAN-2002-1380: mmap() on /proc/pid/mem
> > - CVE-2002-0429: lcall7 DoS
> 
> Will do.  However I can't guarantee that the patches will apply to
> older versions of 2.4.

That is no problem; at this point we need all of the information that we can
get.  It helps greatly to see how the fix was made.

> > [arch-specific patch packages]
> > 
> > Shall I file bugs for these so that they can be fixed for sarge?
> 
> I don't really mind the arch-specific kernel-patch packages depending
> on them since they are and should always built against the Debian source.
> I was mostly referring to the other kernel-patch packages.

Ah, I did not realize you did not mean to include these.  Never mind, then.

> > Pushing a new kernel release is essentially punting to the user, forcing
> > them to decide between old bugs and new bugs.
> 
> Firstly I'm not suggesting the uploading new versions in direct response
> to a security alert.  I'm saying that we should be keeping up with new
> versions so that we can be better prepared for subsequent security alerts.

That is, you are saying that we should try to run the most recent kernels so
that we can patch them more easily?  I'm wary of introducing churn in stable
for problems which we don't yet know about.

> And please don't exaggerate about the possibility of breaking systems.
> New versions are only uploaded to proposed-updates after extensive testing
> by the maintainers, the users in unstable as well as the general Linux
> user base.  If you have a particular breakage in the 2.4 line in mind,
> please bring it up so we can see how it could've be dealt with better.

I did not mean to disregard the testing that you do, but kernel upgrades are
problematic enough as it is, considering that the entire system must be
brought down.

One example which comes to mind immediately is the grossly incompatible
TUN/TAP chan

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Sun, May 25, 2003 at 07:37:10AM +1000, Herbert Xu wrote:

> Matt Zimmerman <[EMAIL PROTECTED]> wrote:
> > 
> > Given an explicit kernel-patch-debian, containing architecture-agnostic
> > differences between kernel.org source and Debian's kernel source,
> > arch-specific merges could be mostly automated, and security fixes could be
> > made in one place.
> 
> How does this automate the arch-specific merges where conflicts arise?

1. unpack pristine kernel source
2. apply Debian patch
3. dry-run arch-specific patch
4. if no conflicts, test and release, otherwise:
5. apply arch-specific patch to pristine kernel source
6. 3-way merge between pristine kernel source, Debian tree, and
   arch-specific tree
7. resolve marked conflicts by hand

Obviously conflicts must be resolved manually, hence the "mostly" above.
This is relatively straightforward to do within a source package, given
direct access to the (separate) pristine kernel source and Debian patch.

-- 
 - mdz

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Sat, May 24, 2003 at 08:44:26PM +0200, Guido Guenther wrote:

> On Sat, May 24, 2003 at 01:42:22PM -0400, Matt Zimmerman wrote:
> > So this means that maintainers of the architecture patches must be sure
> > to merge in these fixes, otherwise they may inherit security
> > vulnerabilities (for example)?  How can we track when this has happened
> > when there are so many different patches?
> The situation won't change much over the current one. You currently can't
> be sure that an arch doesn't back out security fixes in our kernel-source
> with it's kernel-patch diff (intentionally or not).

In most cases, it's much easier for a maintainer to unintentionally leave
something out (especially if they are unaware of it) than to revert it
(unintentionally or not).

-- 
 - mdz

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Sat, May 24, 2003 at 08:42:39PM +0200, Christoph Hellwig wrote:

> On Sat, May 24, 2003 at 02:34:17PM -0400, Matt Zimmerman wrote:
> > What benefit is there in not announcing these problems?  Security
> > through obscurity?  How can we inform our users of their exposure when
> > we are not informed ourselves about security problems?
> 
> Noise.  You can's accnounce every possibly security-related fix found by
> an audit - note that it's not clear whether it actually _is_
> security-relevant at this point and certainly no one wrote an exploit for
> it.

You certainly can; other projects do.  The presence of an exploit is
irrelevant; we fix vulnerabilities all the time for which no exploit
necessarily exists.

It's not noise at all when it's something that we and others (desperately!)
want to know about.

> > It is infortunate if this must sometimes happen, but hopefully it is an
> > exception, and in those cases we will need to rebuild modules and
> > provide for both kernel images to be installed at once.
> 
> It's not an exception.  Fixes can and will change the ABI all the time.
> You should not expect to be able to load a binary kernel module into _any_
> other one than the one it was compiled against.  Sometimes security fixes
> may even break the source API.  (remember the dcache issues in
> 2.2.?).

Compatibility is often broken for many other reasons as well; this does not
mean that it is necessary for our purposes.

-- 
 - mdz

Re: Maintaining kernel source in sarge

[Herbert Xu]

Christoph Hellwig <[EMAIL PROTECTED]> wrote:
> 
> Why can't Debian have just one tree for multiple architectures like
> SuSE and RedHat (sometimes) do.  Okay suse supports 'only' i386,
> x86_64,ppc,ppc64,s390,s390x,ia64 but their kernel also has patches
> for sparc,sparc64,mips and m68k although I can't guarantee that these
> really work in the relased tree (but last time I visted their office
> people were playing with those ports in their spare time).

I don't think we can go all the way yet, but let's make a start.  If
the architecture maintainers send me patches which clearly don't affect
other archs or otherwise cause build problems, I will merge them.

I don't think we can guarantee a tree that builds on all or most
architectures, but we should be able to keep the difference to a
minimum.
-- 
Debian GNU/Linux 3.0 is out! ( http://www.debian.org/ )
Email:  Herbert Xu ~{PmV>HI~} <[EMAIL PROTECTED]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Maintaining kernel source in sarge

[Herbert Xu]

Matt Zimmerman <[EMAIL PROTECTED]> wrote:
> 
> I think that we can live with multiple source packages given a little bit of
> infrastructure which more closely follows the way that kernel development
> happens.  We need to be able to introduce specific patches to all of our
> kernels while minimizing the possibility of errors.  Errors include breaking
> kernel builds, breaking module packages, missing some kernel packages and
> patches, etc.  We also need to consider users who need to build their own
> kernels.

Agreed.

>> Here is my rough idea as to how it can work:
>> 
>> 1. The architectures should be treated as independent packages.
> 
> By 'independent packages', do you mean that we should have separate
> kernel-source source packages for each architecture?  This would seem to be
> a step backward.

No, they are independent packages already in that each architecture is
built from a different Debian source package.  I'm just saying that the
Security Team should treat it as different packages for the purposes of
release DSAs.

> This is already the case.  However, look at the mess which results:  s390
> and mips have integrated one of the ptrace patches into their respective
> kernel-patch packages (DSA-276 and DSA-270).  Now, how are we going to fix
> i386?  If we integrate the patch into kernel-source-2.4.17 or
> kernel-source-2.4.19, then s390 and mips are now broken because they will
> conflict.

This is a mistake that we can avoid in future.  The release of a
kernel-source or a general kernel-patch package with the security fix
should precede all architecture uploads.

>> The main advantage to this is in fact security.  It is routine for small
>> security fixes to enter a stable kernel unannounced.  For instance,
>> between 2.4.18 and 2.4.19, dozens of unannounced small security which only
>> affect specific drivers were fixed.  It also minimises any back-porting
>> that has to be done when a security alert is raised.
> 
> I believe that we should not do anything to encourage this kind of
> irresponsible behaviour.  If an upstream is not willing to share information
> about security fixes with us, we should not reward them by doing extra
> integration work to try to push a new upstream release into stable.

Well it's not a question of irresponsibility.  It's simply impossible to
for either the upstream or us to vet every single patch that passes through
the system and decide whether it has security implications and raise an
alert as a result.

>> In fact, due to the use of modules, it is often impossible to make
>> security fixes which are module ABI compatible with previous kernels.  For
>> example, the last two security holes (ptrace and net hash) both changed
>> the modules ABI.
> 
> It is infortunate if this must sometimes happen, but hopefully it is an
> exception, and in those cases we will need to rebuild modules and provide
> for both kernel images to be installed at once.

It's going to be more common than you think due to the way the Linux
module ABI works.  A small change in any structure that is either directly
or indirectly referred to by a commonly used one will result in a change.

> I don't see why the ptrace fix necessitated a change in the module ABI
> however, only that it was implemented that way upstream.

If you mean that it could have been fixed without changing the bitfields
in task_struct then perhaps so assuming that we care about it enough to
do it in such a way.  Otherwise I don't see your point.

>> Let me make it a bit more concrete as to what we can do about woody
>> right now.
> 
> Yes, on that note: could you send any separate patches that you have for
> these issues to [EMAIL PROTECTED]
> 
> - CAN-2003-0001: etherleak
> - CAN-2003-0127: ptrace
> - CAN-2003-0244: ip route cache hash
> - CAN-2003-0246: ioperm issue
> - CAN-2002-1380: mmap() on /proc/pid/mem
> - CVE-2002-0429: lcall7 DoS

Will do.  However I can't guarantee that the patches will apply to
older versions of 2.4.

> Here is a quick list for unstable:
> 
> mizar:[...deb/notmine/security/kernel] apt-cache dumpavail | grep-dctrl 
> -nsPackage -FDepends kernel-source-
> kernel-patch-2.2.20-powerpc
> kernel-patch-2.4.19-powerpc
> kernel-patch-2.4.20-hppa
> kernel-patch-2.4.20-ia64
> kernel-patch-2.4.20-powerpc
> 
> Shall I file bugs for these so that they can be fixed for sarge?

I don't really mind the arch-specific kernel-patch packages depending
on them since they are and should always built against the Debian source.
I was mostly referring to the other kernel-patch packages.

In fact, I don't see a single generic kernel-patch package in unstable
that depends on kernel-source.  That's a great thing.

>> Solution: Remove offending kernel-patch packages and kernel-source
>> packages.  Users of those kernel-source packages should be encouraged to
>> upgrade to a later version.  I'd recommend that we keep 2.4.18 and inject
>> 2.4.19/2.4.20 as soon as possible.  All architectures have kernel images
>> in eithe

Re: Maintaining kernel source in sarge

[Herbert Xu]

Matt Zimmerman <[EMAIL PROTECTED]> wrote:
> 
> Given an explicit kernel-patch-debian, containing architecture-agnostic
> differences between kernel.org source and Debian's kernel source,
> arch-specific merges could be mostly automated, and security fixes could be
> made in one place.

How does this automate the arch-specific merges where conflicts arise?
-- 
Debian GNU/Linux 3.0 is out! ( http://www.debian.org/ )
Email:  Herbert Xu ~{PmV>HI~} <[EMAIL PROTECTED]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Maintaining kernel source in sarge

[Sven Luther]

On Sun, May 25, 2003 at 04:29:13AM +1000, Russell Coker wrote:
> On Sun, 25 May 2003 04:18, Bernd Eckenfels wrote:
> > On Sat, May 24, 2003 at 07:55:08PM +0200, Christoph Hellwig wrote:
> > > Some m68k architectures might be a hard, I agree.  But having a package
> > > that works on as many machines as possible would be very cool.
> >
> > well, I there is a shared debian-kernel cvs then all architecture
> > maintainers can commit, and the package can be build for those who are
> > ready. This might generate a autobuilder and testing-transition problem :(
> 
> The problem here is when architecture specific patches require patches to 
> core 
> code.  If we have a single kernel source tree that everyone commits to then 
> it will be changing daily, it will be difficult to determine what source was 
> used to compile a particular kernel (and getting two kernels compiled from 
> the same source will be a neat trick).
> 
> I think that the thing to do is to have a base kernel source package that is 
> the standard Linus tree plus fixes that are really important.  That means 
> security fixes, fixes to file systems to fix data-loss issues, etc.  The VM 
> fix for 2.4.20 which stops machines with 4G of RAM from choking under load 
> would be a border-line case.

Why not have a standard linus tree + a set of security fix patches + a
per arch arch specific patch.

This would have the advantage of easily isolating the security patches
from the other stuff, and make work easier for people of the different
arches to check if they would cause problems or not on their arch.

Ideally, the security patches could even be isolated for each problem
they solve, a bit like the patches to the xfree86 package Branden has
been using all this time.

Friendly,

Sven Luther

Re: Maintaining kernel source in sarge

[Christoph Hellwig]

On Sat, May 24, 2003 at 11:37:09AM -0400, Daniel Jacobowitz wrote:
> Guido, you're not going about it the right way.  It's a three-way
> merge.  You take a kernel.org tree, diff it against the architecture
> tree that you're interested in, and then wiggle it into applying to the
> kernel source package that comes with Debian.  It's not all that hard,
> and there's a number of tools to help you (dirdiff, for instance; but
> all I ever use is diff, patch, a text editor, and CVS/BK).

These diff some random CVS tree vs kernel-source approach also
has other problems.  At least kernel-patch-2.4.20-m68k reintroduces
the oh so unfree firmware the kernel-source package removes..

Re: Maintaining kernel source in sarge

[Guido Guenther]

On Sat, May 24, 2003 at 01:42:22PM -0400, Matt Zimmerman wrote:
> So this means that maintainers of the architecture patches must be sure to
> merge in these fixes, otherwise they may inherit security vulnerabilities
> (for example)?  How can we track when this has happened when there are so
> many different patches?
The situation won't change much over the current one. You currently
can't be sure that an arch doesn't back out security fixes in our
kernel-source with it's kernel-patch diff (intentionally or not).

Herbert did a great job of keeping the kernel-patch maintainers up to
date about pending security issues. I certainly hope that splitting out
i386 will not change that. Having a separate kernel-patch-i386 will
make it even easier to pull these changes into the different
architectures.
 -- Guido

Re: Maintaining kernel source in sarge

[Christoph Hellwig]

On Sat, May 24, 2003 at 02:34:17PM -0400, Matt Zimmerman wrote:
> What benefit is there in not announcing these problems?  Security through
> obscurity?  How can we inform our users of their exposure when we are not
> informed ourselves about security problems?

Noise.  You can's accnounce every possibly security-related fix found
by an audit - note that it's not clear whether it actually _is_
security-relevant at this point and certainly no one wrote an exploit
for it.

> It is infortunate if this must sometimes happen, but hopefully it is an
> exception, and in those cases we will need to rebuild modules and provide
> for both kernel images to be installed at once.

It's not an exception.  Fixes can and will change the ABI all the time.
You should not expect to be able to load a binary kernel module into
_any_ other one than the one it was compiled against.  Sometimes
security fixes may even break the source API.  (remember the dcache
issues in 2.2.?).

Re: Maintaining kernel source in sarge

[Russell Coker]

On Sun, 25 May 2003 04:18, Bernd Eckenfels wrote:
> On Sat, May 24, 2003 at 07:55:08PM +0200, Christoph Hellwig wrote:
> > Some m68k architectures might be a hard, I agree.  But having a package
> > that works on as many machines as possible would be very cool.
>
> well, I there is a shared debian-kernel cvs then all architecture
> maintainers can commit, and the package can be build for those who are
> ready. This might generate a autobuilder and testing-transition problem :(

The problem here is when architecture specific patches require patches to core 
code.  If we have a single kernel source tree that everyone commits to then 
it will be changing daily, it will be difficult to determine what source was 
used to compile a particular kernel (and getting two kernels compiled from 
the same source will be a neat trick).

I think that the thing to do is to have a base kernel source package that is 
the standard Linus tree plus fixes that are really important.  That means 
security fixes, fixes to file systems to fix data-loss issues, etc.  The VM 
fix for 2.4.20 which stops machines with 4G of RAM from choking under load 
would be a border-line case.

Then patches that strictly affect one architecture will be separate, patches 
that are only really needed for one architecture (EG JFFS2 patches may be 
required for embedded devices but for most machines - including all Alpha's 
there would be no need for it) would be kept separate.

Also for the architecture-specific patches we should avoid the temptation to 
merge in other patches.  When people merge in patches for things such as XFS 
into more generic kernel patches (as SuSe appears to do) it makes things very 
painful for anyone who wants to extract things and use a sub-set of the 
patches.

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

Re: Maintaining kernel source in sarge

[Christoph Hellwig]

On Sat, May 24, 2003 at 08:18:40PM +0200, Bernd Eckenfels wrote:
> On Sat, May 24, 2003 at 07:55:08PM +0200, Christoph Hellwig wrote:
> > Some m68k architectures might be a hard, I agree.  But having a package
> > that works on as many machines as possible would be very cool. 
> 
> well, I there is a shared debian-kernel cvs then all architecture
> maintainers can commit, and the package can be build for those who are
> ready. This might generate a autobuilder and testing-transition problem :(

Yes, same as for gcc,glibc,xfree86.  The real question is 'should the
kernel be different' and if yes why?

Re: Maintaining kernel source in sarge

[Guido Guenther]

On Sat, May 24, 2003 at 11:37:09AM -0400, Daniel Jacobowitz wrote:
> Guido, you're not going about it the right way.  It's a three-way
> merge.  You take a kernel.org tree, diff it against the architecture
> tree that you're interested in, and then wiggle it into applying to the
> kernel source package that comes with Debian.  It's not all that hard,
> and there's a number of tools to help you (dirdiff, for instance; but
> all I ever use is diff, patch, a text editor, and CVS/BK).
That helps with the merges but doesn't give me a source tree that is as
close to the linux-mips.org tree as possible. If we have a vanilla
upstream source I can still add Herbert's patches on top of that later
if I desire.
 -- Guido

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Fri, May 23, 2003 at 09:20:28PM +1000, Herbert Xu wrote:

> So unless someone can come up with a solution to this problem,
> we will have to live with multiple Debian source packages for now.
> 
> This does make security fixes more difficult than it would be otherwise,
> however, I do not think it is unsurmountable given enough cooperation
> from the people involved.

I think that we can live with multiple source packages given a little bit of
infrastructure which more closely follows the way that kernel development
happens.  We need to be able to introduce specific patches to all of our
kernels while minimizing the possibility of errors.  Errors include breaking
kernel builds, breaking module packages, missing some kernel packages and
patches, etc.  We also need to consider users who need to build their own
kernels.

> Here is my rough idea as to how it can work:
> 
> 1. The architectures should be treated as independent packages.

By 'independent packages', do you mean that we should have separate
kernel-source source packages for each architecture?  This would seem to be
a step backward.

> We should not be shy of releasing DSAs for some architectures before
> others.  Since we cannot assume that all architectures are at the same
> version, some will inevitably take longer to fix due to the back-porting
> involved.

This is already the case.  However, look at the mess which results:  s390
and mips have integrated one of the ptrace patches into their respective
kernel-patch packages (DSA-276 and DSA-270).  Now, how are we going to fix
i386?  If we integrate the patch into kernel-source-2.4.17 or
kernel-source-2.4.19, then s390 and mips are now broken because they will
conflict.

> 2. We should be more liberal about adding new kernels to the stable
> archive and getting rid of old ones.
> 
> The main advantage to this is in fact security.  It is routine for small
> security fixes to enter a stable kernel unannounced.  For instance,
> between 2.4.18 and 2.4.19, dozens of unannounced small security which only
> affect specific drivers were fixed.  It also minimises any back-porting
> that has to be done when a security alert is raised.

I believe that we should not do anything to encourage this kind of
irresponsible behaviour.  If an upstream is not willing to share information
about security fixes with us, we should not reward them by doing extra
integration work to try to push a new upstream release into stable.

What benefit is there in not announcing these problems?  Security through
obscurity?  How can we inform our users of their exposure when we are not
informed ourselves about security problems?

> The disadvantage is of course the potential to break existing systems.
> However, I believe this can be minimised through careful management and
> thorough testing.  It is also mitigated by the fact that we allow multiple
> kernels to be installed simultaneously so it is easy to roll back.
>
> In fact, due to the use of modules, it is often impossible to make
> security fixes which are module ABI compatible with previous kernels.  For
> example, the last two security holes (ptrace and net hash) both changed
> the modules ABI.

It is infortunate if this must sometimes happen, but hopefully it is an
exception, and in those cases we will need to rebuild modules and provide
for both kernel images to be installed at once.

I don't see why the ptrace fix necessitated a change in the module ABI
however, only that it was implemented that way upstream.

> Let me make it a bit more concrete as to what we can do about woody
> right now.

Yes, on that note: could you send any separate patches that you have for
these issues to [EMAIL PROTECTED]

- CAN-2003-0001: etherleak
- CAN-2003-0127: ptrace
- CAN-2003-0244: ip route cache hash
- CAN-2003-0246: ioperm issue
- CAN-2002-1380: mmap() on /proc/pid/mem
- CVE-2002-0429: lcall7 DoS

> Problem: Too many kernel-source packages.
> 
> This is caused in part by gratuitous kernel-patch dependencies.
> Kernel-patch packages should *never* depend on a kernel-source
> package since the user can always use a kernel tar ball.

Here is a quick list for unstable:

mizar:[...deb/notmine/security/kernel] apt-cache dumpavail | grep-dctrl 
-nsPackage -FDepends kernel-source-
kernel-patch-2.2.20-powerpc
kernel-patch-2.4.19-powerpc
kernel-patch-2.4.20-hppa
kernel-patch-2.4.20-ia64
kernel-patch-2.4.20-powerpc

Shall I file bugs for these so that they can be fixed for sarge?

> Solution: Remove offending kernel-patch packages and kernel-source
> packages.  Users of those kernel-source packages should be encouraged to
> upgrade to a later version.  I'd recommend that we keep 2.4.18 and inject
> 2.4.19/2.4.20 as soon as possible.  All architectures have kernel images
> in either stable or proposed-updates that is at least 2.4.18 or higher.

Removing kernel-patch package from stable does not help users who are
running kernels built from them, and this is not the least of the problems,
either

Re: Maintaining kernel source in sarge

[Christoph Hellwig]

On Sat, May 24, 2003 at 07:51:17PM +0200, Karsten Merker wrote:
> Because it simply did not work out - not all architectures are in sync with
> Linus' tree

Oh, I know that well enough.

> and if I understood "our" port maintainer correctly, there are
> some architecture-specific things Linus does not accept for his tree

some architectures have code that can't go into a stable branch
anymore, yes.  ia64 is the only major example I know.

> or that
> are incompatible between other architectures.

Umm, no.  Sometimes you get conflicts when applying the plain
architecture patches, bat that's not due to incompatible requirements
but because they touch the same area.

> I doubt that a standard Suse kernel tree produces a working kernel on mips,
> mipsel or m68k; can't say for the other architectures, though.

Even Marcelo's tree seems to get a working m68k kernel nowdays -
although just the less stranger m68k variants.  Well, the mips tree is
total mess, you'd have to ask the guys playing with it at suse (Andreas
Jaeger IIRC). OTOH to the diff to a 2.4.20 XFS tree (i.e. almost like
Marcelo's tree) to get it working on my indy was rather small.

Re: Maintaining kernel source in sarge

[Bernd Eckenfels]

On Sat, May 24, 2003 at 07:55:08PM +0200, Christoph Hellwig wrote:
> Some m68k architectures might be a hard, I agree.  But having a package
> that works on as many machines as possible would be very cool. 

well, I there is a shared debian-kernel cvs then all architecture
maintainers can commit, and the package can be build for those who are
ready. This might generate a autobuilder and testing-transition problem :(

Greetings
Bernd
-- 
  (OO)  -- [EMAIL PROTECTED] --
 ( .. )  [EMAIL PROTECTED],linux.de,debian.org} http://home.pages.de/~eckes/
  o--o *plush*  2048/93600EFD  [EMAIL PROTECTED]  +497257930613  BE5-RIPE
(OO)  When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl!

Re: Maintaining kernel source in sarge

[Christoph Hellwig]

On Sat, May 24, 2003 at 01:25:38PM -0400, Daniel Jacobowitz wrote:
> > Sure, it's more work but I think it's worth it.
> 
> Because no one's done it?

Hey, if that was an argument.  The question is whether people want it..

> We can't count on it because the architecture ports become available at
> different times; m68k still hasn't caught up IIRC.  It would be nice,
> though.

Some m68k architectures might be a hard, I agree.  But having a package
that works on as many machines as possible would be very cool. 

Note that although I'm not a DD (and don't have plans to become one),
I'd be happy to invest time to help out wherever needed for such a
kernel-unified package.

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Sat, May 24, 2003 at 11:37:09AM -0400, Daniel Jacobowitz wrote:

> Guido, you're not going about it the right way.  It's a three-way merge.
> You take a kernel.org tree, diff it against the architecture tree that
> you're interested in, and then wiggle it into applying to the kernel
> source package that comes with Debian.  It's not all that hard, and
> there's a number of tools to help you (dirdiff, for instance; but all I
> ever use is diff, patch, a text editor, and CVS/BK).

This is how I thought it worked already, but I don't have the experience in
maintaining the various arch-specific patches to speak first-hand about it.

The only thing that would seem to complicate this is that the Debian patches
to kernel source are implicit (kernel-source .diff.gz) rather than explicit
(kernel-patch-debian).

Given an explicit kernel-patch-debian, containing architecture-agnostic
differences between kernel.org source and Debian's kernel source,
arch-specific merges could be mostly automated, and security fixes could be
made in one place.

-- 
 - mdz

Re: Maintaining kernel source in sarge

[Daniel Jacobowitz]

On Sat, May 24, 2003 at 07:03:22PM +0200, Christoph Hellwig wrote:
> On Sat, May 24, 2003 at 11:37:09AM -0400, Daniel Jacobowitz wrote:
> > Guido, you're not going about it the right way.  It's a three-way
> > merge.  You take a kernel.org tree, diff it against the architecture
> > tree that you're interested in, and then wiggle it into applying to the
> > kernel source package that comes with Debian.  It's not all that hard,
> > and there's a number of tools to help you (dirdiff, for instance; but
> > all I ever use is diff, patch, a text editor, and CVS/BK).
> 
> Why can't Debian have just one tree for multiple architectures like
> SuSE and RedHat (sometimes) do.  Okay suse supports 'only' i386,
> x86_64,ppc,ppc64,s390,s390x,ia64 but their kernel also has patches
> for sparc,sparc64,mips and m68k although I can't guarantee that these
> really work in the relased tree (but last time I visted their office
> people were playing with those ports in their spare time).
> 
> Sure, it's more work but I think it's worth it.

Because no one's done it?

We can't count on it because the architecture ports become available at
different times; m68k still hasn't caught up IIRC.  It would be nice,
though.

-- 
Daniel Jacobowitz
MontaVista Software Debian GNU/Linux Developer

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Sat, May 24, 2003 at 09:24:16PM +1000, Herbert Xu wrote:

> Guido Guenther <[EMAIL PROTECTED]> wrote:
> >
> > It's very hard to get these bug fixes anyway since when I do a
> > _complete_ diff between kernel-source-2.X.Y in the archive and the
> > kernel source for architecture foo I'll _always_ (accidentally) pull out
> > all the bug fixes you made. Only diffing specific parts of the
> 
> OK, barring any major objections, that's how it will be for 2.4.21.  I
> will make kernel-source-2.4.21 be identical to the upstream tar ball
> except for the non-free bits.  A kernel-patch-i386 package will be
> introduced.

So this means that maintainers of the architecture patches must be sure to
merge in these fixes, otherwise they may inherit security vulnerabilities
(for example)?  How can we track when this has happened when there are so
many different patches?

-- 
 - mdz

Re: Maintaining kernel source in sarge

[Christoph Hellwig]

On Sat, May 24, 2003 at 11:37:09AM -0400, Daniel Jacobowitz wrote:
> Guido, you're not going about it the right way.  It's a three-way
> merge.  You take a kernel.org tree, diff it against the architecture
> tree that you're interested in, and then wiggle it into applying to the
> kernel source package that comes with Debian.  It's not all that hard,
> and there's a number of tools to help you (dirdiff, for instance; but
> all I ever use is diff, patch, a text editor, and CVS/BK).

Why can't Debian have just one tree for multiple architectures like
SuSE and RedHat (sometimes) do.  Okay suse supports 'only' i386,
x86_64,ppc,ppc64,s390,s390x,ia64 but their kernel also has patches
for sparc,sparc64,mips and m68k although I can't guarantee that these
really work in the relased tree (but last time I visted their office
people were playing with those ports in their spare time).

Sure, it's more work but I think it's worth it.

And imagine how many people would scream if debian had $BIGNUM
XFree86 or glibc packages..

Re: Maintaining kernel source in sarge

[Daniel Jacobowitz]

On Sat, May 24, 2003 at 04:06:30PM +0200, Guido Guenther wrote:
> On Sat, May 24, 2003 at 09:24:16PM +1000, Herbert Xu wrote:
> > OK, barring any major objections, that's how it will be for 2.4.21.
> > I will make kernel-source-2.4.21 be identical to the upstream tar ball
> > except for the non-free bits.  A kernel-patch-i386 package will be
> > introduced.
> You won't here me objecting ;)
> Thanks a lot,

I'll object!

Guido, you're not going about it the right way.  It's a three-way
merge.  You take a kernel.org tree, diff it against the architecture
tree that you're interested in, and then wiggle it into applying to the
kernel source package that comes with Debian.  It's not all that hard,
and there's a number of tools to help you (dirdiff, for instance; but
all I ever use is diff, patch, a text editor, and CVS/BK).

For the PowerPC kernels this is how I've been doing it (sporadically)
for years.  It's never been hard; nowadays the diffs are only on the
order of 70K, but they used to be multi-megabyte and I still rarely had
conflicts with Herbert's patches.

If you do have problems, do the merge the other way: kernel.org tarball
diffed against Herbert's packages, wiggle those diffs on top of your
architecture tree, diff the result against the kernel-source package,
and call that your archictecture patch.

-- 
Daniel Jacobowitz
MontaVista Software Debian GNU/Linux Developer

Re: Maintaining kernel source in sarge

[Guido Guenther]

On Sat, May 24, 2003 at 09:24:16PM +1000, Herbert Xu wrote:
> OK, barring any major objections, that's how it will be for 2.4.21.
> I will make kernel-source-2.4.21 be identical to the upstream tar ball
> except for the non-free bits.  A kernel-patch-i386 package will be
> introduced.
You won't here me objecting ;)
Thanks a lot,
 -- Guido

Re: Maintaining kernel source in sarge

[Herbert Xu]

Guido Guenther <[EMAIL PROTECTED]> wrote:
>
> It's very hard to get these bug fixes anyway since when I do a
> _complete_ diff between kernel-source-2.X.Y in the archive and the
> kernel source for architecture foo I'll _always_ (accidentally)
> pull out all the bug fixes you made. Only diffing specific parts of the

OK, barring any major objections, that's how it will be for 2.4.21.
I will make kernel-source-2.4.21 be identical to the upstream tar ball
except for the non-free bits.  A kernel-patch-i386 package will be
introduced.
-- 
Debian GNU/Linux 3.0 is out! ( http://www.debian.org/ )
Email:  Herbert Xu ~{PmV>HI~} <[EMAIL PROTECTED]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Maintaining kernel source in sarge

[Guido Guenther]

On Sat, May 24, 2003 at 08:44:48AM +1000, Herbert Xu wrote:
> Guido Guenther <[EMAIL PROTECTED]> wrote:
> >
> > This only works if we have a _clean_ kernel-source-2.X.Y package. One of
> > the reasons why maintaining kernel-patch-2.X.Y- packages is such a
> > pain is the asymmetry between i386 and other arches - almost every time
> > a new kernel-source package is uploaded the kernel-patch-2.X.Y-mips
> > package has to be rediffed. So the first and maybe most important step
> 
> I can understand your pain.  However, most of the changes made to the
> kernel-source are not i386-specific.  In fact, if they were, they would
> not be causing these patch conflicts that you're seeing.
> 
> So essentially throwing them out means that your architecture will miss
> out on all bug fixes.  If this turns out to be what most of you want, then
> that's fine and I will do just that.
It's very hard to get these bug fixes anyway since when I do a
_complete_ diff between kernel-source-2.X.Y in the archive and the
kernel source for architecture foo I'll _always_ (accidentally)
pull out all the bug fixes you made. Only diffing specific parts of the
tree is unfortunately out of question (at least for mips(el)) since the
diff is just too big to make this feasible. So having a separate i386
kernel-patch package would make things even easier since I can then
first diff against a clean Linus tree (giving a minimal diff) and then
look into your kernel-patch-*-i386 and apply all the fixes.
What worries me a bit is that the majority of our users would have to
pull an additional kernel-patch-*-i386 to build custom kernels but I'm
not sure how many of the folks who build custom i386 kernels get the
source packages from our archives, many of them get the source from
kernel.org directly anyway, I'd guess.
Regards,
 -- Guido

Re: Maintaining kernel source in sarge

[Herbert Xu]

Guido Guenther <[EMAIL PROTECTED]> wrote:
>
> This only works if we have a _clean_ kernel-source-2.X.Y package. One of
> the reasons why maintaining kernel-patch-2.X.Y- packages is such a
> pain is the asymmetry between i386 and other arches - almost every time
> a new kernel-source package is uploaded the kernel-patch-2.X.Y-mips
> package has to be rediffed. So the first and maybe most important step

I can understand your pain.  However, most of the changes made to the
kernel-source are not i386-specific.  In fact, if they were, they would
not be causing these patch conflicts that you're seeing.

So essentially throwing them out means that your architecture will miss
out on all bug fixes.  If this turns out to be what most of you want, then
that's fine and I will do just that.
-- 
Debian GNU/Linux 3.0 is out! ( http://www.debian.org/ )
Email:  Herbert Xu ~{PmV>HI~} <[EMAIL PROTECTED]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Fri, May 23, 2003 at 09:04:05AM +0200, Martin Schulze wrote:

> To make it more interesting, Matt Zimmerman revealed[2] that merging all
> kernel source packages would save space of one CD from our archive and our
> CD images.

I was probably exaggerating slightly; I did not do the calculations.
However, each kernel-source package represents 25M of binary and 30M of
source packages, which is a significant amount of space both in the main
archive and on the CD set, not to mention the benefits in simplicity and
maintainability.

> Since sarge is far from being released, now would be a perfect time to
> start working on consolidating kernel and source packages in order to have
> a new and much better working system ready for sarge when it is time to
> release.

Another thing that concerns me is the upgrade process.  The practice of
overwriting the running kernel and modules results in a risky, interactive
installation process, and makes it impossible to back out to the old kernel
unless the user takes appropriate measures.

I noticed that the current 2.4.20 kernel-image packages in unstable have
taken a different approach.  When the i386 kernel was updated to include the
ptrace fix, it was built as a separate binary package, 2.4.20-1-
rather than 2.4.20-.  This allowed both kernels to be installed
simultaneously, and the upgrade easily reverted.  Users who had installed
the kernel-image-2.4-686 metapackage were still automatically upgraded.

-- 
 - mdz

Re: Maintaining kernel source in sarge

[Guido Guenther]

On Fri, May 23, 2003 at 05:58:15PM +1000, Russell Coker wrote:
> That's not a problem, you just have to run diff between the source tree for 
> the platform in question and the Linus tree.
This only works if we have a _clean_ kernel-source-2.X.Y package. One of
the reasons why maintaining kernel-patch-2.X.Y- packages is such a
pain is the asymmetry between i386 and other arches - almost every time
a new kernel-source package is uploaded the kernel-patch-2.X.Y-mips
package has to be rediffed. So the first and maybe most important step
is to have a _clean_ kernel-source-2.X.Y- package to patch against
(which won't be removed from the archive as long as a single
kernel-patch-2.X.Y- depends on it). i386 would then just simply be
kernel-patch-2.X.Y-i386.
Regards,
 -- Guido

Re: Maintaining kernel source in sarge

[Herbert Xu]

Martin Schulze <[EMAIL PROTECTED]> wrote:
> 
> Manoj emphasized[1] that using one single kernel source package per
> kernel version and maintaining several patch packages for each
> architecture which finally build our kernel-image-$version packages is
> possible.

It seems that there is some confusion here as to what the term kernel source
package means.

I believe Martin is referring to Debian source packages, i.e., the things
that you get via apt-get source and make binary packages out of.  It
would be advantageous for security fixes if there were only one such
package so that multiple kernel-image packages can be built from it.

Manoj on the hand (correct me if I'm wrong) is referring to the source
code of the Linux kernel which is an entirely different beast.  There
is no doubt that we can get away with just one kernel-source binary
package per version.  Indeed, this is already the case bar two
architectures, ia64 and hppa.

The goal that Martin is trying to achieve is indeed a noble one.
Unfortunately I personally believe that it is not realistic.  It will
not become achievable until one of the following happens:

1. We reduce the number of architectures that we support.
2. The upstream kernel starts supporting multiple architectures by default.
3. We do not release kernel packages until all architectures are up-to-date.
   This could take up to a year in some cases and would require a lot of
   man-power.

1 is politically infeasible.  2 is out of our reach.  3 can be done but
in my opinion it is undesirable as the changes that cause delays in
architectures from being up-to-date are probably going to be the ones
that make back-porting security fixes difficult or slow.  And yes we
will have to back-port them if we went down that route.

So unless someone can come up with a solution to this problem,
we will have to live with multiple Debian source packages for now.

This does make security fixes more difficult than it would be otherwise,
however, I do not think it is unsurmountable given enough cooperation
from the people involved.


Here is my rough idea as to how it can work:

1. The architectures should be treated as independent packages.

We should not be shy of releasing DSAs for some architectures before
others.  Since we cannot assume that all architectures are at the same
version, some will inevitably take longer to fix due to the back-porting
involved.

When a security alert is raised, the Security Team can either start
building kernel images themselves or notify the kernel maintainers to
do it.  Then multiple DSAs can be released as the images are built
and uploaded.

2. We should be more liberal about adding new kernels to the stable
archive and getting rid of old ones.

The main advantage to this is in fact security.  It is routine for small
security fixes to enter a stable kernel unannounced.  For instance,
between 2.4.18 and 2.4.19, dozens of unannounced small security which only
affect specific drivers were fixed.  It also minimises any back-porting
that has to be done when a security alert is raised.

The disadvantage is of course the potential to break existing systems.
However, I believe this can be minimised through careful management and
thorough testing.  It is also mitigated by the fact that we allow multiple
kernels to be installed simultaneously so it is easy to roll back.

In fact, due to the use of modules, it is often impossible to make
security fixes which are module ABI compatible with previous kernels.
For example, the last two security holes (ptrace and net hash) both
changed the modules ABI.

3. All kernel-image maintainers should make sure that their packages
can be rebuilt on a system other than their own.  This needs to be
stressed since these packages aren't checked by the autobuilders so we
need to be extra careful.


Let me make it a bit more concrete as to what we can do about woody
right now.

Problem: Too many kernel-source packages.

This is caused in part by gratuitous kernel-patch dependencies.
Kernel-patch packages should *never* depend on a kernel-source
package since the user can always use a kernel tar ball.

Solution: Remove offending kernel-patch packages and kernel-source packages.
Users of those kernel-source packages should be encouraged to upgrade to
a later version.  I'd recommend that we keep 2.4.18 and inject 2.4.19/2.4.20
as soon as possible.  All architectures have kernel images in either
stable or proposed-updates that is at least 2.4.18 or higher.

For 2.2 I suggest that we move to 2.2.25 ASAP and move the one remaining
architecture that only work on 2.2 (m68k) to it.

Problem: Too many kernel-image souce packages.

We will have to live with this one.

Solution: None.  However, if we treat each architecture indenpendently,
the task for releasing security fixes becomes more manageble.  This
is also easier if we regularly update the kernel-image packages in stable.

Problem: Binary module packages.

This is actually not too bad for woody as we

Re: Maintaining kernel source in sarge

[Russell Coker]

On Fri, 23 May 2003 17:04, Martin Schulze wrote:
> I wonder if some people (Maybe Manoj and Russell, who are both quite
> competent) could help with this effort.

I would love to contribute, however at the moment I am over-committed already.

Progress towards getting SE Linux support in main is going well, when that is 
sorted out I'll have a lot more spare time for such things.

> I have to admit, that having several architectures use the same plain
> kernel source from Linus Torvalds aka kernel.org could be problematic
> since many architecture ports use their own kernel repository which is
> not always synced with the main kernel tree for whatever reasons.

That's not a problem, you just have to run diff between the source tree for 
the platform in question and the Linus tree.

It will make for some big patches (for example the patch for the kernel used 
on Familiar is extensive and has not been ported to 2.4.20 because it's too 
much work).  Which raises another issue.  Often (always?) some platforms 
don't have support for the latest kernel.  We can probably expect to have the 
last three Linus kernels in our source tree for this reason.

The proceedures for determining which kernels to keep and which to drop will 
also need to be worked out.  We have ia64, hppa, and Familiar-ARM (*) 
platforms with 2.4.19 support but no 2.4.20.  When 2.4.21 comes out we can't 
remove 2.4.20 source until after those platforms have support for 2.4.21 as 
they may just upgrade from 2.4.19 to 2.4.20.

We will probably end up having a minimum of support for three different kernel 
versions at all times.  This is a lot better than supporting three source 
trees for three architectures of the same kernel version!  Even users of i386 
often want old versions of the kernel source because of bugs (I recently had 
a request for a 2.4.18 version of one of my kernel patches because 2.4.19 and 
2.4.20 wouldn't work for someone).

Probably the first thing we should do is make some standards for kernel patch 
packages.  I suggest kernel-patch-2.x-NAME as the package name, and having 
the three latest kernel versions supported for a stable kernel (except when 
there aren't three versions of the patch with the same functionality and when 
it's too difficult to back-port the code) and up to five patches for 2.5.x 
kernels from the last 10 kernel versions.

(*)  Familiar kernel isn't in Debian yet, but hopefully soon will be.

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

Re: Maintaining kernel source in sarge

[Joel Baker]

On Fri, May 23, 2003 at 09:04:05AM +0200, Martin Schulze wrote:
> Martin Schulze wrote:
> > Only a few people will probably have noticed the mess resulting from
> > tons of different kernel packages in the stable (and unstable)
> > distribution.  Not only there are several versions of kernel source in
> > each architecture, they are also different for most architectures.
> > Only mips and mipsel share the same kernel source.
> > 
> > To make it worse, there are also third party kernel modules that
> > depend on a particular version of the kernel source (they don't depend
> > on the particular Debian revision, though, I hope).
> > 
> > As a result of this, it is almost impossible to update the kernel in a
> > released Debian distribution.
> 
> Manoj emphasized[1] that using one single kernel source package per
> kernel version and maintaining several patch packages for each
> architecture which finally build our kernel-image-$version packages is
> possible.
> 
> However, Herbert Xu hasn't contributed to this thread yet and most of
> our architecture maintainers haven't raised a word either.  These are
> most probably the people who will continue to do the work, and hence,
> need our support if the kernel source tree should be consolidated.

Well, okay, this is only semi-ontopic for the thread, but as the person
currently working in ~/Debian-Packages/netbsd-kernel-*, I'll speak up:

I'm trying very hard to arrange that we don't repeat the misfeatures of
this particular past. This is, granted, far easier than for Linux, since
there isn't really a concept of a 'separate arch repository' to be out of
sync, on NetBSD.

(No, this isn't pimping it; I don't even have it compiling yet.)

That said, I firmly believe the idea is one with merit, even if it won't
be easy.
-- 
Joel Baker <[EMAIL PROTECTED]>


pgpifBkogNNq0.pgp
Description: PGP signature

Re: Maintaining kernel source in sarge

[Martin Schulze]

Martin Schulze wrote:
> Only a few people will probably have noticed the mess resulting from
> tons of different kernel packages in the stable (and unstable)
> distribution.  Not only there are several versions of kernel source in
> each architecture, they are also different for most architectures.
> Only mips and mipsel share the same kernel source.
> 
> To make it worse, there are also third party kernel modules that
> depend on a particular version of the kernel source (they don't depend
> on the particular Debian revision, though, I hope).
> 
> As a result of this, it is almost impossible to update the kernel in a
> released Debian distribution.

Manoj emphasized[1] that using one single kernel source package per
kernel version and maintaining several patch packages for each
architecture which finally build our kernel-image-$version packages is
possible.

However, Herbert Xu hasn't contributed to this thread yet and most of
our architecture maintainers haven't raised a word either.  These are
most probably the people who will continue to do the work, and hence,
need our support if the kernel source tree should be consolidated.

I wonder if some people (Maybe Manoj and Russell, who are both quite
competent) could help with this effort.

I have to admit, that having several architectures use the same plain
kernel source from Linus Torvalds aka kernel.org could be problematic
since many architecture ports use their own kernel repository which is
not always synced with the main kernel tree for whatever reasons.

However, I do believe that it should be a goal of Debian to share as
much code as possible and hence share as much of the kernel through
various architectures as well.

> Removing one kernel version and including another without rebuilding
> all modules packages will break several installations.  Not removing
> the old packages will make the archive grow through time which will
> cause problems with CD build scripts.

To make it more interesting, Matt Zimmerman revealed[2] that merging
all kernel source packages would save space of one CD from our archive
and our CD images.

Ola Lundqvist added[3] that building modules packages is a non-trivial
task since the build process often depends on having the full kernel
source installed and already compiled.  He even provided some code
that could help to cope with this.  He also added a proposed kernel
policy.  Please read and comment.

I wonder whether the build scripts of modules packages can be modified
to not require the compiled kernel source anymore.  This would at
least remove this requirement and turn module building into a less
painful task.

Links:
 1. http://lists.debian.org/debian-devel/2003/debian-devel-200305/msg01348.html
 2. http://lists.debian.org/debian-devel/2003/debian-devel-200305/msg01368.html
 3. http://lists.debian.org/debian-devel/2003/debian-devel-200305/msg01390.html

Since sarge is far from being released, now would be a perfect time to
start working on consolidating kernel and source packages in order to
have a new and much better working system ready for sarge when it is
time to release.

Regards,

Joey

-- 
WARNING: Do not execute!  This call violates patent DE10108564.
http://www.elug.de/projekte/patent-party/patente/DE10108564

wget -O patinfo-`date +"%Y%m%d"`.html http://patinfo.ffii.org/

Please always Cc to me when replying to me on the lists.

Re: Maintaining kernel source in sarge

[Josselin Mouette]

Le dim 18/05/2003 à 16:52, Martin Schulze a écrit :
> I also wonder if there are efforts in progress to unify the kernel
> source through more than two architectures?  This would require a
> group or architecture maintainers (current kernel package mantainers)
> to work collaboratively towards this goal.

Maybe the modules situation is even worse, as rebuilding the kernel may
or may not break some of them.
Is it possible to reliably guarantee the binary-compatibility of a newer
(patched) kernel-image with modules built against the previous one ?
Using explicit dependencies for the compiler, for example.

Otherwise, maybe it would be better to include *all* extra modules we
want to build in the kernel build process, and just provide them in
separate packages, as we already do for pcmcia-modules.
This would make these modules more difficult to maintain in sid, but it
would vastly improve the maintainability of the kernel itself.
-- 
 .''`.   Josselin Mouette/\./\
: :' :   [EMAIL PROTECTED]
`. `'[EMAIL PROTECTED]
  `-  Debian GNU/Linux -- The power of freedom


signature.asc
Description: PGP signature

Re: Maintaining kernel source in sarge

[Ola Lundqvist]

Hello

On Mon, May 19, 2003 at 12:43:02PM -0400, David Z Maze wrote:
> Ola Lundqvist <[EMAIL PROTECTED]> writes:
> 
> > Kernel module policy:
> > -
> >
> > * Kernel modules must be provided as a "binary source" package.
> > * Module source packages should provide a debian/rules file.
> > * The debian/rules file must compile the module if KSRC=kernelsourcedir
> >   and KVERS=versionname is priovided.
> 
> I'd be slightly happier if the targets kernel-package used were
> supported here, 'debian/rules kdist-image' and such.  (This is to
> accomodate "binary source" packages that have a single debian/rules
> file that's copied verbatim from the source package to the binary
> package; both lm-sensors and i2c work this way, don't know about other
> packages.)

Hmm, that was what I wanted too. It was just too long time ago I fiddled
with the real options. I use my scripts too much I think.

> > * The debian/rules file may fail if an unsupported version of the kernel is
> >   provided by the environment.
> > * The debian/rules file may fail if no kernel-headers is in that location.
> > * The debian/rules file should handke KMAINT and KEMAIL env variables.
> 
> ...in fact, this looks a lot like what kernel-package currently
> documents.  Is a separate policy from the kernel-package documentation
> needed?

Well the kernel-package documentation is what I want to be highlighted.
But they are not at mandatory (or similar) and some packages do not follow
them. They are also not complete (in my opinion at least).

> (FWIW, i2c and lm-sensors both successfully build against only the
> kernel headers, via the kernel-build-* packages.)

The kernel build-packages is a good step.

Regards,

// Ola

> -- 
> David Maze [EMAIL PROTECTED]  http://people.debian.org/~dmaze/
> "Theoretical politics is interesting.  Politicking should be illegal."
>   -- Abra Mitchell
> 
> 
> -- 
> To UNSUBSCRIBE, email to [EMAIL PROTECTED]
> with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
> 

-- 
 - Ola Lundqvist ---
/  [EMAIL PROTECTED] Annebergsslingan 37  \
|  [EMAIL PROTECTED] 654 65 KARLSTAD  |
|  +46 (0)54-10 14 30  +46 (0)70-332 1551   |
|  http://www.opal.dhs.org UIN/icq: 4912500 |
\  gpg/f.p.: 7090 A92B 18FE 7994 0C36  4FE4 18A1 B1CF 0FE5 3DD9 /
 ---

Re: Maintaining kernel source in sarge

[Ola Lundqvist]

Hello

On Mon, May 19, 2003 at 11:06:16AM -0400, Matt Zimmerman wrote:
> On Mon, May 19, 2003 at 10:02:58AM +0200, Ola Lundqvist wrote:
> 
> > I'll start here:
> > 
> > Kernel package policy:
>   "kernel image" to avoid confusion between kernel source, kernel headers,

Agreed.

> kernel modules, etc.
> > --
> > 
> > * It should only exist one kernel-source package.
> > * Every modification to the kernel should be added as a patch package.
> > * Modifications may be separated to make it easier to administrate and
> >   for other people/packages to use it.
> 
> Kernel image packages must include a list of patches which have been
> applied, and the packages from which they came.

Agreed.

> > Kernel module policy:
> > -
> > 
> > * Kernel modules must be provided as a "binary source" package.
> > * Module source packages should provide a debian/rules file.
> > * The debian/rules file must compile the module if KSRC=kernelsourcedir
> >   and KVERS=versionname is priovided.
> > * The debian/rules file may fail if an unsupported version of the kernel is
> >   provided by the environment.
> > * The debian/rules file may fail if no kernel-headers is in that location.
> > * The debian/rules file should handke KMAINT and KEMAIL env variables.
> 
> It would be a significant gain if kernel modules could always be built
> against kernel-headers, without requiring full kernel-source.  Is there any
> situation where this is not feasible, or could it be made a requirement?

At least the pcmcia and freeswan needs some parts of kernel source to build.
At least last time I tried. I have been bitten by this a lot of times.

But yes that would be really great if they just have to depend on the
kernel source.

Regards,

// Ola

PS. I accidentily deleted two other mails in this thread. Was they
just for me or has they not yet been delivered to the mailinglist?
DS.

> -- 
>  - mdz
> 
> 
> -- 
> To UNSUBSCRIBE, email to [EMAIL PROTECTED]
> with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
> 

-- 
 - Ola Lundqvist ---
/  [EMAIL PROTECTED] Annebergsslingan 37  \
|  [EMAIL PROTECTED] 654 65 KARLSTAD  |
|  +46 (0)54-10 14 30  +46 (0)70-332 1551   |
|  http://www.opal.dhs.org UIN/icq: 4912500 |
\  gpg/f.p.: 7090 A92B 18FE 7994 0C36  4FE4 18A1 B1CF 0FE5 3DD9 /
 ---

Re: Maintaining kernel source in sarge

[David Z Maze]

Ola Lundqvist <[EMAIL PROTECTED]> writes:

> Kernel module policy:
> -
>
> * Kernel modules must be provided as a "binary source" package.
> * Module source packages should provide a debian/rules file.
> * The debian/rules file must compile the module if KSRC=kernelsourcedir
>   and KVERS=versionname is priovided.

I'd be slightly happier if the targets kernel-package used were
supported here, 'debian/rules kdist-image' and such.  (This is to
accomodate "binary source" packages that have a single debian/rules
file that's copied verbatim from the source package to the binary
package; both lm-sensors and i2c work this way, don't know about other
packages.)

> * The debian/rules file may fail if an unsupported version of the kernel is
>   provided by the environment.
> * The debian/rules file may fail if no kernel-headers is in that location.
> * The debian/rules file should handke KMAINT and KEMAIL env variables.

...in fact, this looks a lot like what kernel-package currently
documents.  Is a separate policy from the kernel-package documentation
needed?

(FWIW, i2c and lm-sensors both successfully build against only the
kernel headers, via the kernel-build-* packages.)

-- 
David Maze [EMAIL PROTECTED]  http://people.debian.org/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
-- Abra Mitchell

Re: Maintaining kernel source in sarge

[Steve Langasek]

On Mon, May 19, 2003 at 11:06:16AM -0400, Matt Zimmerman wrote:

> > Kernel module policy:
> > -

> > * Kernel modules must be provided as a "binary source" package.
> > * Module source packages should provide a debian/rules file.
> > * The debian/rules file must compile the module if KSRC=kernelsourcedir
> >   and KVERS=versionname is priovided.
> > * The debian/rules file may fail if an unsupported version of the kernel is
> >   provided by the environment.
> > * The debian/rules file may fail if no kernel-headers is in that location.
> > * The debian/rules file should handke KMAINT and KEMAIL env variables.

> It would be a significant gain if kernel modules could always be built
> against kernel-headers, without requiring full kernel-source.  Is there any
> situation where this is not feasible, or could it be made a requirement?

I think it's safe to say that if a kernel module package requires
something not present in the kernel-headers package for building,
there's a bug somewhere.  Which is not to say there aren't bugs, of
course.

-- 
Steve Langasek
postmodern programmer


pgpwDosniEoMl.pgp
Description: PGP signature

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Mon, May 19, 2003 at 10:02:58AM +0200, Ola Lundqvist wrote:

> I'll start here:
> 
> Kernel package policy:
  "kernel image" to avoid confusion between kernel source, kernel headers,
kernel modules, etc.
> --
> 
> * It should only exist one kernel-source package.
> * Every modification to the kernel should be added as a patch package.
> * Modifications may be separated to make it easier to administrate and
>   for other people/packages to use it.

Kernel image packages must include a list of patches which have been
applied, and the packages from which they came.

> Kernel module policy:
> -
> 
> * Kernel modules must be provided as a "binary source" package.
> * Module source packages should provide a debian/rules file.
> * The debian/rules file must compile the module if KSRC=kernelsourcedir
>   and KVERS=versionname is priovided.
> * The debian/rules file may fail if an unsupported version of the kernel is
>   provided by the environment.
> * The debian/rules file may fail if no kernel-headers is in that location.
> * The debian/rules file should handke KMAINT and KEMAIL env variables.

It would be a significant gain if kernel modules could always be built
against kernel-headers, without requiring full kernel-source.  Is there any
situation where this is not feasible, or could it be made a requirement?

-- 
 - mdz

Re: Maintaining kernel source in sarge

[Ola Lundqvist]

Hello

On Sun, May 18, 2003 at 04:52:54PM +0200, Martin Schulze wrote:
*SNIP*
> Removing one kernel version and including another without rebuilding
> all modules packages will break several installations.  Not removing
> the old packages will make the archive grow through time which will
> cause problems with CD build scripts.
> 
> Hence, it is important that when a new kernel is added (e.g. for
> security reasons) an older package is removed *and* all relevant
> modules packages are rebuilt and included as well.

Rebuilding module pacakges is not always trivial. I hope that code
can help some. The reason is that most of them depend on a kernel-tree
that is already compiled. The following code fixes that (at least for my 
situations).
I use it to compile modules for custom kernels. If you want me to make a command
line tool, just tell me. :)

Modify it as you wish.



# This file should be sourced by a file that sets the following variables. 
#revision=01
#kernver=2.4.19
#arch=686
#append=freeswan+vlan+mppe+ctx+netboot+raid
#clean=no
#prepmods=

# Reasonable defaults
#E=${revision:=01}

# Some other settings.
srcdir=/usr/src
kernbdir=kernel-source-$kernver
curdir=`pwd`
modsrcdir=modules
kernsrcdir=$curdir/$kernbdir

FAKE=
if [ "$UID" != "0" ] ; then
FAKE=fakeroot
fi

if [ -z "$maint" ] ; then
I="[^:][^:]*"
maint=$(grep "^$USER:" /etc/passwd | \
sed "s|$I:$I:$I:$I:||;s|[,:].*||;")
fi
if [ -z "$email" ] ; then
email="$EMAIL"
fi

if [ "$cleanall" = "yes" -a -d "$modsrcdir" ] ; then
for d in $(find $modsrcdir -type d -mindepth 1 -maxdepth 1) ; do
rm -Rf $d
done
fi

if [ ! -d $kernbdir ] ; then
echo "Unpacking source kernel-source-$kernver.tar.bz2"
tar xfj $srcdir/kernel-source-$kernver.tar.bz2
else
echo "CD $kernbdir"
cd $kernbdir
if [ -d "debian" ] ; then
$FAKE debian/rules clean < I would be glad if somebody could investigate the modules situation
> and describe which modules packages require which kernel versions
> (and/or depend on which other packages).
> 
> I also wonder if there are efforts in progress to unify the kernel
> source through more than two architectures?  This would require a
> group or architecture maintainers (current kernel package mantainers)
> to work collaboratively towards this goal.

A policy for kernel, kernel-patcha and kernel-module packages should be
great.

I'll start here:

Kernel package policy:
--

* It should only exist one kernel-source package.
* Every modification to the kernel should be added as a patch package.
* Modifications may be separated to make it easier to administrate and
  for other people/packages to use it.

Kernel module policy:
-

* Kernel modules must be provided as a "binary source" package.
* Module source packages should provide a debian/rules file.
* The debian/rules file must compile the module if KSRC=kernelsourcedir
  and KVERS=versionname is priovided.
* The debian/rules file may fail if an unsupported version of the kernel is
  provided by the environment.
* The debian/rules file may fail if no kernel-headers is in that location.
* The debian/rules file should handke KMAINT and KEMAIL env variables.

What do you think?

Regards,

// Ola

> Regards,
> 
>   Joey
> 
> -- 
> Ten years and still binary compatible.  -- XFree86
> 
> Please always Cc to me when replying to me on the lists.
> 
> 
> -- 
> To UNSUBSCRIBE, email to [EMAIL PROTECTED]
> with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
> 

-- 
 - Ola Lundqvist ---
/  [EMAIL PROTECTED] Annebergsslingan 37  \
|  [EMAIL PROTECTED] 654 65 KARLSTAD  |
|  +46 (0)54-10 14 30  +46 (0)70-332 1551   |
|  http://www.opal.dhs.org UIN/icq: 4912500 |
\  gpg/f.p.: 7090 A92B 18FE 7994 0C36  4FE4 18A1 B1CF 0FE5 3DD9 /
 ---

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Mon, May 19, 2003 at 05:13:54AM +1000, Russell Coker wrote:

> Definately.  This should be done if only to avoid multiple copies of a 27M
> bzip2 archive wasting everyone's disk space and network bandwidth.
> 
> Also regarding the i386 arch, multiple patches would be good.  Something
> in the i386 patch gives me lots of 'D' state processes, if the patch was
> separated into a few smaller patches then it would be easier to track down
> the cause.

The kernel-source diff used to be quite manageable; this iteration seems to
be much heavier than in the past (compare README.Debian).  IMO, our base
kernel-source diff should contain only things that absolutely everyone
wants, like security and critical fixes, and things necessary for our boot
process and kernel-image packages to work.  Fixing typos and compiler
warnings seems like it causes more problems than it solves, especially with
regard to patch conflicts.  "Value added" modifications would be more
manageable in a separate kernel-patch package.

> Next we need a mechanism for tracking which version of each patch was used in 
> the compilation.  Was your kernel compiled with version 0 of the security 
> patch or version 1 which has the ptrace fix?

See my notes about dh-kpatches elsewhere in this thread.

-- 
 - mdz

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Sun, May 18, 2003 at 12:06:21PM -0500, Manoj Srivastava wrote:

>   There is also a mechanism to order the order in which
>  kernel-patches are applied -- so if, say, a m68k kernel image
>  maintainer wanted to create a patch relative to the i386 patches,
>  they could depend on that patch, and order the m68k kernel-pacth to
>  be applied later in the chain than the i386 patch. 
> 
>   This dependency-and-ordering mechanism could be extended to
>  third party modules.
> 
>   People interested in hammering out details of this mechanism,
>  and kernel image maintainers, please contact me; perhaps it is time
>  to create policy for kernel patches. 

dh-kpatches provides a dependency/ordering facility which has worked well
for me in my packages.  It also provides
/usr/share/doc/kernel-image-foo/applied-patches documenting the package and
version for each patch that is applied.  I think this would be a good
starting point for such a policy, since it is already being applied in
Debian.

-- 
 - mdz

Re: Maintaining kernel source in sarge

[Matt Zimmerman]

On Mon, May 19, 2003 at 09:33:48AM +1000, Brian May wrote:

> Looking at woody in fact, it appears to only exceptions appear to be
> HPPA and IA64:
> 
> kernel-source-2.2.22 - Linux kernel source for version 2.2.22
> kernel-source-2.4.10 - Linux kernel source for version 2.4.10
> kernel-source-2.4.14 - Linux kernel source for version 2.4.14
> kernel-source-2.4.16 - Linux kernel source for version 2.4.16
> kernel-source-2.4.17 - Linux kernel source for version 2.4.17
> kernel-source-2.4.17-hppa - Linux kernel source for version 2.4.17 on HPPA
> kernel-source-2.4.17-ia64 - Linux kernel source for version 2.4.17 on IA-64
> kernel-source-2.4.18 - Linux kernel source for version 2.4.18
> kernel-source-2.4.18-hppa - Linux kernel source for version 2.4.18 on HPPA
> kernel-source-2.4.19 - Linux kernel source for version 2.4.19
> kernel-source-2.4.20 - Linux kernel source for version 2.4.20 with Debian 
> patches

2.2.22 has i386-{compact,idepci} and some alpha kernel images
2.4.10 has no reason to even be in the archive as far as I can tell
2.4.14 has an associated m68k patch, but no kernel images
2.4.16 has i386 and ARM kernel images
2.4.17 has powerpc, hppa, ia64, mips and mipsel kernel images
2.4.18 has i386, alpha, hppa, powerpc, and sparc kernel images
2.4.19 has mips and sun4u kernel images
2.4.20 did not ship with woody

We could practically shrink woody by one CD if we could consolidate some of
these.

In addition, we have:

kernel-image-2.2.10-apus |  2.2.10-13 |stable | powerpc
kernel-image-2.2.10-powerpc-apus |  2.2.10-13 |stable | source
kernel-image-2.2.19-netwinder | 20011109.1 |stable | source, arm
kernel-image-2.2.19-riscpc |   20011109 |stable | source, arm
kernel-image-2.2.20 |   2.2.20-5 |stable | i386
kernel-image-2.2.20-amiga |   2.2.20-2 |stable | source, m68k
kernel-image-2.2.20-atari |   2.2.20-1 |stable | source, m68k
kernel-image-2.2.20-bvme6000 |   2.2.20-1 |stable | source, m68k
kernel-image-2.2.20-chrp |   2.2.20-3 |stable | powerpc
kernel-image-2.2.20-compact |   2.2.20-5 |stable | i386
kernel-image-2.2.20-i386 |   2.2.20-5 |stable | source
kernel-image-2.2.20-idepci |   2.2.20-5 |stable | i386
kernel-image-2.2.20-mac |   2.2.20-1 |stable | source, m68k
kernel-image-2.2.20-mvme147 |   2.2.20-1 |stable | source, m68k
kernel-image-2.2.20-mvme16x |   2.2.20-1 |stable | source, m68k
kernel-image-2.2.20-pmac |   2.2.20-3 |stable | powerpc
kernel-image-2.2.20-prep |   2.2.20-3 |stable | powerpc
kernel-image-2.2.20-reiserfs |   2.2.20-4 |stable | i386
kernel-image-2.2.20-reiserfs-i386 |   2.2.20-4 |stable | source
kernel-image-2.2.20-sun4cdm |  9 |stable | sparc
kernel-image-2.2.20-sun4dm-smp |  9 |stable | sparc
kernel-image-2.2.20-sun4u |  9 |stable | sparc
kernel-image-2.2.20-sun4u-smp |  9 |stable | sparc

for which there is no corresponding kernel-source in woody.  This means that
these kernels cannot be built on woody, and this is very problematic for
support (specifically security fixes).  There seem to be about 85
kernel-image packages in woody, and I have no idea how many of them can
actually be built, much less patched and supported in any sane way.

To fix the ptrace bug, the s390 and mips architectures rolled the ptrace
security fix into kernel-patch-2.4.17-s390 and
kernel-patch-2.4.{17,19}-mips.  This makes things even worse, because if
kernel-source-2.4.{17,19} are patched to contain the fix, it will almost
certainly make these architectures' kernel images fail to build because of
patch conflicts, and require that the -patch packages be updated _again_ to
undo this.

> user-mode-linux builds by depending on "kernel-source-2.4.20" and
> automatically applying the UML patch, I wonder if the same thing could
> also be doine for -hppa and/or -ia64?

UML doesn't build a kernel-source package at all; it works essentially like
kernel-image-x.y.z-i386.  That is, it build-depends on kernel-source-x.y.z
and kernel-patch-uml, and builds with those.

This approach has its share of problems, of course.  In fact, to use UML as
an example again, I believe user-mode-linux will not build in woody because
it build-depends on the exact version of kernel-patch-uml that it expects,
and 'testing' apparently does not care about build-dependencies.

The ideal solution would be to be able to share tarballs between source
packages.  Then, all of the kernel-image packages could be built as if they
had a complete kernel source tree as their source package (which simplifies
things a lot), and yet we would only need one such tarball in the archive.
Of course, I have little idea how much work this would be to implement,
except that it would touch a lot of different tools.  I don't have the time
anyway.

Making 'testing' aware of build-dependencies seems like it should be
significantly more straig

Re: Maintaining kernel source in sarge

[Brian May]

On Sun, May 18, 2003 at 04:52:54PM +0200, Martin Schulze wrote:
> I also wonder if there are efforts in progress to unify the kernel
> source through more than two architectures?  This would require a
> group or architecture maintainers (current kernel package mantainers)
> to work collaboratively towards this goal.

I thought this was already the case?

Looking at woody in fact, it appears to only exceptions appear to be
HPPA and IA64:

kernel-source-2.2.22 - Linux kernel source for version 2.2.22
kernel-source-2.4.10 - Linux kernel source for version 2.4.10
kernel-source-2.4.14 - Linux kernel source for version 2.4.14
kernel-source-2.4.16 - Linux kernel source for version 2.4.16
kernel-source-2.4.17 - Linux kernel source for version 2.4.17
kernel-source-2.4.17-hppa - Linux kernel source for version 2.4.17 on HPPA
kernel-source-2.4.17-ia64 - Linux kernel source for version 2.4.17 on IA-64
kernel-source-2.4.18 - Linux kernel source for version 2.4.18
kernel-source-2.4.18-hppa - Linux kernel source for version 2.4.18 on HPPA
kernel-source-2.4.19 - Linux kernel source for version 2.4.19
kernel-source-2.4.20 - Linux kernel source for version 2.4.20 with Debian 
patches

user-mode-linux builds by depending on "kernel-source-2.4.20" and
automatically applying the UML patch, I wonder if the same thing could
also be doine for -hppa and/or -ia64?
-- 
Brian May <[EMAIL PROTECTED]>

Re: Maintaining kernel source in sarge

[Russell Coker]

On Mon, 19 May 2003 03:06, Manoj Srivastava wrote:
>   When I first envisaged the kernel source and kernel-patch
>  system, I always figured there should be a single source package per
>  version -- the one you get from kernel.org. *EVERY* arch, including
>  i386, should provided a kernel-patch package with all the changes for
>  their arch based on the pristine kernel source.

Definately.  This should be done if only to avoid multiple copies of a 27M 
bzip2 archive wasting everyone's disk space and network bandwidth.

Also regarding the i386 arch, multiple patches would be good.  Something in 
the i386 patch gives me lots of 'D' state processes, if the patch was 
separated into a few smaller patches then it would be easier to track down 
the cause.

>   There is also a mechanism to order the order in which
>  kernel-patches are applied -- so if, say, a m68k kernel image
>  maintainer wanted to create a patch relative to the i386 patches,
>  they could depend on that patch, and order the m68k kernel-pacth to
>  be applied later in the chain than the i386 patch.

Or if there's a patch that everyone needs such as the ptrace patch then only 
one package needs to provide it and everything else can depend on it.  Why 
not have a "security" kernel patch package which starts out as a zero byte 
file and gets bigger as the need dictates.

Next we need a mechanism for tracking which version of each patch was used in 
the compilation.  Was your kernel compiled with version 0 of the security 
patch or version 1 which has the ptrace fix?

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

Re: Maintaining kernel source in sarge

[Manoj Srivastava]

On Sun, 18 May 2003 16:52:54 +0200, Martin Schulze <[EMAIL PROTECTED]> said: 

> I also wonder if there are efforts in progress to unify the kernel
> source through more than two architectures?  This would require a
> group or architecture maintainers (current kernel package
> mantainers) to work collaboratively towards this goal.

When I first envisaged the kernel source and kernel-patch
 system, I always figured there should be a single source package per
 version -- the one you get from kernel.org. *EVERY* arch, including
 i386, should provided a kernel-patch package with all the changes for
 their arch based on the pristine kernel source.

There is also a mechanism to order the order in which
 kernel-patches are applied -- so if, say, a m68k kernel image
 maintainer wanted to create a patch relative to the i386 patches,
 they could depend on that patch, and order the m68k kernel-pacth to
 be applied later in the chain than the i386 patch. 

This dependency-and-ordering mechanism could be extended to
 third party modules.

People interested in hammering out details of this mechanism,
 and kernel image maintainers, please contact me; perhaps it is time
 to create policy for kernel patches. 

manoj
-- 
Chip Salzenberg sent me a complete patch to add System V IPC (msg, sem
and shm calls), so I added them.  If that bothers you, you can always
undefine them in config.sh.  :-) --Larry Wall in <[EMAIL PROTECTED]>
Manoj Srivastava   <[EMAIL PROTECTED]>  
1024R/C7261095 print CB D9 F4 12 68 07 E4 05  CC 2D 27 12 1D F5 E8 6E
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C