Re: lintian for Emdebian

[Russ Allbery]

Hi Neil,

I'm really sorry that it took me nine months to respond to this message.
It had been sitting around waiting for me to find some time to think about
the issues that you raise with adding overrides with Lintian custom checks
and trying to figure out how to approach this problem.

Neil Williams  writes:

> No, the problem appears to be only when running the checks due to the
> arbitrary way that perl indexes hashes:

> # perform checks
> for my $check (keys %checks) {
> my $ci = $check_info{$check};

> Sorting those keys puts manpages behind emdebian but then puts copyright
> ahead of emdebian which means that the copyright overrides are missed
> instead. :-(

> I can hack around it by then making the emdebian checks to be 00emdebian
> etc. but that is ugly.

I think I'm going to toss a sort in there for future versions of Lintian
just to make the results predictable (it's always much easier to test when
things happen in predictable sequences), but I agree that relying on the
order in which checks are loaded and using file names to force that is
ugly.  I think that's a sign that we'd be going down the wrong path.

> Even that needs some flexibility because I currently use
> Tags::add_override to selectively override tags dependent on the type of
> package so that I don't get too many unused overrides.

Hm, yes, that's a fundamental problem with using overrides for this.
Lintian wants people to clean up overrides.  What you really want is
something akin to an override, but which isn't treated the same way, isn't
processed as part of the package, and isn't expected to be active in any
particular case.

> What the code does not achieve is only loading this global override file
> upon request of the check script or under defined circumstances.

> I don't want to remove these warnings from lintian when, e.g. sponsoring
> packages for Debian - only when building packages for Emdebian.

> To do that, I'd need this code to be wrapped in a command-line option
> because whenever the check script is loaded it will be too late to
> prevent the existing checks from being removed.

This is a really good point.  I hadn't thought about this.

> It would be a lot cleaner if I could disable other checks within the
> check script itself by making Tags::add_override reliable and
> predictable. I wouldn't need a global override directory if there was a
> hook in lintian to set overrides *before* any other &run() subroutines
> are actually run (including my own). i.e. a second function in the check
> script Package::foo that lintian calls when the package has been
> unpacked but before any &run() functions are called.

The ongoing concern that I have about this approach is that it feels
unclean to me to have particular Lintian check modules be able to modify
the results of other check modules.  It feels like a layering violation,
and like the check concept is being a bit "abused" in order to introduce
code that isn't really related to that specific set of checks.

For the next version of Lintian, I've just implemented something a bit
different that works at a different level.  I'm not sure if you'll like it
very much, but I think it will accomplish what you want to accomplish.

What I've done is add two flags.  --suppress-tags takes a comma-separated
list of tags.  --suppress-tags-from-file takes the name of a file listing
tags (or comments).  Any suppressed tags will be treated as if they don't
exist (although they will still be run and the corresponding collect
scripts will still be run, since there's no good way to avoid that).  They
will not be output and will not change Lintian's exit status.

I think this will accomplish what you need by way of:

lintian --suppress-tags-from-file /path/to/emdebian/suppress 

Obviously this can be an alias, like emdebian-lintian.

What this doesn't do is selectively suppress tags based on the version of
the package, of course.  One has to remember to run the emdebian version
of the Lintian alias on the package instead of the regular Lintian.
However, it would be possible to do this in a wrapper that examines the
file names that are being checked and selectively adds that flag based on
the versions extracted from the file names, and that would work in most
cases.

The other advantage of this approach is that it's useful for other
purposes, such as using Lintian to check an internal package repository at
some site that wants to selectively ignore some of Lintian's tags as not
useful for their local policies.

If you don't think this will work for your problem, please do let me know
and we can talk some more about other alternatives, such as a separate
layer of code that can be run before the checks.

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-lint-maint-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: lintian for Emdebian

[Neil Williams]

On Thu, 2008-04-03 at 16:59 -0700, Russ Allbery wrote:
> Neil Williams <[EMAIL PROTECTED]> writes:
> 
> We should support a global overrides file or directory.  I'd be happy to
> add that functionality, but it does probably require code changes now that
> I think about this some more.

Yes, it would need to either be duplicated into %info under the $file
key of each package or held as a separate reference and checked in
check_overrides() of Tags.pm or simply duplicate the existing code to
read package overrides:
unless ($no_override) {
if (open(O, '<', "$base/override")) {
while () 
which means reading the global override file more than once.

I've played with a global override directory and I've included some
working perl code below. However, there are problems with using this as
a global and I've also included sample code for an alternative method
that does not involve a global directory but actually retains complete
control over the overrides within the specific check script itself. (So
this email is a bit long.) :-)

> > However, I'm finding it impossible to override the manpage warnings
> > without removing all the manpages checks with -X :
> >
> > I don't actually need the manpages checks but I would like to not have
> > to alias lintian to lintian -X manpages.
> 
> Where are you adding the overrides?  Is it possible that the manpage check
> script is happening before your overrides are added?

> > Is there a way of implementing '-X' within the check scripts? (It would
> > save time if I could drop entire scripts like manpages and replace with
> > a simple check that /usr/share/man/ is empty).
> 
> You could divert the manpages file, but there isn't any way to skip an
> entire check from within another check (in part because the order in which
> the checks are run is not, I think, fully deterministic, although I could
> be wrong -- I think it runs them in glob order).

Not AFAICT at the readdir stage:
$ perl -le 'opendir DIR, "/usr/share/lintian/checks"; print join " ", readdir 
DIR'

emdebian appears there before manpages.
. .. changelog-file.desc po-debconf.desc emdebian init.d scripts
copyright-file.desc emdebian.desc menu-format description.desc
common_data.pm shared-libs md5sums etcfiles menu-format.desc cruft.desc
debconf files standards-version menus.desc fields conffiles.desc
standards-version.desc manpages.desc conffiles control-file.desc
etcfiles.desc nmu.desc huge-usr-share.desc debconf.desc infofiles
files.desc control-files description fields.desc cruft init.d.desc nmu
debian-readme.desc copyright-file menus infofiles.desc debhelper rules
manpages

No, the problem appears to be only when running the checks due to the
arbitrary way that perl indexes hashes:

# perform checks
for my $check (keys %checks) {
my $ci = $check_info{$check};

Sorting those keys puts manpages behind emdebian but then puts copyright
ahead of emdebian which means that the copyright overrides are missed
instead. :-(

I can hack around it by then making the emdebian checks to be 00emdebian
etc. but that is ugly.

> I think the right approach may be to create a directory
> /usr/share/lintian/overrides/global and load all files in that directory
> in addition to the regular overrides.

Even that needs some flexibility because I currently use
Tags::add_override to selectively override tags dependent on the type of
package so that I don't get too many unused overrides.

My current override set is:

binary-or-shlib-defines-rpath
binary-without-manpage
build-depends-indep-without-arch-indep
changelog-should-mention-nmu
debian-files-list-in-source
debian-rules-missing-required-target
extended-description-is-empty
native-package-with-dash-version
no-copyright-file
no-md5sums-control-file
python-script-but-no-python-dep
source-nmu-has-incorrect-version-number

Of those, some apply only to the source package, some only to binaries,
some only to the Emdebian TDebs that are a completely different issue
(eventually, those will need something like the current lintian support
for udebs).

e.g.

elsif (($info =~ /GNU message catalog/) and ($tdeb > 0))
{
Tags::add_override ("extended-description-is-empty");
Tags::add_override ("no-md5sums-control-file");
Tags::add_override ("no-copyright-file");
# need TDeb checks here.
Tags::add_override ("debian-rules-missing-required-target *");
# might want to fix this one.
Tags::add_override ("debian-files-list-in-source");
Tags::add_override ("native-package-with-dash-version");

This code works for me (without the extra flexibility):
Adding to the existing code at:

unless ($no_override) {
if (open(O, '<', "$base/override")) {
while () {
chomp;
next if m,^\s*(\#|\z),o;
s/^\s+//o;
s/\s+$//o;
s/\s+/ /go;
my $override = $_;

Re: lintian for Emdebian

[Russ Allbery]

Neil Williams <[EMAIL PROTECTED]> writes:

> Hmmm, can't find any docs on a "global lintian override file" - where
> would that live and how would it cope with unknown/unbuilt packages?

Ah, hm, we may not have this functionality right now, now that I think
about it.  I think the current override format always requires the package
patch the file name.

We should support a global overrides file or directory.  I'd be happy to
add that functionality, but it does probably require code changes now that
I think about this some more.

> So far, I'm working with:
> Tags::add_override ("source-nmu-has-incorrect-version-number");
> Tags::add_override ("changelog-should-mention-nmu");
> Tags::add_override ("binary-or-shlib-defines-rpath");
> tag "binary-or-shlib-omits-rpath", "$file $RPATH{$file}";
> and others

Good idea and good workaround.

> However, I'm finding it impossible to override the manpage warnings
> without removing all the manpages checks with -X :
>
> Tags::add_override ("binary-without-manpage");
> Tags::add_override ("binary-without-manpage apt-cache");
> Tags::add_override ("apt", "binary-without-manpage apt-cache");
> Tags::add_override ("binary-without-manpage usr/bin/apt-cache");
> Tags::add_override ("apt", "binary-without-manpage usr/bin/apt-cache");
> Tags::add_override ("apt", "binary-without-manpage ./usr/bin/apt-cache");
> Tags::add_override ("binary-without-manpage apt-cache");
>
> Still, I get:
> W: apt: binary-without-manpage usr/bin/apt-cache
> W: apt: binary-without-manpage usr/bin/apt-cdrom
> W: apt: binary-without-manpage usr/bin/apt-config
> W: apt: binary-without-manpage usr/bin/apt-get
> W: apt: binary-without-manpage usr/bin/apt-key
> W: apt: binary-without-manpage usr/bin/apt-mark
> I: apt: unused-override binary-without-manpage
> I: apt: unused-override binary-without-manpage apt-cache
> I: apt: unused-override binary-without-manpage usr/bin/apt-cache
>
> I don't actually need the manpages checks but I would like to not have
> to alias lintian to lintian -X manpages.

Where are you adding the overrides?  Is it possible that the manpage check
script is happening before your overrides are added?

> Is there a way of implementing '-X' within the check scripts? (It would
> save time if I could drop entire scripts like manpages and replace with
> a simple check that /usr/share/man/ is empty).

You could divert the manpages file, but there isn't any way to skip an
entire check from within another check (in part because the order in which
the checks are run is not, I think, fully deterministic, although I could
be wrong -- I think it runs them in glob order).

I think the right approach may be to create a directory
/usr/share/lintian/overrides/global and load all files in that directory
in addition to the regular overrides.

-- 
Russ Allbery ([EMAIL PROTECTED])   


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

Re: lintian for Emdebian

[Neil Williams]

On Tue, 2008-04-01 at 14:09 -0700, Russ Allbery wrote:
> Neil Williams <[EMAIL PROTECTED]> writes:
> 
> My recommendation is to try to tackle this on two fronts:
> 
> * For the new checks specific to emdebian, add new check files.  You can
>   do this in a completely separate package and just depend on lintian.

OK, that's working, thanks Russ.

Current test code:
http://buildd.emdebian.org/svn/browser/current/host/trunk/emdebian-tools/trunk/checks

>   Lintian when run will pick up any files in /usr/share/lintian/checks, so
>   you can just drop more checks and *.desc files in there.  You can have
>   those checks do nothing if they detect they're not running in the
>   emdebian mode somehow.
> 
> * For the Lintian checks that are wrong for emdebian, I would just add a
>   global override file for emdebian. 

Hmmm, can't find any docs on a "global lintian override file" - where
would that live and how would it cope with unknown/unbuilt packages?

So far, I'm working with:
Tags::add_override ("source-nmu-has-incorrect-version-number");
Tags::add_override ("changelog-should-mention-nmu");
Tags::add_override ("binary-or-shlib-defines-rpath");
tag "binary-or-shlib-omits-rpath", "$file $RPATH{$file}";
and others

Those all work fine.

However, I'm finding it impossible to override the manpage warnings
without removing all the manpages checks with -X :

Tags::add_override ("binary-without-manpage");
Tags::add_override ("binary-without-manpage apt-cache");
Tags::add_override ("apt", "binary-without-manpage apt-cache");
Tags::add_override ("binary-without-manpage usr/bin/apt-cache");
Tags::add_override ("apt", "binary-without-manpage usr/bin/apt-cache");
Tags::add_override ("apt", "binary-without-manpage ./usr/bin/apt-cache");
Tags::add_override ("binary-without-manpage apt-cache");

Still, I get:
W: apt: binary-without-manpage usr/bin/apt-cache
W: apt: binary-without-manpage usr/bin/apt-cdrom
W: apt: binary-without-manpage usr/bin/apt-config
W: apt: binary-without-manpage usr/bin/apt-get
W: apt: binary-without-manpage usr/bin/apt-key
W: apt: binary-without-manpage usr/bin/apt-mark
I: apt: unused-override binary-without-manpage
I: apt: unused-override binary-without-manpage apt-cache
I: apt: unused-override binary-without-manpage usr/bin/apt-cache

I don't actually need the manpages checks but I would like to not have
to alias lintian to lintian -X manpages.

Is there a way of implementing '-X' within the check scripts? (It would
save time if I could drop entire scripts like manpages and replace with
a simple check that /usr/share/man/ is empty). (Have I got the overrides
wrong? I've tried various formats.)

W: apt: binary-without-manpage usr/bin/apt-cache
I: apt: unused-override binary-without-manpage
I: apt: unused-override binary-without-manpage apt-cache
I: apt: unused-override binary-without-manpage usr/bin/apt-cache

Tags::add_override ("binary-without-manpage");
Tags::add_override ("binary-without-manpage *");
Tags::add_override ("binary-without-manpage apt-cache");
Tags::add_override ("binary-without-manpage usr/bin/apt-cache");

You can get an example .deb from the Emdebian repository:
http://www.emdebian.org/packages/pool/main/a/apt/

$ lintian -q -C emdebian apt_0.7.9em2_arm.deb 
E: apt: unsupported-interpreter-in-binary usr/bin/apt-mark
E: apt: unsupported-interpreter-in-binary usr/lib/dpkg/methods/apt/setup

(These are errors in the current Emdebian package which I need to fix -
a failure with 'lintian -q -C emdebian' will cause a FTBFS in Emdebian
once this support is implemented in emdebian-tools 1.0.0 - that call
could easily be 'lintian -q -X manpages -C emdebian' but I'd like to
know if I've done something wrong in my manpage overrides.)

$ lintian -I apt_0.7.9em2_arm.deb 
W: apt: binary-without-manpage usr/bin/apt-cache
W: apt: binary-without-manpage usr/bin/apt-cdrom
W: apt: binary-without-manpage usr/bin/apt-config
W: apt: binary-without-manpage usr/bin/apt-get
W: apt: binary-without-manpage usr/bin/apt-key
W: apt: binary-without-manpage usr/bin/apt-mark
E: apt: unsupported-interpreter-in-binary usr/bin/apt-mark
E: apt: unsupported-interpreter-in-binary usr/lib/dpkg/methods/apt/setup
W: apt: package-name-doesnt-match-sonames libapt-pkg-libc6.6-6-4.6
I: apt: unused-override binary-or-shlib-defines-rpath
I: apt: unused-override binary-without-manpage
I: apt: unused-override build-depends-indep-without-arch-indep
N: 2 tags overridden (2 errors)


>  If you need additional Lintian
>   support for override files that are more overridey than normal
>   (suppressing tags even with --show-overrides, for example) or are
>   selectively loaded, we can try to work out generic changes to the driver
>   scripts for this.

Thanks, I'll work through the current package set and build up a list of
lintian messages and overrides.

-- 


Neil Williams
=
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/




signature.asc
Description: This is a digitally 

Re: lintian for Emdebian

[Russ Allbery]

Neil Williams <[EMAIL PROTECTED]> writes:

> Now I'm quite familiar with Perl (all the emdebian build tools are in
> Perl), I could come up with compatible Perl modules along the lines of
> Lintian::binaries::Emdebian or Lintian::Cross but I want to be able to
> fold these checks into the standard lintian checks so that I don't just
> create a fork that will lag behind lintian and eventually be too
> incompatible to maintain. (We did that with dpkg-dev and dpkg-cross and
> it took nearly 10 years to get our changes back into dpkg-dev.)
>
> So I'm looking for ideas. I want to be able to selectively disable
> checks *within* lintian packages - e.g. the RPATH issue - whilst
> retaining the other checks in that package, modifying the importance of
> some and adding new ones. Simply disabling the entire set of checks is
> not worthwhile - I'd have to maintain 90% of the original check file as
> a fork.

My recommendation is to try to tackle this on two fronts:

* For the new checks specific to emdebian, add new check files.  You can
  do this in a completely separate package and just depend on lintian.
  Lintian when run will pick up any files in /usr/share/lintian/checks, so
  you can just drop more checks and *.desc files in there.  You can have
  those checks do nothing if they detect they're not running in the
  emdebian mode somehow.

* For the Lintian checks that are wrong for emdebian, I would just add a
  global override file for emdebian.  If you need additional Lintian
  support for override files that are more overridey than normal
  (suppressing tags even with --show-overrides, for example) or are
  selectively loaded, we can try to work out generic changes to the driver
  scripts for this.

-- 
Russ Allbery ([EMAIL PROTECTED])   


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]

lintian for Emdebian

[Neil Williams]

I've been looking over the lintian source code with a view to
implementing some form of lintian support for crossbuilt packages in
Emdebian.

Common differences between Emdebian packages and standard Debian policy
include:

1. Removal of documentation, even those files mandated by Debian policy
like manpages, leading to empty virtual packages and empty (or entirely
missing) -doc packages. /usr/share/doc should be almost empty.
2. Architecture-checks on all files in ./*/?bin and ./*/lib to ensure
that the build has completed successfully without the native compiler
being called accidentally (quite common) or due to a patch failure.
3. Removal of all /usr/share/locale/ content unless the package is an
Emdebian TDeb.
4. Complete ban on the use of perl in maintainer scripts, including the
use of any command that is implemented in perl like update-alternatives,
until such time as Emdebian has a C replacement.
5. Complete ban on trying to process any file in a maintainer script
that has been removed for Emdebian (like install-info).
6. Checks on Emdebian TDebs to ensure the presence of the POT file in
the source, the presence of only .mo files in the package and a ban on
empty TDeb packages.
7. No "Essential" tags in debian/control, ever.
8. No unsupported interpreters (python etc.) and no expectation of
running /bin/bash - all /bin/sh only (actually must be able to use
busybox dash but I'll deal with that on a package-specific basis via the
BTS.)
9. Mandatory use of RPATH - any package that has removed RPATH for
Debian will find it exists again after a cross build and this is
essential. A lack of an RPATH may indicate a crossbuild failure.
10. No attempts to use network access during package configuration -
this one is harder but it is important because a lot of embedded devices
will have no possible network connection during installation - serial
yes, network, no. (Installation media == USB keys or SD cards.)

Those are just the top 10. :-)

There are probably another 25 that I can think of.

Now I'm quite familiar with Perl (all the emdebian build tools are in
Perl), I could come up with compatible Perl modules along the lines of
Lintian::binaries::Emdebian or Lintian::Cross but I want to be able to
fold these checks into the standard lintian checks so that I don't just
create a fork that will lag behind lintian and eventually be too
incompatible to maintain. (We did that with dpkg-dev and dpkg-cross and
it took nearly 10 years to get our changes back into dpkg-dev.)

So I'm looking for ideas. I want to be able to selectively disable
checks *within* lintian packages - e.g. the RPATH issue - whilst
retaining the other checks in that package, modifying the importance of
some and adding new ones. Simply disabling the entire set of checks is
not worthwhile - I'd have to maintain 90% of the original check file as
a fork.

I want to be able to do this largely independently of lintian itself so
that I don't have to bug you for changes in lintian every time Emdebian
makes a new Policy decision. (Emdebian Policy is extremely fluid at this
time but will be mostly compatible with Debian Policy within the
constraints of an embedded device.)

Within the above 10 items, some are even more subtle because if, for
example, someone uses Emdebian patches and Emdebian build tools to
prepare an i386 package on i386 (e.g. for Eeee PC etc.) then the RPATH
issue may disappear and the check would need to revert to Debian
behaviour. Necessarily, the actual code running these tests should
remain in the Emdebian build tools packages (emdebian-tools) so that
these can be maintained without expecting lintian maintainers to test
with cross builds.

A basic subset of these checks is already implemented in the
emdebian-tools build script but wider and more subtle support is
necessary, along with proper support for the rest of the lintian checks.

It may also be a good idea if Emdebian can modify the importance of
various existing lintian checks so that we can downplay errors and
warnings that really only matter within the wider Debian context.

I don't want to spend too much time discussing the actual checks, I
described the 10 above to give some flavour of the extent of the changes
and the required granularity of the changes. 

I'm willing to do the work, all I need from you is some direction and
possible means of creating a suitable interface or wrapper. I'm quite
happy writing 'emlintian' if I know how it could remain a wrapper
without becoming a fork.

How could this be achieved?

-- 


Neil Williams
=
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/




signature.asc
Description: This is a digitally signed message part