Bug#964723: Ordering of entries: the calibre opens everything problem

[Vincent Lefevre]

Hi,

For security reasons, I don't think that xdg-open should be used until
one can force the MIME type (which is provided in mail messages).

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Bug#964723: Ordering of entries: the calibre opens everything problem

[Enrico Zini]

Package: mime-support
Version: 3.62
Severity: important

Hello,

thank you for maintaining mime-support.

>From what I understand, entries in /etc/mailcap are sorted by package
name. As a consequence, people who install, for example, calibre, will
find that it opens quite a lot of file types (which it can do), with a
higher priority over the software one would expect: calibre comes
alphabetically before evince, okular, libreoffice, and most other
packages.

There is only a limited way to control this system-wide with
/etc/mailcap.order, and no way for users to control ordering.

As a user, I can grep /etc/mailcap and paste my preferred entries in
~/.mailcap, but at that point I have hardcoded those command lines in a
way outside the control of package maintainers.

Suppose that a package has an exploitable command line, and given
#928037 it might well be, and a new version fixes it in its mime
information, my ~/.mailcap will be stuck with the exploitable version
forever.

Suppose that a package changes command line options and updates its mime
information correctly, my ~/.mailcap will be stuck with a nonfunctioning
command line, and I'll spend an afternoon trying to figure out why mutt
suddenly can't open PDFs anymore, retracing and relearning the chain of
commands and configuration files that go on behind the scenes. I
personally found it really nontrivial to figure out.

I'm trying to resist the temptation to write and package a tool called
aaanything that uses aalib to display as many file types as possible,
and provides mime informations accordingly, so that it will sort before
pretty much any other package.

What I did as a workaround in my system[1], was to add entries in my
~/.mailcap that delegate to xdg-open, like:

  application/pdf; xdg-open %s && sleep 0.3s; test=test -n "$DISPLAY"

Is there a reason this kind of behaviour could not be a default?

I'm opening this issue to document this kind of surprising and possibly
dangerous behaviour coming out of the way mime-support works.

I can't think of many options besides delegating to xdg-open: there
doesn't seem a way for packages to declare a priority for their mime
support, and even if there was, it would be arbitrary.

I could think of packaging something that diverted run-mailcap to a
wrapper to xdg-open, falling back to the original run-mailcap if
xdg-open doesn't have a mapping, or if DISPLAY is not set.

I could think of packaging a program called 0xdg-mime that depends on
xdg-utils and registers xdg-open as mime handler for as many entries as
possible.

I really don't feel good about the current situation being the standard
user interface for mime handling in Debian.


Enrico

[1] https://www.enricozini.org/blog/2020/debian/mime-type-associations/


-- System Information:
Debian Release: 10.4
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.6.0-0.bpo.2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_IE.UTF-8, LC_CTYPE=en_IE.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_IE:en (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

mime-support depends on no packages.

Versions of packages mime-support recommends:
ii  bzip2 1.0.6-9.2~deb10u1
ii  file  1:5.35-4+deb10u1
ii  xz-utils  5.2.4-1

mime-support suggests no packages.

-- no debconf information

Bug#964723: Ordering of entries: the calibre opens everything problem

[Charles Plessy]

Hi Enrico,

thanks a lot for your email, you are touching something important.

Frankly speaking, mailcap is old.  We are therefore stuck in a state
where changing it is very difficult and involves taking decisions that
are very "cleaving".  What is our priority: portability ? compatibility ?
ability to run on extermely low-powered processors ?  Or are we ready
to rewrite it, to innovate, to make major changes that may not be
adopted by other distros, to replace it by something entirely different ?

My own take on that problem is to make it "optional" instead of
"standard", so that people can develop alternatives, or try to entirely
live without it (probably using XDG as a replacement).

One problem is that the mime-support ships /etc/mime.types, which is
needed by many programs beyond the mailcap system.  I proposed to move
it to base-files and the answer was "I don't think it would be a good
idea".  So maybe we need a mime.types package, priority standard, that
contains this single file.  Or maybe you can help me to make Santiago
change his mind.  http://bugs.debian.org/787571

Second problem: I proposed on debian-devel to make mailcap optional in
2019 and the only answer I got was: "I don't really see a benefit to this."
https://lists.debian.org/debian-devel/2019/08/msg00306.html
Frankly speaking, this makes me fear that resistance can spark after I
would have invested my time in working on it.

(I also proposed to hijack /usr/bin/open so that Mac users would feel
at home, but as you can guess, the answer was kind of "no", or more
precisely: wait 5 years before seing it happen.  Actually, I wish I had
done it. https://lists.debian.org/debian-devel/2014/04/msg00822.html)

In summary, I think xdg-open could be the standard default on all
systems even if they do not have a Destkop environment installed.
Actually, xdg-open falls back on mailcap in these cases and I think that
a bunch of tools are already calling xdg-open directly.  So we would
need a tool that draws information from /usr/share/applications instead
of /usr/lib/mime, and voilà we would just have to make xdg-open aware
of it.

Thanks again for your thoughts; it is long overdue that the mailcap
systems evolves and at least stops standing in the way of change.  If I
can count on your support and on Debian's benevolence, I am more than
happy to push major changes.

Have a nice week-end,

Charles

-- 
Charles Plessy  Akano, Uruma, Okinawa, Japan
Debian Med packaging team http://www.debian.org/devel/debian-med
Tooting from work,   https://mastodon.technology/@charles_plessy
Tooting from home, https://framapiaf.org/@charles_plessy

Bug#964723: Ordering of entries: the calibre opens everything problem

[Charles Plessy]

Le Fri, Jul 10, 2020 at 02:02:31PM +0900, Charles Plessy a écrit :
> 
> need a tool that draws information from /usr/share/applications instead
> of /usr/lib/mime, and voilà we would just have to make xdg-open aware
> of it.

I just remembered it is more complicated: the FreeDesktop menu entries
do not contain priority values.  The informations used by the Desktop
Environment systems to sort entries in their menus are somewhere else,
and I have not found yet...

Bug#964723: Ordering of entries: the calibre opens everything problem

[Enrico Zini]

On Fri, Jul 10, 2020 at 02:02:31PM +0900, Charles Plessy wrote:

Thanks Charles for your reply!

> My own take on that problem is to make it "optional" instead of
> "standard", so that people can develop alternatives, or try to entirely
> live without it (probably using XDG as a replacement).

This sounds sensible to me.


> One problem is that the mime-support ships /etc/mime.types, which is
> needed by many programs beyond the mailcap system.  I proposed to move
> it to base-files and the answer was "I don't think it would be a good
> idea".  So maybe we need a mime.types package, priority standard, that
> contains this single file.  Or maybe you can help me to make Santiago
> change his mind.  http://bugs.debian.org/787571

I do understand Santiago's reasoning, and I don't see a problem in
having mime-support only provide /etc/mime.types, and depend or
recommend a mailcap package with lower priority, that other packages can
provide.

The mailcap package could Provides: run-mailcap, opening the way for
other packages to provide alternative implementations, like via
xdg-open.

In this way, packages that depend on mime-support for /etc/mime.types
would keep working, and I understand they are the majority. Packages
that depend on mime-support for run-mailcap or /etc/mailcap can keep
working because of the dependencies, and can get warned to depend on
run-mailcap or mailcap depending on what interfaces they need of it.


> Second problem: I proposed on debian-devel to make mailcap optional in
> 2019 and the only answer I got was: "I don't really see a benefit to this."
> https://lists.debian.org/debian-devel/2019/08/msg00306.html
> Frankly speaking, this makes me fear that resistance can spark after I
> would have invested my time in working on it.

My impression was, rather than of hostility, of not many people
understood the situation or cared much about it. Personally, I ignored
the existance of mime-support for 20 years, and got aware of it only
when I started to dig on why mutt opened files with the most annoying
possible program depending on machines.

I had had other mailcap related issues in the past, which were solved by
a quick google search and copypaste of some reasonable sounding line
from stackoverflow to ~/.mailcap without digging further.


> (I also proposed to hijack /usr/bin/open so that Mac users would feel
> at home, but as you can guess, the answer was kind of "no", or more
> precisely: wait 5 years before seing it happen.  Actually, I wish I had
> done it. https://lists.debian.org/debian-devel/2014/04/msg00822.html)

That seems like worth doing to me in general. In my system I do indeed
have openvt but not open, and I see 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=732796
has been fixed, liberating open for just this.


> In summary, I think xdg-open could be the standard default on all
> systems even if they do not have a Destkop environment installed.
> Actually, xdg-open falls back on mailcap in these cases and I think that
> a bunch of tools are already calling xdg-open directly.  So we would
> need a tool that draws information from /usr/share/applications instead
> of /usr/lib/mime, and voilà we would just have to make xdg-open aware
> of it.

Indeed!


> Thanks again for your thoughts; it is long overdue that the mailcap
> systems evolves and at least stops standing in the way of change.  If I
> can count on your support and on Debian's benevolence, I am more than
> happy to push major changes.

I'm happy to support improvements on this. Nobody can guarantee Debian's
benevolence, although I think discussions have become somewhat easier
since 2014.


On Fri, Jul 10, 2020 at 02:57:24PM +0900, Charles Plessy wrote:

> I just remembered it is more complicated: the FreeDesktop menu entries
> do not contain priority values.  The informations used by the Desktop
> Environment systems to sort entries in their menus are somewhere else,
> and I have not found yet...

I'm surprised hearing that priority values exist in mailcap, and PDFs
still open in Calibre. I had thought it just used alphabetical ordering.

Admittedly, the user experience is somewhat confused by some programs
using run-mailcap and some xdg-open. For example, right now in my system
I have a sane default for JPEG files in mailcap but not in xdg-mime:

   $ run-mailcap --norun /usr/share/gimp/2.0/scripts/images/texture1.jpg
   geeqie '/usr/share/gimp/2.0/scripts/images/texture1.jpg'
   $ xdg-mime query default image/jpeg
   darktable.desktop

But when I open a file using mutt, mc, or a file manager, it's not clear
which of the two backends is being making the wrong choice. So if I
opened a PDF in mutt, run-mailcap would frustrate me with calibre, and
if I clicked on a JPEG in pcmanfm, xdg-open would frustrate me with
darktable, but the UI hides from me what it is that I could debug.


How do priority values work at the moment, and are they being
effectively used?

I looked into https://wiki.debian.org/MimeTypesSuppor

Bug#964723: Ordering of entries: the calibre opens everything problem

[Charles Plessy]

Le Fri, Jul 10, 2020 at 11:19:48AM +0200, Enrico Zini a écrit :
> 
> I see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=732796
> has been fixed, liberating open

Thanks for the great news !  I did not realise before.

> I'm surprised hearing that priority values exist in mailcap, and PDFs
> still open in Calibre. I had thought it just used alphabetical ordering.

Quick answer: mailcap priority values, which might be an extension
specific to Debian, are explained in update-mime(8).

But many mailcap entries are generated from Desktop files, which can not
contain priority values.  As a consequence of the current Policy that
"packages already using desktop entries should not use the mailcap
system directly", mailcap files are being removed from packages, causing
priority information to be lost.

I do not necessarly disagree with the Policy.  The more packages Debian
has, the harder it would be to coordinate priority settings at this
level anyway.  The best solution will be to improve mailcap so that it
can find the FreeDesktop information where it is, or to add an
additional configuration file that provides robust system-wide defaults.

I will go ahead with the splitting of the mime-support package to
facilitate that work.  Please do not hesitate to ping me if I am too
slow: I still am very limited in the amount of free time I have in front
of the computer.

Cheers,

-- 
Charles Plessy
Debian Med packaging team,
http://www.debian.org/devel/debian-med
Akano, Uruma, Okinawa, Japan

Bug#964723: Ordering of entries: the calibre opens everything problem

[smcv]

On Fri, 10 Jul 2020 at 14:57:24 +0900, Charles Plessy wrote:
> I just remembered it is more complicated: the FreeDesktop menu entries
> do not contain priority values.

As far as I'm aware, this is deliberate, for two reasons:

* The authors and maintainers of individual viewers (evince, calibre,
  okular, etc.) are not well-placed to choose priorities, because
  *obviously* they think theirs is the best, otherwise they wouldn't be
  continuing to develop it. The best you're likely to get from priorities
  is that the maintainers of things that know they *shouldn't* be the
  default, like gimp opening PDFs (which it can, but that's rarely what
  anyone wants), can choose an artificially low priority.

* To get well-integrated desktop environments, the default choice of viewer
  ought to be desktop-specific: if a GNOME user and a KDE user share a
  computer, neither user has expressed any preference about their PDF
  viewer, and they both open a PDF, then the GNOME user ought to get
  evince and the KDE user ought to get okular. (Obviously the user's
  choice, *if they have expressed a choice*, should be prioritized higher
  than this - but we need sensible defaults.)

> The informations used by the Desktop
> Environment systems to sort entries in their menus are somewhere else,
> and I have not found yet...

/usr/share/applications/${desktop}-mimeapps.list, where ${desktop} is
one of the colon-separated values in $XDG_CURRENT_DESKTOP, transformed
into lower case. For example, GNOME sets XDG_CURRENT_DESKTOP=GNOME, and
gnome-session-common provides /usr/share/applications/gnome-mimeapps.list.

I think Ubuntu's Unity used to set XDG_CURRENT_DESKTOP=Unity:GNOME (read
as "Unity, which is similar to/based on GNOME"), which means it would use
/usr/share/applications/unity-mimeapps.list, falling back to the GNOME
list for anything not found there.

Full specification:
https://specifications.freedesktop.org/mime-apps-spec/latest/

smcv