Bug#633799: getmail4: Mboxrd format is not supported
Hi, On Thu, Oct 25, 2012 at 03:44:26PM +0200, Christoph Anton Mitterer wrote: > Hi. > > On Thu, 2012-10-25 at 21:26 +0900, Osamu Aoki wrote: > > But if you read history of bug report including patches, you could have > > written a bit kinder tone message. I feel a bit sad to see this message. > Well it wasn't meant particularly personally or offensive,... I just > think the issue is quite serious. > > I see now, that you considered this just to be a documentation > problem... > > IMHO, one needs to look throughout all Debian, to find any places where > mboxo is still used. Python ! http://docs.python.org/2/library/mailbox.html#mbox and links from there. It argues why it does so. > The problem is, that using mboxo itself (even if documented) is IMHO a > serious bug, as the format is utterly broken. H... I thought differently ... But I take this as a chance to improve after your good work with upstream. > Especially no user expects that when he stores mail it's being > irrecoverably cluttered up (which is what mboxo does). > Actually I'd say that most people even don't know that there are > different subformats of mbox. ... I know this is old discussion we do not repeat here... > > > really must warn our users on that issue. > > > And even if upstream would fix it, we still would need to warn our users > > > at least in the NEWS file / release notes... that all their mail from > > > previous years is likely corrupted. > > mboxo has been always so and have been widely used. > I know, and this is actually quite a problem. As I wrote above, most > people don't know this... and AFAIU the corruption inherent to this > format can't be undone. > > > mboxrd is technically superior. > Yes,... an alternative is mboxcl2... but it has also it's drawbacks. > > > > anyone who stores file in mbox should know there are risks as you > > describe. > Phew... I mean I wouldn't call myself uneducated ;-) ... and I was > really shocked when I learned about this recently. > I asked around at my friends, all studied computer scientists and decent > sysadmins... noone knew. I am not arguing this with you. Not all smart people know how much stupidity has been commited by human before. I do not expect it either. My point is it was knowen and documented issue as seen on python. This package only used python as is ... thus suffered. > > I think you are a bit exxagurating severity of trivial part of data > > change. > Actually most people seem to see it like this. I can't join that opinion > however. > I mean the change is little, arguably, but a) it can't be undone > automatically and b) given, that storing mail is the core functionality > of e.g. getmail, it think it's quite severe. That would be the same if > your paint program changes all the colours (just a tiny bit) when you > save your image. > > > > Let's ask release manager how this should be handled now. > Saw your mail :) Thanks for your efforts :) I realize that ML is not for discussion. I post this message to mark this bug actively cared. This bug fix will be my next work within a week :-) Osamu -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#633799: getmail4: Mboxrd format is not supported
Hi. On Thu, 2012-10-25 at 21:26 +0900, Osamu Aoki wrote: > But if you read history of bug report including patches, you could have > written a bit kinder tone message. I feel a bit sad to see this message. Well it wasn't meant particularly personally or offensive,... I just think the issue is quite serious. I see now, that you considered this just to be a documentation problem... IMHO, one needs to look throughout all Debian, to find any places where mboxo is still used. The problem is, that using mboxo itself (even if documented) is IMHO a serious bug, as the format is utterly broken. Especially no user expects that when he stores mail it's being irrecoverably cluttered up (which is what mboxo does). Actually I'd say that most people even don't know that there are different subformats of mbox. > No. This is not news. So not an appropriate place. > Right place should be more like README.Debian or BUGS. Mhh well in principle that's true, but to my experience, NEWS is the only place (apart from release-notes and debconfi dialgues) that is likely read by most people (well at least by those that have apt-listchanges installed). Does Debian expect people to read README.Debian? > > - if possibly in a priority-high dialogue via debconf > This is not right place per policy. Is that forbidden? I used to know several packages that put informative dialogues there (even though not with priority high, I guess). > > If upstream is stupid and doesn't want to fix this, then we really > Please do not call upstream stupid ... especially in public like this. > I think you are bright but this is not very polite. As I already told upstream himself,... I said _if_ (perhaps I should have written "even if")... So when I wrote this, I haven't thought you'd just consider this as a documentation issue. Therefore I wanted to make clear, that even when an upstream would be stupid/not cooperating/etc. ... we (in Debian) should not just lean back. > > really must warn our users on that issue. > > And even if upstream would fix it, we still would need to warn our users > > at least in the NEWS file / release notes... that all their mail from > > previous years is likely corrupted. > mboxo has been always so and have been widely used. I know, and this is actually quite a problem. As I wrote above, most people don't know this... and AFAIU the corruption inherent to this format can't be undone. > mboxrd is technically superior. Yes,... an alternative is mboxcl2... but it has also it's drawbacks. > anyone who stores file in mbox should know there are risks as you > describe. Phew... I mean I wouldn't call myself uneducated ;-) ... and I was really shocked when I learned about this recently. I asked around at my friends, all studied computer scientists and decent sysadmins... noone knew. > I think you are a bit exxagurating severity of trivial part of data > change. Actually most people seem to see it like this. I can't join that opinion however. I mean the change is little, arguably, but a) it can't be undone automatically and b) given, that storing mail is the core functionality of e.g. getmail, it think it's quite severe. That would be the same if your paint program changes all the colours (just a tiny bit) when you save your image. > Let's ask release manager how this should be handled now. Saw your mail :) Thanks for your efforts :) Best wishes, Chris. smime.p7s Description: S/MIME cryptographic signature
Bug#633799: getmail4: Mboxrd format is not supported
Hi, Problem is getmail documentation pretends it impliments mboxrd while it actually use mboxo from python. Mboxo may not be your favorite (nor mine), but python decided to use mboxo with reason. python manual goes: | 18.4.1.2. mbox | ... | Several variations of the mbox format exist to address perceived | shortcomings in the original. In the interest of compatibility, mbox | implements the original format, which is sometimes referred to as mboxo. | This means that the Content-Length header, if present, is ignored and | that any occurrences of “From ” at the beginning of a line in a message | body are transformed to “>From ” when storing the message, although | occurrences of “>From ” are not transformed to “From ” when reading the | message. On Wed, Oct 24, 2012 at 03:29:26AM +0200, Christoph Anton Mitterer wrote: > severity 633799 critical > forwarded 633799 http://article.gmane.org/gmane.mail.getmail.user/4576 > stop > Osamu, I can't believe my eyes why you severity-normal-ised such an > issue, if you'd not... I (and others) would have had the chance to see > it e.g. via apt-listbugs. Thank you for your attention and success in making upstream accept patch to realize mboxrd format instead of mboxo. (Have you read entire bug report including patches.) But if you read history of bug report including patches, you could have written a bit kinder tone message. I feel a bit sad to see this message. Maybe I should have applied my documentation patch in BTS. But since upstream did not take this patch, I left it here. > Before I saw this bug at Debian, I reported the same issue again > upstream: > http://article.gmane.org/gmane.mail.getmail.user/4576 > which does of course not mean that now they're smarter. > > > But from Debian side this clearly needs to be handled better. What I > suggested at the Debian bug on the analogous Evolution issue is: > big fat warnings in: > - the NEWS file No. This is not news. So not an appropriate place. Right place should be more like README.Debian or BUGS. > - the package descripting This is not right place. > - if possibly in a priority-high dialogue via debconf This is not right place per policy. > If upstream is stupid and doesn't want to fix this, then we really Please do not call upstream stupid ... especially in public like this. I think you are bright but this is not very polite. > really must warn our users on that issue. > And even if upstream would fix it, we still would need to warn our users > at least in the NEWS file / release notes... that all their mail from > previous years is likely corrupted. mboxo has been always so and have been widely used. mboxrd is technically superior. anyone who stores file in mbox should know there are risks as you describe. > For the above reasons, reseting the severity again to critical, as it > causes serious (because irrecoverable) data loss. I think important point is not above reason only but the fact upstream applied patch to make this program to function as mboxrd. I think you are a bit exxagurating severity of trivial part of data change. > If you don't want to apply the solving patch in only debian (if upstream > rejects it) I'd be fined with closing this bug (as wontfix) if the above > warnings (NEWS file, package description, debconf dialogue) are > installed. > > Again, I need to point out that I'm really disturbed and sad that such a > bug got hidden away in Debian :( Yah, I had homework to update README.Debian. Let's ask release manager how this should be handled now. If they are OK to accept patch only to fix this bug (inconsistence between documentation and code.) causing mbox file to be not the most technically safe one. Regards, Osamu -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#633799: getmail4: Mboxrd format is not supported
severity 633799 critical forwarded 633799 http://article.gmane.org/gmane.mail.getmail.user/4576 stop Guys... I really can't believe what you did here,... talking about marking such a highly severe issue that leads to irrecoverable data corruption back to severity normal. This is really outrageous. I recently found the same issue in Evolution: Debian bug #690741 upstream bug https://bugzilla.gnome.org/show_bug.cgi?id=686258 And it made me really sick how they dealt with it, just to see now that Debian, from which I'd really have thought it would do better in such a situation does the same and more or less "hide" things away. Not aware of this issue, I used both Evolution and when on vacation getmail and I have now a total of nearly 17000 mails in my archive that are corrupted. Osamu, I can't believe my eyes why you severity-normal-ised such an issue, if you'd not... I (and others) would have had the chance to see it e.g. via apt-listbugs. Before I saw this bug at Debian, I reported the same issue again upstream: http://article.gmane.org/gmane.mail.getmail.user/4576 which does of course not mean that now they're smarter. But from Debian side this clearly needs to be handled better. What I suggested at the Debian bug on the analogous Evolution issue is: big fat warnings in: - the NEWS file - the package descripting - if possibly in a priority-high dialogue via debconf If upstream is stupid and doesn't want to fix this, then we really really must warn our users on that issue. And even if upstream would fix it, we still would need to warn our users at least in the NEWS file / release notes... that all their mail from previous years is likely corrupted. For the above reasons, reseting the severity again to critical, as it causes serious (because irrecoverable) data loss. If you don't want to apply the solving patch in only debian (if upstream rejects it) I'd be fined with closing this bug (as wontfix) if the above warnings (NEWS file, package description, debconf dialogue) are installed. Again, I need to point out that I'm really disturbed and sad that such a bug got hidden away in Debian :( Cheers, Chris. smime.p7s Description: S/MIME cryptographic signature
Bug#633799: getmail4: Mboxrd format is not supported
Package: getmail4 Version: 4.20.3-2 Severity: normal Although getmail has a destination named "Mboxrd", and although the documentation explicitly says that it produces mboxrd-format files, this does not in fact seem to be the case. The Mboxrd destination produces mboxo files. For instance, I had the following test file being served by dovecot. Dovecot's mbox backend uses mboxcl2, i.e. it relies only on the Content-Length header to delimit messages, and leaves the body of the message unescaped. |From l...@iki.fi Wed Jul 13 22:04:07 2011 |From: me |To: whoever |Content-Length: 80 | |This is some text. |From 0 |>From 1 |>>From 2 |>>>From 3 |Some more text at the end. | (I prepended | to the lines to prevent further mangling while the bug report is in transit.) The file produced by getmail's Mboxrd destination looks like the following: |From unknown Wed Jul 13 23:12:36 2011 |Return-Path: |Delivered-To: unknown |Received: from localhost (127.0.0.1:993) by bq.la.iki.fi with IMAP4-SSL; 13 | Jul 2011 20:12:36 - |From: me |To: whoever | |This is some text. |>From 0 |>From 1 |>>From 2 |>>>From 3 |Some more text at the end. | Note that only the line "From 0" has been escaped, not any of the lines following it. This format is mboxo, not mboxrd. As is obvious, mboxo is not a very good format since from the fetched mail we cannot reconstruct the original message accurately, as the distinction between "From 0" and "From 1" has been lost. This is arguably a documentation bug, but a severe one, as the documentation explicitly gives the wrong information several times, and assuming the wrong mbox format can cause corruption of the messages. -- System Information: Debian Release: squeeze/sid APT prefers stable APT policy: (900, 'stable'), (2, 'unstable'), (1, 'experimental') Architecture: amd64 (x86_64) Kernel: Linux 2.6.32-3-amd64 (SMP w/2 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Versions of packages getmail4 depends on: ii python 2.6.6-3+squeeze6 interactive high-level object-orie ii python-support 1.0.8automated rebuilding support for P getmail4 recommends no packages. getmail4 suggests no packages. -- no debconf information -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
