Bug#859478: offlineimap: 'maxage' comments are wrong, offlineimap DELETES your mails

[Cyril Brulebois]

Ilias Tsitsimpis  (2017-04-10):
> On Tue, Apr 04, 2017 at 07:10PM, Cyril Brulebois wrote:
> > > This should be fixed in newer versions of OfflineIMAP. Could you please
> > > give it a try?
> > 
> > Well, trying out new versions is something I can do, but it really doesn't
> > help with the fact that offlineimap in stable is responsible for data loss.
> 
> You are right, that's why I didn't close this bug report.
> 
> The patches that fix this bug should be:
> 
> https://github.com/OfflineIMAP/offlineimap/commit/25513e90387
> https://github.com/OfflineIMAP/offlineimap/commit/8096f6cd5bf
> 
> Unfortunately, backporting them is not trivial since they introduce
> changes in the core logic of OfflineIMAP.
> 
> Right now, I am considering uploading an updated version in stable,
> where the docs will mention that this feature should not be used, and
> there will also be a warning message in case it is used. Another option
> would be to completely disable this feature in stable (since it is
> broken and doesn't work as expected), but this may break older setups
> (and I would like to avoid this). What do you think?

Yeah, I initially thought about updating the docs, but that doesn't help
with longstanding ~/.offlineimaprc copied/edited from the example one. I
think updated docs + warning when this setting is used would be a good
compromise. This warning could get unnoticed though (because one might
run offlineimap from a loop, cron, screen)… but I think it'd leave users
a chance to avoid running into mail deletion. Would have worked for me
at least. :D


KiBi.


signature.asc
Description: Digital signature

Bug#859478: offlineimap: 'maxage' comments are wrong, offlineimap DELETES your mails

[Ilias Tsitsimpis]

On Tue, Apr 04, 2017 at 07:10PM, Cyril Brulebois wrote:
> > This should be fixed in newer versions of OfflineIMAP. Could you please
> > give it a try?
> 
> Well, trying out new versions is something I can do, but it really doesn't
> help with the fact that offlineimap in stable is responsible for data loss.

You are right, that's why I didn't close this bug report.

The patches that fix this bug should be:

https://github.com/OfflineIMAP/offlineimap/commit/25513e90387
https://github.com/OfflineIMAP/offlineimap/commit/8096f6cd5bf

Unfortunately, backporting them is not trivial since they introduce
changes in the core logic of OfflineIMAP.

Right now, I am considering uploading an updated version in stable,
where the docs will mention that this feature should not be used, and
there will also be a warning message in case it is used. Another option
would be to completely disable this feature in stable (since it is
broken and doesn't work as expected), but this may break older setups
(and I would like to avoid this). What do you think?

-- 
Ilias

Bug#859478: offlineimap: 'maxage' comments are wrong, offlineimap DELETES your mails

[Cyril Brulebois]

Ilias Tsitsimpis  (2017-04-04):
> Control: fixed -1 6.6.0~rc3+dfsg1-1
> 
> Hi Cyril,
> 
> On Tue, Apr 04, 2017 at 01:28AM, Cyril Brulebois wrote:
> > Looking at the code, it seems “is that older than maxage?” is
> > implemented by looking at the timestamp in the filename for local
> > mails, which happens to be October 2016 for a mail received in 2015!
> 
> Your analysis is correct. Here is a blog post which explains in depth
> the above bug:
> 
> http://www.offlineimap.org/devel/2015/04/14/why-we-changed-maxage.html

Yes I've read this.

> This should be fixed in newer versions of OfflineIMAP. Could you please
> give it a try?

Well, trying out new versions is something I can do, but it really doesn't
help with the fact that offlineimap in stable is responsible for data loss.

> You may also be interested in the 'sync_deletes' feature, introduced in
> version 7.0.6+dfsg1-1. Copying from the docs:
> 
> This option stands in the [Repository RemoteExample] section.
> 
> Propagate deletions from remote to local. Messages deleted in this 
> repository
> won't get deleted on the local repository if set to "no". Default is yes.

This looks like an ideal solution to my initial problem, thanks!


KiBi.


signature.asc
Description: Digital signature

Bug#859478: offlineimap: 'maxage' comments are wrong, offlineimap DELETES your mails

[Ilias Tsitsimpis]

Control: fixed -1 6.6.0~rc3+dfsg1-1

Hi Cyril,

On Tue, Apr 04, 2017 at 01:28AM, Cyril Brulebois wrote:
> Looking at the code, it seems “is that older than maxage?” is
> implemented by looking at the timestamp in the filename for local
> mails, which happens to be October 2016 for a mail received in 2015!

Your analysis is correct. Here is a blog post which explains in depth
the above bug:

http://www.offlineimap.org/devel/2015/04/14/why-we-changed-maxage.html

This should be fixed in newer versions of OfflineIMAP. Could you please
give it a try?

You may also be interested in the 'sync_deletes' feature, introduced in
version 7.0.6+dfsg1-1. Copying from the docs:

This option stands in the [Repository RemoteExample] section.

Propagate deletions from remote to local. Messages deleted in this 
repository
won't get deleted on the local repository if set to "no". Default is yes.

Best,

-- 
Ilias

Bug#859478: offlineimap: 'maxage' comments are wrong, offlineimap DELETES your mails

[Cyril Brulebois]

Cyril Brulebois  (2017-04-04):
> Except… deleting a few mails from 2015 on the server with a little
> python script using imaplib, and running offlineimap again leads to…
> the same mails getting deleted from my laptop! Given the maxage
> param was set to 90, I *think* 31-Dec-2015 happened before 90 days
> ago, so clearly not a timezone thing.
> 
> 
> Expected results: mails aren't on the server anymore (I deleted
> them explicitly) BUT they're still on my laptop.

Looking at the code, it seems “is that older than maxage?” is
implemented by looking at the timestamp in the filename for local
mails, which happens to be October 2016 for a mail received in 2015!

I'm not sure what offlineimap did with the timestamps, but that's
the bit of software responsible for all mails fetched over IMAP,
so either timestamp handling is buggy or maxage implementation is
buggy.

Either way, it might make sense to be extra cautious when there are
such bold claims in the documentation, we're talking about deletions
here!


KiBi.


signature.asc
Description: Digital signature

Bug#859478: offlineimap: 'maxage' comments are wrong, offlineimap DELETES your mails

[Cyril Brulebois]

Package: offlineimap
Version: 6.3.4-1+kibi1
Severity: critical
Justification: causes serious data loss

Hi,

Looking at a way to stay below my mail quota, I've thought of using
offlineimap's "maxage" feature, which is advertised like this:

# Messages older than maxage days will not be synced, their flags will
# not be changed, they will not be deleted etc.  For offlineimap it will be 
like these
# messages do not exist.  This will perform an IMAP search in the case of IMAP 
or Gmail
# and therefor requires that the server support server side searching.  This 
will
# calculate the earliest day that would be included in the search and include 
all 
# messages from that day until today.   e.g. maxage = 3 to sync only the last 3 
days mail

Except… deleting a few mails from 2015 on the server with a little
python script using imaplib, and running offlineimap again leads to…
the same mails getting deleted from my laptop! Given the maxage
param was set to 90, I *think* 31-Dec-2015 happened before 90 days
ago, so clearly not a timezone thing.


Expected results: mails aren't on the server anymore (I deleted
them explicitly) BUT they're still on my laptop.


KiBi.