Re: WebDAV folders internally have hundreds of copies of the same few files

2018-11-19 Thread Anatoli

Hi Vladislav,

Yes, I don't actually have the expunge_mode option in the configs, so 
the default "delayed" is applied.


I've just tried some experiments with cyr_expire, the -X n option alone 
clears everything, including the shared DAV folders, but I can't find 
how to clear only some shared DAV folder. The -p and -u options seem not 
to work as intented on the shared DAV folders (user-owned DAV folders 
are cleared with -p  param). Any idea how to accomplish this?


Regards,
Anatoli

*From:* Vladislav Kurz
*Sent:* Monday, November 19, 2018 05:50
*To:* Info-cyrus
*Subject:* Re: WebDAV folders internally have hundreds of copies of the 
same few files


On 11/14/18 2:54 PM, Ken Murchison wrote:


On 11/13/18 10:15 PM, Anatoli wrote:

Hi,

I'm not sure this is due to some configuration option, bug or feature,
but I'm observing some folders on Cyrus HTTP WebDAV server having
hundreds (995 at this moment to be precise) internal files in the
format "NNN." that correspond to the same file but different versions
in time.

There are 2-3 files (xls) in the folder that are edited constantly
during the day and it looks like each save operation creates a new
file. The files are of some 3-5Mb each. In the explorer/web view there
are only a couple of files with a total size of 17.5Mb, but the
reported disk usage for the folder is of 1.6Gb.

Could someone please shed some light on what's going on and how to
make each file visible to the users to be stored in only one internal
file?

Thanks,
Anatoli


Because *DAV is layered on top of an IMAP store, we have to abide by
IMAP semantics in which messages (in this case DAV resources) are
immutable.  Therefore, we can NOT overwrite an existing message in the
mailbox.  Each change MUST result in a new message.  However, the server
does mark the previous version(s) as deleted and expunged, which means
that they will eventually be removed by cyr_expire.  If you aren't
running cyr_expire, you should consider adding an event to cyrus.conf to
remove expunged messages (see -X option).


Hello,

you probably have "expunge_mode: delayed". That's why deleted mails (and
*dav files) stay in place, until cleaned by cyr_expire as mentioned above.



Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: WebDAV folders internally have hundreds of copies of the same few files

2018-11-19 Thread Anatoli

> Enable the imapmagicplus option in imapd.conf

With this option I could login with imtest specifying "+DAV" for the 
user, both as the admin and as a regular user, but then when trying to 
select a folder, I got "NO Permission denied" for the admin and "NO 
Mailbox does not exist" for a common user. This looks like undocumented 
magic and mbexamine actually provides all needed information, so I will 
get file details with it.


Thanks for you help, Ken.

*From:* Ken Murchison
*Sent:* Friday, November 16, 2018 13:17
*To:* Info-cyrus
*Subject:* Re: WebDAV folders internally have hundreds of copies of the 
same few files



On 11/16/18 1:19 AM, Anatoli wrote:

Ken,

> If you login as an admin, you should be able to SELECT the mailbox 
and use normal IMAP commands.  Alternatively, if you add "+DAV" to a 
regular userid, this will also allow SELECTing of DAV collection 
mailboxes.


When I do imtest -a mailadmin and then issue a002 select 
#drive/folder, I get:


a002 NO Invalid mailbox type



It appears that just logging in as an admin won't do the trick.





When I do imtest -a mailadmin+DAV, I get (XX replaces the real auth 
value):


C: A01 AUTHENTICATE PLAIN XXX
S: A01 NO authentication failure
Authentication failed. generic failure

I've also tried these combinations with the same result: 
"u...@domain.tld+DAV" and "user+...@domain.tld". Am I doing it wrong?



Enable the imapmagicplus option in imapd.conf





Independently of this, is there a way to obtain the details about the 
flags for each message in a folder directly from the db files? My idea 
is to rsync the Cyrus store where the WebDAV is located to some other 
folder (probably on another server), purge there the files marked as 
deleted and backup the remaining files, all this without altering the 
Cyrus store. Is it possible?



> If the append of the resource into the mailbox, or updating the DAV 
db entry fails, the operation should be reverted, with partial saving 
done.  Which version of Cyrus are you using?


Currently it is 3.0.5, but I can upgrade if there were any related 
changes in the later versions. What I was observing is that if a PC 
(LibreOffice Calc on Ubuntu 16.04, the folder mounted with davfs2) 
crashes in the middle of saving a file on the server, the result was a 
damaged file: some 400 bytes instead of 3-4Mb of a real file.


I was thinking that maybe when the TCP stream hangs in the middle, 
Cyrus would interpret it as an end of data and write a file, but I've 
just tried to make some tests killing mount.davfs2 process and/or 
blocking the stream with iptables in the middle of file transmission 
and Cyrus responded correctly. For killing mount.davfs2 it showed in 
the corresponding log file the bytestream followed by:


HTTP/1.1 400 Bad Request

Unable to read body data

And for blocking the stream with iptables it was showing the 
bytestream ending where the transmission stopped, without the 400 Bad 
Request, but in both cases the original file was not modified by the 
partial upload. So it appears the problem is elsewhere.



Regards,
Anatoli

*From:* Ken Murchison
*Sent:* Thursday, November 15, 2018 14:09
*To:* Info-cyrus
*Subject:* Re: WebDAV folders internally have hundreds of copies of 
the same few files



On 11/15/18 1:48 AM, Anatoli wrote:

Hi Ken,

Thanks a lot for the clarification, everything makes sense now. How 
can I list the files marked for deletion and those that are not 
marked? unexpunge can provide the list of files marked for deletion, 
is there any better way to list them, directly reading the DB? How to 
list those that are not marked?



If you login as an admin, you should be able to SELECT the mailbox and 
use normal IMAP commands.  Alternatively, if you add "+DAV" to a 
regular userid, this will also allow SELECTing of DAV collection 
mailboxes.



One more question, related: we got a client who's PC was crashing 
exactly during the modify operation (some issue with the PC hardware 
triggered by Excel save operation, probably a RAM spike touching some 
bad blocks). As a result, the file in Cyrus was becoming damaged, 
i.e. partially saved. Is it expected?


Shouldn't Cyrus update the db with the pointer to the new file (a new 
message in the store) only if the operation completes successfully 
(e.g. the WebDAV messages exchange completes and the connection is 
closed at the right time or something similar)?



If the append of the resource into the mailbox, or updating the DAV db 
entry fails, the operation should be reverted, with partial saving 
done.  Which version of Cyrus are you using?





*From:* Ken Murchison
*Sent:* Wednesday, November 14, 2018 10:54
*To:* Info-cyrus
*Subject:* Re: WebDAV folders internally have hundreds of copies of 
the same few files



On 11/13/18 10:15 PM, Anatoli wrote:

Hi,

I'm not sure this is due to some configuration option, bug or 
feature, but I'm observing some folders on Cyrus HTTP WebDAV server 
having hundreds (995 at this 

Re: WebDAV folders internally have hundreds of copies of the same few files

2018-11-19 Thread Vladislav Kurz
On 11/14/18 2:54 PM, Ken Murchison wrote:
> 
> On 11/13/18 10:15 PM, Anatoli wrote:
>> Hi,
>>
>> I'm not sure this is due to some configuration option, bug or feature,
>> but I'm observing some folders on Cyrus HTTP WebDAV server having
>> hundreds (995 at this moment to be precise) internal files in the
>> format "NNN." that correspond to the same file but different versions
>> in time.
>>
>> There are 2-3 files (xls) in the folder that are edited constantly
>> during the day and it looks like each save operation creates a new
>> file. The files are of some 3-5Mb each. In the explorer/web view there
>> are only a couple of files with a total size of 17.5Mb, but the
>> reported disk usage for the folder is of 1.6Gb.
>>
>> Could someone please shed some light on what's going on and how to
>> make each file visible to the users to be stored in only one internal
>> file?
>>
>> Thanks,
>> Anatoli
> 
> 
> Because *DAV is layered on top of an IMAP store, we have to abide by
> IMAP semantics in which messages (in this case DAV resources) are
> immutable.  Therefore, we can NOT overwrite an existing message in the
> mailbox.  Each change MUST result in a new message.  However, the server
> does mark the previous version(s) as deleted and expunged, which means
> that they will eventually be removed by cyr_expire.  If you aren't
> running cyr_expire, you should consider adding an event to cyrus.conf to
> remove expunged messages (see -X option).

Hello,

you probably have "expunge_mode: delayed". That's why deleted mails (and
*dav files) stay in place, until cleaned by cyr_expire as mentioned above.

-- 
Best Regards
Vladislav Kurz


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus