Re: Cyrus doesn't preserve hard-links on replication
Sorry for the multiples emails... The option "provide_uuid=1" I found in my last message seems to be unrecognized by cyrus now (It's probably on by default). It's probably only useful for new messages anyway. So I'm back to square one and my google-fu failed me. If someone know how to solve that problem any help would be greatly appreciated. Cheers, Adrien Le dim. 17 nov. 2019 à 18:00, a écrit : > Send Info-cyrus mailing list submissions to > info-cyrus@lists.andrew.cmu.edu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > or, via email, send a message with subject or body 'help' to > info-cyrus-requ...@lists.andrew.cmu.edu > > You can reach the person managing the list at > info-cyrus-ow...@lists.andrew.cmu.edu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Info-cyrus digest..." > > > Today's Topics: > >1. Cyrus doesn't preserve hard-links on replication > (Adrien Remillieux) > > > -- > > Message: 1 > Date: Sun, 17 Nov 2019 13:54:11 +0100 > From: Adrien Remillieux > To: info-cyrus@lists.andrew.cmu.edu > Subject: Cyrus doesn't preserve hard-links on replication > Message-ID: > ak+k8ez8lisz+cw83g1smzu9q...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello, > > I set up replication between two cyrus servers (master runs 2.5.10 and > slave 3.0.8) with plans to decommission the old server once everything is > working. I noticed that the mail spool takes 950GB instead of ~300GB on the > old server. I suspected the hardlinks for message deduplication weren't > recreated so I ran rdfind on the mail spool and the tool found many > identical files. Is there a cyrus tool to recreate the hardlinks ? I looked > at the admin tools but I didn't find anything. Rdfind should work but it > also matched metadata such as cyrus.annotations for example. So I need to > go through the 600 mB dry-run log file to exclude unwanted files. > > 99% of the log file look like that : Are those safe to hardlink ? > > # duptype id depth size device inode priority name > DUPTYPE_FIRST_OCCURRENCE 1030757 3 842 2065 330967752 3 > /var/spool/cyrus/mail/c/user/user1/15384. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 317750405 3 > /var/spool/cyrus/mail/m/user/user2/19262. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 323550741 3 > /var/spool/cyrus/mail/r/user/user3/96106. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 316733834 3 > /var/spool/cyrus/mail/m/user/user4/41168. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 314623377 3 > /var/spool/cyrus/mail/m/user/user5/25377. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 316201219 3 > /var/spool/cyrus/mail/m/user/user6/49119. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 321991878 3 > /var/spool/cyrus/mail/q/user/user7/46487. > > Cheers, > Adrien > -- next part -- > An HTML attachment was scrubbed... > URL: < > http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20191117/60161ca7/attachment-0001.html > > > > -- > > Subject: Digest Footer > > ___ > Info-cyrus mailing list > Info-cyrus@lists.andrew.cmu.edu > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > -- > > End of Info-cyrus Digest, Vol 172, Issue 14 > *** > Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus doesn't preserve hard-links on replication
By shuffling the keywords on my google searches I was able to find this: https://lists.andrew.cmu.edu/pipermail/info-cyrus/2006-March/021405.html Apparently there is a few settings to set to avoid copying the same message multiple times. This would be a nice addition to the cyrus docs on replication ! Le dim. 17 nov. 2019 à 18:00, a écrit : > Send Info-cyrus mailing list submissions to > info-cyrus@lists.andrew.cmu.edu > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > or, via email, send a message with subject or body 'help' to > info-cyrus-requ...@lists.andrew.cmu.edu > > You can reach the person managing the list at > info-cyrus-ow...@lists.andrew.cmu.edu > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Info-cyrus digest..." > > > Today's Topics: > >1. Cyrus doesn't preserve hard-links on replication > (Adrien Remillieux) > > > -- > > Message: 1 > Date: Sun, 17 Nov 2019 13:54:11 +0100 > From: Adrien Remillieux > To: info-cyrus@lists.andrew.cmu.edu > Subject: Cyrus doesn't preserve hard-links on replication > Message-ID: > ak+k8ez8lisz+cw83g1smzu9q...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello, > > I set up replication between two cyrus servers (master runs 2.5.10 and > slave 3.0.8) with plans to decommission the old server once everything is > working. I noticed that the mail spool takes 950GB instead of ~300GB on the > old server. I suspected the hardlinks for message deduplication weren't > recreated so I ran rdfind on the mail spool and the tool found many > identical files. Is there a cyrus tool to recreate the hardlinks ? I looked > at the admin tools but I didn't find anything. Rdfind should work but it > also matched metadata such as cyrus.annotations for example. So I need to > go through the 600 mB dry-run log file to exclude unwanted files. > > 99% of the log file look like that : Are those safe to hardlink ? > > # duptype id depth size device inode priority name > DUPTYPE_FIRST_OCCURRENCE 1030757 3 842 2065 330967752 3 > /var/spool/cyrus/mail/c/user/user1/15384. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 317750405 3 > /var/spool/cyrus/mail/m/user/user2/19262. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 323550741 3 > /var/spool/cyrus/mail/r/user/user3/96106. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 316733834 3 > /var/spool/cyrus/mail/m/user/user4/41168. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 314623377 3 > /var/spool/cyrus/mail/m/user/user5/25377. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 316201219 3 > /var/spool/cyrus/mail/m/user/user6/49119. > DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 321991878 3 > /var/spool/cyrus/mail/q/user/user7/46487. > > Cheers, > Adrien > -- next part -- > An HTML attachment was scrubbed... > URL: < > http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20191117/60161ca7/attachment-0001.html > > > > -- > > Subject: Digest Footer > > ___ > Info-cyrus mailing list > Info-cyrus@lists.andrew.cmu.edu > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > -- > > End of Info-cyrus Digest, Vol 172, Issue 14 > *** > Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Cyrus doesn't preserve hard-links on replication
Hello, I set up replication between two cyrus servers (master runs 2.5.10 and slave 3.0.8) with plans to decommission the old server once everything is working. I noticed that the mail spool takes 950GB instead of ~300GB on the old server. I suspected the hardlinks for message deduplication weren't recreated so I ran rdfind on the mail spool and the tool found many identical files. Is there a cyrus tool to recreate the hardlinks ? I looked at the admin tools but I didn't find anything. Rdfind should work but it also matched metadata such as cyrus.annotations for example. So I need to go through the 600 mB dry-run log file to exclude unwanted files. 99% of the log file look like that : Are those safe to hardlink ? # duptype id depth size device inode priority name DUPTYPE_FIRST_OCCURRENCE 1030757 3 842 2065 330967752 3 /var/spool/cyrus/mail/c/user/user1/15384. DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 317750405 3 /var/spool/cyrus/mail/m/user/user2/19262. DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 323550741 3 /var/spool/cyrus/mail/r/user/user3/96106. DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 316733834 3 /var/spool/cyrus/mail/m/user/user4/41168. DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 314623377 3 /var/spool/cyrus/mail/m/user/user5/25377. DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 316201219 3 /var/spool/cyrus/mail/m/user/user6/49119. DUPTYPE_WITHIN_SAME_TREE -1030757 3 842 2065 321991878 3 /var/spool/cyrus/mail/q/user/user7/46487. Cheers, Adrien Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
