Re: Mailboxes are in Maildir format. Any good backup tips? Had success with version control?
deoren wrote: > > Right now I'm using LVM snapshots + tarballs for daily backups, but > I'd like to get better coverage for incremental changes that occur > throughout the day. The size of existing content is low, but (small) > changes are frequent. If you actually want to preserve those increments (as opposed to just keeping an rsync mirror up-to-date), I like rdiff-backup. It handles maildirs well because of the one-message-per-file design. Charles -- ----------- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ ---
Re: [Dovecot] Passing info from mail process to mail_filter plugin script?
Timo Sirainen wrote: > On 25.9.2013, at 3.06, Charles Cazabon wrote: > > > > I hacked an additional %variable (I used %q) into a copy of > > mail-user.c:mail_user_var_expand_table () [...] > Better to not give a one character name, but only the long name so it won't > conflict with any future Dovecot additions. That was my plan. I added the single character key when I was trying to debug why it wasn't working. > > The %q does not appear to be getting replaced; it's just skipped over. > > It's not even passing an empty string in its place. [...] > It looks like this should be correct. The plugin settings expansion i done > by mail-user.c:mail_user_expand_plugins_envs(). Aha, thanks. My code to set this information was still not being called quite early enough, so the table didn't contain a value for my variable when mail_user_expand_plugins_envs() was called. I've rejigged it some more, and now it is indeed working. Thanks very much for your help with this. > The other possibility would be that you just modify mail-filter plugin and > add the extra parameter without any %variable changes. Probably better since > then you don't need to patch Dovecot core itself. Ah, perhaps that wasn't clear - this isn't a constant or anything like that. The extra argument getting passed to mail_filter is taken from data the client sends to the IMAP server -- so I needed to be able to pull it out of the IMAP session and pass it to the mail_filter script. Charles -- --- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ ---
Re: [Dovecot] Passing info from mail process to mail_filter plugin script?
Timo Sirainen wrote:
>
> Without modifications the only way to pass data is via the plugin {
> mail_filter } parameters, such as the %u expanding to username in the
> example.
I hacked an additional %variable (I used %q) into a copy of
mail-user.c:mail_user_var_expand_table () (and called this modified version
instead of the original, in the same place it is normally called), and moved
my data collection around in imap/main.c:settings_var_expand() so that my info
was available at the time mail_user->var_expand_table is populated.
This appears to be working thus far. If I iterate over that table, my custom
variable is present and has a correct key, value, and long_key before
client_create() is called.
But when I put %q into the mail_filter config like so:
mail_plugins = $mail_plugins mail_filter
plugin {
mail_filter = mail-filter %q foo %u
mail_filter_out = mail-filter-out %u
}
... and restart Dovecot, I find that the filter script is only being passed 2
arguments, the constant "foo" in the above and the username. The %q does not
appear to be getting replaced; it's just skipped over. It's not even passing
an empty string in its place.
Am I modifying the correct var_expand_table here? Is there any step I've
missed in making the new variable get substituted properly? lib/var-expand.c
doesn't appear to require anything else, but I may have missed something...
Any help appreciated,
Charles
--
-----------
Charles Cazabon
GPL'ed software available at: http://pyropus.ca/software/
---
Re: [Dovecot] Passing info from mail process to mail_filter plugin script?
Timo Sirainen wrote: > On 21.9.2013, at 20.32, Charles Cazabon wrote: > > > > However, my mail-filter script (for filtering messages on the way from the > > mail store through the IMAP process to a remote client) needs some extra > > info > > from the IMAP mail process. I'm having difficulty figuring out how this can > > be accomplished. > > > > Ideally, I'd like to pass another open file descriptor (opened in the IMAP > > process) to the script, but I'm not sure if this is possible. [...] > imap process communicates with the mail_filter process via UNIX socket. It > would be possible to pass a file descriptor, but it would need motifications > to both the mail-filter/[io]stream-ext-filter.c and to src/util/script.c to > use fd_send() and fd_recv(). Thanks for the response, Timo. Okay, given that this looks a little more complex than the other route, I decided to try your suggestion of adding a new % variable which I can supply in the mail_filter plugin configuration as a commandline argument to the filter script. > If you want some other parameters that don't exist in %variables > (they get expanded immediately when the imap process starts), you need to > modify for example mail-filter-plugin.c where it passes muser->args and > muser->out_args to [io]_stream_create_ext_filter(). I *think*, if I'm reading the code correctly, that it might be simpler in my particular case to modify the var_expand_table directly. The info I want to pass to the filter becomes available in the imap code shortly after login (actually immediately after client_add_input()), so unfortunately it's just after settings_var_expand() is called in src/imap/main.c:client_create_from_input(). I hacked in some code to change the table at that point (i.e. just before client_create_from_input() returns), but it seems the table has already been used to expand the vars by then, i.e. I'm just too late and my new variable isn't recognized/expanded. Would a simple, ugly hack like calling settings_var_expand() a second time be sufficient to get my new % variable added here? Charles -- --- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ ---
[Dovecot] Passing info from mail process to mail_filter plugin script?
Hi, all, The mail_filter plugin allows you to pass messages through an external script/command on the way into and out of the Maildir/mbox/etc mail store. With the recent/new version of the plugin, this seems to work correctly. However, my mail-filter script (for filtering messages on the way from the mail store through the IMAP process to a remote client) needs some extra info from the IMAP mail process. I'm having difficulty figuring out how this can be accomplished. Ideally, I'd like to pass another open file descriptor (opened in the IMAP process) to the script, but I'm not sure if this is possible. Opening the file in the IMAP main.c fails, presumably due to file descriptor limits, but I don't see where restrict_fd_limit() is called there. My second choice for passing this info would be in an environment variable or argument to the mail filter script. Adding env_put() to the IMAP code doesn't seem to get the value passed to the script, presumably because the environment is getting cleared, but again I'm not certain where the list of env vars to keep is set here. There may be another way to accomplish this that I'm not thinking of? Any assistance would be appreciated. Charles -- ----------- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ ---
Re: [Dovecot] mail-filter plugin v2.2 - corrupts IMAP indexes?
Charles Cazabon wrote: > > I'm trying out the mail-filter plugin with Dovecot 2.2.5 built from source, > and it seems like any nontrivial filter causes some sort of index corruption > that breaks IMAP use. For the list archives: Timo has released a new version of this plugin that fixes this. Charles -- ----------- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ ---
[Dovecot] mail-filter plugin v2.2 - corrupts IMAP indexes?
Hi, I'm trying out the mail-filter plugin (v2.2, with Dovecot 2.2.5 built from source), and it seems like any nontrivial filter causes some sort of index corruption that breaks IMAP use. i.e. using a null filter (basically `cat -`) for mail-filter.sh and mail-filter-out.sh appears to work fine, and the debug logging shows it's using the filters (though it reports the wrong Unix socket for one of them - not sure if that's an actual bug or just a copy&paste error where it logs it; there's a similar accidental use of the same Unix socket path in the readme for the plugin): Sep 15 14:24:03 imap: Debug: Loading modules from directory: .../dovecot/lib/dovecot Sep 15 14:24:03 imap: Debug: Module loaded: .../dovecot/lib/dovecot/mail_filter_plugin.so Sep 15 14:24:03 imap(charlesc): Debug: Effective uid=1000, gid=1000, home=/home/charlesc Sep 15 14:24:03 imap(charlesc): Debug: mail_filter: Filtering mail_filter via socket .../dovecot/var/run/dovecot/mail-filter Sep 15 14:24:03 imap(charlesc): Debug: mail_filter: Filtering mail_filter_out via socket .../dovecot/var/run/dovecot/mail-filter If I use a slightly more complex pair of filter scripts that just swap the case of all alpha chars in the message body (not headers), so this pair of filters results in the same output as the original message, something seems to get corrupted in the indexes, and the IMAP process fails to respond correctly to FETCH requests, and logs this: Sep 15 14:24:03 imap(charlesc): Debug: maildir++: root=.../Mail-test, index=, indexpvt=, control=, inbox=.../Mail-test, alt= Sep 15 14:24:03 imap(charlesc): Error: Cached message size smaller than expected (0 < 1092) Sep 15 14:24:03 imap(charlesc): Error: Maildir filename has wrong S value, renamed the file from .../Mail-test/new/1379276625.M25480P28541.ludmilla,S=0,W=1122 to .../Mail-test/new/1379276625.M25480P28541.ludmilla,S=1092 Sep 15 14:24:03 imap(charlesc): Error: Corrupted index cache file .../Mail-test/dovecot.index.cache: Broken physical size for mail UID 5 Sep 15 14:24:03 imap(charlesc): Error: read(BODY[]) failed: Invalid argument (FETCH for mailbox INBOX UID 5) Sep 15 14:24:03 imap(charlesc): Info: Disconnected: FETCH failed in=105 out=1176 It reports the size of the message changes, too, which is odd, as running the filters manually confirms the output is identical to the input. mail-filter.sh attached; mail-filter-out.sh is identical except using a different temp file name to avoid clashes. Am I doing something wrong? How can I prevent this "Corrupted index cache file" problem? Thanks, Charles -- ----------- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ --- mail-filter.sh Description: Bourne shell script
