Re: [Dovecot] Deleting metadata smashes file dovecot.dict
On 08/08/12 14:23, Andre Gröbe wrote:
I am using dovecot (see dovecot -n output below) with metadata plugin in
my own plugin. I want to insert and delete some large metadata (4KiB).
It seems, that the delete in a second call of my plugin smashes the file
dovecot.dict in users dir:
That's an error in dict-file.c, file_dict_refresh():
while ((key = i_stream_read_next_line(input)) != NULL
(value = i_stream_read_next_line(input)) != NULL) {
If reading the value requires reading from the stream (because the value
doesn't fit into the input buffer), the key is overwritten. The attached
patch duplicates the key before reading the value.
Index: dict-file.c
===
--- dict-file.c (revision 156809)
+++ dict-file.c (working copy)
@@ -186,9 +186,14 @@
if (dict-fd != -1) {
input = i_stream_create_fd(dict-fd, (size_t)-1, FALSE);
- while ((key = i_stream_read_next_line(input)) != NULL
- (value = i_stream_read_next_line(input)) != NULL) {
+
+ while ((key = i_stream_read_next_line(input)) != NULL) {
key = p_strdup(dict-hash_pool, key);
+
+ if ((value = i_stream_read_next_line(input)) == NULL) {
+break;
+ }
+
value = p_strdup(dict-hash_pool, value);
hash_table_insert(dict-hash, key, value);
}
Re: [Dovecot] Deleting metadata smashes file dovecot.dict
On 28.8.2012, at 17.42, Ewald Dieterich wrote:
On 08/08/12 14:23, Andre Gröbe wrote:
I am using dovecot (see dovecot -n output below) with metadata plugin in
my own plugin. I want to insert and delete some large metadata (4KiB).
It seems, that the delete in a second call of my plugin smashes the file
dovecot.dict in users dir:
That's an error in dict-file.c, file_dict_refresh():
while ((key = i_stream_read_next_line(input)) != NULL
(value = i_stream_read_next_line(input)) != NULL) {
If reading the value requires reading from the stream (because the value
doesn't fit into the input buffer), the key is overwritten. The attached
patch duplicates the key before reading the value.
Committed: http://hg.dovecot.org/dovecot-2.1/rev/6e53209030f6
Re: [Dovecot] Deleting metadata smashes file dovecot.dict
Hi Timo, [Timo Sirainen; Di 21 Aug 2012 12:22:04 CEST] Unfortunatly this can't be a fix because in file_set_size() dovecot defines an array 'char block[IO_BLOCK_SIZE]'. On the other hand the default block size is predestined for file ops. Does the attached patch fix it? (not a very good fix) No, it doesn't. Still got no response on large metadata and writing destroys the dictionary. Sorry for bumping - is there a chance to get it work with dovecot 2.1.7? TIA Andre
Re: [Dovecot] Deleting metadata smashes file dovecot.dict
Hi Timo, thx for reply. [t...@iki.fi; Mo 20 Aug 2012 20:22:58 CEST] Is there a fix so dovecot reads a line until EOL or EOF (if this is the real reason)? Without having actually looked at the code, I'd guess you can just replace IO_BLOCK_SIZE with (size_t)-1. Unfortunatly this can't be a fix because in file_set_size() dovecot defines an array 'char block[IO_BLOCK_SIZE]'. On the other hand the default block size is predestined for file ops. Best regards Andre smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Deleting metadata smashes file dovecot.dict
On 21.8.2012, at 13.10, Andre Gröbe wrote: [t...@iki.fi; Mo 20 Aug 2012 20:22:58 CEST] Is there a fix so dovecot reads a line until EOL or EOF (if this is the real reason)? Without having actually looked at the code, I'd guess you can just replace IO_BLOCK_SIZE with (size_t)-1. Unfortunatly this can't be a fix because in file_set_size() dovecot defines an array 'char block[IO_BLOCK_SIZE]'. On the other hand the default block size is predestined for file ops. Does the attached patch fix it? (not a very good fix) diff Description: Binary data
Re: [Dovecot] Deleting metadata smashes file dovecot.dict
Hi Timo, [t...@iki.fi; Di 21 Aug 2012 12:22:04 CEST] Unfortunatly this can't be a fix because in file_set_size() dovecot defines an array 'char block[IO_BLOCK_SIZE]'. On the other hand the default block size is predestined for file ops. Does the attached patch fix it? (not a very good fix) No, it doesn't. Still got no response on large metadata and writing destroys the dictionary. Best Regards Andre smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Deleting metadata smashes file dovecot.dict
Hi it seems to me, that it depends on how dovecot handles line reading. Dovecot presumes that a line is of max length of IO_BLOCK_SIZE (=8192) Bytes. Is there a fix so dovecot reads a line until EOL or EOF (if this is the real reason)? Regards Andre smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] Deleting metadata smashes file dovecot.dict
On 20.8.2012, at 16.35, Andre Gröbe wrote: it seems to me, that it depends on how dovecot handles line reading. Dovecot presumes that a line is of max length of IO_BLOCK_SIZE (=8192) Bytes. Is there a fix so dovecot reads a line until EOL or EOF (if this is the real reason)? Without having actually looked at the code, I'd guess you can just replace IO_BLOCK_SIZE with (size_t)-1.
Re: [Dovecot] Deleting metadata smashes file dovecot.dict
On reading 'big' metadata entries i got a similiar behaviour (metadata_maxsize = 10240). The command getmetadata returns no data. I have created a dovecot.dict with a single entry (key=priv/mailbox/fba9aa2cf41721501f34d6254462/private/vendor/test/10458d10cf1f2a502375d6254462/2) to reproduce that. I expected to get the value with command: . getmetadata inbox /private/vendor/test/10458d10cf1f2a502375d6254462/2 . OK Completed. It seems to me that there is a bug in metadata plugin. Anyone any ideas? TIA Andre dovecot.dict.bz2 Description: application/bzip
[Dovecot] Deleting metadata smashes file dovecot.dict
Hi, I am using dovecot (see dovecot -n output below) with metadata plugin in my own plugin. I want to insert and delete some large metadata (4KiB). It seems, that the delete in a second call of my plugin smashes the file dovecot.dict in users dir: 1st call: *dovecot.dict is empty -delete metadata 1 -delete metadata 2 -insert metadata 1 ~8KiB -insert metadata 2 1KiB *dovecot.dict contains two entries 2nd call: -delete metadata 1 -delete metadata 2 -insert metadata 1 ~8KiB -insert metadata 2 1KiB *dovecot.dict contains the two entries + some waste If all eight steps are called together (in the same plugin context) the file dovecot.dict seems to be ok. I have created a test plugin to reproduce this behaviour: http://temp-share.com/show/dPf3myu5W. It needs the notify and (patched) metadata plugin to run. To get it work i copy an email from one IMAP folder to another (to trigger the copy notifier). It seems, that there is an issue on writing metadata/dict, can anybody reproduce it or knows what is wrong? TIA -- Andre
