Hello again,
I have created a parser script, a little bit more advanced than the one
provided with Dovecot. The main feature is probably to index documents
inside zip/rar/tgz archives...
I am using Ansible, swaks and doveadm to run automatic tests for each
supported content. For specific reasons, I am not yet able to add Apache
Tika to the distribution. However, I already made some tests with it.
For now, I want to talk about the indexing script.
I also have noticed a few weird behaviours. I will mention them at the
end, albeit I am not 100% sure where they are coming from. I realised
last week that using QEMU snapshots was not working as expected, so I am
now more careful with this feature.
For the developers or users who would be interested and the Dovecot team
members to understand my questions, here how the tests are working:
To run my tests, I have a set of files in various formats, with a UUID
inside. They are office files, text files, or even archives with a text
file inside...
The first test I am running is the script alone. I check that the script
can convert the file to text, and then I use grep to check the UUID is
present. This works *perfectly* for all the content, except ppt, but
it's minor.
The second test is full:
- I use swaks to send the email with an attachment and the appropriate
mime type.
- I then refresh the index using doveadm rescan.
- I check that fts search returns a line, with doveadm fts search.
- I then expunge the mailbox to be sure that the next test is valid.
For the second test, it works almost all the time, except in the
following situations:
- When the attachment is an email (mime type message/rfc822)
- RTF (could be a bug in my script)
- Text file in UTF16 (Even if this file is converted to UTF8)
*Questions:*
1 - Is there any limitation or special case for the mime message/rfc822
2 - Is the mime type received coming from the email headers?
3 - When the script is called without arguments, what is the purpose of
the extension at the end of each supported mime types?
4 - Can I return a wildcard in the supported mime types, for instance
"text/* *" ?
5 - I would like to handle attachments of types
application/octet-stream. I have added "application/octet-stream *", but
I am not sure if dovecot will pass the attachments with these mime type
or not.
*Notes:*
1 - I used netcat to monitor the solr server. I realise that sometimes,
the data sent to the solr server only contains the headers of the email,
not the text returned by the parser. Especially with rfc822 messages. I
will do more tests.
2 - I just finished to write the script, it's not yet refactored, but at
list it is well documented. I will do a full security audit later. I am
actually testing an associated AppArmor profile.
3 - I will do more intensive test on the script on bigger mail boxes
with more attachments.
4 - I may rewrite the script in Python
5 - Suggestions welcome.
I initially attached the current version of the script, but the email is
probably pending for review...In this case, the last development version
is on Github:
https://github.com/progmaticltd/homebox/blob/dev/install/playbooks/roles/dovecot/files/fts/decode2text
The configuration of supported mime types is a simple file, accessible
on github as well:
https://github.com/progmaticltd/homebox/blob/dev/install/playbooks/roles/dovecot/templates/fts/mime-supported.conf
Thanks for your advices or suggestions.
