Re: [Dbmail] Optimizing Dbmail Database

2009-12-11 Thread Paul J Stevens
Michael Monnerie wrote: On Donnerstag, 10. Dezember 2009 Tomas Kuliavas wrote: DBMail might find its niche in some setups, but large mailboxes are not in that niche. 750 GB DB proves it. You can't do text search raw email sources. There is no point of storing them in DB. And you believe

Re: [Dbmail] Optimizing Dbmail Database

2009-12-11 Thread Michael Monnerie
On Freitag, 11. Dezember 2009 Josh Marshall wrote: I have found that since linux kernel 2.6 series, LVM snapshots have caused system lockups. I used it happily in the 2.4 series. That's why LVM snapshots are not used in XenServer 5.x. They also said it's unstable, especially under high load.

Re: [Dbmail] Optimizing Dbmail Database

2009-12-11 Thread Michael Monnerie
On Freitag, 11. Dezember 2009 Daniel Urstöger wrote: Well, one can also do that with a filesystem based storage, you just need something similar to the MySQL replication for flat files. DRDB for example. DRBD puts a burden on the server all the time. For a secure replication you need to

Re: [Dbmail] Optimizing Dbmail Database

2009-12-11 Thread Michael Monnerie
On Freitag, 11. Dezember 2009 Michael Monnerie wrote: LVM snapshots Another thing to remember: You can only do a snapshot of a single filesystem at a time. So if you have your DB and attachments in different volumes, snapshots are not transactions anymore. Some people may be happy to live

Re: [Dbmail] Optimizing Dbmail Database

2009-12-11 Thread Michael Monnerie
On Freitag, 11. Dezember 2009 Paul J Stevens wrote: Currently, a full body text search will do a full table scan of the mimeparts table and pull in all mimeparts part of the messages in the mailbox being searched. If we want to skip all non text/* mimeparts (as allowed by the imap rfc), we'd

Re: [Dbmail] Optimizing Dbmail Database

2009-12-11 Thread Michael Monnerie
On Freitag, 11. Dezember 2009 Tomas Kuliavas wrote: Emails are not raw text. There are at least two ways to write test in email and if you go to 8bit text, number of same text variations multiplies. SQL can't search emails stored in DB, because SQL does not know about encodings, mime

Re: [Dbmail] Optimizing Dbmail Database

2009-12-11 Thread Tomas Kuliavas
2009.12.11 13:14 Michael Monnerie rašė: On Freitag, 11. Dezember 2009 Tomas Kuliavas wrote: Emails are not raw text. There are at least two ways to write test in email and if you go to 8bit text, number of same text variations multiplies. SQL can't search emails stored in DB, because SQL

Re: [Dbmail] Optimizing Dbmail Database

2009-12-11 Thread Michael Monnerie
On Freitag, 11. Dezember 2009 Tomas Kuliavas wrote: Are you sure that syntax of your select query is correct? No, that was pseudo code to demonstrate you can search for variations within one query. how complex select call you will make in order to cover all variations? flowed format,

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Tomas Kuliavas
Correct solution is not to store email data in DB. I think any sane DBA could say you that. Don't store binary data in DB. In large mailbox setups you get best performance by storing emails in filesystem (one email per file in hashed directory structure) and caching email headers in DB.

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Reindl Harald
Am 10.12.2009 20:23, schrieb Tomas Kuliavas: Correct solution is not to store email data in DB. I think any sane DBA could say you that. Don't store binary data in DB. What stupid statement in context of DBmail signature.asc Description: OpenPGP digital signature

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Tomas Kuliavas
2009.12.10 21:55 Reindl Harald rašė: Am 10.12.2009 20:23, schrieb Tomas Kuliavas: Correct solution is not to store email data in DB. I think any sane DBA could say you that. Don't store binary data in DB. What stupid statement in context of DBmail I am not DBmail developer or user. Maybe I

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread tabris
Reindl Harald wrote: Am 10.12.2009 20:23, schrieb Tomas Kuliavas: Correct solution is not to store email data in DB. I think any sane DBA could say you that. Don't store binary data in DB. What stupid statement in context of DBmail Incidentally, if one were to do as he suggests, it

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread tabris
Reindl Harald wrote: Am 10.12.2009 20:23, schrieb Tomas Kuliavas: Correct solution is not to store email data in DB. I think any sane DBA could say you that. Don't store binary data in DB. What stupid statement in context of DBmail Only because dbmail already does hte opposite.

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Alan Hodgson
On Thursday 10 December 2009, Tomas Kuliavas to...@users.sourceforge.net wrote: DBMail might find its niche in some setups, but large mailboxes are not in that niche. 750 GB DB proves it. You can't do text search raw email sources. There is no point of storing them in DB. DBMail does store

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Josh Marshall
I'd like to point out a few things: * The added complexity of storing and synchronising files on disk with records in tables, especially in a load-balanced, high available situation, is much more work than any returns you'll ever get. * Whether the emails are stored in the database or the

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread tabris
Josh Marshall wrote: I'd like to point out a few things: * The added complexity of storing and synchronising files on disk with records in tables, especially in a load-balanced, high available situation, is much more work than any returns you'll ever get. * Whether the emails are stored in

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Michael Monnerie
On Donnerstag, 10. Dezember 2009 Tomas Kuliavas wrote: DBMail might find its niche in some setups, but large mailboxes are not in that niche. 750 GB DB proves it. You can't do text search raw email sources. There is no point of storing them in DB. And you believe doing a raw text search on a

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Daniel Urstöger
I do not want to add to this quite hot situation, but there are two things worth mentioning: * I'd like to see impact-free daily backups for filesystem-based systems. With dbmail, just have a slave replica you can pause replication on to get a perfect snapshot, with no impact on the live

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Michael Monnerie
On Freitag, 11. Dezember 2009 Daniel Urstöger wrote: the Full Text Index ( FTI ) is quite bad for searches dbmail doesn't use FTI. mfg zmi -- // Michael Monnerie, Ing.BSc- http://it-management.at // Tel: 0660 / 415 6531 .network.your.ideas. // // Wir haben

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Josh Marshall
Hi, I didn't think it was hot. I have seen this argument a few times before, comparing apples and oranges and suggesting that dbmail does it the wrong way and should change. Yes dbmail has a niche and it excels in situations where filesystem-based mail systems can't cut it, and that's why I'm

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Daniel Urstöger
Never claimed it does ;) Just saying that for bigger datasets it becomes more and more useless but with Sphinx it remains blazing fast. So, for dbmail that would be a nice to have feature ... Am 11.12.2009 um 01:07 schrieb Michael Monnerie michael.monne...@is.it-management.at : On

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Daniel Urstöger
Oranges and Apples, I agree to that. I happily use dbmail as well as my qmail/vpopmail setup. Every system has it quirks and shortcommings. To address your two points though: I have found that since linux kernel 2.6 series, LVM snapshots have caused system lockups. I used it happily in the

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Josh Marshall
On Fri, 2009-12-11 at 01:38 +0100, Daniel Urstöger wrote: Yes, but after backing to the snapshot to some place one can remove it and the speed will be back to normal. So, running a db slave and using mysqldump for backups is not much different. Not quite. Having a separate slave database

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Reindl Harald
Am 11.12.2009 01:38, schrieb Daniel Urstöger dan...@gosi.at: Yes, but after backing to the snapshot to some place one can remove it and the speed will be back to normal. So, running a db slave and using mysqldump for backups is not much different. a) the slave can yun on one or more other

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Daniel Urstöger
My message was not quite finished, sadly there isnt an App for that ;) I have found that since linux kernel 2.6 series, LVM snapshots have caused system lockups. I used it happily in the 2.4 series. Besides that, I did mention *impact-free*. Adding a snapshot and reading from a snapshot

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Daniel Urstöger
Not quite. Having a separate slave database server to do the heavy work of backups has no impact on the master database during the backup period. Therefore the master database is always at normal speed. Well, one can also do that with a filesystem based storage, you just need something

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Josh Marshall
On Fri, 2009-12-11 at 02:01 +0100, Daniel Urstöger wrote: Not quite. Having a separate slave database server to do the heavy work of backups has no impact on the master database during the backup period. Therefore the master database is always at normal speed. Well, one can also do

Re: [Dbmail] Optimizing Dbmail Database

2009-12-10 Thread Tomas Kuliavas
2009.12.11 01:33 Michael Monnerie rašė: On Donnerstag, 10. Dezember 2009 Tomas Kuliavas wrote: DBMail might find its niche in some setups, but large mailboxes are not in that niche. 750 GB DB proves it. You can't do text search raw email sources. There is no point of storing them in DB.

[Dbmail] Optimizing Dbmail Database

2009-12-06 Thread Daniel Mejia
I was thinking to run the OPTIMIZE TABLE command in the dbmail database, bcoz right now the database size of my organization dbmail database is currently around 750 GB. Another reason was that my dbmail-util -ay command (which runs on 2am and 6am everyday) keeps giving me this error: [code] Dec

Re: [Dbmail] Optimizing Dbmail Database

2009-12-06 Thread Josh Marshall
On Sun, 2009-12-06 at 20:13 -0800, Daniel Mejia wrote: I was thinking to run the OPTIMIZE TABLE command in the dbmail database, bcoz right now the database size of my organization dbmail database is currently around 750 GB. Is there much free space in that table? If not, optimize table wont

Re: [Dbmail] Optimizing Dbmail Database

2009-12-06 Thread Daniel Mejia
Is there much free space in that table? If not, optimize table wont free any space and probably wont improve access speed etc. with the SHOW TABLE STATUS command, i can see that the free space is not adequate. i have around 2 TB of free space on the hard disk, but im not so sure about the table