Re: Multiple Databases?
Hi Jerry, It's a common need for big mail providers, and it make complete sense to have cold/hot storage depending on the access pattern. There are several solutions to that problem. One is to implement your business logic (about what is hot or cold) in your mailbox implementation and delegate to the right components. Another is to introduce an Archive concept in james and define a new interface for that in a way that doesn't hide that concern: it's probably a much better idea to be able to handle archiving process, etc. In both solution, you need to instanciate more than one MailboxManager to handle the different kind of storage. We once implemented an archive tool (it's here https://github.com/linag ora/OBM/tree/master/java/sync/imap-archive ) that handle the process of taking emails from people imap tree and put them into an "archive/year" tree that is stored on a cheaper storage. It's not as simple as it seems so I would be more confident doing it as a first class concept in James. My 2 cents. -- Matthieu On Tue, 2018-05-08 at 14:08 -0500, Jerry Malcolm wrote: > I realize this is a completely off-the-wall question... but has > there > been any discussion about breaking the JAMES db into multiple > databases? I have a bunch of clients, each with a bunch of > accounts. > And they all want to archive all of their mail and be able to access > it > through the same mail client. As the years go by, the db just keeps > growing and growing. This became acutely obvious when I was > recently > forced by my server provider to migrate to newer hardware, and I > realized I had to have mail down for a long time while I transferred > a > 60gB+ file across a slow connection to the new server. The db size > is > also making daily backups a problem. > > The reality is that 90% of the mail is archived into 'year' folders > for > each of the accounts which are basically "read-only" now. Only a > relatively small amount of mail is truly in dynamically updated > folders. If there was a way to store "/archives/2002" folders > through > "/archives/2017" folders in one db and all of the other folders in > another db, it would make backup and migration a much simpler task. > > Ok, I'm pretty sure that isn't in the immediate plan. But just let > me > put my vote in. Alternatively, is there any alternative to having > one > ever-growing mail db? Is there some trick with the db server that > can > present one logical db from multiple db files? (I know a lot about > databases... but there's still a bunch I don't know). Has anyone > else > faced/addressed this problem? Or is the answer to just live with it? > > Thanks. > > Jerry > > > - > To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org > For additional commands, e-mail: server-user-h...@james.apache.org > > - To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org For additional commands, e-mail: server-user-h...@james.apache.org
Re: Multiple Databases?
Using James IMAP library, I had implemented a solution where I was showing the mails from archive in mail clients like outlook and stuff. So yes, it is pretty much possible to do that (composite or stuff, whatever you want to call it.) On Wed, May 9, 2018 at 12:39 PM Rami Ojares wrote: > I remember at some point in James' codebase there was a set of interfaces. > And if you implemented them you could fully determine how James persists > data. > In this case storing to any amount of databases with your > organisation-specific criteria would be no problem. > You would just implement the storage related interfaces in any way you > like. > > If I wish to implement a custom backend is it enough to provide > implementations to all interfaces under org.apache.james.mailbox package? > > - Rami > > On 09-May-18 09:56, Ioan Eugen Stan wrote: > > Hi, > > > > I think id does make sense to have such a functionality. No reason to > > keep archived emails in the main storage if hey are rarely accessed. Not > > sure on what solution to adopt as there are many possible ways to > > achieve this. > > > > One thing that I am thinking of is to provide a Composite Mailbox > > implementation. > > > > A rough sketch is bellow: > > > > This composite mailbox implementation could delegate actual processing > > to a couple of implementations. Each implementation could manage a group > > of folders. > > > > This solution could solve this use case by: > > > > - keep normal folders inside a database using jpa implementation > > > > - keep archived emails in a Maildir or S3 storage etc, possibly on > > slower, less expensive systems. > > > > I haven't given a lot of thought to this so I'm not sure if it is even > > possible at the moment, but I think the pattern has some merit. > > > > Regards, > > > > Eugen > > > > > > On 09.05.2018 07:46, Benoit Tellier wrote: > >> Hi Jerry, > >> > >> There is currently no solution to do what you describe. There is no > >> "custom storage treatment" for archived mails. > >> > >> I did not address nor faced this problem, thus can not really help you. > >> > >> Cheers, > >> > >> Benoit Tellier > >> > >> Le 09/05/2018 à 02:08, Jerry Malcolm a écrit : > >>> I realize this is a completely off-the-wall question... but has there > >>> been any discussion about breaking the JAMES db into multiple > >>> databases? I have a bunch of clients, each with a bunch of accounts. > >>> And they all want to archive all of their mail and be able to access it > >>> through the same mail client. As the years go by, the db just keeps > >>> growing and growing. This became acutely obvious when I was recently > >>> forced by my server provider to migrate to newer hardware, and I > >>> realized I had to have mail down for a long time while I transferred a > >>> 60gB+ file across a slow connection to the new server. The db size is > >>> also making daily backups a problem. > >>> > >>> The reality is that 90% of the mail is archived into 'year' folders for > >>> each of the accounts which are basically "read-only" now. Only a > >>> relatively small amount of mail is truly in dynamically updated > >>> folders. If there was a way to store "/archives/2002" folders through > >>> "/archives/2017" folders in one db and all of the other folders in > >>> another db, it would make backup and migration a much simpler task. > >>> > >>> Ok, I'm pretty sure that isn't in the immediate plan. But just let me > >>> put my vote in. Alternatively, is there any alternative to having one > >>> ever-growing mail db? Is there some trick with the db server that can > >>> present one logical db from multiple db files? (I know a lot about > >>> databases... but there's still a bunch I don't know). Has anyone else > >>> faced/addressed this problem? Or is the answer to just live with it? > >>> > >>> Thanks. > >>> > >>> Jerry > >>> > >>> > >>> - > >>> To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org > >>> For additional commands, e-mail: server-user-h...@james.apache.org > >>> > >> - > >> To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org > >> For additional commands, e-mail: server-user-h...@james.apache.org > >> > > > > > > - > > To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org > > For additional commands, e-mail: server-user-h...@james.apache.org > > > > > - > To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org > For additional commands, e-mail: server-user-h...@james.apache.org > > -- Akhilesh Tyagi
Re: Multiple Databases?
I remember at some point in James' codebase there was a set of interfaces. And if you implemented them you could fully determine how James persists data. In this case storing to any amount of databases with your organisation-specific criteria would be no problem. You would just implement the storage related interfaces in any way you like. If I wish to implement a custom backend is it enough to provide implementations to all interfaces under org.apache.james.mailbox package? - Rami On 09-May-18 09:56, Ioan Eugen Stan wrote: Hi, I think id does make sense to have such a functionality. No reason to keep archived emails in the main storage if hey are rarely accessed. Not sure on what solution to adopt as there are many possible ways to achieve this. One thing that I am thinking of is to provide a Composite Mailbox implementation. A rough sketch is bellow: This composite mailbox implementation could delegate actual processing to a couple of implementations. Each implementation could manage a group of folders. This solution could solve this use case by: - keep normal folders inside a database using jpa implementation - keep archived emails in a Maildir or S3 storage etc, possibly on slower, less expensive systems. I haven't given a lot of thought to this so I'm not sure if it is even possible at the moment, but I think the pattern has some merit. Regards, Eugen On 09.05.2018 07:46, Benoit Tellier wrote: Hi Jerry, There is currently no solution to do what you describe. There is no "custom storage treatment" for archived mails. I did not address nor faced this problem, thus can not really help you. Cheers, Benoit Tellier Le 09/05/2018 à 02:08, Jerry Malcolm a écrit : I realize this is a completely off-the-wall question... but has there been any discussion about breaking the JAMES db into multiple databases? I have a bunch of clients, each with a bunch of accounts. And they all want to archive all of their mail and be able to access it through the same mail client. As the years go by, the db just keeps growing and growing. This became acutely obvious when I was recently forced by my server provider to migrate to newer hardware, and I realized I had to have mail down for a long time while I transferred a 60gB+ file across a slow connection to the new server. The db size is also making daily backups a problem. The reality is that 90% of the mail is archived into 'year' folders for each of the accounts which are basically "read-only" now. Only a relatively small amount of mail is truly in dynamically updated folders. If there was a way to store "/archives/2002" folders through "/archives/2017" folders in one db and all of the other folders in another db, it would make backup and migration a much simpler task. Ok, I'm pretty sure that isn't in the immediate plan. But just let me put my vote in. Alternatively, is there any alternative to having one ever-growing mail db? Is there some trick with the db server that can present one logical db from multiple db files? (I know a lot about databases... but there's still a bunch I don't know). Has anyone else faced/addressed this problem? Or is the answer to just live with it? Thanks. Jerry - To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org For additional commands, e-mail: server-user-h...@james.apache.org - To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org For additional commands, e-mail: server-user-h...@james.apache.org - To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org For additional commands, e-mail: server-user-h...@james.apache.org - To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org For additional commands, e-mail: server-user-h...@james.apache.org
Re: Multiple Databases?
Hi, I think id does make sense to have such a functionality. No reason to keep archived emails in the main storage if hey are rarely accessed. Not sure on what solution to adopt as there are many possible ways to achieve this. One thing that I am thinking of is to provide a Composite Mailbox implementation. A rough sketch is bellow: This composite mailbox implementation could delegate actual processing to a couple of implementations. Each implementation could manage a group of folders. This solution could solve this use case by: - keep normal folders inside a database using jpa implementation - keep archived emails in a Maildir or S3 storage etc, possibly on slower, less expensive systems. I haven't given a lot of thought to this so I'm not sure if it is even possible at the moment, but I think the pattern has some merit. Regards, Eugen On 09.05.2018 07:46, Benoit Tellier wrote: > Hi Jerry, > > There is currently no solution to do what you describe. There is no > "custom storage treatment" for archived mails. > > I did not address nor faced this problem, thus can not really help you. > > Cheers, > > Benoit Tellier > > Le 09/05/2018 à 02:08, Jerry Malcolm a écrit : >> I realize this is a completely off-the-wall question... but has there >> been any discussion about breaking the JAMES db into multiple >> databases? I have a bunch of clients, each with a bunch of accounts. >> And they all want to archive all of their mail and be able to access it >> through the same mail client. As the years go by, the db just keeps >> growing and growing. This became acutely obvious when I was recently >> forced by my server provider to migrate to newer hardware, and I >> realized I had to have mail down for a long time while I transferred a >> 60gB+ file across a slow connection to the new server. The db size is >> also making daily backups a problem. >> >> The reality is that 90% of the mail is archived into 'year' folders for >> each of the accounts which are basically "read-only" now. Only a >> relatively small amount of mail is truly in dynamically updated >> folders. If there was a way to store "/archives/2002" folders through >> "/archives/2017" folders in one db and all of the other folders in >> another db, it would make backup and migration a much simpler task. >> >> Ok, I'm pretty sure that isn't in the immediate plan. But just let me >> put my vote in. Alternatively, is there any alternative to having one >> ever-growing mail db? Is there some trick with the db server that can >> present one logical db from multiple db files? (I know a lot about >> databases... but there's still a bunch I don't know). Has anyone else >> faced/addressed this problem? Or is the answer to just live with it? >> >> Thanks. >> >> Jerry >> >> >> - >> To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org >> For additional commands, e-mail: server-user-h...@james.apache.org >> > - > To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org > For additional commands, e-mail: server-user-h...@james.apache.org > - To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org For additional commands, e-mail: server-user-h...@james.apache.org
Re: Multiple Databases?
Hi Jerry, There is currently no solution to do what you describe. There is no "custom storage treatment" for archived mails. I did not address nor faced this problem, thus can not really help you. Cheers, Benoit Tellier Le 09/05/2018 à 02:08, Jerry Malcolm a écrit : > I realize this is a completely off-the-wall question... but has there > been any discussion about breaking the JAMES db into multiple > databases? I have a bunch of clients, each with a bunch of accounts. > And they all want to archive all of their mail and be able to access it > through the same mail client. As the years go by, the db just keeps > growing and growing. This became acutely obvious when I was recently > forced by my server provider to migrate to newer hardware, and I > realized I had to have mail down for a long time while I transferred a > 60gB+ file across a slow connection to the new server. The db size is > also making daily backups a problem. > > The reality is that 90% of the mail is archived into 'year' folders for > each of the accounts which are basically "read-only" now. Only a > relatively small amount of mail is truly in dynamically updated > folders. If there was a way to store "/archives/2002" folders through > "/archives/2017" folders in one db and all of the other folders in > another db, it would make backup and migration a much simpler task. > > Ok, I'm pretty sure that isn't in the immediate plan. But just let me > put my vote in. Alternatively, is there any alternative to having one > ever-growing mail db? Is there some trick with the db server that can > present one logical db from multiple db files? (I know a lot about > databases... but there's still a bunch I don't know). Has anyone else > faced/addressed this problem? Or is the answer to just live with it? > > Thanks. > > Jerry > > > - > To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org > For additional commands, e-mail: server-user-h...@james.apache.org > - To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org For additional commands, e-mail: server-user-h...@james.apache.org
Multiple Databases?
I realize this is a completely off-the-wall question... but has there been any discussion about breaking the JAMES db into multiple databases? I have a bunch of clients, each with a bunch of accounts. And they all want to archive all of their mail and be able to access it through the same mail client. As the years go by, the db just keeps growing and growing. This became acutely obvious when I was recently forced by my server provider to migrate to newer hardware, and I realized I had to have mail down for a long time while I transferred a 60gB+ file across a slow connection to the new server. The db size is also making daily backups a problem. The reality is that 90% of the mail is archived into 'year' folders for each of the accounts which are basically "read-only" now. Only a relatively small amount of mail is truly in dynamically updated folders. If there was a way to store "/archives/2002" folders through "/archives/2017" folders in one db and all of the other folders in another db, it would make backup and migration a much simpler task. Ok, I'm pretty sure that isn't in the immediate plan. But just let me put my vote in. Alternatively, is there any alternative to having one ever-growing mail db? Is there some trick with the db server that can present one logical db from multiple db files? (I know a lot about databases... but there's still a bunch I don't know). Has anyone else faced/addressed this problem? Or is the answer to just live with it? Thanks. Jerry - To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org For additional commands, e-mail: server-user-h...@james.apache.org
