Re: RabbitMQ as a MailQueue
Hi Matthieu, My answers are inlined. On 20/04/2020 16:25, Matthieu Baechler wrote: > Hi, > > [...] > > We studied a lot of solutions, ranging from having a standalone ActiveMQ server to implementing the MailQueue using RabbitMQ or Kafka. Why we chose RabbitMQ deserves an ADR don't you think? > [...] > > However, there are big drawbacks: > > * the complexity of the code is a bit crazy I don't by it. A time serie with applicative tombstones is not something complicated to me. > * the complexity of tuning is higher than we would like Maybe the settings correspond to the code technicity and not correlated to the administrator experience thus requiring the administrator to understand the way the solution works. A way around this issue could be having configuration reflecting admin expectations like: - Count of Cassandra servers - Maximum mail throughtput. Our technical parameters could then been derived from the supplied values. Technical values could still be exposed for advanced users BTW. > * RabbitMQ is not easy to put in cluster and has deciving > characteristics (like not support network partition very well) > * we didn't implement "delay" feature for RabbitMQ because there's no > easy solution to that, what prevents this MailQueue implementation to > be used for SMTP out (it would not handle delivery retries) I proposed https://issues.apache.org/jira/browse/JAMES-2896 which sounds achievable given reasonable enhancement. Can you explain to me why such a solution would not be doable to you? Maybe discussing it in the issue > > We discussed recenlty about adding delay feature to it but as it would > not make other points better and cost a lot to implement, I'm willing > to debate alternatives. > > > My opinion is that the messaging technology should not be a limiting > factor now that we put the blobs out of it (it's a refactoring we did > for RabbitMQ). We can probably handle thousand mails per second with a > standalone ActiveMQ without any issue. > > > Once the MailQueue is an scalability issue again, we could either > evaluate back RabbitMQ or switch to Kafka or Pulsar. I don't know pulsar enough but I don't think Kafka could handle all administrative operations (especially arbitrary deletion) by itself. IMO the gain on the complexity aspect would be limited. > WDYT about that plan? I'm never against people willing to explore alternatives. However we might then cosider fully remove rabbitMQ support, thus for the EventBus implementation and for the distributed task manager, in order not to ship a product with two messaging dependencies. Regards Benoit - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3153) Upgrade reactor to make use of retryWhen and RetrySpec
[ https://issues.apache.org/jira/browse/JAMES-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthieu Baechler updated JAMES-3153: - Description: Latest reactor version include some new features regarding retry backoff (see https://github.com/reactor/reactor-core/pull/1979 ) As we were limited by the previous API, we did copy some code from reactor to workaround the limitations. Now that they implement a RetrySpec, we should be able to get rid of this code duplication and rely on their well-tested feature. h2. Definition of done: * All reactive code should still work as before * The code should use RetryWhen with RetrySpec instead of previous solutions was: Latest reactor version include some new features regarding retry backoff (see https://github.com/reactor/reactor-core/pull/1979 ) As we were limited by the previous API, we did copy some code from reactor to workaround the limitations. Now that they implement a RetrySpec, we should be able to get rid of this code duplication and rely on their well-tested feature. > Upgrade reactor to make use of retryWhen and RetrySpec > -- > > Key: JAMES-3153 > URL: https://issues.apache.org/jira/browse/JAMES-3153 > Project: James Server > Issue Type: Improvement >Reporter: Matthieu Baechler >Priority: Major > > Latest reactor version include some new features regarding retry backoff (see > https://github.com/reactor/reactor-core/pull/1979 ) > As we were limited by the previous API, we did copy some code from reactor to > workaround the limitations. > Now that they implement a RetrySpec, we should be able to get rid of this > code duplication and rely on their well-tested feature. > h2. Definition of done: > * All reactive code should still work as before > * The code should use RetryWhen with RetrySpec instead of previous solutions -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Created] (MAILBOX-397) getting a mapper should not throw
Benoit Tellier created MAILBOX-397: -- Summary: getting a mapper should not throw Key: MAILBOX-397 URL: https://issues.apache.org/jira/browse/MAILBOX-397 Project: James Mailbox Issue Type: Improvement Components: store Reporter: Benoit Tellier https://github.com/linagora/james-project/pull/3332#discussion_r416464020 A quick analyses shows Exception is never thrown upon mapper instanciation. I propose to remove that boiler plate annoying code in a side pull request. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Created] (JAMES-3158) Deprecation of HybridBlobStore
Lan Khuat created JAMES-3158: Summary: Deprecation of HybridBlobStore Key: JAMES-3158 URL: https://issues.apache.org/jira/browse/JAMES-3158 Project: James Server Issue Type: Improvement Reporter: Lan Khuat After the discussion in [https://github.com/linagora/james-project/pull/3331], we decided that the HybridBlobStore should be deprecated first before being completely removed from James. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Commented] (JAMES-3156) Removal of HybridBlobStore
[ https://issues.apache.org/jira/browse/JAMES-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094398#comment-17094398 ] Lan Khuat commented on JAMES-3156: -- See: https://issues.apache.org/jira/browse/JAMES-3158 > Removal of HybridBlobStore > -- > > Key: JAMES-3156 > URL: https://issues.apache.org/jira/browse/JAMES-3156 > Project: James Server > Issue Type: Improvement >Reporter: Lan Khuat >Priority: Major > > HybridBlobStore will be replaced by CacheBlobStore in the upcoming version, > and thus we need to: > * Remove the HybridBlobStore along with all of its tests & configurations. > * Remove related documentation. > * Add an upgrade instruction. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Created] (JAMES-3159) Implement web key discovery protocol based on draft from Werner Koch
Manuel Blechschmidt created JAMES-3159: -- Summary: Implement web key discovery protocol based on draft from Werner Koch Key: JAMES-3159 URL: https://issues.apache.org/jira/browse/JAMES-3159 Project: James Server Issue Type: New Feature Components: protocols Affects Versions: 3.6.0 Reporter: Manuel Blechschmidt Attachments: Screenshot from 2020-04-28 13-07-11.png This issue is for tracking the implementation of the OpenPGP Web Key Directory [https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-07.html] It will first implement the direct method on the url the mail servers runs on: https://example.org/.well-known/openpgpkey/ hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe Here is an overview diagram: !Screenshot from 2020-04-28 13-07-11.png! The implementation will be done in the following repository: [https://github.com/ManuelB/james-project] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3161) Create wkd maven project in server/protocols
[ https://issues.apache.org/jira/browse/JAMES-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Blechschmidt updated JAMES-3161: --- Summary: Create wkd maven project in server/protocols (was: Create wkd project in server/protocols) > Create wkd maven project in server/protocols > > > Key: JAMES-3161 > URL: https://issues.apache.org/jira/browse/JAMES-3161 > Project: James Server > Issue Type: Sub-task >Reporter: Manuel Blechschmidt >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Created] (JAMES-3160) Create branch feature/JAMES-3159-web-discovery-protocol based on tag james-project-3.5.0
Manuel Blechschmidt created JAMES-3160:
--
Summary: Create branch feature/JAMES-3159-web-discovery-protocol
based on tag james-project-3.5.0
Key: JAMES-3160
URL: https://issues.apache.org/jira/browse/JAMES-3160
Project: James Server
Issue Type: Sub-task
Reporter: Manuel Blechschmidt
{code}
git clone https://git.apache.org/repos/asf/james-project.git
git checkout james-project-3.5.0
git checkout -b feature/JAMES-3159-web-discovery-protocol
# Forked repository on github
git remote add ManuelB https://github.com/ManuelB/james-project.git
git push --set-upstream ManuelB feature/JAMES-3159-web-discovery-protocol
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Resolved] (JAMES-3160) Create branch feature/JAMES-3159-web-discovery-protocol based on tag james-project-3.5.0
[
https://issues.apache.org/jira/browse/JAMES-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manuel Blechschmidt resolved JAMES-3160.
Resolution: Fixed
> Create branch feature/JAMES-3159-web-discovery-protocol based on tag
> james-project-3.5.0
>
>
> Key: JAMES-3160
> URL: https://issues.apache.org/jira/browse/JAMES-3160
> Project: James Server
> Issue Type: Sub-task
>Reporter: Manuel Blechschmidt
>Priority: Major
>
> {code}
> git clone https://git.apache.org/repos/asf/james-project.git
> git checkout james-project-3.5.0
> git checkout -b feature/JAMES-3159-web-discovery-protocol
> # Forked repository on github
> git remote add ManuelB https://github.com/ManuelB/james-project.git
> git push --set-upstream ManuelB feature/JAMES-3159-web-discovery-protocol
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Created] (JAMES-3161) Create wkd project in server/protocols
Manuel Blechschmidt created JAMES-3161: -- Summary: Create wkd project in server/protocols Key: JAMES-3161 URL: https://issues.apache.org/jira/browse/JAMES-3161 Project: James Server Issue Type: Sub-task Reporter: Manuel Blechschmidt -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Created] (JAMES-3162) Create WebKeyDisoveryServer with routes
Manuel Blechschmidt created JAMES-3162: -- Summary: Create WebKeyDisoveryServer with routes Key: JAMES-3162 URL: https://issues.apache.org/jira/browse/JAMES-3162 Project: James Server Issue Type: Sub-task Reporter: Manuel Blechschmidt The server has to serve the following GET requests: https://example.org/.well-known/openpgpkey/hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe https://example.org/.well-known/openpgpkey/policy https://example.org/.well-known/openpgpkey/submission-address -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Created] (JAMES-3163) Create WebKeyDiscoveryStore including File and Memory implementation
Manuel Blechschmidt created JAMES-3163: -- Summary: Create WebKeyDiscoveryStore including File and Memory implementation Key: JAMES-3163 URL: https://issues.apache.org/jira/browse/JAMES-3163 Project: James Server Issue Type: Sub-task Reporter: Manuel Blechschmidt * WebKeyDiscoveryStore * MemoryWebKeyDiscoveryStore * FileWebKeyDiscoveryStore -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Created] (JAMES-3164) Create WebKeyDiscoveryMailet that can process and create emails Web Key Directory Update Protocol
Manuel Blechschmidt created JAMES-3164: -- Summary: Create WebKeyDiscoveryMailet that can process and create emails Web Key Directory Update Protocol Key: JAMES-3164 URL: https://issues.apache.org/jira/browse/JAMES-3164 Project: James Server Issue Type: Sub-task Reporter: Manuel Blechschmidt https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-07.html#rfc.section.4 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Created] (JAMES-3165) Create wkd data project james-server-data-wkd
Manuel Blechschmidt created JAMES-3165: -- Summary: Create wkd data project james-server-data-wkd Key: JAMES-3165 URL: https://issues.apache.org/jira/browse/JAMES-3165 Project: James Server Issue Type: Sub-task Reporter: Manuel Blechschmidt -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Resolved] (JAMES-3165) Create wkd data project james-server-data-wkd
[ https://issues.apache.org/jira/browse/JAMES-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Blechschmidt resolved JAMES-3165. Resolution: Fixed > Create wkd data project james-server-data-wkd > - > > Key: JAMES-3165 > URL: https://issues.apache.org/jira/browse/JAMES-3165 > Project: James Server > Issue Type: Sub-task >Reporter: Manuel Blechschmidt >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3166) Create james-server-guice-wkd project with WebKeyDiscoveryModule
[ https://issues.apache.org/jira/browse/JAMES-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Blechschmidt updated JAMES-3166: --- Summary: Create james-server-guice-wkd project with WebKeyDiscoveryModule (was: Create james-server-guice-wkd) > Create james-server-guice-wkd project with WebKeyDiscoveryModule > > > Key: JAMES-3166 > URL: https://issues.apache.org/jira/browse/JAMES-3166 > Project: James Server > Issue Type: Sub-task >Reporter: Manuel Blechschmidt >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Created] (JAMES-3166) Create james-server-guice-wkd
Manuel Blechschmidt created JAMES-3166: -- Summary: Create james-server-guice-wkd Key: JAMES-3166 URL: https://issues.apache.org/jira/browse/JAMES-3166 Project: James Server Issue Type: Sub-task Reporter: Manuel Blechschmidt -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Resolved] (JAMES-3161) Create wkd maven project in server/protocols
[ https://issues.apache.org/jira/browse/JAMES-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Blechschmidt resolved JAMES-3161. Resolution: Fixed > Create wkd maven project in server/protocols > > > Key: JAMES-3161 > URL: https://issues.apache.org/jira/browse/JAMES-3161 > Project: James Server > Issue Type: Sub-task >Reporter: Manuel Blechschmidt >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3160) Create branch feature/JAMES-3159-web-directory-protocol based on tag james-project-3.5.0
[
https://issues.apache.org/jira/browse/JAMES-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manuel Blechschmidt updated JAMES-3160:
---
Summary: Create branch feature/JAMES-3159-web-directory-protocol based on
tag james-project-3.5.0 (was: Create branch
feature/JAMES-3159-web-discovery-protocol based on tag james-project-3.5.0)
> Create branch feature/JAMES-3159-web-directory-protocol based on tag
> james-project-3.5.0
>
>
> Key: JAMES-3160
> URL: https://issues.apache.org/jira/browse/JAMES-3160
> Project: James Server
> Issue Type: Sub-task
>Reporter: Manuel Blechschmidt
>Priority: Major
>
> {code}
> git clone https://git.apache.org/repos/asf/james-project.git
> git checkout james-project-3.5.0
> git checkout -b feature/JAMES-3159-web-discovery-protocol
> # Forked repository on github
> git remote add ManuelB https://github.com/ManuelB/james-project.git
> git push --set-upstream ManuelB feature/JAMES-3159-web-discovery-protocol
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3159) Implement web key directory protocol based on draft from Werner Koch
[ https://issues.apache.org/jira/browse/JAMES-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Blechschmidt updated JAMES-3159: --- Attachment: Screenshot from 2020-04-28 13-46-53.png > Implement web key directory protocol based on draft from Werner Koch > > > Key: JAMES-3159 > URL: https://issues.apache.org/jira/browse/JAMES-3159 > Project: James Server > Issue Type: New Feature > Components: protocols >Affects Versions: 3.6.0 >Reporter: Manuel Blechschmidt >Priority: Major > Attachments: Screenshot from 2020-04-28 13-46-53.png > > > This issue is for tracking the implementation of the OpenPGP Web Key Directory > [https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-07.html] > It will first implement the direct method on the url the mail servers runs on: > https://example.org/.well-known/openpgpkey/ > hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe > Here is an overview diagram: > !Screenshot from 2020-04-28 13-07-11.png! > The implementation will be done in the following repository: > [https://github.com/ManuelB/james-project] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3159) Implement web key directory protocol based on draft from Werner Koch
[ https://issues.apache.org/jira/browse/JAMES-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Blechschmidt updated JAMES-3159: --- Summary: Implement web key directory protocol based on draft from Werner Koch (was: Implement web key discovery protocol based on draft from Werner Koch) > Implement web key directory protocol based on draft from Werner Koch > > > Key: JAMES-3159 > URL: https://issues.apache.org/jira/browse/JAMES-3159 > Project: James Server > Issue Type: New Feature > Components: protocols >Affects Versions: 3.6.0 >Reporter: Manuel Blechschmidt >Priority: Major > Attachments: Screenshot from 2020-04-28 13-46-53.png > > > This issue is for tracking the implementation of the OpenPGP Web Key Directory > [https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-07.html] > It will first implement the direct method on the url the mail servers runs on: > https://example.org/.well-known/openpgpkey/ > hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe > Here is an overview diagram: > !Screenshot from 2020-04-28 13-07-11.png! > The implementation will be done in the following repository: > [https://github.com/ManuelB/james-project] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3159) Implement web key directory protocol based on draft from Werner Koch
[ https://issues.apache.org/jira/browse/JAMES-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Blechschmidt updated JAMES-3159: --- Attachment: (was: Screenshot from 2020-04-28 13-07-11.png) > Implement web key directory protocol based on draft from Werner Koch > > > Key: JAMES-3159 > URL: https://issues.apache.org/jira/browse/JAMES-3159 > Project: James Server > Issue Type: New Feature > Components: protocols >Affects Versions: 3.6.0 >Reporter: Manuel Blechschmidt >Priority: Major > Attachments: Screenshot from 2020-04-28 13-46-53.png > > > This issue is for tracking the implementation of the OpenPGP Web Key Directory > [https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-07.html] > It will first implement the direct method on the url the mail servers runs on: > https://example.org/.well-known/openpgpkey/ > hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe > Here is an overview diagram: > !Screenshot from 2020-04-28 13-07-11.png! > The implementation will be done in the following repository: > [https://github.com/ManuelB/james-project] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3159) Implement web key directory protocol based on draft from Werner Koch
[ https://issues.apache.org/jira/browse/JAMES-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Blechschmidt updated JAMES-3159: --- Description: This issue is for tracking the implementation of the OpenPGP Web Key Directory [https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-07.html] It will first implement the direct method on the url the mail servers runs on: https://example.org/.well-known/openpgpkey/ hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe Here is an overview diagram: !Screenshot from 2020-04-28 13-46-53.png! The implementation will be done in the following repository: [https://github.com/ManuelB/james-project] was: This issue is for tracking the implementation of the OpenPGP Web Key Directory [https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-07.html] It will first implement the direct method on the url the mail servers runs on: https://example.org/.well-known/openpgpkey/ hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe Here is an overview diagram: !Screenshot from 2020-04-28 13-07-11.png! The implementation will be done in the following repository: [https://github.com/ManuelB/james-project] > Implement web key directory protocol based on draft from Werner Koch > > > Key: JAMES-3159 > URL: https://issues.apache.org/jira/browse/JAMES-3159 > Project: James Server > Issue Type: New Feature > Components: protocols >Affects Versions: 3.6.0 >Reporter: Manuel Blechschmidt >Priority: Major > Attachments: Screenshot from 2020-04-28 13-46-53.png > > > This issue is for tracking the implementation of the OpenPGP Web Key Directory > [https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-07.html] > It will first implement the direct method on the url the mail servers runs on: > https://example.org/.well-known/openpgpkey/ > hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe > Here is an overview diagram: > !Screenshot from 2020-04-28 13-46-53.png! > The implementation will be done in the following repository: > [https://github.com/ManuelB/james-project] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3160) Create branch feature/JAMES-3159-web-directory-protocol based on tag james-project-3.5.0
[
https://issues.apache.org/jira/browse/JAMES-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manuel Blechschmidt updated JAMES-3160:
---
Description:
{code}
git clone https://git.apache.org/repos/asf/james-project.git
git checkout james-project-3.5.0
git checkout -b feature/JAMES-3159-web-directory-protocol
# Forked repository on github
git remote add ManuelB https://github.com/ManuelB/james-project.git
git push --set-upstream ManuelB feature/JAMES-3159-web-directory-protocol
{code}
was:
{code}
git clone https://git.apache.org/repos/asf/james-project.git
git checkout james-project-3.5.0
git checkout -b feature/JAMES-3159-web-discovery-protocol
# Forked repository on github
git remote add ManuelB https://github.com/ManuelB/james-project.git
git push --set-upstream ManuelB feature/JAMES-3159-web-discovery-protocol
{code}
> Create branch feature/JAMES-3159-web-directory-protocol based on tag
> james-project-3.5.0
>
>
> Key: JAMES-3160
> URL: https://issues.apache.org/jira/browse/JAMES-3160
> Project: James Server
> Issue Type: Sub-task
>Reporter: Manuel Blechschmidt
>Priority: Major
>
> {code}
> git clone https://git.apache.org/repos/asf/james-project.git
> git checkout james-project-3.5.0
> git checkout -b feature/JAMES-3159-web-directory-protocol
> # Forked repository on github
> git remote add ManuelB https://github.com/ManuelB/james-project.git
> git push --set-upstream ManuelB feature/JAMES-3159-web-directory-protocol
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3160) Create branch feature/JAMES-3159-web-directory-protocol based on tag james-project-3.5.0
[
https://issues.apache.org/jira/browse/JAMES-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manuel Blechschmidt updated JAMES-3160:
---
Description:
{code}
git clone https://git.apache.org/repos/asf/james-project.git
git checkout james-project-3.5.0
git checkout -b feature/JAMES-3159-web-key--directory-protocol
# Forked repository on github
git remote add ManuelB https://github.com/ManuelB/james-project.git
git push --set-upstream ManuelB feature/JAMES-3159-web-key--directory-protocol
{code}
was:
{code}
git clone https://git.apache.org/repos/asf/james-project.git
git checkout james-project-3.5.0
git checkout -b feature/JAMES-3159-web-directory-protocol
# Forked repository on github
git remote add ManuelB https://github.com/ManuelB/james-project.git
git push --set-upstream ManuelB feature/JAMES-3159-web-directory-protocol
{code}
> Create branch feature/JAMES-3159-web-directory-protocol based on tag
> james-project-3.5.0
>
>
> Key: JAMES-3160
> URL: https://issues.apache.org/jira/browse/JAMES-3160
> Project: James Server
> Issue Type: Sub-task
>Reporter: Manuel Blechschmidt
>Priority: Major
>
> {code}
> git clone https://git.apache.org/repos/asf/james-project.git
> git checkout james-project-3.5.0
> git checkout -b feature/JAMES-3159-web-key--directory-protocol
> # Forked repository on github
> git remote add ManuelB https://github.com/ManuelB/james-project.git
> git push --set-upstream ManuelB feature/JAMES-3159-web-key--directory-protocol
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3160) Create branch feature/JAMES-3159-web-directory-protocol based on tag james-project-3.5.0
[
https://issues.apache.org/jira/browse/JAMES-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manuel Blechschmidt updated JAMES-3160:
---
Description:
{code}
git clone https://git.apache.org/repos/asf/james-project.git
git checkout james-project-3.5.0
git checkout -b feature/JAMES-3159-web-key-directory-protocol
# Forked repository on github
git remote add ManuelB https://github.com/ManuelB/james-project.git
git push --set-upstream ManuelB feature/JAMES-3159-web-key-directory-protocol
{code}
was:
{code}
git clone https://git.apache.org/repos/asf/james-project.git
git checkout james-project-3.5.0
git checkout -b feature/JAMES-3159-web-key--directory-protocol
# Forked repository on github
git remote add ManuelB https://github.com/ManuelB/james-project.git
git push --set-upstream ManuelB feature/JAMES-3159-web-key--directory-protocol
{code}
> Create branch feature/JAMES-3159-web-directory-protocol based on tag
> james-project-3.5.0
>
>
> Key: JAMES-3160
> URL: https://issues.apache.org/jira/browse/JAMES-3160
> Project: James Server
> Issue Type: Sub-task
>Reporter: Manuel Blechschmidt
>Priority: Major
>
> {code}
> git clone https://git.apache.org/repos/asf/james-project.git
> git checkout james-project-3.5.0
> git checkout -b feature/JAMES-3159-web-key-directory-protocol
> # Forked repository on github
> git remote add ManuelB https://github.com/ManuelB/james-project.git
> git push --set-upstream ManuelB feature/JAMES-3159-web-key-directory-protocol
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3163) Create WebKeyDirectoryStore including File and Memory implementation
[ https://issues.apache.org/jira/browse/JAMES-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Blechschmidt updated JAMES-3163: --- Summary: Create WebKeyDirectoryStore including File and Memory implementation (was: Create WebKeyDiscoveryStore including File and Memory implementation) > Create WebKeyDirectoryStore including File and Memory implementation > > > Key: JAMES-3163 > URL: https://issues.apache.org/jira/browse/JAMES-3163 > Project: James Server > Issue Type: Sub-task >Reporter: Manuel Blechschmidt >Priority: Major > > * WebKeyDiscoveryStore > * MemoryWebKeyDiscoveryStore > * FileWebKeyDiscoveryStore -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3162) Create WebKeyDirectoryServer with routes
[ https://issues.apache.org/jira/browse/JAMES-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Blechschmidt updated JAMES-3162: --- Summary: Create WebKeyDirectoryServer with routes (was: Create WebKeyDisoveryServer with routes) > Create WebKeyDirectoryServer with routes > > > Key: JAMES-3162 > URL: https://issues.apache.org/jira/browse/JAMES-3162 > Project: James Server > Issue Type: Sub-task >Reporter: Manuel Blechschmidt >Priority: Major > > The server has to serve the following GET requests: > https://example.org/.well-known/openpgpkey/hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe > https://example.org/.well-known/openpgpkey/policy > https://example.org/.well-known/openpgpkey/submission-address -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3160) Create branch feature/JAMES-3159-web-key-directory-protocol based on tag james-project-3.5.0
[
https://issues.apache.org/jira/browse/JAMES-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manuel Blechschmidt updated JAMES-3160:
---
Summary: Create branch feature/JAMES-3159-web-key-directory-protocol based
on tag james-project-3.5.0 (was: Create branch
feature/JAMES-3159-web-directory-protocol based on tag james-project-3.5.0)
> Create branch feature/JAMES-3159-web-key-directory-protocol based on tag
> james-project-3.5.0
>
>
> Key: JAMES-3160
> URL: https://issues.apache.org/jira/browse/JAMES-3160
> Project: James Server
> Issue Type: Sub-task
>Reporter: Manuel Blechschmidt
>Priority: Major
>
> {code}
> git clone https://git.apache.org/repos/asf/james-project.git
> git checkout james-project-3.5.0
> git checkout -b feature/JAMES-3159-web-key-directory-protocol
> # Forked repository on github
> git remote add ManuelB https://github.com/ManuelB/james-project.git
> git push --set-upstream ManuelB feature/JAMES-3159-web-key-directory-protocol
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3163) Create WebKeyDirectoryStore including File and Memory implementation
[ https://issues.apache.org/jira/browse/JAMES-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Blechschmidt updated JAMES-3163: --- Description: * WebKeyDirectoryStore * MemoryWebKeyDirectoryStore * FileWebKeyDirectoryStore was: * WebKeyDiscoveryStore * MemoryWebKeyDiscoveryStore * FileWebKeyDiscoveryStore > Create WebKeyDirectoryStore including File and Memory implementation > > > Key: JAMES-3163 > URL: https://issues.apache.org/jira/browse/JAMES-3163 > Project: James Server > Issue Type: Sub-task >Reporter: Manuel Blechschmidt >Priority: Major > > * WebKeyDirectoryStore > * MemoryWebKeyDirectoryStore > * FileWebKeyDirectoryStore -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3164) Create WebKeyDirectoryMailet that can process and create emails Web Key Directory Update Protocol
[ https://issues.apache.org/jira/browse/JAMES-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Blechschmidt updated JAMES-3164: --- Summary: Create WebKeyDirectoryMailet that can process and create emails Web Key Directory Update Protocol (was: Create WebKeyDiscoveryMailet that can process and create emails Web Key Directory Update Protocol) > Create WebKeyDirectoryMailet that can process and create emails Web Key > Directory Update Protocol > - > > Key: JAMES-3164 > URL: https://issues.apache.org/jira/browse/JAMES-3164 > Project: James Server > Issue Type: Sub-task >Reporter: Manuel Blechschmidt >Priority: Major > > https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-07.html#rfc.section.4 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3166) Create james-server-guice-wkd project with WebKeyDirectoryModule
[ https://issues.apache.org/jira/browse/JAMES-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Blechschmidt updated JAMES-3166: --- Summary: Create james-server-guice-wkd project with WebKeyDirectoryModule (was: Create james-server-guice-wkd project with WebKeyDiscoveryModule) > Create james-server-guice-wkd project with WebKeyDirectoryModule > > > Key: JAMES-3166 > URL: https://issues.apache.org/jira/browse/JAMES-3166 > Project: James Server > Issue Type: Sub-task >Reporter: Manuel Blechschmidt >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-3159) Implement web key directory protocol based on draft from Werner Koch
[ https://issues.apache.org/jira/browse/JAMES-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Blechschmidt updated JAMES-3159: --- Description: This issue is for tracking the implementation of the OpenPGP Web Key Directory [https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-07.html] It will first implement the direct method on the url the mail servers runs on: https://example.org/.well-known/openpgpkey/ hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe Here is an overview diagram: !Screenshot from 2020-04-28 13-46-53.png! The implementation will be done in the following repository: [https://github.com/ManuelB/james-project/tree/feature/JAMES-3159-web-key-directory-protocol] was: This issue is for tracking the implementation of the OpenPGP Web Key Directory [https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-07.html] It will first implement the direct method on the url the mail servers runs on: https://example.org/.well-known/openpgpkey/ hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe Here is an overview diagram: !Screenshot from 2020-04-28 13-46-53.png! The implementation will be done in the following repository: [https://github.com/ManuelB/james-project] > Implement web key directory protocol based on draft from Werner Koch > > > Key: JAMES-3159 > URL: https://issues.apache.org/jira/browse/JAMES-3159 > Project: James Server > Issue Type: New Feature > Components: protocols >Affects Versions: 3.6.0 >Reporter: Manuel Blechschmidt >Priority: Major > Attachments: Screenshot from 2020-04-28 13-46-53.png > > > This issue is for tracking the implementation of the OpenPGP Web Key Directory > [https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-07.html] > It will first implement the direct method on the url the mail servers runs on: > https://example.org/.well-known/openpgpkey/ > hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe > Here is an overview diagram: > !Screenshot from 2020-04-28 13-46-53.png! > The implementation will be done in the following repository: > [https://github.com/ManuelB/james-project/tree/feature/JAMES-3159-web-key-directory-protocol] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Commented] (JAMES-1394) Mails stuck in queue/spool
[ https://issues.apache.org/jira/browse/JAMES-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094449#comment-17094449 ] Raphael Ouazana commented on JAMES-1394: This should be fixed on the oncoming 3.5.0. You can already try it see [https://www.mail-archive.com/server-dev@james.apache.org/msg65145.html] . > Mails stuck in queue/spool > -- > > Key: JAMES-1394 > URL: https://issues.apache.org/jira/browse/JAMES-1394 > Project: James Server > Issue Type: Bug > Components: Queue, Remote Delivery >Affects Versions: 3.0-beta3 > Environment: James 3.0 Beta3 > linux 2.6.24-23-xen #1 SMP Mon Jan 26 03:09:12 UTC 2009 x86_64 GNU/Linux >Reporter: Thomas Eichmann >Priority: Major > Attachments: jstack-output-stuck.txt, out.txt > > > Issue created based on mailing list discussion here: > http://www.mail-archive.com/server-user@james.apache.org/msg13171.html > Citing the mail content: > First a short background info, what our company would like to achieve using > James. We have several clients that need mass-mailing capability, thus we > want to create a system that allows us to track these mails and especially > store errors from bounces (also by using VERP), so we can block unreachable > addresses. I created a few custom mailets to handle those requirements and a > custom RemoteDelivery implementation for creating VERP style sender > addresses. The storage is all done in a MySQL database via JPA. Currently we > use it for one client which is not sending huge amounts of mails, around > 80,000 per month. > The system works well, but it happens after a bigger batch of mails (around > 4,000 mails) that the mail queueing system gets somehow "stuck". The SMTP > server is able to retrieve new mails, but somehow they don't get send out, > until I restart the server. Sometimes a flush() on queue/spool helps, but > most of the times a restart is necessary. > I added logging to my mailets when they start and finish to see if there is a > problem, but they seem to run fine (no exceptions). It seems that the mails > that arrive after James gets stuck get processed normally by the mailets, but > somehow are not dequeued and sent out. > Server info > James 3.0 Beta3 > linux 2.6.24-23-xen #1 SMP Mon Jan 26 03:09:12 UTC 2009 x86_64 GNU/Linux > Frankly I'm not sure what is the best way to debug this kind of problem with > James. I could find a few other threads with people describing similar > problems, but I couldn't get closer to the problem. > Here some data from JMX that I took while the server was in "stuck" status. > === > INFO org.apache.james:type=component,name=queue,queue=outgoing > === > Size == 80 > === > INFO org.apache.james:type=component,name=queue,queue=spool > === > Size == 0 > === > INFO > org.apache.james:type=component,component=mailetcontainer,name=mailspooler > === > CurrentSpoolCount == 0 > ThreadCount == 20 > === > INFO > org.apache.james:type=component,component=mailetcontainer,name=mailspooler,sub-type=threadpool,threadpool=dequeuer > === > TotalTasks == 0 > AverageTaskTime == 0.0 > ActiveThreads == 2 > ActiveTasks == 2 > QueuedTasks == 0 > MaximalThreads == 2 > === > INFO > org.apache.james:type=component,component=mailetcontainer,name=mailspooler,sub-type=threadpool,threadpool=spooler > === > TotalTasks == 4653 > AverageTaskTime == 145.0 > ActiveThreads == 20 > ActiveTasks == 0 > QueuedTasks == 0 > MaximalThreads == 20 > I also created a jstack output, as it was suggested in one of the threads I > found. I can post it here as well if it could be helpful (I didn't do it now, > because the file is quite large). > This is the logging output when a new mail is sent while the queue is stuck > (IgnoreBlockedMailet and SendAttemptLogMailet are my custom mailets): > DEBUG 03:54:46,819 | james.mailspooler | Begin processing mail > Mail1322625286815-aa8034cb-4d50-4044-926f-10bbe8b637e5 > DEBUG 03:54:46,819 | james.mailprocessor | Call MailProcessor root > INFO 03:54:46,827 | james.mailetcontext | IgnoreBlockedMailet called - > recipients=[t.eichm...@eggsist.com] > INFO 03:54:46,847 | james.mailetcontext | IgnoreBlockedMailet finished - > recipients=[t.eichm...@eggsist.com] > DEBUG 03:54:46,847 | james.mailprocessor | Call MailProcessor transport >
[jira] [Updated] (JAMES-3159) Implement web key directory protocol based on draft from Werner Koch
[ https://issues.apache.org/jira/browse/JAMES-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Blechschmidt updated JAMES-3159: --- Description: This issue is for tracking the implementation of the OpenPGP Web Key Directory [https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-07.html] It will first implement the direct method on the url the mail servers runs on: https://example.org/.well-known/openpgpkey/ hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe Here is an overview diagram: !Screenshot from 2020-04-28 13-46-53.png! The implementation will be done in the following repository: [https://github.com/ManuelB/james-project/tree/feature/JAMES-3159-web-key-directory-protocol] In the first iteration only auth-submit will be implemented. This means that the public key will directly be published without waiting for the confirmation mail. was: This issue is for tracking the implementation of the OpenPGP Web Key Directory [https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-07.html] It will first implement the direct method on the url the mail servers runs on: https://example.org/.well-known/openpgpkey/ hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe Here is an overview diagram: !Screenshot from 2020-04-28 13-46-53.png! The implementation will be done in the following repository: [https://github.com/ManuelB/james-project/tree/feature/JAMES-3159-web-key-directory-protocol] > Implement web key directory protocol based on draft from Werner Koch > > > Key: JAMES-3159 > URL: https://issues.apache.org/jira/browse/JAMES-3159 > Project: James Server > Issue Type: New Feature > Components: protocols >Affects Versions: 3.6.0 >Reporter: Manuel Blechschmidt >Priority: Major > Attachments: Screenshot from 2020-04-28 13-46-53.png > > > This issue is for tracking the implementation of the OpenPGP Web Key Directory > [https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-07.html] > It will first implement the direct method on the url the mail servers runs on: > https://example.org/.well-known/openpgpkey/ > hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe > Here is an overview diagram: > !Screenshot from 2020-04-28 13-46-53.png! > The implementation will be done in the following repository: > [https://github.com/ManuelB/james-project/tree/feature/JAMES-3159-web-key-directory-protocol] > In the first iteration only auth-submit will be implemented. This means that > the public key will directly be published without waiting for the > confirmation mail. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Commented] (JAMES-3159) Implement web key directory protocol based on draft from Werner Koch
[ https://issues.apache.org/jira/browse/JAMES-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094499#comment-17094499 ] Raphael Ouazana commented on JAMES-3159: Very nice! > Implement web key directory protocol based on draft from Werner Koch > > > Key: JAMES-3159 > URL: https://issues.apache.org/jira/browse/JAMES-3159 > Project: James Server > Issue Type: New Feature > Components: protocols >Affects Versions: 3.6.0 >Reporter: Manuel Blechschmidt >Priority: Major > Attachments: Screenshot from 2020-04-28 13-46-53.png > > > This issue is for tracking the implementation of the OpenPGP Web Key Directory > [https://tools.ietf.org/id/draft-koch-openpgp-webkey-service-07.html] > It will first implement the direct method on the url the mail servers runs on: > https://example.org/.well-known/openpgpkey/ > hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe > Here is an overview diagram: > !Screenshot from 2020-04-28 13-46-53.png! > The implementation will be done in the following repository: > [https://github.com/ManuelB/james-project/tree/feature/JAMES-3159-web-key-directory-protocol] > In the first iteration only auth-submit will be implemented. This means that > the public key will directly be published without waiting for the > confirmation mail. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Resolved] (JAMES-3162) Create WebKeyDirectoryServer with routes
[ https://issues.apache.org/jira/browse/JAMES-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manuel Blechschmidt resolved JAMES-3162. Resolution: Fixed > Create WebKeyDirectoryServer with routes > > > Key: JAMES-3162 > URL: https://issues.apache.org/jira/browse/JAMES-3162 > Project: James Server > Issue Type: Sub-task >Reporter: Manuel Blechschmidt >Priority: Major > > The server has to serve the following GET requests: > https://example.org/.well-known/openpgpkey/hu/iy9q119eutrkn8s1mk4r39qejnbu3n5q?l=Joe.Doe > https://example.org/.well-known/openpgpkey/policy > https://example.org/.well-known/openpgpkey/submission-address -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 27/27: JAMES-3153 use reactor.Retry insteaf of reactor-extra version
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit b4de77a1e58708d380f8d7b669fd75f93970125c
Author: Matthieu Baechler
AuthorDate: Fri Apr 24 12:14:20 2020 +0200
JAMES-3153 use reactor.Retry insteaf of reactor-extra version
---
server/blob/blob-objectstorage/pom.xml| 4
.../objectstorage/StreamCompatibleBlobPutter.java | 19 +--
.../blob/objectstorage/aws/AwsS3ObjectStorage.java| 11 +--
3 files changed, 14 insertions(+), 20 deletions(-)
diff --git a/server/blob/blob-objectstorage/pom.xml
b/server/blob/blob-objectstorage/pom.xml
index 4d2f297..5729d35 100644
--- a/server/blob/blob-objectstorage/pom.xml
+++ b/server/blob/blob-objectstorage/pom.xml
@@ -87,10 +87,6 @@
guava
-io.projectreactor.addons
-reactor-extra
-
-
org.apache.jclouds.api
openstack-swift
${jclouds.version}
diff --git
a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/StreamCompatibleBlobPutter.java
b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/StreamCompatibleBlobPutter.java
index 46d1d2e..35da9c8 100644
---
a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/StreamCompatibleBlobPutter.java
+++
b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/StreamCompatibleBlobPutter.java
@@ -34,7 +34,7 @@ import org.jclouds.http.HttpResponseException;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;
-import reactor.retry.Retry;
+import reactor.util.retry.Retry;
public class StreamCompatibleBlobPutter implements BlobPutter {
@@ -54,15 +54,14 @@ public class StreamCompatibleBlobPutter implements
BlobPutter {
public Mono putDirectly(ObjectStorageBucketName bucketName, Blob
blob) {
return Mono.fromRunnable(() ->
blobStore.putBlob(bucketName.asString(), blob))
.publishOn(Schedulers.elastic())
-.retryWhen(Retry.onlyIf(retryContext ->
needToCreateBucket(retryContext.exception(), bucketName))
-.exponentialBackoff(FIRST_BACK_OFF, FOREVER)
-.withBackoffScheduler(Schedulers.elastic())
-.retryMax(MAX_RETRIES)
-.doOnRetry(retryContext ->
blobStore.createContainerInLocation(DEFAULT_LOCATION, bucketName.asString(
-.retryWhen(Retry.onlyIf(RetryContext ->
isPutMethod(RetryContext.exception()))
-.withBackoffScheduler(Schedulers.elastic())
-.exponentialBackoff(FIRST_BACK_OFF, FOREVER)
-.retryMax(RETRY_ONE_LAST_TIME_ON_CONCURRENT_SAVING))
+.retryWhen(Retry
+.backoff(MAX_RETRIES, FIRST_BACK_OFF)
+.filter(throwable -> needToCreateBucket(throwable, bucketName))
+.doBeforeRetry(retryContext ->
blobStore.createContainerInLocation(DEFAULT_LOCATION, bucketName.asString(
+.retryWhen(Retry
+.backoff(RETRY_ONE_LAST_TIME_ON_CONCURRENT_SAVING,
FIRST_BACK_OFF)
+.filter(this::isPutMethod)
+.scheduler(Schedulers.elastic()))
.then();
}
diff --git
a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/aws/AwsS3ObjectStorage.java
b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/aws/AwsS3ObjectStorage.java
index fa58add..8cf22da 100644
---
a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/aws/AwsS3ObjectStorage.java
+++
b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/aws/AwsS3ObjectStorage.java
@@ -65,7 +65,7 @@ import com.google.inject.Module;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;
-import reactor.retry.Retry;
+import reactor.util.retry.Retry;
public class AwsS3ObjectStorage {
@@ -180,11 +180,10 @@ public class AwsS3ObjectStorage {
return Mono.fromRunnable(puttingAttempt)
.publishOn(Schedulers.elastic())
.retryWhen(Retry
-.onlyIf(retryContext ->
needToCreateBucket(retryContext.exception()))
-.exponentialBackoff(FIRST_BACK_OFF, FOREVER)
-.withBackoffScheduler(Schedulers.elastic())
-.retryMax(MAX_RETRY_ON_EXCEPTION)
-.doOnRetry(retryContext ->
s3Client.createBucket(bucketName.asString(;
+.backoff(MAX_RETRY_ON_EXCEPTION, FIRST_BACK_OFF)
+.filter(throwable -> needToCreateBucket(throwable))
+.doBeforeRetry(retryContext ->
s3Client.createBucket(bucketName.asString()))
+.scheduler(Schedu
[james-project] 25/27: JAMES-3065 s/HealthCheck::checkReactive/HealthCheck::check/
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit f5b2483c3895c648e15bff319eb53b406d071bb2
Author: Benoit Tellier
AuthorDate: Fri Apr 24 09:53:39 2020 +0700
JAMES-3065 s/HealthCheck::checkReactive/HealthCheck::check/
---
.../james/backends/cassandra/utils/CassandraHealthCheck.java | 2 +-
.../james/backends/cassandra/utils/CassandraHealthCheckTest.java | 8
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git
a/backends-common/cassandra/src/main/java/org/apache/james/backends/cassandra/utils/CassandraHealthCheck.java
b/backends-common/cassandra/src/main/java/org/apache/james/backends/cassandra/utils/CassandraHealthCheck.java
index c502d57..07f575b 100644
---
a/backends-common/cassandra/src/main/java/org/apache/james/backends/cassandra/utils/CassandraHealthCheck.java
+++
b/backends-common/cassandra/src/main/java/org/apache/james/backends/cassandra/utils/CassandraHealthCheck.java
@@ -51,7 +51,7 @@ public class CassandraHealthCheck implements HealthCheck {
}
@Override
-public Mono checkReactive() {
+public Mono check() {
// execute a simple query to check if cassandra is responding
// idea from: https://stackoverflow.com/questions/10246287
return queryExecutor.execute(SAMPLE_QUERY)
diff --git
a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/utils/CassandraHealthCheckTest.java
b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/utils/CassandraHealthCheckTest.java
index 1796033..dbe115e 100644
---
a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/utils/CassandraHealthCheckTest.java
+++
b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/utils/CassandraHealthCheckTest.java
@@ -49,7 +49,7 @@ class CassandraHealthCheckTest {
@Test
void checkShouldReturnHealthyWhenCassandraIsRunning() {
-Result check = healthCheck.checkReactive().block();
+Result check = healthCheck.check().block();
assertThat(check.isHealthy()).isTrue();
}
@@ -58,7 +58,7 @@ class CassandraHealthCheckTest {
void
checkShouldReturnUnhealthyWhenCassandraIsNotRunning(DockerCassandraExtension.DockerCassandra
cassandraServer) {
try {
cassandraServer.getContainer().pause();
-Result check = healthCheck.checkReactive().block();
+Result check = healthCheck.check().block();
assertThat(check.isUnHealthy()).isTrue();
} finally {
@@ -72,12 +72,12 @@ class CassandraHealthCheckTest {
try {
cassandraServer.getContainer().pause();
-healthCheck.checkReactive().block();
+healthCheck.check().block();
} finally {
cassandraServer.getContainer().unpause();
}
-Result check = healthCheck.checkReactive().block();
+Result check = healthCheck.check().block();
assertThat(check.isHealthy()).isTrue();
}
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 04/27: JAMES-3143 Add context & objects describing inconsistencies
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 01cb8d1af106486bf60924ed1dfa622880cfe74f
Author: LanKhuat
AuthorDate: Fri Apr 17 01:57:50 2020 +0700
JAMES-3143 Add context & objects describing inconsistencies
---
.../task/SolveMessageInconsistenciesService.java | 446 +
.../SolveMessageInconsistenciesServiceTest.java| 265 +---
2 files changed, 593 insertions(+), 118 deletions(-)
diff --git
a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMessageInconsistenciesService.java
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMessageInconsistenciesService.java
index f67ff73..8518f1e 100644
---
a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMessageInconsistenciesService.java
+++
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMessageInconsistenciesService.java
@@ -19,7 +19,11 @@
package org.apache.james.mailbox.cassandra.mail.task;
+import java.util.Collection;
+import java.util.Objects;
import java.util.Optional;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.atomic.AtomicLong;
import javax.inject.Inject;
@@ -27,105 +31,413 @@ import org.apache.james.mailbox.cassandra.ids.CassandraId;
import org.apache.james.mailbox.cassandra.ids.CassandraMessageId;
import org.apache.james.mailbox.cassandra.mail.CassandraMessageIdDAO;
import org.apache.james.mailbox.cassandra.mail.CassandraMessageIdToImapUidDAO;
+import org.apache.james.mailbox.model.ComposedMessageId;
import org.apache.james.mailbox.model.ComposedMessageIdWithMetaData;
import org.apache.james.task.Task;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.MoreObjects;
+import com.google.common.collect.ImmutableList;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
public class SolveMessageInconsistenciesService {
+
+@FunctionalInterface
+interface Inconsistency {
+Mono fix(Context context, CassandraMessageIdToImapUidDAO
imapUidDAO, CassandraMessageIdDAO messageIdDAO);
+}
+
+private static Inconsistency NO_INCONSISTENCY = (context, imapUidDAO,
messageIdDAO) -> Mono.just(Task.Result.COMPLETED);
+
+private static class FailedToRetrieveRecord implements Inconsistency {
+private final ComposedMessageIdWithMetaData message;
+
+private FailedToRetrieveRecord(ComposedMessageIdWithMetaData message) {
+this.message = message;
+}
+
+@Override
+public Mono fix(Context context,
CassandraMessageIdToImapUidDAO imapUidDAO, CassandraMessageIdDAO messageIdDAO) {
+context.addErrors(message.getComposedMessageId());
+LOGGER.error("Failed to retrieve record: {}",
message.getComposedMessageId());
+return Mono.just(Task.Result.PARTIAL);
+}
+}
+
+private static class OrphanImapUidEntry implements Inconsistency {
+private final ComposedMessageIdWithMetaData message;
+
+private OrphanImapUidEntry(ComposedMessageIdWithMetaData message) {
+this.message = message;
+}
+
+@Override
+public Mono fix(Context context,
CassandraMessageIdToImapUidDAO imapUidDAO, CassandraMessageIdDAO messageIdDAO) {
+return messageIdDAO.insert(message)
+.doOnSuccess(any -> notifySuccess(context))
+.thenReturn(Task.Result.COMPLETED)
+.onErrorResume(error -> {
+notifyFailure(context);
+return Mono.just(Task.Result.PARTIAL);
+});
+}
+
+private void notifyFailure(Context context) {
+context.addErrors(message.getComposedMessageId());
+LOGGER.error("Failed to fix inconsistency for orphan message in
ImapUid: {}", message.getComposedMessageId());
+}
+
+private void notifySuccess(Context context) {
+LOGGER.info("Inconsistency fixed for orphan message in ImapUid:
{}", message.getComposedMessageId());
+context.incrementAddedMessageIdEntries();
+context.addFixedInconsistency(message.getComposedMessageId());
+}
+}
+
+private static class OutdatedMessageIdEntry implements Inconsistency {
+private final ComposedMessageIdWithMetaData messageFromMessageId;
+private final ComposedMessageIdWithMetaData messageFromImapUid;
+
+private OutdatedMessageIdEntry(ComposedMessageIdWithMetaData message,
ComposedMessageIdWithMetaData messageFromImapUid) {
+this.messageFromMessageId = message;
+this.messageFromImapUid = messageFromImapUid;
+}
+
+@Override
+public Mono f
[james-project] 24/27: JAMES-3065 s/HealthCheck::checkReactive/HealthCheck::check/
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit e78132260994e0557b8add5e3883818f8d3ad7b5
Author: Benoit Tellier
AuthorDate: Fri Apr 24 09:27:16 2020 +0700
JAMES-3065 s/HealthCheck::checkReactive/HealthCheck::check/
---
.../backends/es/ElasticSearchHealthCheck.java | 2 +-
.../es/ElasticSearchHealthCheckConnectionTest.java | 4 +--
.../backends/rabbitmq/RabbitMQHealthCheck.java | 2 +-
.../backends/rabbitmq/RabbitMQHealthCheckTest.java | 28 ++--
.../apache/james/core/healthcheck/HealthCheck.java | 2 +-
.../events/EventDeadLettersHealthCheck.java| 2 +-
.../EventDeadLettersHealthCheckContract.java | 30 +++---
.../apache/james/GuiceLifecycleHealthCheck.java| 2 +-
.../org/apache/james/PeriodicalHealthChecks.java | 2 +-
.../apache/james/PeriodicalHealthChecksTest.java | 16 ++--
.../MessageFastViewProjectionHealthCheck.java | 2 +-
.../MessageFastViewProjectionHealthCheckTest.java | 26 +--
.../james/jpa/healthcheck/JPAHealthCheck.java | 2 +-
.../james/jpa/healthcheck/JPAHealthCheckTest.java | 4 +--
.../james/webadmin/routes/HealthCheckRoutes.java | 4 +--
.../webadmin/routes/HealthCheckRoutesTest.java | 2 +-
16 files changed, 65 insertions(+), 65 deletions(-)
diff --git
a/backends-common/elasticsearch/src/main/java/org/apache/james/backends/es/ElasticSearchHealthCheck.java
b/backends-common/elasticsearch/src/main/java/org/apache/james/backends/es/ElasticSearchHealthCheck.java
index 4dded30..1d3f92e 100644
---
a/backends-common/elasticsearch/src/main/java/org/apache/james/backends/es/ElasticSearchHealthCheck.java
+++
b/backends-common/elasticsearch/src/main/java/org/apache/james/backends/es/ElasticSearchHealthCheck.java
@@ -55,7 +55,7 @@ public class ElasticSearchHealthCheck implements HealthCheck {
}
@Override
-public Mono checkReactive() {
+public Mono check() {
String[] indices = indexNames.stream()
.map(IndexName::getValue)
.toArray(String[]::new);
diff --git
a/backends-common/elasticsearch/src/test/java/org/apache/james/backends/es/ElasticSearchHealthCheckConnectionTest.java
b/backends-common/elasticsearch/src/test/java/org/apache/james/backends/es/ElasticSearchHealthCheckConnectionTest.java
index 0dc6530..70bfa3b 100644
---
a/backends-common/elasticsearch/src/test/java/org/apache/james/backends/es/ElasticSearchHealthCheckConnectionTest.java
+++
b/backends-common/elasticsearch/src/test/java/org/apache/james/backends/es/ElasticSearchHealthCheckConnectionTest.java
@@ -44,7 +44,7 @@ class ElasticSearchHealthCheckConnectionTest {
@Test
void checkShouldSucceedWhenElasticSearchIsRunning() {
-
assertThat(elasticSearchHealthCheck.checkReactive().block().isHealthy()).isTrue();
+
assertThat(elasticSearchHealthCheck.check().block().isHealthy()).isTrue();
}
@Test
@@ -53,7 +53,7 @@ class ElasticSearchHealthCheckConnectionTest {
elasticSearch.getDockerElasticSearch().pause();
try {
-
assertThat(elasticSearchHealthCheck.checkReactive().block().isUnHealthy()).isTrue();
+
assertThat(elasticSearchHealthCheck.check().block().isUnHealthy()).isTrue();
} finally {
elasticSearch.getDockerElasticSearch().unpause();
}
diff --git
a/backends-common/rabbitmq/src/main/java/org/apache/james/backends/rabbitmq/RabbitMQHealthCheck.java
b/backends-common/rabbitmq/src/main/java/org/apache/james/backends/rabbitmq/RabbitMQHealthCheck.java
index 098dc26..fff049a 100644
---
a/backends-common/rabbitmq/src/main/java/org/apache/james/backends/rabbitmq/RabbitMQHealthCheck.java
+++
b/backends-common/rabbitmq/src/main/java/org/apache/james/backends/rabbitmq/RabbitMQHealthCheck.java
@@ -49,7 +49,7 @@ public class RabbitMQHealthCheck implements HealthCheck {
}
@Override
-public Mono checkReactive() {
+public Mono check() {
try {
return Flux.concat(connectionPool.tryConnection(),
rabbitChannelPoolImpl.tryChannel())
diff --git
a/backends-common/rabbitmq/src/test/java/org/apache/james/backends/rabbitmq/RabbitMQHealthCheckTest.java
b/backends-common/rabbitmq/src/test/java/org/apache/james/backends/rabbitmq/RabbitMQHealthCheckTest.java
index f205bc7..fa2eef4 100644
---
a/backends-common/rabbitmq/src/test/java/org/apache/james/backends/rabbitmq/RabbitMQHealthCheckTest.java
+++
b/backends-common/rabbitmq/src/test/java/org/apache/james/backends/rabbitmq/RabbitMQHealthCheckTest.java
@@ -46,21 +46,21 @@ class RabbitMQHealthCheckTest {
@Test
void checkShouldReturnHealthyWhenRabbitMQIsRunning() {
-Result check = healthCheck.checkReactive().block();
+Result check = healthCheck.check().block();
assertThat(check.isHea
[james-project] 05/27: JAMES-3143 SolveMessageInconsistenciesTask/DTO
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 84374e53305d80483b09799537f22fc97c6fb2b9
Author: LanKhuat
AuthorDate: Tue Apr 21 00:09:51 2020 +0700
JAMES-3143 SolveMessageInconsistenciesTask/DTO
---
.../mail/task/MessageInconsistenciesEntry.java | 92 +
.../mail/task/SolveMessageInconsistenciesTask.java | 143 +
...nconsistenciesTaskAdditionalInformationDTO.java | 135 +++
.../task/SolveMessageInconsistenciesTaskDTO.java | 60 +
...essageInconsistenciesTaskSerializationTest.java | 78 +++
...ssageInconsistencies.additionalInformation.json | 28
.../json/solveMessageInconsistencies.task.json | 3 +
.../SolveMessageInconsistenciesRequestToTask.java | 36 ++
8 files changed, 575 insertions(+)
diff --git
a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/MessageInconsistenciesEntry.java
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/MessageInconsistenciesEntry.java
new file mode 100644
index 000..0dec7f2
--- /dev/null
+++
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/MessageInconsistenciesEntry.java
@@ -0,0 +1,92 @@
+/
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information*
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the*
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the*
+ * specific language governing permissions and limitations *
+ * under the License. *
+ /
+
+package org.apache.james.mailbox.cassandra.mail.task;
+
+import java.util.Objects;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public class MessageInconsistenciesEntry {
+
+public interface Builder {
+@FunctionalInterface
+interface RequireMailboxId {
+RequireMessageId mailboxId(String mailboxId);
+}
+
+@FunctionalInterface
+interface RequireMessageId {
+RequireMessageUid messageId(String messageId);
+}
+
+@FunctionalInterface
+interface RequireMessageUid {
+MessageInconsistenciesEntry messageUid(Long messageUid);
+}
+}
+
+public static Builder.RequireMailboxId builder() {
+return mailboxId -> messageId -> messageUid -> new
MessageInconsistenciesEntry(mailboxId, messageId, messageUid);
+}
+
+private final String mailboxId;
+private final String messageId;
+private final Long messageUid;
+
+private MessageInconsistenciesEntry(@JsonProperty("mailboxId") String
mailboxId,
+@JsonProperty("messageId") String
messageId,
+ @JsonProperty("uid") Long messageUid) {
+this.mailboxId = mailboxId;
+this.messageId = messageId;
+this.messageUid = messageUid;
+}
+
+@JsonProperty("mailboxId")
+public String getMailboxId() {
+return mailboxId;
+}
+
+@JsonProperty("messageId")
+public String getMessageId() {
+return messageId;
+}
+
+@JsonProperty("uid")
+public Long getMessageUid() {
+return messageUid;
+}
+
+@Override
+public final boolean equals(Object o) {
+if (o instanceof MessageInconsistenciesEntry) {
+MessageInconsistenciesEntry that = (MessageInconsistenciesEntry) o;
+
+return Objects.equals(this.mailboxId, that.mailboxId)
+&& Objects.equals(this.messageId, that.messageId)
+&& Objects.equals(this.messageUid, that.messageUid);
+}
+return false;
+}
+
+@Override
+public final int hashCode() {
+return Objects.hash(mailboxId, messageId, messageUid);
+}
+}
diff --git
a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMessageInconsistenciesTask.java
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/tas
[james-project] 26/27: JAMES-3153 make use of RetryWhen for backoff on reactor
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 7cb8ba9e88c76718f62ec32b2ded8c1d35420569
Author: Matthieu Baechler
AuthorDate: Thu Apr 23 12:34:56 2020 +0200
JAMES-3153 make use of RetryWhen for backoff on reactor
---
.../backends/cassandra/init/ResilientClusterProvider.java | 4 ++--
.../main/java/org/apache/james/backends/es/ClientProvider.java | 4 ++--
.../james/backends/rabbitmq/RabbitMQConnectionFactory.java | 4 ++--
.../james/backends/rabbitmq/ReactorRabbitMQChannelPool.java| 6 +++---
.../apache/james/backends/rabbitmq/SimpleConnectionPool.java | 4 ++--
.../apache/james/mailbox/events/RetryBackoffConfiguration.java | 1 -
.../james/mailbox/cassandra/mail/CassandraMailboxMapper.java | 7 ---
.../james/mailbox/cassandra/mail/CassandraMessageIdMapper.java | 3 ++-
.../james/mailbox/cassandra/mail/CassandraMessageMapper.java | 3 ++-
.../james/mailbox/cassandra/mail/CassandraModSeqProvider.java | 4 ++--
.../james/mailbox/cassandra/mail/CassandraUidProvider.java | 4 ++--
.../apache/james/mailbox/events/delivery/EventDelivery.java| 6 ++
.../org/apache/james/mailbox/events/GroupRegistration.java | 4 ++--
.../apache/james/mailbox/events/KeyRegistrationHandler.java| 10 +-
pom.xml| 2 +-
.../james/transport/mailets/delivery/MailDispatcher.java | 3 ++-
.../src/test/java/org/apache/james/webadmin/WebAdminUtils.java | 3 ++-
server/queue/queue-activemq/pom.xml| 2 +-
.../task/eventsourcing/distributed/RabbitMQWorkQueue.java | 8
19 files changed, 42 insertions(+), 40 deletions(-)
diff --git
a/backends-common/cassandra/src/main/java/org/apache/james/backends/cassandra/init/ResilientClusterProvider.java
b/backends-common/cassandra/src/main/java/org/apache/james/backends/cassandra/init/ResilientClusterProvider.java
index f91e68d..4eeb502 100644
---
a/backends-common/cassandra/src/main/java/org/apache/james/backends/cassandra/init/ResilientClusterProvider.java
+++
b/backends-common/cassandra/src/main/java/org/apache/james/backends/cassandra/init/ResilientClusterProvider.java
@@ -40,6 +40,7 @@ import com.google.common.collect.ImmutableList;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;
+import reactor.util.retry.Retry;
@Singleton
public class ResilientClusterProvider implements Provider {
@@ -51,10 +52,9 @@ public class ResilientClusterProvider implements
Provider {
@Inject
ResilientClusterProvider(ClusterConfiguration configuration) {
Duration waitDelay = Duration.ofMillis(configuration.getMinDelay());
-Duration forever = Duration.ofMillis(Long.MAX_VALUE);
cluster = Mono.fromCallable(getClusterRetryCallable(configuration))
.doOnError(e -> LOGGER.warn("Error establishing Cassandra
connection. Next retry scheduled in {} ms", waitDelay, e))
-.retryBackoff(configuration.getMaxRetry(), waitDelay, forever,
Schedulers.elastic())
+.retryWhen(Retry.backoff(configuration.getMaxRetry(),
waitDelay).scheduler(Schedulers.elastic()))
.publishOn(Schedulers.elastic())
.block();
}
diff --git
a/backends-common/elasticsearch/src/main/java/org/apache/james/backends/es/ClientProvider.java
b/backends-common/elasticsearch/src/main/java/org/apache/james/backends/es/ClientProvider.java
index 247ed50..b73fbd3 100644
---
a/backends-common/elasticsearch/src/main/java/org/apache/james/backends/es/ClientProvider.java
+++
b/backends-common/elasticsearch/src/main/java/org/apache/james/backends/es/ClientProvider.java
@@ -56,6 +56,7 @@ import com.google.common.annotations.VisibleForTesting;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;
+import reactor.util.retry.Retry;
public class ClientProvider implements Provider {
@@ -180,13 +181,12 @@ public class ClientProvider implements
Provider {
private RestHighLevelClient connect(ElasticSearchConfiguration
configuration) {
Duration waitDelay = Duration.ofMillis(configuration.getMinDelay());
-Duration forever = Duration.ofMillis(Long.MAX_VALUE);
boolean suppressLeadingZeroElements = true;
boolean suppressTrailingZeroElements = true;
return Mono.fromCallable(() -> connectToCluster(configuration))
.doOnError(e -> LOGGER.warn("Error establishing ElasticSearch
connection. Next retry scheduled in {}",
DurationFormatUtils.formatDurationWords(waitDelay.toMillis(),
suppressLeadingZeroElements, suppressTrailingZeroElements), e))
-.retryBackoff(configuration.getMaxRetries(), waitDelay, forever,
Schedulers.elastic())
+.retryWhen(Retry.backoff(configuration.getMaxRetries(),
waitDelay).schedu
[james-project] 19/27: JAMES-3138 InMemoryCurrentQuotaManager: Improve reactor code style
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit e73dfb8e9696c2e9b7975d756a16965a56426826
Author: Benoit Tellier
AuthorDate: Sat Apr 25 11:19:29 2020 +0700
JAMES-3138 InMemoryCurrentQuotaManager: Improve reactor code style
- get rid of redundant Mono::from
- filter rather than nested if
---
.../mailbox/inmemory/quota/InMemoryCurrentQuotaManager.java | 12
1 file changed, 4 insertions(+), 8 deletions(-)
diff --git
a/mailbox/memory/src/main/java/org/apache/james/mailbox/inmemory/quota/InMemoryCurrentQuotaManager.java
b/mailbox/memory/src/main/java/org/apache/james/mailbox/inmemory/quota/InMemoryCurrentQuotaManager.java
index 34a9be6..8521557 100644
---
a/mailbox/memory/src/main/java/org/apache/james/mailbox/inmemory/quota/InMemoryCurrentQuotaManager.java
+++
b/mailbox/memory/src/main/java/org/apache/james/mailbox/inmemory/quota/InMemoryCurrentQuotaManager.java
@@ -85,14 +85,10 @@ public class InMemoryCurrentQuotaManager implements
CurrentQuotaManager {
@Override
public Mono setCurrentQuotas(QuotaOperation quotaOperation) {
-return Mono.from(getCurrentQuotas(quotaOperation.quotaRoot()))
-.flatMap(storedQuotas -> {
-if (!storedQuotas.equals(CurrentQuotas.from(quotaOperation))) {
-return Mono.from(decrease(new
QuotaOperation(quotaOperation.quotaRoot(), storedQuotas.count(),
storedQuotas.size(
-.then(Mono.from(increase(quotaOperation)));
-}
-return Mono.empty();
-});
+return getCurrentQuotas(quotaOperation.quotaRoot())
+.filter(storedQuotas ->
!storedQuotas.equals(CurrentQuotas.from(quotaOperation)))
+.flatMap(storedQuotas -> decrease(new
QuotaOperation(quotaOperation.quotaRoot(), storedQuotas.count(),
storedQuotas.size()))
+.then(increase(quotaOperation)));
}
private Mono updateQuota(QuotaRoot quotaRoot,
UnaryOperator quotaFunction) {
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 16/27: JAMES-3138 Integration tests for consistency tasks
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 6e597252351c0d5986f97be41ece6cd501a2b737
Author: Benoit Tellier
AuthorDate: Fri Apr 24 20:12:05 2020 +0700
JAMES-3138 Integration tests for consistency tasks
---
.../james/backends/cassandra/TestingSession.java | 2 +-
.../org/apache/james/modules/MailboxProbeImpl.java | 6 +
.../rabbitmq/ConsistencyTasksIntegrationTest.java | 314 +
3 files changed, 321 insertions(+), 1 deletion(-)
diff --git
a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSession.java
b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSession.java
index b229b46..cbb7ae7 100644
---
a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSession.java
+++
b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSession.java
@@ -35,7 +35,7 @@ public class TestingSession implements Session {
private final Session delegate;
private volatile Scenario scenario;
-TestingSession(Session delegate) {
+public TestingSession(Session delegate) {
this.delegate = delegate;
this.scenario = Scenario.NOTHING;
}
diff --git
a/server/container/guice/mailbox/src/main/java/org/apache/james/modules/MailboxProbeImpl.java
b/server/container/guice/mailbox/src/main/java/org/apache/james/modules/MailboxProbeImpl.java
index 97e0c16..1d5936a 100644
---
a/server/container/guice/mailbox/src/main/java/org/apache/james/modules/MailboxProbeImpl.java
+++
b/server/container/guice/mailbox/src/main/java/org/apache/james/modules/MailboxProbeImpl.java
@@ -36,6 +36,7 @@ import org.apache.james.mailbox.SubscriptionManager;
import org.apache.james.mailbox.exception.MailboxException;
import org.apache.james.mailbox.exception.MailboxNotFoundException;
import org.apache.james.mailbox.model.ComposedMessageId;
+import org.apache.james.mailbox.model.MailboxCounters;
import org.apache.james.mailbox.model.MailboxId;
import org.apache.james.mailbox.model.MailboxMetaData;
import org.apache.james.mailbox.model.MailboxPath;
@@ -146,6 +147,11 @@ public class MailboxProbeImpl implements GuiceProbe,
MailboxProbe {
}
}
+public MailboxCounters retrieveCounters(MailboxPath path) throws
MailboxException {
+MailboxSession systemSession =
mailboxManager.createSystemSession(path.getUser());
+return mailboxManager.getMailbox(path,
systemSession).getMailboxCounters(systemSession);
+}
+
@Override
public ComposedMessageId appendMessage(String username, MailboxPath
mailboxPath, InputStream message, Date internalDate, boolean isRecent, Flags
flags)
throws MailboxException {
diff --git
a/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/ConsistencyTasksIntegrationTest.java
b/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/ConsistencyTasksIntegrationTest.java
new file mode 100644
index 000..f821ced
--- /dev/null
+++
b/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/ConsistencyTasksIntegrationTest.java
@@ -0,0 +1,314 @@
+/
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information*
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the*
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the*
+ * specific language governing permissions and limitations *
+ * under the License. *
+ /
+
+package org.apache.james.webadmin.integration.rabbitmq;
+
+import static io.restassured.RestAssured.when;
+import static io.restassured.RestAssured.with;
+import static org.apache.james.backends.cassandra.Scenario.Builder.awaitOn;
+import static org.apache.jame
[james-project] 11/27: JAMES-3138 Task for recomputing current quotas
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit ab4e4e3b1df2b4be221dfd233990582b0801a606
Author: Rene Cordier
AuthorDate: Mon Apr 13 17:12:02 2020 +0700
JAMES-3138 Task for recomputing current quotas
---
.../org/apache/james/mailbox/model/QuotaRoot.java | 5 +
mailbox/tools/quota-recompute/pom.xml | 20
.../quota/task/RecomputeCurrentQuotasTask.java | 101 +
...eCurrentQuotasTaskAdditionalInformationDTO.java | 87 ++
.../quota/task/RecomputeCurrentQuotasTaskDTO.java | 57
...ecomputeCurrentQuotasTaskSerializationTest.java | 61 +
6 files changed, 331 insertions(+)
diff --git
a/mailbox/api/src/main/java/org/apache/james/mailbox/model/QuotaRoot.java
b/mailbox/api/src/main/java/org/apache/james/mailbox/model/QuotaRoot.java
index c7a825a..1ad8d20 100644
--- a/mailbox/api/src/main/java/org/apache/james/mailbox/model/QuotaRoot.java
+++ b/mailbox/api/src/main/java/org/apache/james/mailbox/model/QuotaRoot.java
@@ -73,4 +73,9 @@ public class QuotaRoot {
.add("domain", domain)
.toString();
}
+
+public String asString() {
+return domain.map(domainValue -> value + "@" + domainValue.asString())
+.orElse(value);
+}
}
diff --git a/mailbox/tools/quota-recompute/pom.xml
b/mailbox/tools/quota-recompute/pom.xml
index 34c21a7..2c59165 100644
--- a/mailbox/tools/quota-recompute/pom.xml
+++ b/mailbox/tools/quota-recompute/pom.xml
@@ -38,13 +38,33 @@
${james.groupId}
+james-json
+test
+test-jar
+
+
+${james.groupId}
james-server-data-api
${james.groupId}
+james-server-task-json
+
+
+${james.groupId}
testing-base
test
+
+net.javacrumbs.json-unit
+json-unit-assertj
+test
+
+
+org.mockito
+mockito-core
+test
+
\ No newline at end of file
diff --git
a/mailbox/tools/quota-recompute/src/main/java/org/apache/james/mailbox/quota/task/RecomputeCurrentQuotasTask.java
b/mailbox/tools/quota-recompute/src/main/java/org/apache/james/mailbox/quota/task/RecomputeCurrentQuotasTask.java
new file mode 100644
index 000..050737b
--- /dev/null
+++
b/mailbox/tools/quota-recompute/src/main/java/org/apache/james/mailbox/quota/task/RecomputeCurrentQuotasTask.java
@@ -0,0 +1,101 @@
+/
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information*
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the*
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the*
+ * specific language governing permissions and limitations *
+ * under the License. *
+ /
+
+package org.apache.james.mailbox.quota.task;
+
+import java.time.Clock;
+import java.time.Instant;
+import java.util.Optional;
+
+import org.apache.james.mailbox.model.QuotaRoot;
+import
org.apache.james.mailbox.quota.task.RecomputeCurrentQuotasService.Context;
+import
org.apache.james.mailbox.quota.task.RecomputeCurrentQuotasService.Context.Snapshot;
+import org.apache.james.task.Task;
+import org.apache.james.task.TaskExecutionDetails;
+import org.apache.james.task.TaskType;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.github.steveash.guavate.Guavate;
+import com.google.common.collect.ImmutableList;
+
+import reactor.core.scheduler.Schedulers;
+
+public class RecomputeCurrentQuotasTask implements Task {
+static final TaskType RECOMPUTE_CURRENT_QUOTAS =
TaskType.of("recompute-current-quotas");
+
+public static class Details implements
TaskExecutionDetails.AdditionalInformation {
+private final Instant instant;
+private final long processedQuotaRoots;
+private final ImmutableList failedQuotaRoot
[james-project] 13/27: JAMES-3138 Changelog entry
This is an automated email from the ASF dual-hosted git repository. btellier pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/james-project.git commit 5178d1a1ced30548c7987497d1a65a34fda02b05 Author: Rene Cordier AuthorDate: Tue Apr 14 16:05:03 2020 +0700 JAMES-3138 Changelog entry --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6d1986f..a5d57c9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,7 @@ of tasks being currently executed. - JAMES-3072 WebAdmin endpoint to export mailbox backup - JAMES-3117 Add PeriodicalHealthChecks for periodical calling all health checks - JAMES-3143 WebAdmin endpoint to solve Cassandra message inconsistencies +- JAMES-3138 Webadmin endpoint to recompute users current quotas on top of Guice products ### Changed - Switch to Java 11 for build and run - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 14/27: JAMES-3138 Documentation for Recomputing current quotas endpoint
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 251a73d1a7543e98610c3025528dfbe218d6013a
Author: Rene Cordier
AuthorDate: Tue Apr 14 16:18:23 2020 +0700
JAMES-3138 Documentation for Recomputing current quotas endpoint
---
src/site/markdown/server/manage-webadmin.md | 29 +
1 file changed, 29 insertions(+)
diff --git a/src/site/markdown/server/manage-webadmin.md
b/src/site/markdown/server/manage-webadmin.md
index 7efda46..384d4f6 100644
--- a/src/site/markdown/server/manage-webadmin.md
+++ b/src/site/markdown/server/manage-webadmin.md
@@ -974,6 +974,7 @@ Response codes:
- [Updating the quota size for a user](#Updating_the_quota_size_for_a_user)
- [Deleting the quota size for a user](#Deleting_the_quota_size_for_a_user)
- [Searching user by quota ratio](#Searching_user_by_quota_ratio)
+ - [Recomputing current quotas for
users](#Recomputing_current_quotas_for_users)
### Getting the quota for a user
@@ -1226,6 +1227,34 @@ Response codes:
- 200: List of users had successfully been returned.
- 400: Validation issues with parameters
+
+### Recomputing current quotas for users
+
+This task is available on top of Cassandra & JPA products.
+
+```
+curl -XPOST /quota/users?task=RecomputeCurrentQuotas
+```
+
+Will recompute current quotas (count and size) for all users stored in James.
+
+James maintains per quota a projection for current quota count and size. As
with any projection, it can
+go out of sync, leading to inconsistent results being returned to the client.
+
+[More details about endpoints returning a task](#Endpoints_returning_a_task).
+
+The scheduled task will have the following type `recompute-current-quotas` and
the following `additionalInformation`:
+
+```
+{
+ "type":"recompute-current-quotas",
+ "processedQuotaRoots": 3,
+ "failedQuotaRoots": ["#private&bob@localhost"]
+}
+```
+
+**WARNING**: this task do not take into account concurrent modifications upon
a single current quota recomputation.
+Rerunning the task will *eventually* provide the consistent result.
## Administrating quotas by domains
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 01/27: [Refactoring] Move MailboxACLJsonConverter from mailbox/store to mailbox/cassandra
This is an automated email from the ASF dual-hosted git repository. btellier pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/james-project.git commit d563e4afa12afa7f844874ead5f854c1405c3977 Author: Benoit Tellier AuthorDate: Fri Apr 24 14:47:17 2020 +0700 [Refactoring] Move MailboxACLJsonConverter from mailbox/store to mailbox/cassandra This is the only place where it is being used. --- .../apache/james/mailbox/cassandra}/json/MailboxACLJsonConverter.java | 2 +- .../org/apache/james/mailbox/cassandra/mail/CassandraACLMapper.java | 2 +- .../james/mailbox/cassandra}/json/MailboxACLJsonConverterTest.java | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/mailbox/store/src/main/java/org/apache/james/mailbox/store/json/MailboxACLJsonConverter.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/json/MailboxACLJsonConverter.java similarity index 98% rename from mailbox/store/src/main/java/org/apache/james/mailbox/store/json/MailboxACLJsonConverter.java rename to mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/json/MailboxACLJsonConverter.java index 8fdfc83..d6b650f 100644 --- a/mailbox/store/src/main/java/org/apache/james/mailbox/store/json/MailboxACLJsonConverter.java +++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/json/MailboxACLJsonConverter.java @@ -17,7 +17,7 @@ * under the License. * / -package org.apache.james.mailbox.store.json; +package org.apache.james.mailbox.cassandra.json; import java.io.IOException; diff --git a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraACLMapper.java b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraACLMapper.java index eccabcd..43eeef8 100644 --- a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraACLMapper.java +++ b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraACLMapper.java @@ -35,12 +35,12 @@ import org.apache.james.backends.cassandra.init.configuration.CassandraConfigura import org.apache.james.backends.cassandra.utils.CassandraAsyncExecutor; import org.apache.james.mailbox.acl.ACLDiff; import org.apache.james.mailbox.cassandra.ids.CassandraId; +import org.apache.james.mailbox.cassandra.json.MailboxACLJsonConverter; import org.apache.james.mailbox.cassandra.table.CassandraACLTable; import org.apache.james.mailbox.cassandra.table.CassandraMailboxTable; import org.apache.james.mailbox.exception.MailboxException; import org.apache.james.mailbox.exception.UnsupportedRightException; import org.apache.james.mailbox.model.MailboxACL; -import org.apache.james.mailbox.store.json.MailboxACLJsonConverter; import org.apache.james.util.FunctionalUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/mailbox/store/src/test/java/org/apache/james/mailbox/store/json/MailboxACLJsonConverterTest.java b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/json/MailboxACLJsonConverterTest.java similarity index 99% rename from mailbox/store/src/test/java/org/apache/james/mailbox/store/json/MailboxACLJsonConverterTest.java rename to mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/json/MailboxACLJsonConverterTest.java index b15fed2..3da7d4b 100644 --- a/mailbox/store/src/test/java/org/apache/james/mailbox/store/json/MailboxACLJsonConverterTest.java +++ b/mailbox/cassandra/src/test/java/org/apache/james/mailbox/cassandra/json/MailboxACLJsonConverterTest.java @@ -17,7 +17,7 @@ * under the License. * / -package org.apache.james.mailbox.store.json; +package org.apache.james.mailbox.cassandra.json; import static net.javacrumbs.jsonunit.assertj.JsonAssertions.assertThatJson; import static org.apache.james.mailbox.model.MailboxACL.Right.CreateMailbox; - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] branch master updated (77e4f75 -> b4de77a)
This is an automated email from the ASF dual-hosted git repository. btellier pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/james-project.git. from 77e4f75 JAMES-3145 document metrics logger new d563e4a [Refactoring] Move MailboxACLJsonConverter from mailbox/store to mailbox/cassandra new 10e1a44 JAMES-3143 Add select all query for MessageId/ImapUid new e31d789 JAMES-3143 SolveMessageInconsistenciesService/Test new 01cb8d1 JAMES-3143 Add context & objects describing inconsistencies new 84374e5 JAMES-3143 SolveMessageInconsistenciesTask/DTO new 4db5a1f JAMES-3143 Guice binding for task new 6ceec97 JAMES-3143 Exposes task in MessageRoutes new c638eff JAMES-3143 Add Webadmin integration test new a7d33f7 JAMES-3143 Update webadmin documentation new e0ce72b JAMES-3143 Update CHANGELOG new ab4e4e3 JAMES-3138 Task for recomputing current quotas new 1b0bc6e JAMES-3138 Webadmin endpoint and guice bindings for recomputing current quotas new 5178d1a JAMES-3138 Changelog entry new 251a73d JAMES-3138 Documentation for Recomputing current quotas endpoint new 7f5a694 JAMES-3138 Task for recomputing current quotas new 6e59725 JAMES-3138 Integration tests for consistency tasks new 1157a7b JAMES-3138 Use concatMap upon quota computation new 00160a4 JAMES-3138 CassandraCurrentQuotaManager: Improve Reactor code style new e73dfb8 JAMES-3138 InMemoryCurrentQuotaManager: Improve reactor code style new fc528ae JAMES-3138 Cassandra test instumentation: inject failure after awaitOn new 9266b93 JAMES-3138 VarArg for Cassandra instrumentation scenari registration new d018a04 JAMES-3138 Add fix nothing integration tests new 12a5987 JAMES-3065 remove remaining usages of Reactor types in public api new e781322 JAMES-3065 s/HealthCheck::checkReactive/HealthCheck::check/ new f5b2483 JAMES-3065 s/HealthCheck::checkReactive/HealthCheck::check/ new 7cb8ba9 JAMES-3153 make use of RetryWhen for backoff on reactor new b4de77a JAMES-3153 use reactor.Retry insteaf of reactor-extra version The 27 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGELOG.md | 4 +- .../cassandra/init/ResilientClusterProvider.java | 4 +- .../cassandra/utils/CassandraHealthCheck.java | 2 +- .../apache/james/backends/cassandra/Scenario.java | 23 +- .../james/backends/cassandra/TestingSession.java | 6 +- .../backends/cassandra/TestingSessionTest.java | 31 +- .../cassandra/utils/CassandraHealthCheckTest.java | 8 +- .../apache/james/backends/es/ClientProvider.java | 4 +- .../backends/es/ElasticSearchHealthCheck.java | 2 +- .../es/ElasticSearchHealthCheckConnectionTest.java | 4 +- .../rabbitmq/RabbitMQConnectionFactory.java| 4 +- .../backends/rabbitmq/RabbitMQHealthCheck.java | 2 +- .../rabbitmq/ReactorRabbitMQChannelPool.java | 6 +- .../backends/rabbitmq/SimpleConnectionPool.java| 4 +- .../backends/rabbitmq/RabbitMQHealthCheckTest.java | 28 +- core/pom.xml | 4 + .../apache/james/core/healthcheck/HealthCheck.java | 10 +- mailbox/api/pom.xml| 4 + .../events/EventDeadLettersHealthCheck.java| 2 +- .../mailbox/events/RetryBackoffConfiguration.java | 1 - .../org/apache/james/mailbox/model/QuotaRoot.java | 5 + .../EventDeadLettersHealthCheckContract.java | 30 +- .../apache/james/mailbox/model/QuotaRootTest.java | 24 + .../cassandra}/json/MailboxACLJsonConverter.java | 2 +- .../mailbox/cassandra/mail/CassandraACLMapper.java | 2 +- .../cassandra/mail/CassandraMailboxMapper.java | 7 +- .../cassandra/mail/CassandraMessageIdDAO.java | 12 + .../cassandra/mail/CassandraMessageIdMapper.java | 3 +- .../mail/CassandraMessageIdToImapUidDAO.java | 11 + .../cassandra/mail/CassandraMessageMapper.java | 3 +- .../cassandra/mail/CassandraModSeqProvider.java| 4 +- .../cassandra/mail/CassandraUidProvider.java | 4 +- .../mail/task/MessageInconsistenciesEntry.java | 92 .../task/SolveMessageInconsistenciesService.java | 443 .../mail/task/SolveMessageInconsistenciesTask.java | 143 ++ ...nconsistenciesTaskAdditionalInformationDTO.java | 135 + ...ava => SolveMessageInconsistenciesTaskDTO.java} | 26 +- .../quota/CassandraCurrentQuotaManager.java| 12 +- .../json/MailboxACLJsonConverterTest.java | 2 +- .../cassandra/mail/CassandraACLMapperTest.java | 6 +- .../cassandra/mail/CassandraMailboxDAOTest.java| 2 +- .../SolveMessageInconsistencie
[james-project] 21/27: JAMES-3138 VarArg for Cassandra instrumentation scenari registration
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 9266b93a73e9aa2c862f681f3530c51fa37a8024
Author: Benoit Tellier
AuthorDate: Tue Apr 28 07:40:44 2020 +0700
JAMES-3138 VarArg for Cassandra instrumentation scenari registration
---
.../java/org/apache/james/backends/cassandra/TestingSession.java| 4 ++--
.../org/apache/james/backends/cassandra/TestingSessionTest.java | 5 ++---
.../integration/rabbitmq/ConsistencyTasksIntegrationTest.java | 6 ++
3 files changed, 6 insertions(+), 9 deletions(-)
diff --git
a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSession.java
b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSession.java
index cbb7ae7..c8a7a54 100644
---
a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSession.java
+++
b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSession.java
@@ -44,8 +44,8 @@ public class TestingSession implements Session {
this.scenario = scenario;
}
-public void registerScenario(Scenario.ExecutionHook hook) {
-this.scenario = Scenario.combine(hook);
+public void registerScenario(Scenario.ExecutionHook... hooks) {
+this.scenario = Scenario.combine(hooks);
}
@Override
diff --git
a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSessionTest.java
b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSessionTest.java
index 5140e14..6c8dfc5 100644
---
a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSessionTest.java
+++
b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSessionTest.java
@@ -23,7 +23,6 @@ import static
com.datastax.driver.core.querybuilder.QueryBuilder.select;
import static org.apache.james.backends.cassandra.Scenario.Builder.awaitOn;
import static
org.apache.james.backends.cassandra.Scenario.Builder.executeNormally;
import static org.apache.james.backends.cassandra.Scenario.Builder.fail;
-import static org.apache.james.backends.cassandra.Scenario.combine;
import static
org.apache.james.backends.cassandra.versions.table.CassandraSchemaVersionTable.TABLE_NAME;
import static
org.apache.james.backends.cassandra.versions.table.CassandraSchemaVersionTable.VALUE;
import static org.assertj.core.api.Assertions.assertThat;
@@ -154,13 +153,13 @@ class TestingSessionTest {
@Test
void scenarioShouldDefiningSeveralHooks(CassandraCluster cassandra) {
cassandra.getConf()
-.registerScenario(combine(
+.registerScenario(
executeNormally()
.times(1)
.whenQueryStartsWith("SELECT value FROM schemaVersion;"),
fail()
.times(1)
-.whenQueryStartsWith("SELECT value FROM schemaVersion;")));
+.whenQueryStartsWith("SELECT value FROM schemaVersion;"));
SoftAssertions.assertSoftly(softly -> {
assertThatCode(() -> dao.getCurrentSchemaVersion().block())
diff --git
a/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/ConsistencyTasksIntegrationTest.java
b/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/ConsistencyTasksIntegrationTest.java
index f821ced..bb69945 100644
---
a/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/ConsistencyTasksIntegrationTest.java
+++
b/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/ConsistencyTasksIntegrationTest.java
@@ -44,7 +44,6 @@ import org.apache.james.DockerElasticSearchExtension;
import org.apache.james.GuiceJamesServer;
import org.apache.james.JamesServerBuilder;
import org.apache.james.JamesServerExtension;
-import org.apache.james.backends.cassandra.Scenario;
import org.apache.james.backends.cassandra.Scenario.Barrier;
import org.apache.james.backends.cassandra.TestingSession;
import
org.apache.james.backends.cassandra.init.SessionWithInitializedTablesFactory;
@@ -68,7 +67,6 @@ import
org.apache.james.webadmin.routes.CassandraMappingsRoutes;
import org.apache.james.webadmin.routes.TasksRoutes;
import org.eclipse.jetty.http.HttpStatus;
import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.RepeatedTest;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
@@ -272,7 +270,7 @@ class Cons
[james-project] 08/27: JAMES-3143 Add Webadmin integration test
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit c638effd63116e7cda98a7127469ae81f63412b6
Author: LanKhuat
AuthorDate: Wed Apr 22 11:05:10 2020 +0700
JAMES-3143 Add Webadmin integration test
---
.../RabbitMQWebAdminServerIntegrationTest.java | 33 ++
1 file changed, 33 insertions(+)
diff --git
a/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/RabbitMQWebAdminServerIntegrationTest.java
b/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/RabbitMQWebAdminServerIntegrationTest.java
index 0f7b1e8..c107aa6 100644
---
a/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/RabbitMQWebAdminServerIntegrationTest.java
+++
b/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/RabbitMQWebAdminServerIntegrationTest.java
@@ -199,6 +199,39 @@ class RabbitMQWebAdminServerIntegrationTest extends
WebAdminServerIntegrationTes
}
@Test
+void solveMessageInconsistenciesTasksShouldBeExposed() {
+String taskId = with().post(UPGRADE_TO_LATEST_VERSION)
+.jsonPath()
+.get("taskId");
+
+with()
+.get("/tasks/" + taskId + "/await")
+.then()
+.body("status", is("completed"));
+
+taskId = with()
+.queryParam("task", "SolveInconsistencies")
+.post("/messages")
+.jsonPath()
+.get("taskId");
+
+given()
+.basePath(TasksRoutes.BASE)
+.when()
+.get(taskId + "/await")
+.then()
+.body("status", is("completed"))
+.body("type", is("solve-message-inconsistencies"))
+.body("additionalInformation.processedImapUidEntries", is(0))
+.body("additionalInformation.processedMessageIdEntries", is(0))
+.body("additionalInformation.addedMessageIdEntries", is(0))
+.body("additionalInformation.updatedMessageIdEntries", is(0))
+.body("additionalInformation.removedMessageIdEntries", is(0))
+.body("additionalInformation.fixedInconsistencies", hasSize(0))
+.body("additionalInformation.errors", hasSize(0));
+}
+
+@Test
void getSwaggerShouldContainDistributedEndpoints() {
when()
.get(SwaggerRoutes.SWAGGER_ENDPOINT)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 07/27: JAMES-3143 Exposes task in MessageRoutes
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 6ceec974c780faa1895da90c47ac9bd94e90bb15
Author: LanKhuat
AuthorDate: Wed Apr 22 11:03:14 2020 +0700
JAMES-3143 Exposes task in MessageRoutes
---
.../james/modules/server/MessagesRoutesModule.java | 5 ++-
.../james/webadmin/routes/MessagesRoutes.java | 38 +-
.../james/webadmin/routes/MessageRoutesTest.java | 4 ++-
3 files changed, 44 insertions(+), 3 deletions(-)
diff --git
a/server/container/guice/protocols/webadmin-mailbox/src/main/java/org/apache/james/modules/server/MessagesRoutesModule.java
b/server/container/guice/protocols/webadmin-mailbox/src/main/java/org/apache/james/modules/server/MessagesRoutesModule.java
index df23a93..5e0ec7a 100644
---
a/server/container/guice/protocols/webadmin-mailbox/src/main/java/org/apache/james/modules/server/MessagesRoutesModule.java
+++
b/server/container/guice/protocols/webadmin-mailbox/src/main/java/org/apache/james/modules/server/MessagesRoutesModule.java
@@ -19,11 +19,14 @@
package org.apache.james.modules.server;
+import static
org.apache.james.webadmin.tasks.TaskFromRequestRegistry.TaskRegistration;
+
import org.apache.james.webadmin.Routes;
import org.apache.james.webadmin.routes.MessagesRoutes;
import com.google.inject.AbstractModule;
import com.google.inject.multibindings.Multibinder;
+import com.google.inject.name.Names;
public class MessagesRoutesModule extends AbstractModule {
@Override
@@ -31,6 +34,6 @@ public class MessagesRoutesModule extends AbstractModule {
Multibinder routesMultibinder =
Multibinder.newSetBinder(binder(), Routes.class);
routesMultibinder.addBinding().to(MessagesRoutes.class);
-
+Multibinder.newSetBinder(binder(), TaskRegistration.class,
Names.named(MessagesRoutes.ALL_MESSAGES_TASKS));
}
}
diff --git
a/server/protocols/webadmin/webadmin-mailbox/src/main/java/org/apache/james/webadmin/routes/MessagesRoutes.java
b/server/protocols/webadmin/webadmin-mailbox/src/main/java/org/apache/james/webadmin/routes/MessagesRoutes.java
index 45d235f..da42211 100644
---
a/server/protocols/webadmin/webadmin-mailbox/src/main/java/org/apache/james/webadmin/routes/MessagesRoutes.java
+++
b/server/protocols/webadmin/webadmin-mailbox/src/main/java/org/apache/james/webadmin/routes/MessagesRoutes.java
@@ -21,7 +21,11 @@ package org.apache.james.webadmin.routes;
import static org.apache.james.webadmin.routes.MailboxesRoutes.TASK_PARAMETER;
+import java.util.Optional;
+import java.util.Set;
+
import javax.inject.Inject;
+import javax.inject.Named;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
@@ -58,13 +62,18 @@ public class MessagesRoutes implements Routes {
private final MessageId.Factory messageIdFactory;
private final MessageIdReIndexer reIndexer;
private final JsonTransformer jsonTransformer;
+private final Set
allMessagesTaskRegistration;
+
+public static final String ALL_MESSAGES_TASKS = "allMessagesTasks";
@Inject
-MessagesRoutes(TaskManager taskManager, MessageId.Factory
messageIdFactory, MessageIdReIndexer reIndexer, JsonTransformer
jsonTransformer) {
+MessagesRoutes(TaskManager taskManager, MessageId.Factory
messageIdFactory, MessageIdReIndexer reIndexer, JsonTransformer jsonTransformer,
+ @Named(ALL_MESSAGES_TASKS)
Set allMessagesTaskRegistration) {
this.taskManager = taskManager;
this.messageIdFactory = messageIdFactory;
this.reIndexer = reIndexer;
this.jsonTransformer = jsonTransformer;
+this.allMessagesTaskRegistration = allMessagesTaskRegistration;
}
@Override
@@ -75,6 +84,8 @@ public class MessagesRoutes implements Routes {
@Override
public void define(Service service) {
service.post(MESSAGE_PATH, reIndexMessage(), jsonTransformer);
+allMessagesOperations()
+.ifPresent(route -> service.post(BASE_PATH, route,
jsonTransformer));
}
@POST
@@ -121,4 +132,29 @@ public class MessagesRoutes implements Routes {
.haltError();
}
}
+
+@POST
+@Path("/")
+@ApiOperation(value = "Operation on messages")
+@ApiImplicitParams({
+@ApiImplicitParam(
+required = true,
+name = "task",
+paramType = "query parameter",
+dataType = "String",
+defaultValue = "none",
+example = "?task=SolveInconsistencies",
+value = "Compulsory. Depends on the tasks handled by the product")
+})
+@ApiResponses(value = {
+@ApiResponse(code = HttpStatus.CREATED_201, message = "Task is
created", response = TaskIdDto.class),
+@ApiResponse(code = HttpStatus.INTERNAL_SERVER_ERROR_500, message =
"Internal server error - Something went bad o
[james-project] 22/27: JAMES-3138 Add fix nothing integration tests
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit d018a04bd0634dd4783dc9ec00724d1d1b20b34a
Author: Benoit Tellier
AuthorDate: Tue Apr 28 07:48:11 2020 +0700
JAMES-3138 Add fix nothing integration tests
---
.../rabbitmq/ConsistencyTasksIntegrationTest.java | 134 +
1 file changed, 134 insertions(+)
diff --git
a/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/ConsistencyTasksIntegrationTest.java
b/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/ConsistencyTasksIntegrationTest.java
index bb69945..11faa4b 100644
---
a/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/ConsistencyTasksIntegrationTest.java
+++
b/server/protocols/webadmin-integration-test/distributed-webadmin-integration-test/src/test/java/org/apache/james/webadmin/integration/rabbitmq/ConsistencyTasksIntegrationTest.java
@@ -50,6 +50,7 @@ import
org.apache.james.backends.cassandra.init.SessionWithInitializedTablesFact
import org.apache.james.junit.categories.BasicFeature;
import org.apache.james.mailbox.events.RetryBackoffConfiguration;
import org.apache.james.mailbox.exception.MailboxException;
+import org.apache.james.mailbox.model.MailboxConstants;
import org.apache.james.mailbox.model.MailboxPath;
import org.apache.james.mailbox.model.QuotaRoot;
import org.apache.james.modules.AwsS3BlobStoreExtension;
@@ -309,4 +310,137 @@ class ConsistencyTasksIntegrationTest {
.asLong())
.isEqualTo(1);
}
+
+@Test
+void
solveCassandraMappingInconsistencyShouldSolveNotingWhenNoInconsistencies() {
+with()
+.put(AliasRoutes.ROOT_PATH + SEPARATOR + USERNAME + "/sources/" +
ALIAS_1);
+with()
+.put(AliasRoutes.ROOT_PATH + SEPARATOR + USERNAME + "/sources/" +
ALIAS_2);
+
+String taskId = with()
+.queryParam("action", "SolveInconsistencies")
+.post(CassandraMappingsRoutes.ROOT_PATH)
+.jsonPath()
+.get("taskId");
+
+with()
+.basePath(TasksRoutes.BASE)
+.get(taskId + "/await");
+
+when()
+.get(AliasRoutes.ROOT_PATH + SEPARATOR + USERNAME)
+.then()
+.contentType(ContentType.JSON)
+.statusCode(HttpStatus.OK_200)
+.body("source", hasItems(ALIAS_1, ALIAS_2));
+}
+
+@Test
+void
solveMailboxesInconsistencyShouldSolveNotingWhenNoInconsistencies(GuiceJamesServer
server) {
+MailboxProbeImpl probe = server.getProbe(MailboxProbeImpl.class);
+
+try {
+probe.createMailbox(MailboxPath.inbox(BOB));
+} catch (Exception e) {
+// Failure is expected
+}
+
+// schema version 6 or higher required to run solve mailbox
inconsistencies task
+String taskId = with().post(UPGRADE_TO_LATEST_VERSION)
+.jsonPath()
+.get("taskId");
+
+with()
+.get("/tasks/" + taskId + "/await")
+.then()
+.body("status", is("completed"));
+
+taskId = with()
+.header("I-KNOW-WHAT-I-M-DOING", "ALL-SERVICES-ARE-OFFLINE")
+.queryParam("task", "SolveInconsistencies")
+.post("/mailboxes")
+.jsonPath()
+.get("taskId");
+
+with()
+.basePath(TasksRoutes.BASE)
+.get(taskId + "/await");
+
+// The mailbox is removed as it is not in the mailboxDAO source of
truth.
+assertThat(probe.listUserMailboxes(BOB.asString()))
+.containsOnly(MailboxConstants.INBOX);
+}
+
+@Test
+void
recomputeMailboxCountersShouldSolveNotingWhenNoInconsistencies(GuiceJamesServer
server) throws MailboxException {
+MailboxProbeImpl probe = server.getProbe(MailboxProbeImpl.class);
+MailboxPath inbox = MailboxPath.inbox(BOB);
+probe.createMailbox(inbox);
+
+try {
+probe.appendMessage(BOB.asString(), inbox,
+new ByteArrayInputStream("Subject:
test\r\n\r\ntestmail".getBytes(StandardCharsets.UTF_8)), new Date(), false, new
Flags(Flags.Flag.SEEN));
+} catch (Exception e) {
+// Expected to fail
+}
+
+String taskId = with()
+.basePath("/mailboxes")
+.queryParam("task", "RecomputeMailboxCounters")
+.post()
+.jsonPath()
+.get("taskId");
+
+with()
+.basePath(TasksRoutes.BASE)
+.get(taskId + "/await");
+
+assertThat(probe.retrieveCounters(inbox).getCount()).isEqualTo(1);
+}
+
+@Test
+void
recompu
[james-project] 17/27: JAMES-3138 Use concatMap upon quota computation
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 1157a7b3e34fb837fb26d6019426b2d98b6503d8
Author: Benoit Tellier
AuthorDate: Sat Apr 25 11:13:43 2020 +0700
JAMES-3138 Use concatMap upon quota computation
This limit the parralelism of a background task, limiting its capacity
to overwhelm James and affect live traffic.
---
.../apache/james/mailbox/quota/task/RecomputeCurrentQuotasService.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/mailbox/tools/quota-recompute/src/main/java/org/apache/james/mailbox/quota/task/RecomputeCurrentQuotasService.java
b/mailbox/tools/quota-recompute/src/main/java/org/apache/james/mailbox/quota/task/RecomputeCurrentQuotasService.java
index 8508b8e..dbb5078 100644
---
a/mailbox/tools/quota-recompute/src/main/java/org/apache/james/mailbox/quota/task/RecomputeCurrentQuotasService.java
+++
b/mailbox/tools/quota-recompute/src/main/java/org/apache/james/mailbox/quota/task/RecomputeCurrentQuotasService.java
@@ -141,7 +141,7 @@ public class RecomputeCurrentQuotasService {
public Mono recomputeCurrentQuotas(Context context) {
try {
return Iterators.toFlux(usersRepository.list())
-.flatMap(username -> recomputeUserCurrentQuotas(context,
username))
+.concatMap(username -> recomputeUserCurrentQuotas(context,
username))
.reduce(Task.Result.COMPLETED, Task::combine);
} catch (UsersRepositoryException e) {
LOGGER.error("Error while accessing users from repository", e);
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 06/27: JAMES-3143 Guice binding for task
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 4db5a1f875c815383dde3eec2ad3c3ae5d782838
Author: LanKhuat
AuthorDate: Wed Apr 22 11:01:44 2020 +0700
JAMES-3143 Guice binding for task
---
.../webadmin/InconsistencySolvingRoutesModule.java | 16
.../apache/james/modules/TaskSerializationModule.java| 13 +
.../james/modules/server/MessagesRoutesModule.java | 2 ++
3 files changed, 31 insertions(+)
diff --git
a/server/container/guice/cassandra-guice/src/main/java/org/apache/james/modules/webadmin/InconsistencySolvingRoutesModule.java
b/server/container/guice/cassandra-guice/src/main/java/org/apache/james/modules/webadmin/InconsistencySolvingRoutesModule.java
index baf4402..f5197e1 100644
---
a/server/container/guice/cassandra-guice/src/main/java/org/apache/james/modules/webadmin/InconsistencySolvingRoutesModule.java
+++
b/server/container/guice/cassandra-guice/src/main/java/org/apache/james/modules/webadmin/InconsistencySolvingRoutesModule.java
@@ -21,11 +21,14 @@ package org.apache.james.modules.webadmin;
import
org.apache.james.mailbox.cassandra.mail.task.RecomputeMailboxCountersService;
import
org.apache.james.mailbox.cassandra.mail.task.SolveMailboxInconsistenciesService;
+import
org.apache.james.mailbox.cassandra.mail.task.SolveMessageInconsistenciesService;
import org.apache.james.webadmin.Routes;
import org.apache.james.webadmin.routes.CassandraMappingsRoutes;
import org.apache.james.webadmin.routes.MailboxesRoutes;
+import org.apache.james.webadmin.routes.MessagesRoutes;
import org.apache.james.webadmin.routes.RecomputeMailboxCountersRequestToTask;
import
org.apache.james.webadmin.routes.SolveMailboxInconsistenciesRequestToTask;
+import
org.apache.james.webadmin.routes.SolveMessageInconsistenciesRequestToTask;
import org.apache.james.webadmin.service.CassandraMappingsService;
import org.apache.james.webadmin.tasks.TaskFromRequestRegistry;
@@ -60,9 +63,22 @@ public class InconsistencySolvingRoutesModule extends
AbstractModule {
}
}
+public static class SolveMessageInconsistenciesModules extends
AbstractModule {
+@Override
+protected void configure() {
+
bind(SolveMessageInconsistenciesService.class).in(Scopes.SINGLETON);
+
+Multibinder multiBinder
= Multibinder.newSetBinder(binder(),
+TaskFromRequestRegistry.TaskRegistration.class,
Names.named(MessagesRoutes.ALL_MESSAGES_TASKS));
+
+
multiBinder.addBinding().to(SolveMessageInconsistenciesRequestToTask.class);
+}
+}
+
@Override
protected void configure() {
install(new SolveRRTInconsistenciesModules());
install(new SolveMailboxInconsistenciesModules());
+install(new SolveMessageInconsistenciesModules());
}
}
diff --git
a/server/container/guice/cassandra-rabbitmq-guice/src/main/java/org/apache/james/modules/TaskSerializationModule.java
b/server/container/guice/cassandra-rabbitmq-guice/src/main/java/org/apache/james/modules/TaskSerializationModule.java
index 2b03f57..ab6b867 100644
---
a/server/container/guice/cassandra-rabbitmq-guice/src/main/java/org/apache/james/modules/TaskSerializationModule.java
+++
b/server/container/guice/cassandra-rabbitmq-guice/src/main/java/org/apache/james/modules/TaskSerializationModule.java
@@ -41,6 +41,9 @@ import
org.apache.james.mailbox.cassandra.mail.task.RecomputeMailboxCountersTask
import
org.apache.james.mailbox.cassandra.mail.task.SolveMailboxInconsistenciesService;
import
org.apache.james.mailbox.cassandra.mail.task.SolveMailboxInconsistenciesTaskAdditionalInformationDTO;
import
org.apache.james.mailbox.cassandra.mail.task.SolveMailboxInconsistenciesTaskDTO;
+import
org.apache.james.mailbox.cassandra.mail.task.SolveMessageInconsistenciesService;
+import
org.apache.james.mailbox.cassandra.mail.task.SolveMessageInconsistenciesTaskAdditionalInformationDTO;
+import
org.apache.james.mailbox.cassandra.mail.task.SolveMessageInconsistenciesTaskDTO;
import org.apache.james.mailbox.model.MailboxId;
import org.apache.james.mailbox.model.MessageId;
import org.apache.james.queue.api.MailQueueFactory;
@@ -296,6 +299,11 @@ public class TaskSerializationModule extends
AbstractModule {
}
@ProvidesIntoSet
+public TaskDTOModule
solveMessageInconsistenciesTask(SolveMessageInconsistenciesService
solveMessageInconsistenciesService) {
+return
SolveMessageInconsistenciesTaskDTO.module(solveMessageInconsistenciesService);
+}
+
+@ProvidesIntoSet
public TaskDTOModule
singleMailboxReindexingTask(SingleMailboxReindexingTask.Factory factory) {
return SingleMailboxReindexingTaskDTO.module(factory);
}
@@ -440,6 +448,11 @@ public class TaskSerializationModule extends
AbstractModule {
return
MailboxesExportTaskAd
[james-project] 15/27: JAMES-3138 Task for recomputing current quotas
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 7f5a6944c169e9dd6205a53ca5c1c9e8f259e88c
Author: Rene Cordier
AuthorDate: Fri Apr 24 11:05:54 2020 +0700
JAMES-3138 Task for recomputing current quotas
---
.../apache/james/mailbox/model/QuotaRootTest.java | 24 ++
1 file changed, 24 insertions(+)
diff --git
a/mailbox/api/src/test/java/org/apache/james/mailbox/model/QuotaRootTest.java
b/mailbox/api/src/test/java/org/apache/james/mailbox/model/QuotaRootTest.java
index b9b80fb..76dd31e 100644
---
a/mailbox/api/src/test/java/org/apache/james/mailbox/model/QuotaRootTest.java
+++
b/mailbox/api/src/test/java/org/apache/james/mailbox/model/QuotaRootTest.java
@@ -19,6 +19,11 @@
package org.apache.james.mailbox.model;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Optional;
+
+import org.apache.james.core.Domain;
import org.junit.jupiter.api.Test;
import nl.jqno.equalsverifier.EqualsVerifier;
@@ -28,4 +33,23 @@ class QuotaRootTest {
void shouldMatchBeanContract() {
EqualsVerifier.forClass(QuotaRoot.class).verify();
}
+
+@Test
+void asStringShouldReturnValueWhenNoDomain() {
+String value = "#private&bob";
+QuotaRoot quotaRoot = QuotaRoot.quotaRoot(value, Optional.empty());
+
+assertThat(quotaRoot.asString()).isEqualTo(value);
+}
+
+@Test
+void asStringShouldReturnValueWithDomainWhenHasDomain() {
+String value = "#private&bob";
+Domain domain = Domain.of("apache.org");
+QuotaRoot quotaRoot = QuotaRoot.quotaRoot(value, Optional.of(domain));
+
+String expectedValue = "#private&b...@apache.org";
+
+assertThat(quotaRoot.asString()).isEqualTo(expectedValue);
+}
}
\ No newline at end of file
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 20/27: JAMES-3138 Cassandra test instumentation: inject failure after awaitOn
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit fc528ae112a211339616095d87b85a3366d9eb54
Author: Benoit Tellier
AuthorDate: Sat Apr 25 11:52:28 2020 +0700
JAMES-3138 Cassandra test instumentation: inject failure after awaitOn
This enables synchronized failure injection. New behaviours can be
written then injected at will!
---
.../apache/james/backends/cassandra/Scenario.java | 23 ++-
.../backends/cassandra/TestingSessionTest.java | 26 ++
.../cassandra/mail/CassandraACLMapperTest.java | 6 +++--
.../cassandra/mail/CassandraMailboxDAOTest.java| 2 +-
4 files changed, 49 insertions(+), 8 deletions(-)
diff --git
a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/Scenario.java
b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/Scenario.java
index 0d01c3e..f2f3682 100644
---
a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/Scenario.java
+++
b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/Scenario.java
@@ -41,10 +41,10 @@ public class Scenario {
Behavior EXECUTE_NORMALLY = Session::executeAsync;
-static Behavior awaitOn(Barrier barrier) {
+static Behavior awaitOn(Barrier barrier, Behavior behavior) {
return (session, statement) -> {
barrier.call();
-return session.executeAsync(statement);
+return behavior.execute(session, statement);
};
}
@@ -130,6 +130,19 @@ public class Scenario {
}
}
+@FunctionalInterface
+interface ComposeBehavior {
+RequiresValidity then(Behavior behavior);
+
+default RequiresValidity thenExecuteNormally() {
+return then(Behavior.EXECUTE_NORMALLY);
+}
+
+default RequiresValidity thenFail() {
+return then(Behavior.THROW);
+}
+}
+
static RequiresValidity fail() {
return validity -> statementPredicate -> new ExecutionHook(
statementPredicate,
@@ -144,10 +157,10 @@ public class Scenario {
validity);
}
-static RequiresValidity awaitOn(Barrier barrier) {
-return validity -> statementPredicate -> new ExecutionHook(
+static ComposeBehavior awaitOn(Barrier barrier) {
+return behavior -> validity -> statementPredicate -> new
ExecutionHook(
statementPredicate,
-Behavior.awaitOn(barrier),
+Behavior.awaitOn(barrier, behavior),
validity);
}
}
diff --git
a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSessionTest.java
b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSessionTest.java
index bb08907..5140e14 100644
---
a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSessionTest.java
+++
b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/TestingSessionTest.java
@@ -209,6 +209,7 @@ class TestingSessionTest {
Barrier barrier = new Barrier();
cassandra.getConf()
.registerScenario(awaitOn(barrier)
+.thenExecuteNormally()
.times(1)
.whenQueryStartsWith("INSERT INTO schemaVersion"));
@@ -229,6 +230,7 @@ class TestingSessionTest {
Barrier barrier = new Barrier();
cassandra.getConf()
.registerScenario(awaitOn(barrier)
+.thenExecuteNormally()
.times(1)
.whenQueryStartsWith("INSERT INTO schemaVersion"));
@@ -251,6 +253,7 @@ class TestingSessionTest {
Barrier barrier = new Barrier();
cassandra.getConf()
.registerScenario(awaitOn(barrier)
+.thenExecuteNormally()
.times(1)
.whenQueryStartsWith("INSERT INTO schemaVersion"));
@@ -264,4 +267,27 @@ class TestingSessionTest {
assertThat(dao.getCurrentSchemaVersion().block())
.contains(newVersion);
}
+
+@Test
+void awaitOnShouldBeAbleToInjectFailure(CassandraCluster cassandra) throws
Exception {
+SchemaVersion originalSchemaVersion = new SchemaVersion(32);
+SchemaVersion newVersion = new SchemaVersion(36);
+
+dao.updateVersion(originalSchemaVersion).block();
+Barrier barrier = new Barrier();
+cassandra.getConf()
+.registerScenario(awaitOn(barrier)
+.thenFail()
+.times(1)
+.whenQueryStartsWith("INSERT INTO schemaVersion"));
+
+Mono operation = dao.updateVersion(newVersion).cache();
+
+ope
[james-project] 02/27: JAMES-3143 Add select all query for MessageId/ImapUid
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 10e1a444e3a62ce506d7f349aa84f1a01344cd32
Author: LanKhuat
AuthorDate: Tue Apr 14 10:23:47 2020 +0700
JAMES-3143 Add select all query for MessageId/ImapUid
---
.../james/mailbox/cassandra/mail/CassandraMessageIdDAO.java | 12
.../cassandra/mail/CassandraMessageIdToImapUidDAO.java | 11 +++
2 files changed, 23 insertions(+)
diff --git
a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdDAO.java
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdDAO.java
index a6275e3..c2342b3 100644
---
a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdDAO.java
+++
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdDAO.java
@@ -81,6 +81,7 @@ public class CassandraMessageIdDAO {
private final PreparedStatement selectUidGte;
private final PreparedStatement selectUidRange;
private final PreparedStatement update;
+private final PreparedStatement listStatement;
@Inject
public CassandraMessageIdDAO(Session session, CassandraMessageId.Factory
messageIdFactory) {
@@ -93,6 +94,7 @@ public class CassandraMessageIdDAO {
this.selectAllUids = prepareSelectAllUids(session);
this.selectUidGte = prepareSelectUidGte(session);
this.selectUidRange = prepareSelectUidRange(session);
+this.listStatement = prepareList(session);
}
private PreparedStatement prepareDelete(Session session) {
@@ -146,6 +148,11 @@ public class CassandraMessageIdDAO {
.where(eq(MAILBOX_ID, bindMarker(MAILBOX_ID;
}
+private PreparedStatement prepareList(Session session) {
+return session.prepare(select(new String[] {MESSAGE_ID, MAILBOX_ID,
IMAP_UID})
+.from(TABLE_NAME));
+}
+
private PreparedStatement prepareSelectUidGte(Session session) {
return session.prepare(select(FIELDS)
.from(TABLE_NAME)
@@ -224,6 +231,11 @@ public class CassandraMessageIdDAO {
.map(this::fromRowToComposedMessageIdWithFlags);
}
+public Flux retrieveAllMessages() {
+return cassandraAsyncExecutor.executeRows(listStatement.bind())
+.map(this::fromRowToComposedMessageIdWithFlags);
+}
+
private Flux retrieveRows(CassandraId mailboxId, MessageRange set) {
switch (set.getType()) {
case ALL:
diff --git
a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdToImapUidDAO.java
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdToImapUidDAO.java
index 0725bcc..83099ba 100644
---
a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdToImapUidDAO.java
+++
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdToImapUidDAO.java
@@ -77,6 +77,7 @@ public class CassandraMessageIdToImapUidDAO {
private final PreparedStatement update;
private final PreparedStatement selectAll;
private final PreparedStatement select;
+private final PreparedStatement listStatement;
@Inject
public CassandraMessageIdToImapUidDAO(Session session,
CassandraMessageId.Factory messageIdFactory) {
@@ -87,6 +88,7 @@ public class CassandraMessageIdToImapUidDAO {
this.update = prepareUpdate(session);
this.selectAll = prepareSelectAll(session);
this.select = prepareSelect(session);
+this.listStatement = prepareList(session);
}
private PreparedStatement prepareDelete(Session session) {
@@ -136,6 +138,10 @@ public class CassandraMessageIdToImapUidDAO {
.where(eq(MESSAGE_ID, bindMarker(MESSAGE_ID;
}
+private PreparedStatement prepareList(Session session) {
+return session.prepare(select(new String[] {MESSAGE_ID, MAILBOX_ID,
IMAP_UID}).from(TABLE_NAME));
+}
+
private PreparedStatement prepareSelect(Session session) {
return session.prepare(select(FIELDS)
.from(TABLE_NAME)
@@ -193,6 +199,11 @@ public class CassandraMessageIdToImapUidDAO {
.map(this::toComposedMessageIdWithMetadata);
}
+public Flux retrieveAllMessages() {
+return cassandraAsyncExecutor.executeRows(listStatement.bind())
+.map(row -> toComposedMessageIdWithMetadata(row));
+}
+
private ComposedMessageIdWithMetaData toComposedMessageIdWithMetadata(Row
row) {
return ComposedMessageIdWithMetaData.builder()
.composedMessageId(new ComposedMessageId(
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For a
[james-project] 12/27: JAMES-3138 Webadmin endpoint and guice bindings for recomputing current quotas
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 1b0bc6e1d6caa93a7a155193ac00d1fd09809db2
Author: Rene Cordier
AuthorDate: Wed Apr 22 11:01:21 2020 +0700
JAMES-3138 Webadmin endpoint and guice bindings for recomputing current
quotas
---
.../org/apache/james/CassandraJamesServerMain.java | 2 +
.../guice/cassandra-rabbitmq-guice/pom.xml | 4 ++
.../james/modules/TaskSerializationModule.java | 13 +
.../java/org/apache/james/JPAJamesServerMain.java | 2 +
.../InconsistencyQuotasSolvingRoutesModule.java| 40 +++
.../james/modules/server/MailboxRoutesModule.java | 2 +
...dminServerTaskSerializationIntegrationTest.java | 22 +++-
server/protocols/webadmin/webadmin-mailbox/pom.xml | 4 ++
.../RecomputeCurrentQuotasRequestToTask.java | 37 ++
.../james/webadmin/routes/UserQuotaRoutes.java | 59 +-
.../routes/WebAdminQuotaSearchTestSystem.java | 11 +++-
11 files changed, 192 insertions(+), 4 deletions(-)
diff --git
a/server/container/guice/cassandra-guice/src/main/java/org/apache/james/CassandraJamesServerMain.java
b/server/container/guice/cassandra-guice/src/main/java/org/apache/james/CassandraJamesServerMain.java
index 9bc756c..a7fcc7d 100644
---
a/server/container/guice/cassandra-guice/src/main/java/org/apache/james/CassandraJamesServerMain.java
+++
b/server/container/guice/cassandra-guice/src/main/java/org/apache/james/CassandraJamesServerMain.java
@@ -55,6 +55,7 @@ import org.apache.james.modules.server.DKIMMailetModule;
import org.apache.james.modules.server.DLPRoutesModule;
import org.apache.james.modules.server.DataRoutesModules;
import org.apache.james.modules.server.ElasticSearchMetricReporterModule;
+import org.apache.james.modules.server.InconsistencyQuotasSolvingRoutesModule;
import org.apache.james.modules.server.JMXServerModule;
import org.apache.james.modules.server.JmapTasksModule;
import org.apache.james.modules.server.MailQueueRoutesModule;
@@ -85,6 +86,7 @@ public class CassandraJamesServerMain implements
JamesServerMain {
new DataRoutesModules(),
new DeletedMessageVaultRoutesModule(),
new DLPRoutesModule(),
+new InconsistencyQuotasSolvingRoutesModule(),
new InconsistencySolvingRoutesModule(),
new JmapTasksModule(),
new MailboxesExportRoutesModule(),
diff --git a/server/container/guice/cassandra-rabbitmq-guice/pom.xml
b/server/container/guice/cassandra-rabbitmq-guice/pom.xml
index 29fffab..ade5311 100644
--- a/server/container/guice/cassandra-rabbitmq-guice/pom.xml
+++ b/server/container/guice/cassandra-rabbitmq-guice/pom.xml
@@ -87,6 +87,10 @@
${james.groupId}
+apache-james-mailbox-tools-quota-recompute
+
+
+${james.groupId}
blob-objectstorage
test-jar
test
diff --git
a/server/container/guice/cassandra-rabbitmq-guice/src/main/java/org/apache/james/modules/TaskSerializationModule.java
b/server/container/guice/cassandra-rabbitmq-guice/src/main/java/org/apache/james/modules/TaskSerializationModule.java
index ab6b867..0e8b86f 100644
---
a/server/container/guice/cassandra-rabbitmq-guice/src/main/java/org/apache/james/modules/TaskSerializationModule.java
+++
b/server/container/guice/cassandra-rabbitmq-guice/src/main/java/org/apache/james/modules/TaskSerializationModule.java
@@ -46,6 +46,9 @@ import
org.apache.james.mailbox.cassandra.mail.task.SolveMessageInconsistenciesT
import
org.apache.james.mailbox.cassandra.mail.task.SolveMessageInconsistenciesTaskDTO;
import org.apache.james.mailbox.model.MailboxId;
import org.apache.james.mailbox.model.MessageId;
+import org.apache.james.mailbox.quota.task.RecomputeCurrentQuotasService;
+import
org.apache.james.mailbox.quota.task.RecomputeCurrentQuotasTaskAdditionalInformationDTO;
+import org.apache.james.mailbox.quota.task.RecomputeCurrentQuotasTaskDTO;
import org.apache.james.queue.api.MailQueueFactory;
import org.apache.james.queue.api.ManageableMailQueue;
import org.apache.james.rrt.cassandra.CassandraMappingsSourcesDAO;
@@ -279,6 +282,11 @@ public class TaskSerializationModule extends
AbstractModule {
}
@ProvidesIntoSet
+public TaskDTOModule
recomputeCurrentQuotasTask(RecomputeCurrentQuotasService service) {
+return RecomputeCurrentQuotasTaskDTO.module(service);
+}
+
+@ProvidesIntoSet
public TaskDTOModule
messageIdReindexingTask(MessageIdReIndexingTask.Factory factory) {
return MessageIdReindexingTaskDTO.module(factory);
}
@@ -399,6 +407,11 @@ public class TaskSerializationModule extends
AbstractModule {
}
@ProvidesIntoSet
+public AdditionalInformationDTOModule recomputeCurrentQuotasAdditionalInformation() {
+return RecomputeCurrentQuotasTaskAdd
[james-project] 10/27: JAMES-3143 Update CHANGELOG
This is an automated email from the ASF dual-hosted git repository. btellier pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/james-project.git commit e0ce72b109357de06b31637de0bf22d341f37387 Author: LanKhuat AuthorDate: Thu Apr 23 16:26:28 2020 +0700 JAMES-3143 Update CHANGELOG --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5f70328..6d1986f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,8 +15,9 @@ of tasks being currently executed. - JAMES-3062 EventDeadLettersHealthCheck - JAMES-3058 WebAdmin offline task to correct mailbox inconsistencies on top of Cassandra products - JAMES-3105 WebAdmin offline task to recompute mailbox counters on top of Cassandra products -- JAMES-3072 Webadmin endpoint to export mailbox backup +- JAMES-3072 WebAdmin endpoint to export mailbox backup - JAMES-3117 Add PeriodicalHealthChecks for periodical calling all health checks +- JAMES-3143 WebAdmin endpoint to solve Cassandra message inconsistencies ### Changed - Switch to Java 11 for build and run - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 23/27: JAMES-3065 remove remaining usages of Reactor types in public api
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 12a5987235f60cbbe24c42a1e1411f12105ed441
Author: Matthieu Baechler
AuthorDate: Thu Apr 23 12:13:12 2020 +0200
JAMES-3065 remove remaining usages of Reactor types in public api
---
.../cassandra/utils/CassandraHealthCheckTest.java | 8 +++---
.../es/ElasticSearchHealthCheckConnectionTest.java | 4 +--
.../backends/rabbitmq/RabbitMQHealthCheckTest.java | 28 ++--
core/pom.xml | 4 +++
.../apache/james/core/healthcheck/HealthCheck.java | 10 +---
mailbox/api/pom.xml| 4 +++
.../EventDeadLettersHealthCheckContract.java | 30 +++---
metrics/metrics-tests/pom.xml | 4 +++
.../apache/james/PeriodicalHealthChecksTest.java | 2 +-
server/data/data-jmap/pom.xml | 4 +++
.../MessageFastViewProjectionHealthCheck.java | 20 +--
.../MessageFastViewProjectionHealthCheckTest.java | 26 +--
.../james/jpa/healthcheck/JPAHealthCheck.java | 24 +
.../james/jpa/healthcheck/JPAHealthCheckTest.java | 4 +--
.../james/webadmin/routes/HealthCheckRoutes.java | 17 ++--
.../webadmin/routes/HealthCheckRoutesTest.java | 6 +++--
16 files changed, 107 insertions(+), 88 deletions(-)
diff --git
a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/utils/CassandraHealthCheckTest.java
b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/utils/CassandraHealthCheckTest.java
index c37f641..1796033 100644
---
a/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/utils/CassandraHealthCheckTest.java
+++
b/backends-common/cassandra/src/test/java/org/apache/james/backends/cassandra/utils/CassandraHealthCheckTest.java
@@ -49,7 +49,7 @@ class CassandraHealthCheckTest {
@Test
void checkShouldReturnHealthyWhenCassandraIsRunning() {
-Result check = healthCheck.check();
+Result check = healthCheck.checkReactive().block();
assertThat(check.isHealthy()).isTrue();
}
@@ -58,7 +58,7 @@ class CassandraHealthCheckTest {
void
checkShouldReturnUnhealthyWhenCassandraIsNotRunning(DockerCassandraExtension.DockerCassandra
cassandraServer) {
try {
cassandraServer.getContainer().pause();
-Result check = healthCheck.check();
+Result check = healthCheck.checkReactive().block();
assertThat(check.isUnHealthy()).isTrue();
} finally {
@@ -72,12 +72,12 @@ class CassandraHealthCheckTest {
try {
cassandraServer.getContainer().pause();
-healthCheck.check();
+healthCheck.checkReactive().block();
} finally {
cassandraServer.getContainer().unpause();
}
-Result check = healthCheck.check();
+Result check = healthCheck.checkReactive().block();
assertThat(check.isHealthy()).isTrue();
}
diff --git
a/backends-common/elasticsearch/src/test/java/org/apache/james/backends/es/ElasticSearchHealthCheckConnectionTest.java
b/backends-common/elasticsearch/src/test/java/org/apache/james/backends/es/ElasticSearchHealthCheckConnectionTest.java
index 0f5ddd6..0dc6530 100644
---
a/backends-common/elasticsearch/src/test/java/org/apache/james/backends/es/ElasticSearchHealthCheckConnectionTest.java
+++
b/backends-common/elasticsearch/src/test/java/org/apache/james/backends/es/ElasticSearchHealthCheckConnectionTest.java
@@ -44,7 +44,7 @@ class ElasticSearchHealthCheckConnectionTest {
@Test
void checkShouldSucceedWhenElasticSearchIsRunning() {
-assertThat(elasticSearchHealthCheck.check().isHealthy()).isTrue();
+
assertThat(elasticSearchHealthCheck.checkReactive().block().isHealthy()).isTrue();
}
@Test
@@ -53,7 +53,7 @@ class ElasticSearchHealthCheckConnectionTest {
elasticSearch.getDockerElasticSearch().pause();
try {
-
assertThat(elasticSearchHealthCheck.check().isUnHealthy()).isTrue();
+
assertThat(elasticSearchHealthCheck.checkReactive().block().isUnHealthy()).isTrue();
} finally {
elasticSearch.getDockerElasticSearch().unpause();
}
diff --git
a/backends-common/rabbitmq/src/test/java/org/apache/james/backends/rabbitmq/RabbitMQHealthCheckTest.java
b/backends-common/rabbitmq/src/test/java/org/apache/james/backends/rabbitmq/RabbitMQHealthCheckTest.java
index 769741a..f205bc7 100644
---
a/backends-common/rabbitmq/src/test/java/org/apache/james/backends/rabbitmq/RabbitMQHealthCheckTest.java
+++
b/backends-common/rabbitmq/src/test/java/org/apache/james/backends/rabbitmq/RabbitMQHealthCheckTest.java
@@ -46,21 +46,21 @@ class RabbitMQHealthCheckTest {
[james-project] 18/27: JAMES-3138 CassandraCurrentQuotaManager: Improve Reactor code style
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit 00160a4205d11e0e2a9f51fbbe6f0ed70614148f
Author: Benoit Tellier
AuthorDate: Sat Apr 25 11:15:58 2020 +0700
JAMES-3138 CassandraCurrentQuotaManager: Improve Reactor code style
- get rid of redundant Mono::from
- filter rather than nested if
---
.../cassandra/quota/CassandraCurrentQuotaManager.java| 12
1 file changed, 4 insertions(+), 8 deletions(-)
diff --git
a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/quota/CassandraCurrentQuotaManager.java
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/quota/CassandraCurrentQuotaManager.java
index 92c4200..675b3e2 100644
---
a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/quota/CassandraCurrentQuotaManager.java
+++
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/quota/CassandraCurrentQuotaManager.java
@@ -116,13 +116,9 @@ public class CassandraCurrentQuotaManager implements
CurrentQuotaManager {
@Override
public Mono setCurrentQuotas(QuotaOperation quotaOperation) {
-return Mono.from(getCurrentQuotas(quotaOperation.quotaRoot()))
-.flatMap(storedQuotas -> {
-if (!storedQuotas.equals(CurrentQuotas.from(quotaOperation))) {
-return Mono.from(decrease(new
QuotaOperation(quotaOperation.quotaRoot(), storedQuotas.count(),
storedQuotas.size(
-.then(Mono.from(increase(quotaOperation)));
-}
-return Mono.empty();
-});
+return getCurrentQuotas(quotaOperation.quotaRoot())
+.filter(storedQuotas ->
!storedQuotas.equals(CurrentQuotas.from(quotaOperation)))
+.flatMap(storedQuotas -> decrease(new
QuotaOperation(quotaOperation.quotaRoot(), storedQuotas.count(),
storedQuotas.size()))
+.then(increase(quotaOperation)));
}
}
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[james-project] 03/27: JAMES-3143 SolveMessageInconsistenciesService/Test
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit e31d7894b0dd820553b3dfc86f96536bd02e2dc9
Author: LanKhuat
AuthorDate: Tue Apr 14 10:24:28 2020 +0700
JAMES-3143 SolveMessageInconsistenciesService/Test
---
.../cassandra/mail/CassandraMessageIdDAO.java | 2 +-
.../mail/CassandraMessageIdToImapUidDAO.java | 2 +-
.../task/SolveMessageInconsistenciesService.java | 131 +++
.../SolveMessageInconsistenciesServiceTest.java| 398 +
4 files changed, 531 insertions(+), 2 deletions(-)
diff --git
a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdDAO.java
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdDAO.java
index c2342b3..d15e121 100644
---
a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdDAO.java
+++
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdDAO.java
@@ -149,7 +149,7 @@ public class CassandraMessageIdDAO {
}
private PreparedStatement prepareList(Session session) {
-return session.prepare(select(new String[] {MESSAGE_ID, MAILBOX_ID,
IMAP_UID})
+return session.prepare(select(FIELDS)
.from(TABLE_NAME));
}
diff --git
a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdToImapUidDAO.java
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdToImapUidDAO.java
index 83099ba..91cf3e3 100644
---
a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdToImapUidDAO.java
+++
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/CassandraMessageIdToImapUidDAO.java
@@ -139,7 +139,7 @@ public class CassandraMessageIdToImapUidDAO {
}
private PreparedStatement prepareList(Session session) {
-return session.prepare(select(new String[] {MESSAGE_ID, MAILBOX_ID,
IMAP_UID}).from(TABLE_NAME));
+return session.prepare(select(FIELDS).from(TABLE_NAME));
}
private PreparedStatement prepareSelect(Session session) {
diff --git
a/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMessageInconsistenciesService.java
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMessageInconsistenciesService.java
new file mode 100644
index 000..f67ff73
--- /dev/null
+++
b/mailbox/cassandra/src/main/java/org/apache/james/mailbox/cassandra/mail/task/SolveMessageInconsistenciesService.java
@@ -0,0 +1,131 @@
+/
+ * Licensed to the Apache Software Foundation (ASF) under one *
+ * or more contributor license agreements. See the NOTICE file *
+ * distributed with this work for additional information*
+ * regarding copyright ownership. The ASF licenses this file *
+ * to you under the Apache License, Version 2.0 (the*
+ * "License"); you may not use this file except in compliance *
+ * with the License. You may obtain a copy of the License at *
+ * *
+ * http://www.apache.org/licenses/LICENSE-2.0 *
+ * *
+ * Unless required by applicable law or agreed to in writing, *
+ * software distributed under the License is distributed on an *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY *
+ * KIND, either express or implied. See the License for the*
+ * specific language governing permissions and limitations *
+ * under the License. *
+ /
+
+package org.apache.james.mailbox.cassandra.mail.task;
+
+import java.util.Optional;
+
+import javax.inject.Inject;
+
+import org.apache.james.mailbox.cassandra.ids.CassandraId;
+import org.apache.james.mailbox.cassandra.ids.CassandraMessageId;
+import org.apache.james.mailbox.cassandra.mail.CassandraMessageIdDAO;
+import org.apache.james.mailbox.cassandra.mail.CassandraMessageIdToImapUidDAO;
+import org.apache.james.mailbox.model.ComposedMessageIdWithMetaData;
+import org.apache.james.task.Task;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import reactor.core.publisher.Flux;
+import reactor.core.publisher.Mono;
+
+public class SolveMessageInconsistenciesService {
+public static final Logger LOGGER =
LoggerFactory.getLogger(SolveMessageInconsistenciesService.class);
+
+private final CassandraMessageIdToImapUidDAO idToImapUidDAO;
+private final CassandraMessageIdDAO messageIdDAO;
+
+@Inject
+SolveMessageInconsistenci
[james-project] 09/27: JAMES-3143 Update webadmin documentation
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit a7d33f77dda5a7f029379f81531fc719a08f0321
Author: LanKhuat
AuthorDate: Wed Apr 22 18:42:22 2020 +0700
JAMES-3143 Update webadmin documentation
---
src/site/markdown/server/manage-webadmin.md | 76 -
1 file changed, 75 insertions(+), 1 deletion(-)
diff --git a/src/site/markdown/server/manage-webadmin.md
b/src/site/markdown/server/manage-webadmin.md
index e88532e..7efda46 100644
--- a/src/site/markdown/server/manage-webadmin.md
+++ b/src/site/markdown/server/manage-webadmin.md
@@ -697,6 +697,81 @@ The scheduled task will have the following type
`messageId-reindexing` and the f
Warning: During the re-indexing, the result of search operations might be
altered.
+### Fixing message inconsistencies
+
+This task is only available on top of Guice Cassandra products.
+
+```
+curl -XPOST /messages?task=SolveInconsistencies
+```
+
+Will schedule a task for fixing message inconsistencies created by the message
denormalization process.
+
+Messages are denormalized and stored in separated data tables in Cassandra, so
they can be accessed
+by their unique identifier or mailbox identifier & local mailbox identifier
through different protocols.
+
+Failure in the denormalization process will lead to inconsistencies, for
example:
+
+```
+BOB receives a message
+The denormalization process fails
+BOB can read the message via JMAP
+BOB cannot read the message via IMAP
+
+BOB marks a message as SEEN
+The denormalization process fails
+The message is SEEN via JMAP
+The message is UNSEEN via IMAP
+```
+
+[More details about endpoints returning a task](#Endpoints_returning_a_task).
+
+Response codes:
+
+ - 201: Success. Corresponding task id is returned.
+ - 400: Error in the request. Details can be found in the reported error.
+
+The scheduled task will have the following type
`solve-message-inconsistencies` and the following `additionalInformation`:
+
+```
+{
+ "type":"solve-message-inconsistencies",
+ "timestamp":"2007-12-03T10:15:30Z",
+ "processedImapUidEntries": 2,
+ "processedMessageIdEntries": 1,
+ "addedMessageIdEntries": 1,
+ "updatedMessageIdEntries": 0,
+ "removedMessageIdEntries": 1,
+ "fixedInconsistencies": [
+{
+ "mailboxId": "551f0580-82fb-11ea-970e-f9c83d4cf8c2",
+ "messageId": "d2bee791-7e63-11ea-883c-95b84008f979",
+ "uid": 1
+},
+{
+ "mailboxId": "551f0580-82fb-11ea-970e-f9c83d4cf8c2",
+ "messageId": "d2bee792-7e63-11ea-883c-95b84008f979",
+ "uid": 2
+}
+ ],
+ "errors": [
+{
+ "mailboxId": "551f0580-82fb-11ea-970e-f9c83d4cf8c2",
+ "messageId": "-7e63-11ea-883c-95b84008f979",
+ "uid": 3
+}
+ ]
+}
+```
+
+User actions concurrent to the inconsistency fixing task could result in
concurrency issues. New inconsistencies
+could be created.
+
+However the source of truth will not be impacted, hence rerunning the task
will eventually fix all issues.
+
+This task could be run safely online and can be scheduled on a recurring basis
outside of peak traffic
+by an admin to ensure Cassandra message consistency.
+
## Administrating user mailboxes
- [Creating a mailbox](#Creating_a_mailbox)
@@ -3554,4 +3629,3 @@ Response codes :
- 201: the taskId of the created task
- 400: Invalid action argument for performing operation on mappings data
-
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Commented] (JAMES-3153) Upgrade reactor to make use of retryWhen and RetrySpec
[ https://issues.apache.org/jira/browse/JAMES-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094964#comment-17094964 ] Benoit Tellier commented on JAMES-3153: --- https://github.com/linagora/james-project/pull/3328 contributed this > Upgrade reactor to make use of retryWhen and RetrySpec > -- > > Key: JAMES-3153 > URL: https://issues.apache.org/jira/browse/JAMES-3153 > Project: James Server > Issue Type: Improvement >Reporter: Matthieu Baechler >Priority: Major > > Latest reactor version include some new features regarding retry backoff (see > https://github.com/reactor/reactor-core/pull/1979 ) > As we were limited by the previous API, we did copy some code from reactor to > workaround the limitations. > Now that they implement a RetrySpec, we should be able to get rid of this > code duplication and rely on their well-tested feature. > h2. Definition of done: > * All reactive code should still work as before > * The code should use RetryWhen with RetrySpec instead of previous solutions -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Commented] (JAMES-3065) API projects should not make use of Reactor types
[
https://issues.apache.org/jira/browse/JAMES-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094965#comment-17094965
]
Benoit Tellier commented on JAMES-3065:
---
https://github.com/linagora/james-project/pull/3324 remove remaining usages of
Reactor types in public api
> API projects should not make use of Reactor types
> -
>
> Key: JAMES-3065
> URL: https://issues.apache.org/jira/browse/JAMES-3065
> Project: James Server
> Issue Type: Improvement
>Reporter: Matthieu Baechler
>Priority: Major
> Fix For: 3.5.0
>
>
> James being built using Hexagonal Architecture, we usually don't use types
> from implementation libraries into our API projects.
> We should make sure we use {{Publisher}} instead of {{Mono}}/{{Flux}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Closed] (JAMES-3143) Task to fix message projection inconsistency
[
https://issues.apache.org/jira/browse/JAMES-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoit Tellier closed JAMES-3143.
-
Fix Version/s: 3.6.0
Resolution: Fixed
https://github.com/linagora/james-project/pull/3296 contributed the service and
https://github.com/linagora/james-project/pull/3315 contributed the task
> Task to fix message projection inconsistency
>
>
> Key: JAMES-3143
> URL: https://issues.apache.org/jira/browse/JAMES-3143
> Project: James Server
> Issue Type: Improvement
>Reporter: René Cordier
>Priority: Major
> Fix For: 3.6.0
>
>
> We can create a webadmin to solve these inconsistencies. Iterating
> {{imapUidTable}} entries, we can rewrite entries in {{messageIdTable}}.
> Iterating {{imapUidTable}} we can remove entries not referenced in
> {{messageIdTable}}.
>
> {{XPOST /messages?task=SolveInconsistencies}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Closed] (JAMES-3138) Task to Recompute quotas
[
https://issues.apache.org/jira/browse/JAMES-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoit Tellier closed JAMES-3138.
-
Resolution: Fixed
https://github.com/linagora/james-project/pull/3297 proposes a task and a
endpoint
https://github.com/linagora/james-project/pull/3329 proposes an integration test
> Task to Recompute quotas
>
>
> Key: JAMES-3138
> URL: https://issues.apache.org/jira/browse/JAMES-3138
> Project: James Server
> Issue Type: Improvement
>Reporter: René Cordier
>Priority: Major
>
>
> {code:java}
> POST /mailboxes?task=RecomputeCurrentQuotas{code}
>
> Will recompute mailbox quotas.
> Have a look to {{RecomputeMailboxQuota}}
> DoD: memory, cassandra and JPA implementations of this task with respective
> contract
> ADR:
> https://github.com/apache/james-project/blob/master/src/adr/0028-Recompute-mailbox-quotas.md
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
Against the use of conditional statements in Guice modules
Hello all, While working on Cassandra blob store cache, we discussed our usage of conditional logic within Guice modules. Our conclusion so far is: ``` Should we have the DistributedJamesServerMain doing a lookup on the configuration to know if it should compose the cache module? yes ``` Link: https://github.com/linagora/james-project/pull/3261#issuecomment-613911695 Thus, in an effort to abandon this anti pattern, I took the time to consider why we use such conditional statements in guice, for which purposes, and propose an alternative solution. I wrote both an Architecture Decision Record, that you can find below, as well than a full pull request (https://github.com/linagora/james-project/pull/3261). What is your opinion concerning this (crucial) topic? Regards, Benoit --- ADR Pull request: https://github.com/apache/james-project/pull/188 # 36. Against the use of conditional statements in Guice modules Date: 2019-12-29 ## Status Accepted (lazy consensus) ## Context James products relies historically on Spring for injections. Spring version is outdated (4.3.25 instead of 5.3 release line). Spring enables overriding any component via a configuration file thus endangering overall correctness by giving too much power to the user. James propose several implementation for each of the interfaces we define. We don't run tests for each possible interface combination. We rather run integration tests for combinations that makes sens. By overriding any components, spring defeats this testing logic. Knowing which component implementation combines well with which other one is not intuitive to users, hard to document and test. We thus decided to rely on Guice. Components are defined by code in a static fashion. Overriding components requires explicit code modification and recompilation, thus warning the user about the impact of the choices he does, and lowering the project responsibility. With Guice we expose only supported, well tested combinations of components, thus addressing the combination issue. Instead of having a single big application being able to instantiate each and every component application, we have several product defining their dependencies in a minimalistic way, relying only on the components implementation that are needed. Here is the list of products we rely on: - memory-guice: A memory based James server, mainly for testing purposes - Distributed James: A scalable James server, storing data in various data stores. Cassandra is used for metadata, ElasticSearch for search, RabbitMQ for messaging, and ObjectStorage for byte contents. - Cassandra-guice: An implementation step toward Distributed James. It do not include messaging and ObjectStorage. - JPA-guice: A JPA and Lucene based implementation of James. Only Derby driver is currently supported. - JPA-smtp: A minimalistic SMTP server based on JPA storage technology. Some components however do have several subtly diverging implementation a user might choose to rely on independently of the product he uses. This is the case for: - BlobExport: Exporting a blob from the blobStore to and external user. Two implementation are currently supported: localFiles and LinShare. - Text extraction: Extracting text from attachment to enable attachment search. There is a Tika implementation, but lighter, JSOUP based options are also available. In order to keep the cardinality of Guice products low, we decided to use conditional statements in modules based on the configuration to select which one to enable. Eventually defeating the Guice adoption goals mentioned above. Finally, Blob Storing technology offers a wide combination of technology: - ObjectStorage in itself could implement either Swift APIs or Amazon S3 APIs - We decided to keep supporting Cassandra for blob storing as an upgrade solution from Cassandra-guice to Distributed James for existing users. - Proposals such as [HybridBlobStore](0014-blobstore-storage-policies.md) and then [Cassandra BlobStore cache](0025-cassandra-blob-store-cache.md) proposed to leverage Cassandra as a performance (latency) enhancer for ObjectStorage technologies. Yet again it had been decided to use conditional statements in modules in order to lower the cardinality. However, [Cassandra BlobStore cache](0025-cassandra-blob-store-cache.md) requires expensive resource initialization requiring to perform upgrade procedure (usage of an additional cache keyspace) that represents a cost we don't want to pay if we don't rely on that cache. Not having the cache module thus enables quickly auditing that the caching cassandra session is not initialized. See [this comment](https://github.com/linagora/james-project/pull/3261#pullrequestreview-389804841) as well as [this comment](https://github.com/linagora/james-project/pull/3261#issuecomment-613911695). ### Audit The following modules perform conditional statements upon injection time:
[jira] [Closed] (JAMES-3153) Upgrade reactor to make use of retryWhen and RetrySpec
[ https://issues.apache.org/jira/browse/JAMES-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthieu Baechler closed JAMES-3153. Fix Version/s: 3.6.0 Resolution: Fixed > Upgrade reactor to make use of retryWhen and RetrySpec > -- > > Key: JAMES-3153 > URL: https://issues.apache.org/jira/browse/JAMES-3153 > Project: James Server > Issue Type: Improvement >Reporter: Matthieu Baechler >Priority: Major > Fix For: 3.6.0 > > > Latest reactor version include some new features regarding retry backoff (see > https://github.com/reactor/reactor-core/pull/1979 ) > As we were limited by the previous API, we did copy some code from reactor to > workaround the limitations. > Now that they implement a RetrySpec, we should be able to get rid of this > code duplication and rely on their well-tested feature. > h2. Definition of done: > * All reactive code should still work as before > * The code should use RetryWhen with RetrySpec instead of previous solutions -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
