Re: [PR] JAMES-3997 Netty backpressure for IMAP FETCH [james-project]

[via GitHub]

chibenwa commented on PR #2031:
URL: https://github.com/apache/james-project/pull/2031#issuecomment-1963074605

   Running in prod like wonder, no more OOMs since then.
   
   Merging...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] JAMES-3997 Netty backpressure for IMAP FETCH [james-project]

[via GitHub]

chibenwa merged PR #2031:
URL: https://github.com/apache/james-project/pull/2031


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

(james-project) branch master updated: JAMES-3997 Netty backpressure for IMAP FETCH (#2031)

[btellier]

This is an automated email from the ASF dual-hosted git repository.

btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git


The following commit(s) were added to refs/heads/master by this push:
 new 915bcc33f2 JAMES-3997 Netty backpressure for IMAP FETCH (#2031)
915bcc33f2 is described below

commit 915bcc33f297f44f159a9fac4eead4e42bebd590
Author: Benoit TELLIER 
AuthorDate: Sun Feb 25 22:58:19 2024 +0100

JAMES-3997 Netty backpressure for IMAP FETCH (#2031)

Channel writability is the netty trick for backpressure:
as a server you shall avoid writing to a channel that is not
writable.

In order to apply this Netty best practice to Apache James we
relied on:
 - The ImapSession as a mean to communicate between the
  processor and the channel.
 - A sink to bridge the gap from a push oriented architecture
 (our IMAP stack) to a pull one (netty writability).
---
 .../apache/james/imap/api/process/ImapSession.java | 13 +
 .../james/imap/processor/fetch/FetchProcessor.java | 66 --
 .../netty/ImapChannelUpstreamHandler.java  |  7 +++
 .../james/imapserver/netty/NettyConstants.java |  1 +
 .../james/imapserver/netty/NettyImapSession.java   | 10 
 5 files changed, 92 insertions(+), 5 deletions(-)

diff --git 
a/protocols/imap/src/main/java/org/apache/james/imap/api/process/ImapSession.java
 
b/protocols/imap/src/main/java/org/apache/james/imap/api/process/ImapSession.java
index 5c2bccceb5..94498ef9ce 100644
--- 
a/protocols/imap/src/main/java/org/apache/james/imap/api/process/ImapSession.java
+++ 
b/protocols/imap/src/main/java/org/apache/james/imap/api/process/ImapSession.java
@@ -102,6 +102,19 @@ public interface ImapSession extends 
CommandDetectionSession {
  */
 Mono logout();
 
+/**
+ * Allows implementation to apply back pressure on heavy senders.
+ *
+ * Return true if the sender needs to be throttled.
+ * Return false if backpressure do not need to be applied.
+ * @param restoreBackpressure will be called to restore backpressure to 
its current state when backpressure
+ *is no longer needed.
+ */
+default boolean backpressureNeeded(Runnable restoreBackpressure) {
+// Naive implementation: never backpressure
+return false;
+}
+
 /**
  * Gets the current client state.
  * 
diff --git 
a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchProcessor.java
 
b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchProcessor.java
index 65a437324f..2caf9a0e7d 100644
--- 
a/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchProcessor.java
+++ 
b/protocols/imap/src/main/java/org/apache/james/imap/processor/fetch/FetchProcessor.java
@@ -28,6 +28,7 @@ import java.util.EnumSet;
 import java.util.List;
 import java.util.Optional;
 import java.util.Set;
+import java.util.concurrent.atomic.AtomicReference;
 
 import javax.inject.Inject;
 
@@ -59,6 +60,8 @@ import org.apache.james.metrics.api.MetricFactory;
 import org.apache.james.util.AuditTrail;
 import org.apache.james.util.MDCBuilder;
 import org.apache.james.util.ReactorUtils;
+import org.reactivestreams.Subscriber;
+import org.reactivestreams.Subscription;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -70,8 +73,58 @@ import it.unimi.dsi.fastutil.longs.LongArrayList;
 import it.unimi.dsi.fastutil.longs.LongList;
 import reactor.core.publisher.Flux;
 import reactor.core.publisher.Mono;
+import reactor.core.publisher.Sinks;
 
 public class FetchProcessor extends AbstractMailboxProcessor {
+static class FetchSubscriber implements Subscriber {
+private final AtomicReference subscription = new 
AtomicReference<>();
+private final Sinks.One sink = Sinks.one();
+private final ImapSession imapSession;
+private final Responder responder;
+
+FetchSubscriber(ImapSession imapSession, Responder responder) {
+this.imapSession = imapSession;
+this.responder = responder;
+}
+
+@Override
+public void onSubscribe(Subscription subscription) {
+this.subscription.set(subscription);
+subscription.request(1);
+}
+
+@Override
+public void onNext(FetchResponse fetchResponse) {
+responder.respond(fetchResponse);
+if (imapSession.backpressureNeeded(this::requestOne)) {
+LOGGER.debug("Applying backpressure as we encounter a slow 
reader");
+} else {
+requestOne();
+}
+}
+
+private void requestOne() {
+Optional.ofNullable(subscription.get())
+.ifPresent(s -> s.request(1));
+}
+
+@Override
+public void onError(Throwable throwable) {
+subscription.set(null);
+

Re: [PR] JAMES-4008 JMAP - Email/set - Should be able to save a draft with invalid email address [james-project]

[via GitHub]

chibenwa commented on PR #2040:
URL: https://github.com/apache/james-project/pull/2040#issuecomment-1963077030

   
org.apache.james.jmap.rfc8621.memory.MemoryEmailSubmissionSetMethodTest.emailSubmissionSetCanBeChainedAfterEmailSet(GuiceJamesServer)
   Failing for the past 1 build (Since
   [#4](https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-2040/4/) )
   [Took 10 
sec.](https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-2040/4/testReport/junit/org.apache.james.jmap.rfc8621.memory/MemoryEmailSubmissionSetMethodTest/emailSubmissionSetCanBeChainedAfterEmailSet_GuiceJamesServer_/history)
   Add description
   Error Message
   
   Assertion condition defined as a lambda expression in 
org.apache.james.jmap.rfc8621.contract.EmailSubmissionSetMethodContract that 
uses java.lang.String [Node "methodResponses[0][1].ids"] 
   Expected size: 1 but was: 0 in:
   [] within 10 seconds.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] JAMES-4008 JMAP - Email/set - Should be able to save a draft with invalid email address [james-project]

[via GitHub]

chibenwa commented on PR #2040:
URL: https://github.com/apache/james-project/pull/2040#issuecomment-1963077075

   Related?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

(james-project) branch master updated: JAMES-4010 Do not index body (#2018)

[btellier]

This is an automated email from the ASF dual-hosted git repository.

btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git


The following commit(s) were added to refs/heads/master by this push:
 new 667c27e6b9 JAMES-4010 Do not index body (#2018)
667c27e6b9 is described below

commit 667c27e6b9af0dd72a70679a6bcaee0096e3fda9
Author: Benoit TELLIER 
AuthorDate: Sun Feb 25 23:04:26 2024 +0100

JAMES-4010 Do not index body (#2018)

This relax pressure on the AES blob store and APPEND

Co-authored-by: Rene Cordier 
---
 .../apache/james/mailbox/opensearch/IndexBody.java |  24 +++
 .../opensearch/OpenSearchMailboxConfiguration.java |  37 +++-
 .../OpenSearchListeningMessageSearchIndex.java |  12 +-
 .../mailbox/opensearch/json/IndexableMessage.java  |  17 +-
 .../opensearch/json/MessageToOpenSearchJson.java   |  14 +-
 .../mailbox/opensearch/json/MimePartParser.java|  10 +-
 .../OpenSearchMailboxConfigurationTest.java|  97 ++
 .../OpenSearchNoIndexBodyIntegrationTest.java  | 197 +
 .../store/search/ListeningMessageSearchIndex.java  |  10 +-
 .../modules/ROOT/pages/configure/opensearch.adoc   |   3 +
 src/site/xdoc/server/config-opensearch.xml |   2 +
 11 files changed, 372 insertions(+), 51 deletions(-)

diff --git 
a/mailbox/opensearch/src/main/java/org/apache/james/mailbox/opensearch/IndexBody.java
 
b/mailbox/opensearch/src/main/java/org/apache/james/mailbox/opensearch/IndexBody.java
new file mode 100644
index 00..b5dd068701
--- /dev/null
+++ 
b/mailbox/opensearch/src/main/java/org/apache/james/mailbox/opensearch/IndexBody.java
@@ -0,0 +1,24 @@
+/
+ * Licensed to the Apache Software Foundation (ASF) under one   *
+ * or more contributor license agreements.  See the NOTICE file *
+ * distributed with this work for additional information*
+ * regarding copyright ownership.  The ASF licenses this file   *
+ * to you under the Apache License, Version 2.0 (the*
+ * "License"); you may not use this file except in compliance   *
+ * with the License.  You may obtain a copy of the License at   *
+ *  *
+ *   http://www.apache.org/licenses/LICENSE-2.0 *
+ *  *
+ * Unless required by applicable law or agreed to in writing,   *
+ * software distributed under the License is distributed on an  *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY   *
+ * KIND, either express or implied.  See the License for the*
+ * specific language governing permissions and limitations  *
+ * under the License.   *
+ /
+
+package org.apache.james.mailbox.opensearch;
+
+public enum IndexBody {
+NO, YES
+}
diff --git 
a/mailbox/opensearch/src/main/java/org/apache/james/mailbox/opensearch/OpenSearchMailboxConfiguration.java
 
b/mailbox/opensearch/src/main/java/org/apache/james/mailbox/opensearch/OpenSearchMailboxConfiguration.java
index 094409581e..6fa68d0286 100644
--- 
a/mailbox/opensearch/src/main/java/org/apache/james/mailbox/opensearch/OpenSearchMailboxConfiguration.java
+++ 
b/mailbox/opensearch/src/main/java/org/apache/james/mailbox/opensearch/OpenSearchMailboxConfiguration.java
@@ -35,8 +35,8 @@ public class OpenSearchMailboxConfiguration {
 private Optional writeAliasMailboxName;
 private Optional indexAttachment;
 private Optional indexHeaders;
-
 private Optional optimiseMoves;
+private Optional indexBody;
 
 Builder() {
 indexMailboxName = Optional.empty();
@@ -45,6 +45,7 @@ public class OpenSearchMailboxConfiguration {
 indexAttachment = Optional.empty();
 indexHeaders = Optional.empty();
 optimiseMoves = Optional.empty();
+indexBody = Optional.empty();
 }
 
 public Builder indexMailboxName(Optional indexMailboxName) {
@@ -77,6 +78,11 @@ public class OpenSearchMailboxConfiguration {
 return this;
 }
 
+public Builder indexBody(IndexBody indexBody) {
+this.indexBody = Optional.ofNullable(indexBody);
+return this;
+}
+
 public OpenSearchMailboxConfiguration build() {
 return new OpenSearchMailboxConfiguration(
 
indexMailboxName.orElse(MailboxOpenSearchConstants.DEFAULT_MAILBOX_INDEX),
@@ -84,7 +90,8 @@ public class OpenSearchMailboxConfiguration {
 
writeAliasMailboxName.orElse(MailboxOpenSearchConstants.DEFAULT_MAILBOX_WRITE_ALIAS),
 indexAttachment.orElse(IndexAttachments.YES),
 indexHeaders.orElse(IndexHeaders.YES),
-optimiseMoves.orElse(DEFAULT_OPTIMIZE_MOVES));

Re: [PR] Do not index body [james-project]

[via GitHub]

chibenwa commented on PR #2018:
URL: https://github.com/apache/james-project/pull/2018#issuecomment-1963076095

   https://issues.apache.org/jira/browse/JAMES-4010


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] Do not index body [james-project]

[via GitHub]

chibenwa merged PR #2018:
URL: https://github.com/apache/james-project/pull/2018


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

[PR] JAMES-2586 Fix some jmap postgres integration tests [james-project]

[via GitHub]

hungphan227 opened a new pull request, #2048:
URL: https://github.com/apache/james-project/pull/2048

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] Revert git commit id maven plugin [james-project]

[via GitHub]

chibenwa commented on PR #2049:
URL: https://github.com/apache/james-project/pull/2049#issuecomment-1963472928

   To be honnest I'd be more in favour of removing that plugin from global 
build and configuring it only in `server/apps` modules.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

[PR] [BUILD] Use git-commit-id-maven-plugin in server/apps only [james-project]

[via GitHub]

chibenwa opened a new pull request, #2050:
URL: https://github.com/apache/james-project/pull/2050

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] Revert git commit id maven plugin [james-project]

[via GitHub]

chibenwa commented on PR #2049:
URL: https://github.com/apache/james-project/pull/2049#issuecomment-1963499656

   > I wasn't aware that the James project build used git submodules.
   
   Not in james but the use linagora uses James.
   
   TLDR: it allows repackaging james, cross repository changes without the need 
of a fork.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

(james-project) branch master updated: JAMES-4002 AES blob store tuning to reduce memory pressure (#2021)

[btellier]

This is an automated email from the ASF dual-hosted git repository.

btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git


The following commit(s) were added to refs/heads/master by this push:
 new 66426a8005 JAMES-4002 AES blob store tuning to reduce memory pressure 
(#2021)
66426a8005 is described below

commit 66426a8005be4ecbb174b95450bd945ae0fd2f73
Author: Benoit TELLIER 
AuthorDate: Mon Feb 26 07:27:15 2024 +0100

JAMES-4002 AES blob store tuning to reduce memory pressure (#2021)


 - Reduce gratly memory pressure thanks to buffering
 - Handle OOM - turns out in this situation we might recover from them

Co-authored-by: Trần Hồng Quân 
<55171818+quantranhong1...@users.noreply.github.com>
---
 .../modules/ROOT/pages/configure/blobstore.adoc| 16 +
 .../sample-configuration/jvm.properties| 12 
 server/blob/blob-aes/pom.xml   |  4 ++
 .../org/apache/james/blob/aes/AESBlobStoreDAO.java | 76 +++---
 .../org/apache/james/blob/api/BlobStoreDAO.java| 26 
 .../blob/objectstorage/aws/S3BlobStoreDAO.java | 10 +++
 6 files changed, 135 insertions(+), 9 deletions(-)

diff --git 
a/server/apps/distributed-app/docs/modules/ROOT/pages/configure/blobstore.adoc 
b/server/apps/distributed-app/docs/modules/ROOT/pages/configure/blobstore.adoc
index 3e599f03c5..ca11b9a278 100644
--- 
a/server/apps/distributed-app/docs/modules/ROOT/pages/configure/blobstore.adoc
+++ 
b/server/apps/distributed-app/docs/modules/ROOT/pages/configure/blobstore.adoc
@@ -84,6 +84,22 @@ openssl rand -base64 64
 generate salt with : openssl rand -hex 16
 
 
+AES blob store supports the following system properties that could be 
configured in `jvm.properties`:
+
+
+# Threshold from which we should buffer the blob to a file upon encrypting
+# Unit supported: K, M, G, default to no unit
+james.blob.aes.file.threshold.encrypt=100K
+
+# Threshold from which we should buffer the blob to a file upon decrypting
+# Unit supported: K, M, G, default to no unit
+james.blob.aes.file.threshold.decrypt=256K
+
+# Maximum size of a blob. Larger blobs will be rejected.
+# Unit supported: K, M, G, default to no unit
+james.blob.aes.blob.max.size=100M
+
+
 === Cassandra BlobStore Cache
 
 A Cassandra cache can be enabled to reduce latency when reading small blobs 
frequently.
diff --git a/server/apps/distributed-app/sample-configuration/jvm.properties 
b/server/apps/distributed-app/sample-configuration/jvm.properties
index 8cf160fce6..2e78d542f8 100644
--- a/server/apps/distributed-app/sample-configuration/jvm.properties
+++ b/server/apps/distributed-app/sample-configuration/jvm.properties
@@ -62,3 +62,15 @@ jmx.remote.x.mlet.allow.getMBeansFromURL=false
 
 # Default charset to use in JMAP to present text body parts
 # james.jmap.default.charset=US-ASCII
+
+# Threshold from which we should buffer the blob to a file upon encrypting
+# Unit supported: K, M, G, default to no unit
+#james.blob.aes.file.threshold.encrypt=100K
+
+# Threshold from which we should buffer the blob to a file upon decrypting
+# Unit supported: K, M, G, default to no unit
+#james.blob.aes.file.threshold.decrypt=256K
+
+# Maximum size of a blob. Larger blobs will be rejected.
+# Unit supported: K, M, G, default to no unit
+#james.blob.aes.blob.max.size=100M
diff --git a/server/blob/blob-aes/pom.xml b/server/blob/blob-aes/pom.xml
index 3de703d6af..842bde3316 100644
--- a/server/blob/blob-aes/pom.xml
+++ b/server/blob/blob-aes/pom.xml
@@ -66,6 +66,10 @@
 io.projectreactor
 reactor-core
 
+
+org.apache.james
+james-server-util
+
 
 
 
diff --git 
a/server/blob/blob-aes/src/main/java/org/apache/james/blob/aes/AESBlobStoreDAO.java
 
b/server/blob/blob-aes/src/main/java/org/apache/james/blob/aes/AESBlobStoreDAO.java
index 3d75499fd4..dfd2750f11 100644
--- 
a/server/blob/blob-aes/src/main/java/org/apache/james/blob/aes/AESBlobStoreDAO.java
+++ 
b/server/blob/blob-aes/src/main/java/org/apache/james/blob/aes/AESBlobStoreDAO.java
@@ -23,8 +23,11 @@ import java.io.ByteArrayInputStream;
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.OutputStream;
+import java.nio.channels.Channels;
+import java.nio.channels.WritableByteChannel;
 import java.security.GeneralSecurityException;
 import java.util.Collection;
+import java.util.Optional;
 
 import org.apache.commons.io.IOUtils;
 import org.apache.james.blob.api.BlobId;
@@ -32,20 +35,35 @@ import org.apache.james.blob.api.BlobStoreDAO;
 import org.apache.james.blob.api.BucketName;
 import org.apache.james.blob.api.ObjectNotFoundException;
 import org.apache.james.blob.api.ObjectStoreIOException;
+import org.apache.james.util.Size;
 import org.reactivestreams.Publisher;
+import org.slf4j.LoggerFactory;
 
 import com.github.fge.lambdas.Throwing;
 import com.google.common.base.Preconditions;
 

Re: [PR] Revert git commit id maven plugin [james-project]

[via GitHub]

chibenwa commented on PR #2049:
URL: https://github.com/apache/james-project/pull/2049#issuecomment-1963406487

   Have we tried disactivating it in tmail build?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] JAMES-4003 [FIX] Avoid forwarding bounces as it might create infinite loops [james-project]

[via GitHub]

chibenwa commented on PR #2022:
URL: https://github.com/apache/james-project/pull/2022#issuecomment-1963409069

   I suspect the recent build change to not retrigger the test and report it as 
failed...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

(james-project) branch compress-cves-2 updated (26408466b3 -> 25de96ea4f)

[btellier]

This is an automated email from the ASF dual-hosted git repository.

btellier pushed a change to branch compress-cves-2
in repository https://gitbox.apache.org/repos/asf/james-project.git


omit 26408466b3 [UPDATE] commons-compress 1.24.0 -> 1.26.0
 add 8a9a9d38de [SITE] Blog post for 3.7.5 release
 add fd767d9da5 [SITE] Blog post for 3.8.1 release
 add e9232deaac [SITE] Downloads for 3.8.1 James release
 add 924ce3c29c [SITE] Downloads for 3.7.5 James release
 add f8cd3b0596 [SITE] Downloads for 0.8.10 MIME4J release
 add 701f46bdac [SITE] s/3.8.0/3.8.1
 add d40376550c [SITE] Release notes
 add b0ad23fb06 [SITE] Update upgrade instructions
 add 9cb4bcf949 [SITE] Mime4J 0.8.10 release
 add 169ae5b8e8 JMAP - EmailSubmission/set - Should not invoke 
onSuccessDestroyEmail/onSuccessUpdateEmail when not created (#2041)
 add 232adec907 [FIX] MailReceptionCheck do not send the mail before we 
actively listen to the results (#2038)
 add 9ac709336e [CHANGELOG] Mention 3.7.5 and 3.8.1 releases (#2045)
 add 01f39e7ba7 Document new CVEs (#2046)
 add 3897ad9b99 Update site plugin
 add 7fd6b27741 Update rat plugin
 add 487d9faeea [DOCUMENTATION] CVE-2023-51518 CVE-2023-51747 
CVE-2024-21742 (#2047)
 add 915bcc33f2 JAMES-3997 Netty backpressure for IMAP FETCH (#2031)
 add 667c27e6b9 JAMES-4010 Do not index body (#2018)
 add 7a300ed824 JAMES-4009 StripAttachment should explicitly handle 
duplicates
 add aee6550277 JAMES-3995 Optimize Email/get
 add 66426a8005 JAMES-4002 AES blob store tuning to reduce memory pressure 
(#2021)
 add a2583d4e8a JAMES-4008 JMAP - Email/set - Should be able to save a 
draft with invalid email address (#2040)
 add ba34e7242c JAMES-4007 ImapChannelUpstreamHandler should always cleanup 
INBOUND message
 add 8b9cbf1385 JAMES-4007 Prevent channel inactivity to clear literal 
while IMAP APPEND is in progress
 add 25de96ea4f [UPDATE] commons-compress 1.24.0 -> 1.26.0

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (26408466b3)
\
 N -- N -- N   refs/heads/compress-cves-2 (25de96ea4f)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 CHANGELOG.md   |  16 +-
 docs/antora.yml|   2 +-
 .../{IndexHeaders.java => IndexBody.java}  |   2 +-
 .../opensearch/OpenSearchMailboxConfiguration.java |  37 +++-
 .../OpenSearchListeningMessageSearchIndex.java |  12 +-
 .../mailbox/opensearch/json/IndexableMessage.java  |  17 +-
 .../opensearch/json/MessageToOpenSearchJson.java   |  14 +-
 .../mailbox/opensearch/json/MimePartParser.java|  10 +-
 .../OpenSearchMailboxConfigurationTest.java|  97 ++---
 ...a => OpenSearchNoIndexBodyIntegrationTest.java} | 104 +
 .../store/search/ListeningMessageSearchIndex.java  |  10 +-
 .../james/transport/mailets/StripAttachment.java   |  25 ++-
 .../transport/mailets/StripAttachmentTest.java |  96 ++---
 pom.xml|   4 +-
 .../apache/james/imap/api/process/ImapSession.java |  13 ++
 .../james/imap/processor/fetch/FetchProcessor.java |  66 +-
 server/apps/distributed-app/docs/antora.yml|   2 +-
 .../modules/ROOT/pages/configure/blobstore.adoc|  16 ++
 .../modules/ROOT/pages/configure/opensearch.adoc   |   3 +
 .../docs/modules/ROOT/pages/operate/security.adoc  |  35 +++
 .../docs/modules/ROOT/pages/run/k8s-values.adoc|   2 +-
 .../docs/modules/ROOT/pages/run/run-docker.adoc|   4 +-
 .../sample-configuration/jvm.properties|  12 ++
 server/blob/blob-aes/pom.xml   |   4 +
 .../org/apache/james/blob/aes/AESBlobStoreDAO.java |  76 ++-
 .../org/apache/james/blob/api/BlobStoreDAO.java|  26 +++
 .../blob/objectstorage/aws/S3BlobStoreDAO.java |  10 +
 .../james/healthcheck/MailReceptionCheck.java  |   7 +-
 .../org/apache/james/jmap/api/model/Preview.java   |   2 +
 .../AvoidBinaryBodyBufferingBodyFactory.java   |  98 +++--
 .../apache/james/jmap/mime4j/FakeBinaryBody.java   |  48 +++--
 .../org/apache/james/jmap/mime4j/SizeUtils.java| 152 +
 .../ComputeMessageFastViewProjectionListener.java  |   4 +-
 .../src/main/resources/eml/nested2.eml |  77 +++
 .../rfc8621/contract/EmailGetMethodContract.scala  |  63 +-
 

Re: [PR] JAMES-4002 AES blob store tuning to reduce memory pressure [james-project]

[via GitHub]

chibenwa merged PR #2021:
URL: https://github.com/apache/james-project/pull/2021


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] Revert git commit id maven plugin [james-project]

[via GitHub]

Arsnael commented on PR #2049:
URL: https://github.com/apache/james-project/pull/2049#issuecomment-1963452919

   > Have we tried disactivating it in tmail build?
   
   I tried but wasn't successful. You are free to give it a shot though :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] JAMES-2586 Integration tests for JMAP postgres [james-project]

[via GitHub]

Arsnael commented on PR #2029:
URL: https://github.com/apache/james-project/pull/2029#issuecomment-1963463897

   No sorry didnt realize that the faulty tests in question were not even part 
of this PR. Thinking more of a bad test isolation here..?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

(james-project) branch master updated: JAMES-4008 JMAP - Email/set - Should be able to save a draft with invalid email address (#2040)

[btellier]

This is an automated email from the ASF dual-hosted git repository.

btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git


The following commit(s) were added to refs/heads/master by this push:
 new a2583d4e8a JAMES-4008 JMAP - Email/set - Should be able to save a 
draft with invalid email address (#2040)
a2583d4e8a is described below

commit a2583d4e8afe69b0974085004a92eff15445445b
Author: vttran 
AuthorDate: Mon Feb 26 14:35:39 2024 +0700

JAMES-4008 JMAP - Email/set - Should be able to save a draft with invalid 
email address (#2040)
---
 .../rfc8621/contract/EmailSetMethodContract.scala  | 238 -
 .../james/jmap/json/EmailGetSerializer.scala   |   3 +
 .../james/jmap/json/EmailSetSerializer.scala   |  18 +-
 .../scala/org/apache/james/jmap/mail/Email.scala   |  20 +-
 .../org/apache/james/jmap/mail/EmailSet.scala  |  91 +++-
 .../jmap/method/EmailSetCreatePerformer.scala  |   4 +-
 .../jmap/method/EmailSubmissionSetMethod.scala |  19 ++
 .../james/jmap/json/EmailSetSerializerTest.scala   |  87 
 8 files changed, 451 insertions(+), 29 deletions(-)

diff --git 
a/server/protocols/jmap-rfc-8621-integration-tests/jmap-rfc-8621-integration-tests-common/src/main/scala/org/apache/james/jmap/rfc8621/contract/EmailSetMethodContract.scala
 
b/server/protocols/jmap-rfc-8621-integration-tests/jmap-rfc-8621-integration-tests-common/src/main/scala/org/apache/james/jmap/rfc8621/contract/EmailSetMethodContract.scala
index 32beadf08b..25163a8e8c 100644
--- 
a/server/protocols/jmap-rfc-8621-integration-tests/jmap-rfc-8621-integration-tests-common/src/main/scala/org/apache/james/jmap/rfc8621/contract/EmailSetMethodContract.scala
+++ 
b/server/protocols/jmap-rfc-8621-integration-tests/jmap-rfc-8621-integration-tests-common/src/main/scala/org/apache/james/jmap/rfc8621/contract/EmailSetMethodContract.scala
@@ -45,11 +45,11 @@ import org.apache.james.jmap.http.UserCredential
 import org.apache.james.jmap.rfc8621.contract.DownloadContract.accountId
 import 
org.apache.james.jmap.rfc8621.contract.Fixture.{ACCEPT_RFC8621_VERSION_HEADER, 
ACCOUNT_ID, ANDRE, ANDRE_ACCOUNT_ID, ANDRE_PASSWORD, BOB, BOB_PASSWORD, DOMAIN, 
authScheme, baseRequestSpecBuilder}
 import org.apache.james.jmap.rfc8621.contract.probe.DelegationProbe
-import org.apache.james.mailbox.FlagsBuilder
 import org.apache.james.mailbox.MessageManager.AppendCommand
 import org.apache.james.mailbox.model.MailboxACL.Right
 import org.apache.james.mailbox.model.{ComposedMessageId, MailboxACL, 
MailboxConstants, MailboxId, MailboxPath, MessageId}
 import org.apache.james.mailbox.probe.MailboxProbe
+import org.apache.james.mailbox.{DefaultMailboxes, FlagsBuilder}
 import org.apache.james.mime4j.dom.Message
 import org.apache.james.modules.{ACLProbeImpl, MailboxProbeImpl, 
QuotaProbesImpl}
 import org.apache.james.util.ClassLoaderUtils
@@ -59,7 +59,7 @@ import org.awaitility.Awaitility
 import org.awaitility.Durations.ONE_HUNDRED_MILLISECONDS
 import org.hamcrest.Matchers
 import org.hamcrest.Matchers.{equalTo, not}
-import org.junit.jupiter.api.{BeforeEach, Test}
+import org.junit.jupiter.api.{BeforeEach, Disabled, Test}
 import org.junit.jupiter.params.ParameterizedTest
 import org.junit.jupiter.params.provider.ValueSource
 import play.api.libs.json.{JsNumber, JsString, Json}
@@ -7455,6 +7455,240 @@ trait EmailSetMethodContract {
   |}""".stripMargin)
   }
 
+  @Test
+  def emailSetShouldSucceedWhenInvalidToMailAddressAndHaveDraftKeyword(server: 
GuiceJamesServer): Unit = {
+val bobPath = MailboxPath.inbox(BOB)
+val mailboxId = 
server.getProbe(classOf[MailboxProbeImpl]).createMailbox(bobPath)
+
+val request =
+  s"""{
+ |  "using": ["urn:ietf:params:jmap:core", 
"urn:ietf:params:jmap:mail"],
+ |  "methodCalls": [
+ |["Email/set", {
+ |  "accountId": "$ACCOUNT_ID",
+ |  "create": {
+ |"aa":{
+ |  "mailboxIds": {
+ | "${mailboxId.serialize}": true
+ |  },
+ |  "keywords":{ "$$draft": true },
+ |  "to": [{"email": "invalid1"}],
+ |  "from": [{"email": "${BOB.asString}"}]
+ |}
+ |  }
+ |}, "c1"],
+ |["Email/get",
+ | {
+ |   "accountId": "$ACCOUNT_ID",
+ |   "ids": ["#aa"],
+ |   "properties": ["sentAt", "messageId"]
+ | },
+ | "c2"]]
+ |}""".stripMargin
+
+val response = `given`
+  .header(ACCEPT.toString, ACCEPT_RFC8621_VERSION_HEADER)
+  .body(request)
+.when
+  .post
+.`then`
+  .statusCode(SC_OK)
+  .contentType(JSON)
+  .extract
+  .body
+  .asString
+
+assertThatJson(response)
+  .inPath("methodResponses[1][1].list.[0].sentAt")
+  .isEqualTo("\"${json-unit.ignore}\"")
+   

Re: [PR] JAMES-4008 JMAP - Email/set - Should be able to save a draft with invalid email address [james-project]

[via GitHub]

chibenwa merged PR #2040:
URL: https://github.com/apache/james-project/pull/2040


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] JAMES-4007 Improve IMAP literal lifecycle [james-project]

[via GitHub]

chibenwa merged PR #2035:
URL: https://github.com/apache/james-project/pull/2035


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

(james-project) 01/02: JAMES-4007 ImapChannelUpstreamHandler should always cleanup INBOUND message

[btellier]

This is an automated email from the ASF dual-hosted git repository.

btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git

commit ba34e7242c95fa215b871a2af133e75ee14952ad
Author: Benoit TELLIER 
AuthorDate: Wed Feb 21 14:21:22 2024 +0100

JAMES-4007 ImapChannelUpstreamHandler should always cleanup INBOUND message

It needs to account for unexecuted messages, rejected messages,
cancelled subscriptions.
---
 .../apache/james/imapserver/netty/ImapChannelUpstreamHandler.java | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git 
a/server/protocols/protocols-imap4/src/main/java/org/apache/james/imapserver/netty/ImapChannelUpstreamHandler.java
 
b/server/protocols/protocols-imap4/src/main/java/org/apache/james/imapserver/netty/ImapChannelUpstreamHandler.java
index c0ef774c19..4e9e5985fd 100644
--- 
a/server/protocols/protocols-imap4/src/main/java/org/apache/james/imapserver/netty/ImapChannelUpstreamHandler.java
+++ 
b/server/protocols/protocols-imap4/src/main/java/org/apache/james/imapserver/netty/ImapChannelUpstreamHandler.java
@@ -405,9 +405,6 @@ public class ImapChannelUpstreamHandler extends 
ChannelInboundHandlerAdapter imp
 }
 if (signal.isOnComplete() || signal.isOnError()) {
 afterIDLEUponProcessing(ctx);
-if (message instanceof Closeable) {
-((Closeable) message).close();
-}
 }
 if (signal.hasError()) {
 ctx.fireExceptionCaught(signal.getThrowable());
@@ -419,6 +416,11 @@ public class ImapChannelUpstreamHandler extends 
ChannelInboundHandlerAdapter imp
 .contextWrite(ReactorUtils.context("imap", mdc(session))), 
message)
 // Manage throttling errors
 .doOnError(ctx::fireExceptionCaught)
+.doFinally(Throwing.consumer(any -> {
+if (message instanceof Closeable) {
+((Closeable) message).close();
+}
+}))
 .subscribe();
 disposableAttribute.set(disposable);
 }


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

(james-project) 02/02: JAMES-4007 Prevent channel inactivity to clear literal while IMAP APPEND is in progress

[btellier]

This is an automated email from the ASF dual-hosted git repository.

btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git

commit 8b9cbf138555b9c3c5027f9a0872323da511486a
Author: Benoit TELLIER 
AuthorDate: Wed Feb 21 14:28:29 2024 +0100

JAMES-4007 Prevent channel inactivity to clear literal while IMAP APPEND is 
in progress
---
 .../apache/james/imapserver/netty/ImapRequestFrameDecoder.java| 8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git 
a/server/protocols/protocols-imap4/src/main/java/org/apache/james/imapserver/netty/ImapRequestFrameDecoder.java
 
b/server/protocols/protocols-imap4/src/main/java/org/apache/james/imapserver/netty/ImapRequestFrameDecoder.java
index 4adab34f09..3e3e26d043 100644
--- 
a/server/protocols/protocols-imap4/src/main/java/org/apache/james/imapserver/netty/ImapRequestFrameDecoder.java
+++ 
b/server/protocols/protocols-imap4/src/main/java/org/apache/james/imapserver/netty/ImapRequestFrameDecoder.java
@@ -221,7 +221,12 @@ public class ImapRequestFrameDecoder extends 
ByteToMessageDecoder implements Net
 
 try {
 parseImapMessage(ctx, null, attachment, 
Pair.of(reader, size), readerIndex)
-.ifPresent(ctx::fireChannelRead);
+.ifPresent(message -> {
+ctx.fireChannelRead(message);
+// Remove ongoing subscription: now on 
lifecycle of the message will be managed by ImapChannelUpstreamHandler.
+// Not doing this causes IDLEd IMAP 
connections to clear IMAP append literal while they are processed.
+attachment.remove(SUBSCRIPTION);
+});
 } catch (DecodingException e) {
 ctx.fireExceptionCaught(e);
 }
@@ -237,6 +242,7 @@ public class ImapRequestFrameDecoder extends 
ByteToMessageDecoder implements Net
 
 });
 attachment.put(SUBSCRIPTION, (Disposable) () -> {
+// Clear the file if the connection is reset while buffering 
the litteral.
 subscribe.dispose();
 fileChunkConsumer.discard();
 });


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

(james-project) branch master updated (a2583d4e8a -> 8b9cbf1385)

[btellier]

This is an automated email from the ASF dual-hosted git repository.

btellier pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git


from a2583d4e8a JAMES-4008 JMAP - Email/set - Should be able to save a 
draft with invalid email address (#2040)
 new ba34e7242c JAMES-4007 ImapChannelUpstreamHandler should always cleanup 
INBOUND message
 new 8b9cbf1385 JAMES-4007 Prevent channel inactivity to clear literal 
while IMAP APPEND is in progress

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/james/imapserver/netty/ImapChannelUpstreamHandler.java | 8 +---
 .../apache/james/imapserver/netty/ImapRequestFrameDecoder.java| 8 +++-
 2 files changed, 12 insertions(+), 4 deletions(-)


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] Revert git commit id maven plugin [james-project]

[via GitHub]

jeantil commented on PR #2049:
URL: https://github.com/apache/james-project/pull/2049#issuecomment-1963493574

   Hi,
   I'm on limited connectivity this week.
   
   I wasn't aware that the James project build used git submodules. There are
   several limitations in the plugin with regard to submodules I'm not
   surprised it fails.
   
   You can revert or solve this differently if you want.
   
   I' m interested in the reason for using git submodules if you have the time
   :)
   
   Jean
   
   
   
   Le lun. 26 févr. 2024 à 08:28, Benoit TELLIER ***@***.***> a
   écrit :
   
   > To be honnest I'd be more in favour of removing that plugin from global
   > build and configuring it only in server/apps modules.
   >
   > —
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   > You are receiving this because you were mentioned.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] Revert git commit id maven plugin [james-project]

[via GitHub]

chibenwa commented on PR #2049:
URL: https://github.com/apache/james-project/pull/2049#issuecomment-1963490182

   https://github.com/apache/james-project/pull/2050 ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] JAMES-4009 StripAttachment should explicitly handle duplicates [james-project]

[via GitHub]

Arsnael merged PR #2043:
URL: https://github.com/apache/james-project/pull/2043


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

(james-project) branch master updated: JAMES-4009 StripAttachment should explicitly handle duplicates

[rcordier]

This is an automated email from the ASF dual-hosted git repository.

rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git


The following commit(s) were added to refs/heads/master by this push:
 new 7a300ed824 JAMES-4009 StripAttachment should explicitly handle 
duplicates
7a300ed824 is described below

commit 7a300ed8249365950bde222a4261b9407a5160d8
Author: Benoit TELLIER 
AuthorDate: Fri Feb 23 14:34:56 2024 +0100

JAMES-4009 StripAttachment should explicitly handle duplicates

 - Add a test
 - Explicit handling with an INFO level log
---
 .../james/transport/mailets/StripAttachment.java   | 25 --
 .../transport/mailets/StripAttachmentTest.java | 96 --
 2 files changed, 86 insertions(+), 35 deletions(-)

diff --git 
a/mailet/standard/src/main/java/org/apache/james/transport/mailets/StripAttachment.java
 
b/mailet/standard/src/main/java/org/apache/james/transport/mailets/StripAttachment.java
index f71341a318..459e7fb0d4 100644
--- 
a/mailet/standard/src/main/java/org/apache/james/transport/mailets/StripAttachment.java
+++ 
b/mailet/standard/src/main/java/org/apache/james/transport/mailets/StripAttachment.java
@@ -355,17 +355,26 @@ public class StripAttachment extends GenericMailet {
 }
 
 private void addPartContent(BodyPart bodyPart, Mail mail, String fileName, 
AttributeName attributeName) throws IOException, MessagingException {
-ImmutableMap.Builder> fileNamesToPartContent 
= AttributeUtils
-.getValueAndCastFromMail(mail, attributeName, 
MAP_STRING_BYTES_CLASS)
-.map(ImmutableMap.>builder()::putAll)
-.orElse(ImmutableMap.builder());
+ImmutableMap> result = 
AttributeUtils.getValueAndCastFromMail(mail, attributeName, 
MAP_STRING_BYTES_CLASS)
+.map(Throwing.function(previous -> {
+ImmutableMap.Builder> builder = 
ImmutableMap.>builder()
+.putAll(previous);
+if (!previous.containsKey(fileName)) {
+builder.put(fileName, 
AttributeValue.of(writeAsBytes(bodyPart)));
+} else {
+LOGGER.info("Duplicated file name {} for {}", fileName, 
mail.getName());
+}
+return builder.build();
+}))
+.orElse(ImmutableMap.of(fileName, 
AttributeValue.of(writeAsBytes(bodyPart;
 
+mail.setAttribute(new Attribute(attributeName, 
AttributeValue.of(result)));
+}
+
+private static byte[] writeAsBytes(BodyPart bodyPart) throws IOException, 
MessagingException {
 UnsynchronizedByteArrayOutputStream byteArrayOutputStream = new 
UnsynchronizedByteArrayOutputStream();
 bodyPart.writeTo(byteArrayOutputStream);
-fileNamesToPartContent.put(fileName, 
AttributeValue.of(byteArrayOutputStream.toByteArray()));
-
-Map> build = fileNamesToPartContent.build();
-mail.setAttribute(new Attribute(attributeName, 
AttributeValue.of(build)));
+return byteArrayOutputStream.toByteArray();
 }
 
 private void storeFileNameAsAttribute(Mail mail, AttributeValue 
fileName, boolean hasToBeStored) {
diff --git 
a/mailet/standard/src/test/java/org/apache/james/transport/mailets/StripAttachmentTest.java
 
b/mailet/standard/src/test/java/org/apache/james/transport/mailets/StripAttachmentTest.java
index b602d357fc..c2e4a3efd3 100644
--- 
a/mailet/standard/src/test/java/org/apache/james/transport/mailets/StripAttachmentTest.java
+++ 
b/mailet/standard/src/test/java/org/apache/james/transport/mailets/StripAttachmentTest.java
@@ -68,6 +68,7 @@ class StripAttachmentTest {
 private static Class>> 
MAP_STRING_BYTES_CLASS = (Class>>) (Object) 
Map.class;
 
 private static final String EXPECTED_ATTACHMENT_CONTENT = "#¤ãàé";
+private static final String EXPECTED_ATTACHMENT_CONTENT_2 = "Content 2";
 private static final Optional ABSENT_MIME_TYPE = Optional.empty();
 private static final String CONTENT_TRANSFER_ENCODING_VALUE = "8bit";
 
@@ -93,7 +94,7 @@ class StripAttachmentTest {
 };
 
 @Test
-void serviceShouldNotModifyMailWhenNotMultipart(TemporaryFolder 
temporaryFolder) throws MessagingException, IOException {
+void serviceShouldNotModifyMailWhenNotMultipart(TemporaryFolder 
temporaryFolder) throws Exception, IOException {
 Mailet mailet = initMailet(temporaryFolder);
 MimeMessageBuilder message = MimeMessageBuilder.mimeMessageBuilder()
 .setSubject("test")
@@ -116,7 +117,7 @@ class StripAttachmentTest {
 }
 
 @Test
-void serviceShouldSaveAttachmentInAFolderWhenPatternMatch(TemporaryFolder 
temporaryFolder) throws MessagingException {
+void serviceShouldSaveAttachmentInAFolderWhenPatternMatch(TemporaryFolder 
temporaryFolder) throws Exception {
 Mailet mailet = initMailet(temporaryFolder);
 
 String expectedAttachmentContent = EXPECTED_ATTACHMENT_CONTENT;
@@ 

Re: [PR] JAMES-2586 Fix some jmap postgres integration tests [james-project]

[via GitHub]

hungphan227 commented on code in PR #2048:
URL: https://github.com/apache/james-project/pull/2048#discussion_r1502017082


##
mailbox/postgres/src/main/java/org/apache/james/mailbox/postgres/mail/PostgresAttachmentModule.java:
##
@@ -35,20 +35,22 @@ public interface PostgresAttachmentModule {
 interface PostgresAttachmentTable {
 
 Table TABLE_NAME = DSL.table("attachment");
-Field ID = DSL.field("id", SQLDataType.UUID.notNull());
+Field ID_AS_UUID = DSL.field("id_as_uuid", 
SQLDataType.UUID.notNull());
+Field ID = DSL.field("id", SQLDataType.VARCHAR.notNull());

Review Comment:
   No, this is not "just to pass IT". In 
mailbox/postgres/src/main/java/org/apache/james/mailbox/postgres/mail/dao/PostgresAttachmentDAO.java
 line 87:
   
   ```
   public Flux getAttachments(Collection 
attachmentIds) {
   return postgresExecutor.executeRows(dslContext -> 
Flux.from(dslContext.selectFrom(PostgresAttachmentTable.TABLE_NAME)
   
.where(PostgresAttachmentTable.ID_AS_UUID.in(attachmentIds.stream().map(AttachmentId::asUUID).collect(ImmutableList.toImmutableList())
   .map(row -> AttachmentMetadata.builder()
   
.attachmentId(AttachmentId.from(row.get(PostgresAttachmentTable.ID)))
   .type(row.get(PostgresAttachmentTable.TYPE))
   
.messageId(PostgresMessageId.Factory.of(row.get(PostgresAttachmentTable.MESSAGE_ID)))
   .size(row.get(PostgresAttachmentTable.SIZE))
   .build());
   }
   ```
   
   If we do not have ID_AS_UUID, we have to use a hashmap to save AttachmentId 
- UUID (UUID.nameUUIDFromBytes(id.getBytes(StandardCharsets.UTF_8))) which is 
not good option IMO



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] JAMES-2586 Integration tests for JMAP postgres [james-project]

[via GitHub]

Arsnael commented on PR #2029:
URL: https://github.com/apache/james-project/pull/2029#issuecomment-1963275402

   Potentially racing issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

(james-project) branch master updated: JAMES-3995 Optimize Email/get

[rcordier]

This is an automated email from the ASF dual-hosted git repository.

rcordier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git


The following commit(s) were added to refs/heads/master by this push:
 new aee6550277 JAMES-3995 Optimize Email/get
aee6550277 is described below

commit aee6550277db4e31099867e1d29c7598f9a4dad9
Author: Benoit TELLIER 
AuthorDate: Thu Feb 22 22:47:26 2024 +0100

JAMES-3995 Optimize Email/get
---
 .../org/apache/james/jmap/api/model/Preview.java   |   2 +
 .../AvoidBinaryBodyBufferingBodyFactory.java   | 366 +
 .../apache/james/jmap/mime4j/FakeBinaryBody.java   |  64 
 .../org/apache/james/jmap/mime4j/SizeUtils.java| 152 +
 .../ComputeMessageFastViewProjectionListener.java  |   4 +-
 .../src/main/resources/eml/nested2.eml |  77 +
 .../rfc8621/contract/EmailGetMethodContract.scala  |  63 +++-
 .../contract/EmailParseMethodContract.scala|   2 +-
 .../scala/org/apache/james/jmap/mail/Email.scala   |   6 +-
 .../org/apache/james/jmap/mail/EmailBodyPart.scala |  17 +-
 10 files changed, 734 insertions(+), 19 deletions(-)

diff --git 
a/server/data/data-jmap/src/main/java/org/apache/james/jmap/api/model/Preview.java
 
b/server/data/data-jmap/src/main/java/org/apache/james/jmap/api/model/Preview.java
index 5474b6211c..2ef3ecc876 100644
--- 
a/server/data/data-jmap/src/main/java/org/apache/james/jmap/api/model/Preview.java
+++ 
b/server/data/data-jmap/src/main/java/org/apache/james/jmap/api/model/Preview.java
@@ -29,6 +29,7 @@ import java.util.Objects;
 import javax.inject.Inject;
 
 import org.apache.commons.lang3.StringUtils;
+import org.apache.james.jmap.mime4j.AvoidBinaryBodyBufferingBodyFactory;
 import org.apache.james.mailbox.exception.MailboxException;
 import org.apache.james.mailbox.model.MessageResult;
 import org.apache.james.mime4j.dom.Message;
@@ -81,6 +82,7 @@ public class Preview {
 private Message parse(InputStream inputStream) throws IOException {
 DefaultMessageBuilder defaultMessageBuilder = new 
DefaultMessageBuilder();
 defaultMessageBuilder.setMimeEntityConfig(MimeConfig.PERMISSIVE);
+defaultMessageBuilder.setBodyFactory(new 
AvoidBinaryBodyBufferingBodyFactory());
 return defaultMessageBuilder.parseMessage(inputStream);
 }
 }
diff --git 
a/server/data/data-jmap/src/main/java/org/apache/james/jmap/mime4j/AvoidBinaryBodyBufferingBodyFactory.java
 
b/server/data/data-jmap/src/main/java/org/apache/james/jmap/mime4j/AvoidBinaryBodyBufferingBodyFactory.java
new file mode 100644
index 00..42abd42f2d
--- /dev/null
+++ 
b/server/data/data-jmap/src/main/java/org/apache/james/jmap/mime4j/AvoidBinaryBodyBufferingBodyFactory.java
@@ -0,0 +1,366 @@
+/
+ * Licensed to the Apache Software Foundation (ASF) under one   *
+ * or more contributor license agreements.  See the NOTICE file *
+ * distributed with this work for additional information*
+ * regarding copyright ownership.  The ASF licenses this file   *
+ * to you under the Apache License, Version 2.0 (the*
+ * "License"); you may not use this file except in compliance   *
+ * with the License.  You may obtain a copy of the License at   *
+ *  *
+ *   http://www.apache.org/licenses/LICENSE-2.0 *
+ *  *
+ * Unless required by applicable law or agreed to in writing,   *
+ * software distributed under the License is distributed on an  *
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY   *
+ * KIND, either express or implied.  See the License for the*
+ * specific language governing permissions and limitations  *
+ * under the License.   *
+ /
+
+package org.apache.james.jmap.mime4j;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.Reader;
+import java.io.StringReader;
+import java.io.UnsupportedEncodingException;
+import java.nio.charset.Charset;
+import java.nio.charset.IllegalCharsetNameException;
+import java.nio.charset.UnsupportedCharsetException;
+
+import org.apache.james.mime4j.Charsets;
+import org.apache.james.mime4j.dom.BinaryBody;
+import org.apache.james.mime4j.dom.SingleBody;
+import org.apache.james.mime4j.dom.TextBody;
+import org.apache.james.mime4j.io.InputStreams;
+import org.apache.james.mime4j.message.BasicBodyFactory;
+import org.apache.james.mime4j.message.BodyFactory;
+import org.apache.james.mime4j.util.ByteArrayOutputStreamRecycler;
+import org.apache.james.mime4j.util.ContentUtil;
+
+import com.google.common.io.CountingOutputStream;
+
+/**
+ * Factory for creating message 

Re: [PR] JAMES-3995 Optimize Email/get [james-project]

[via GitHub]

Arsnael merged PR #2005:
URL: https://github.com/apache/james-project/pull/2005


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] JAMES-4003 [FIX] Avoid forwarding bounces as it might create infinite loops [james-project]

[via GitHub]

Arsnael commented on PR #2022:
URL: https://github.com/apache/james-project/pull/2022#issuecomment-1963284254

   This test failed on the last two builds: 
https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-2022/5/testReport/org.apache.james.transport.mailets/RemoteDeliveryErrorHandlingTest/remoteDeliveryShouldStoreTemporaryFailureAsPermanentWhenExceedsMaximumRetries_SMTPMessageSender__DockerMockSmtp_/
   
   Issue? Flaky? Might need to double check


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

[PR] Revert git commit id maven plugin [james-project]

[via GitHub]

Arsnael opened a new pull request, #2049:
URL: https://github.com/apache/james-project/pull/2049

   @jeantil I'm sorry to ask for this, but do you mind if we revert the upgrade 
of the git-commit-id-maven-plugin for now?
   
   It seems tocreate issues with projects using James as a submodule (like 
TMail-backend) and makes the build impossible.
   
   I opened an issue on the plugin backlog if you want to have a look for more 
context/details: 
https://github.com/git-commit-id/git-commit-id-maven-plugin/issues/701
   
   But for now this is a blocker for us unfortunately^^'
   
   Mind that I still put back for `injectAllReactorProjects` to true after the 
revert, hoping it keeps similar gain in build performance :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] JAMES-2586 Fix some jmap postgres integration tests [james-project]

[via GitHub]

vttranlina commented on code in PR #2048:
URL: https://github.com/apache/james-project/pull/2048#discussion_r1502032559


##
mailbox/postgres/src/main/java/org/apache/james/mailbox/postgres/mail/dao/PostgresAttachmentDAO.java:
##
@@ -80,9 +84,21 @@ public Mono> 
getAttachment(AttachmentId attachm
 blobIdFactory.from(row.get(PostgresAttachmentTable.BLOB_ID;
 }
 
+public Flux getAttachments(Collection 
attachmentIds) {
+return postgresExecutor.executeRows(dslContext -> 
Flux.from(dslContext.selectFrom(PostgresAttachmentTable.TABLE_NAME)
+
.where(PostgresAttachmentTable.ID_AS_UUID.in(attachmentIds.stream().map(AttachmentId::asUUID).collect(ImmutableList.toImmutableList())

Review Comment:
   1. Why do we don't use directly "AttachmentId.getId()" in `where` clause?
   2. When using "IN" clause, we should check the size of IN, if large, we 
should partition it, see the same 
`PostgresMailboxMessageDAO#findMessagesByMailboxIdAndUIDs`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] JAMES-2586 Fix some jmap postgres integration tests [james-project]

[via GitHub]

quantranhong1999 commented on code in PR #2048:
URL: https://github.com/apache/james-project/pull/2048#discussion_r1501987659


##
server/protocols/jmap-rfc-8621-integration-tests/postgres-jmap-rfc-8621-integration-tests/src/test/java/org/apache/james/jmap/rfc8621/postgres/PostgresThreadGetTest.java:
##
@@ -88,31 +60,11 @@ public class PostgresThreadGetTest extends PostgresBase 
implements ThreadGetCont
 .overrideWith(new TestJMAPServerModule()))
 .build();
 
-@AfterEach
-void tearDown() throws IOException {
-client.close();
-}
-
 @Override
 public void awaitMessageCount(List mailboxIds, SearchQuery 
query, long messageCount) {
-awaitForOpenSearch(queryConverter.from(mailboxIds, query), 
messageCount);
 }
 
 @Override
 public void initOpenSearchClient() {
-client = MailboxIndexCreationUtil.prepareDefaultClient(
-openSearch.getDockerOpenSearch().clientProvider().get(),
-openSearch.getDockerOpenSearch().configuration());
-}
-
-private void awaitForOpenSearch(Query query, long totalHits) {

Review Comment:
   Why remove this?



##
mailbox/postgres/src/main/java/org/apache/james/mailbox/postgres/mail/PostgresAttachmentModule.java:
##
@@ -35,20 +35,22 @@ public interface PostgresAttachmentModule {
 interface PostgresAttachmentTable {
 
 Table TABLE_NAME = DSL.table("attachment");
-Field ID = DSL.field("id", SQLDataType.UUID.notNull());
+Field ID_AS_UUID = DSL.field("id_as_uuid", 
SQLDataType.UUID.notNull());
+Field ID = DSL.field("id", SQLDataType.VARCHAR.notNull());

Review Comment:
   So we need the "redundant" ID field just to pass IT? Any other way?



##
server/protocols/jmap-rfc-8621-integration-tests/jmap-rfc-8621-integration-tests-common/src/main/scala/org/apache/james/jmap/rfc8621/contract/EmailQueryMethodContract.scala:
##
@@ -7034,22 +7034,24 @@ trait EmailQueryMethodContract {
  |"c1"]]
  |}""".stripMargin
 
-val response = `given`
-  .header(ACCEPT.toString, ACCEPT_RFC8621_VERSION_HEADER)
-  .body(request)
-.when
-  .post
-.`then`
-  .statusCode(SC_OK)
-  .contentType(JSON)
-  .extract
-  .body
-  .asString
+awaitAtMostTenSeconds.untilAsserted { () =>
+  val response = `given`
+.header(ACCEPT.toString, ACCEPT_RFC8621_VERSION_HEADER)
+.body(request)
+.when
+.post
+.`then`

Review Comment:
   indent



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] JAMES-2586 Fix some jmap postgres integration tests [james-project]

[via GitHub]

hungphan227 commented on code in PR #2048:
URL: https://github.com/apache/james-project/pull/2048#discussion_r1502005171


##
server/protocols/jmap-rfc-8621-integration-tests/postgres-jmap-rfc-8621-integration-tests/src/test/java/org/apache/james/jmap/rfc8621/postgres/PostgresThreadGetTest.java:
##
@@ -88,31 +60,11 @@ public class PostgresThreadGetTest extends PostgresBase 
implements ThreadGetCont
 .overrideWith(new TestJMAPServerModule()))
 .build();
 
-@AfterEach
-void tearDown() throws IOException {
-client.close();
-}
-
 @Override
 public void awaitMessageCount(List mailboxIds, SearchQuery 
query, long messageCount) {
-awaitForOpenSearch(queryConverter.from(mailboxIds, query), 
messageCount);
 }
 
 @Override
 public void initOpenSearchClient() {
-client = MailboxIndexCreationUtil.prepareDefaultClient(
-openSearch.getDockerOpenSearch().clientProvider().get(),
-openSearch.getDockerOpenSearch().configuration());
-}
-
-private void awaitForOpenSearch(Query query, long totalHits) {

Review Comment:
   In postgres implementation, Thread feature use postgres instead of opensearch



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org

Re: [PR] JAMES-2586 Fix some jmap postgres integration tests [james-project]

[via GitHub]

Arsnael commented on code in PR #2048:
URL: https://github.com/apache/james-project/pull/2048#discussion_r1502013047


##
mailbox/postgres/src/main/java/org/apache/james/mailbox/postgres/mail/PostgresAttachmentModule.java:
##
@@ -35,20 +35,22 @@ public interface PostgresAttachmentModule {
 interface PostgresAttachmentTable {
 
 Table TABLE_NAME = DSL.table("attachment");
-Field ID = DSL.field("id", SQLDataType.UUID.notNull());
+Field ID_AS_UUID = DSL.field("id_as_uuid", 
SQLDataType.UUID.notNull());
+Field ID = DSL.field("id", SQLDataType.VARCHAR.notNull());

Review Comment:
   If the tests fails because the structure is different, I would prefer here 
that we duplicate the test suite and adapt it to postgres attachment table 
structure instead?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@james.apache.org
For additional commands, e-mail: notifications-h...@james.apache.org